{"thread":{"id":"46468","subject":"[RFC] Git rerere and non-conflicting changes during conflict resolution","startedAt":"2017-07-25T15:10:02Z","lastAt":"2017-07-27T21:01:43Z","messageCount":10,"participants":["Raman Gupta","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"325042","messageId":"17c46229-3b64-34f1-30fa-d40b77e1c054@gmail.com","threadId":"46468","inReplyTo":null,"subject":"[RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Raman Gupta","fromEmail":"rocketraman@gmail.com","sentAt":"2017-07-25T15:09:55Z","receivedAt":"2017-07-25T15:10:02Z","isPatch":false,"sender":{"key":"rocketraman@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53049?v=4"},"body":"I had an interesting situation today: resolving a merge conflict\nrequired modification in other files that were not themselves conflicting.\n\nI just realized that rerere does not remember any changes to these\nadditional files -- only changes to the conflicting files. This makes\nthe end result of rerere obviously incorrect in this situation.\n\nSo my questions are:\n\n1) Is this a known limitation or is there a reason rerere works in\nthis manner?\n\n1b) If it is a limitation/bug, what would be needed to fix it? With\nsome guidance, I might be able to submit a patch...\n\n2) In the meantime, is there a way I can identify these cases, without\nwhich I cannot really trust rerere is doing the right thing?\n\nRegards,\nRaman\n"},{"id":"325045","messageId":"20170725175202.ar4ykqoadbihwb2w@sigill.intra.peff.net","threadId":"46468","inReplyTo":"17c46229-3b64-34f1-30fa-d40b77e1c054@gmail.com","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-25T17:52:02Z","receivedAt":"2017-07-25T17:52:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 25, 2017 at 11:09:55AM -0400, Raman Gupta wrote:\n\n> I had an interesting situation today: resolving a merge conflict\n> required modification in other files that were not themselves conflicting.\n> \n> I just realized that rerere does not remember any changes to these\n> additional files -- only changes to the conflicting files. This makes\n> the end result of rerere obviously incorrect in this situation.\n> \n> So my questions are:\n> \n> 1) Is this a known limitation or is there a reason rerere works in\n> this manner?\n\nYes, it's known. Rerere works by storing a mapping of conflicted hunks\nto their resolutions. If there's no conflicted hunk, I'm not sure how\nwe'd decide what to feed into the mapping to see if there is some\ncontent to be replaced.\n\nThat said, I'm far from an expert on how rerere works. Junio might have\nideas on how we could handle this better. But I do note that for\nrepeated integration runs (like we do for topics in git.git, as they get\nmerged to \"pu\", then \"next\", then \"master\"), he keeps non-conflict\nfixups in a separate commit which gets squashed into the merge\nautomatically. See\n\n  https://github.com/git/git/blob/todo/Reintegrate#L185-L191\n\n> 1b) If it is a limitation/bug, what would be needed to fix it? With\n> some guidance, I might be able to submit a patch...\n\nAs far as I know, something like the Reintegrate script above is the\nstate of the art. IMHO it would be useful if something similar were\nintegrated into rerere, but I'm not sure exactly how it would know when\nto trigger.\n\n> 2) In the meantime, is there a way I can identify these cases, without\n> which I cannot really trust rerere is doing the right thing?\n\nI do think it would be useful if rerere could look at a merge result and\nsay \"OK, I've recorded these bits, but there are other lines that are\nnot part of either parent and which are not part of a conflict\". That\ngives you a warning that such lines need to be part of a fixup (rather\nthan you being surprised when you redo the merge later and have to\nrework the fixup).\n\nBut I don't think even then you can ever trust rerere fully.\nFundamentally you're applying some changes from one merge into another\ncontext. There may be new sites that also need fixing up, and the tool\nhas no way to know. So you should treat a rerere-helped merge as any\nother merge: assume it's a good starting point but use other tools (like\nthe compiler or automated tests) to confirm that the result is sensible.\n\n-Peff\n"},{"id":"325061","messageId":"1d07fa3b-cd3f-42fd-87c1-0627de7c04f2@gmail.com","threadId":"46468","inReplyTo":"20170725175202.ar4ykqoadbihwb2w@sigill.intra.peff.net","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Raman Gupta","fromEmail":"rocketraman@gmail.com","sentAt":"2017-07-25T19:54:32Z","receivedAt":"2017-07-25T19:54:41Z","isPatch":false,"sender":{"key":"rocketraman@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53049?v=4"},"body":"On 25/07/17 01:52 PM, Jeff King wrote:\n> On Tue, Jul 25, 2017 at 11:09:55AM -0400, Raman Gupta wrote:\n> \n>> I had an interesting situation today: resolving a merge conflict\n>> required modification in other files that were not themselves conflicting.\n>>\n>> I just realized that rerere does not remember any changes to these\n>> additional files -- only changes to the conflicting files. This makes\n>> the end result of rerere obviously incorrect in this situation.\n>>\n>> So my questions are:\n>>\n>> 1) Is this a known limitation or is there a reason rerere works in\n>> this manner?\n> \n> Yes, it's known. Rerere works by storing a mapping of conflicted hunks\n> to their resolutions. If there's no conflicted hunk, I'm not sure how\n> we'd decide what to feed into the mapping to see if there is some\n> content to be replaced.\n> \n> That said, I'm far from an expert on how rerere works. Junio might have\n> ideas on how we could handle this better. But I do note that for\n> repeated integration runs (like we do for topics in git.git, as they get\n> merged to \"pu\", then \"next\", then \"master\"), he keeps non-conflict\n> fixups in a separate commit which gets squashed into the merge\n> automatically. See\n> \n>   https://github.com/git/git/blob/todo/Reintegrate#L185-L191\n\nSeems relatively simple to me, at least conceptually.\n\n1) Store the state of the index after the merge.\n\n2) After conflict resolution is complete (i.e. user executes \"git\ncommit\"), diff index @ step 1 with commit.\n\n3) Assume that all changes in that diff are related to conflict\nresolution (as they should be), and save that diff to the rerere cache.\n\nI could be missing something fundamental here though...\n\n>> 1b) If it is a limitation/bug, what would be needed to fix it? With\n>> some guidance, I might be able to submit a patch...\n> \n> As far as I know, something like the Reintegrate script above is the\n> state of the art. IMHO it would be useful if something similar were\n> integrated into rerere, but I'm not sure exactly how it would know when\n> to trigger.\n\nI've seen the Reintegrate script before. It is very specific to the\ngit.git workflow. I think it makes sense to expose this particular\ncapability in git proper, given that rerere itself is exposed. Plus\nthat could actually simplify the Reintegrate script a bit.\n\n>> 2) In the meantime, is there a way I can identify these cases, without\n>> which I cannot really trust rerere is doing the right thing?\n> \n> I do think it would be useful if rerere could look at a merge result and\n> say \"OK, I've recorded these bits, but there are other lines that are\n> not part of either parent and which are not part of a conflict\". That\n> gives you a warning that such lines need to be part of a fixup (rather\n> than you being surprised when you redo the merge later and have to\n> rework the fixup).\n\nAgreed.\n\n> But I don't think even then you can ever trust rerere fully.\n> Fundamentally you're applying some changes from one merge into another\n> context. There may be new sites that also need fixing up, and the tool\n> has no way to know. So you should treat a rerere-helped merge as any\n> other merge: assume it's a good starting point but use other tools (like\n> the compiler or automated tests) to confirm that the result is sensible.\n\nWhile I agree that every case cannot be handled, and this type of\nvalidation is still necessary, I believe git should cover the cases\nwhich it is proper and possible to cover.\n\n> -Peff\n> \n\nRegards,\nRaman\n"},{"id":"325063","messageId":"20170725202550.zasszwoq5z2jhj66@sigill.intra.peff.net","threadId":"46468","inReplyTo":"1d07fa3b-cd3f-42fd-87c1-0627de7c04f2@gmail.com","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-25T20:25:51Z","receivedAt":"2017-07-25T20:26:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 25, 2017 at 03:54:32PM -0400, Raman Gupta wrote:\n\n> > That said, I'm far from an expert on how rerere works. Junio might have\n> > ideas on how we could handle this better. But I do note that for\n> > repeated integration runs (like we do for topics in git.git, as they get\n> > merged to \"pu\", then \"next\", then \"master\"), he keeps non-conflict\n> > fixups in a separate commit which gets squashed into the merge\n> > automatically. See\n> > \n> >   https://github.com/git/git/blob/todo/Reintegrate#L185-L191\n> \n> Seems relatively simple to me, at least conceptually.\n> \n> 1) Store the state of the index after the merge.\n> \n> 2) After conflict resolution is complete (i.e. user executes \"git\n> commit\"), diff index @ step 1 with commit.\n> \n> 3) Assume that all changes in that diff are related to conflict\n> resolution (as they should be), and save that diff to the rerere cache.\n> \n> I could be missing something fundamental here though...\n\nThat describes the \"save\" step. How do we do the apply step? We later do\na merge and see that there are some conflicts. We compute a hash for\neach conflict, and see if it has a resolution in the database.\n\nBut how do we decide which hashes to look up for the parts that are\noutside of a conflict?\n\nIn theory you want those fixups to \"ride along\" with the other\nconflicts. But which fixups ride along with which conflicts? You really\nwant some way to uniquely identify a particular merge. But of course you\ncan't use the content, because the whole point of rerere is that you're\ndoing a _similar_ merge in another context. And that's why Reintegrate\nties it to the branch being merged. That gives us the unique identifier.\nBut rerere normally does not concern itself with that.\n\nI think you could mark the fixups as potential ride-alongs for each\nconflict that's resolved. And then when we apply a resolution, tell the\nuser \"by the way, you might want these fixups, too\" (or maybe even try\nto apply them automatically). That will suggest fixups that may not be\nrelevant anymore, but if the next \"save\" includes a particular\nresolution but _not_ a potential fixup, then that fixup can be culled\nfrom its list.\n\nThere's still a corner case that doesn't handle, though: you might need\nfixups even if you have no conflicts at all.\n\nSo we may be stumbling towards an answer, but it still sounds kind of\nhacky and fragile.\n\n> > As far as I know, something like the Reintegrate script above is the\n> > state of the art. IMHO it would be useful if something similar were\n> > integrated into rerere, but I'm not sure exactly how it would know when\n> > to trigger.\n> \n> I've seen the Reintegrate script before. It is very specific to the\n> git.git workflow. I think it makes sense to expose this particular\n> capability in git proper, given that rerere itself is exposed. Plus\n> that could actually simplify the Reintegrate script a bit.\n\nRight, sorry if I wasn't clear; my \"something similar\" meant only the\nfixup-applying lines I linked to.\n\n> > But I don't think even then you can ever trust rerere fully.\n> > Fundamentally you're applying some changes from one merge into another\n> > context. There may be new sites that also need fixing up, and the tool\n> > has no way to know. So you should treat a rerere-helped merge as any\n> > other merge: assume it's a good starting point but use other tools (like\n> > the compiler or automated tests) to confirm that the result is sensible.\n> \n> While I agree that every case cannot be handled, and this type of\n> validation is still necessary, I believe git should cover the cases\n> which it is proper and possible to cover.\n\nYes, I agree that some help is usually better than none (although we\nneed to be careful that any improvement is honest about its limitations\nfor any given case, and doesn't end up tricking the user into a false\nsense of security).\n\n-Peff\n"},{"id":"325064","messageId":"xmqqo9s8uuth.fsf@gitster.mtv.corp.google.com","threadId":"46468","inReplyTo":"20170725175202.ar4ykqoadbihwb2w@sigill.intra.peff.net","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-25T20:26:34Z","receivedAt":"2017-07-25T20:26:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> 1) Is this a known limitation or is there a reason rerere works in\n>> this manner?\n>\n> Yes, it's known. Rerere works by storing a mapping of conflicted hunks\n> to their resolutions. If there's no conflicted hunk, I'm not sure how\n> we'd decide what to feed into the mapping to see if there is some\n> content to be replaced.\n\nCorrect.  \n\nThis is not even a limitation but is outside the scope of rerere.\nLet's understand that first.\n\nA semantic conflict that requires an evil merge that touches a file\nthat is not involved in any textual conflict during a merge will\nhappen even if there is *NO* textual merge conflict.  \n\nImagine that there is a global variable 'xyzzy' used in many places\nin the code, and then a side branch forks from the mainline.  The\nmainline renames the variable to 'frotz' in the entire codebase,\nwhile the side branch adds one more place that the variable is used\nunder its original name.  Then you merge these two branches.  This\nwill textually merge cleanly if the place the side branch adds a new\nmention of 'xyzzy' is textually far from any block of text in the\ncommon ancestor that has been updated on the mainline while these\ntwo branches diverged.\n\n\"git checkout mainline && git merge side\" will cleanly automerge,\nyet the result is not correct.  The new mention of 'xyzzy' added by\nthe merge needs to be corrected to 'frotz'.\n\nNow we take that as the baseline and further imagine that during the\ntime these two branches diverged, the mainline also updated some\ndocumentation of something totally unrelated to 'xyzzy' vs 'frotz'\nvariable.  Perhaps README was updated, or something.  The side\nbranch also updated the same file in a different way.  This time,\nthe changes to this same file may result in textual conflict.\n\n\"git checkout mainline && git merge side\" will result in a conflict,\nwhose resolution may be recorded by rerere for that file.  It should\nbe crystal clear that this conflict does *not* have anything to do\nwith the semantic conflict between 'xyzzy' vs 'frotz'.  \n\nThe realization we must draw from the above observation is that what\nthe \"merge-fix\" machinery in the Reintegrate script you cited in\nyour message tries to help, which is the semantic conflict,\nfundamentally cannot be tied to any textual merge conflict that may\n(or may not) happen.  That is what makes the issue outside the scope\nof rerere.\n\nThe above is not to say that the need to record and replay such evil\nmerges to solve semantic conflict does not exist.  Far from it.  It\nis just clarifying that it is a wrong approach to try to \"teach\"\nrerere to somehow handle that case as well.\n\nIf we wanted to port the \"merge-fix\" logic, and I do wish it would\nhappen some day, the update belongs to \"git merge\".\n\nYou were too kind to call the \"merge-fix\" logic in Reintegrate \"the\nstate of the art\", but I am not happy about its limitation.  Here\nare the things I wish to have in an ideal version of the \"merge-fix\"\nlogic, which does not exist yet:\n\n * There is a database of \"to be cherry-picked\" commits, keyed by a\n   pair of branch names.  That is, given branches A and B, the\n   database will return 0 or more commits that can be cherry-picked.\n   The order of branch names in the pair is immaterial, i.e. asking\n   the database for cherry-pickable commits keyed by <A, B> and\n   keyed by <B, A> will yield the identical set of commits.\n\n * When merging commit X to commit Y, \"git merge\" in the ideal world\n   does the following:\n\n   - It first does what it currently does, i.e. three-way merge with\n     the merge strategy and applying rerere for re-application of an\n     earlier resolution of textual conflicts.\n\n   - Then, it looks up the database to find the keys <A, B> where\n     A is in X but not in Y, and B is not in X but in Y.\n     These commits are cherry-picked and squashed into the result of\n     the above.\n\nThe intent is that a pair <A, B> represents the mainline and side\nbranch in the above example, where A renamed 'xyzzy' to 'frotz' and\nB added new reference to 'xyzzy'.  And the cherry-pickable commit\nfound in the database is to tweak the 'xyzzy' B adds into 'frotz'.\n\nI said A and B in the above are branch names, but in the ideal\nworld, they can be commit object names (possibly in the middle of a\nbranch), as long as we can reliable update the database's keys every\ntime \"git rebase\" etc. rewrites commits.\n\nTo populate the database, we'd need a reverse.\n\n * When merging branch B into branch A (or the other way around) for\n   the first time, \"git merge\" would do what it currently does.\n\n * The user then concludes the merge to resolve *ONLY* the textual\n   conflict, and commit the result.  It is important that no\n   additional evil merge to correct for semantic conflicts is done\n   in this step.  Note that if the auto-merge cleanly succeeds, this\n   step may not even exist.\n\n * Then the user makes another commit to correct for semantic\n   conflicts (aka \"merge-fix\").\n\n * Then the user tells Git that semantic conflicts were resolved and\n   need to be recorded (just like running \"git rerere\" manually,\n   before \"git commit\" automatically does it for them these days).\n   This will result in the following:\n\n   - The database is updated so that key <A, B> yields the\n     \"merge-fix\" commit;\n\n   - The head is detached at the tip of branch A before the merge;\n\n   - \"git merge B\" is done again, which _should_ reproduce the state\n     immediately after the user committed the \"merge-fix\";\n\n   - The tip of branch A is reset to the result of the above.\n\nThe merge-fix logic in Reintegrate is a poor-man's emulation of the\nabove ideal.  A value its database yields is not a set of commits,\nbut a single commit, and instead of getting keyed by a pair of\nbranch names, the database is keyed by a single branch name\n(i.e. recording \"I had trouble when merging this branch\" without\nsaying \"... to the integration branch that already had this other\nbranch\"), so the look-up does not have to do \"A is in X but not in\nY, and B is not in X but in Y\".  \n\nIt is still usable but the database need to be reorganized every\ntime the order of topics merged to 'pu' is changed.\n\n\n"},{"id":"325066","messageId":"xmqqk22wuu5g.fsf@gitster.mtv.corp.google.com","threadId":"46468","inReplyTo":"xmqqo9s8uuth.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-25T20:40:59Z","receivedAt":"2017-07-25T20:41:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> To populate the database, we'd need a reverse.\n> ...\n>  * Then the user tells Git that semantic conflicts were resolved and\n>    need to be recorded (just like running \"git rerere\" manually,\n>    before \"git commit\" automatically does it for them these days).\n>    This will result in the following:\n>\n>    - The database is updated so that key <A, B> yields the\n>      \"merge-fix\" commit;\n> ...\n\nI probably should have been aiming for stars, as I were outlining\nthe ideal merge-fix logic.  The key <A, B> is merely a default, and\nthe worst one at that.  There should be a way for the user to tell\nwhich exact pair of commits (i.e. another side branch that was\nmerged earlier to the mainline A that renamed 'xyzzy' to 'frotz'\nwholesale, and the exact commit on the side branch B that added an\nextra mention of 'xyzzy').  \n\nIf the logic can figure out what these two commits are without\nuser's help, mechanically by only looking at the merge-fix commit,\nthat would be even better.  But I do not believe in miracles, so...\n\n"},{"id":"325073","messageId":"20170725205843.bi6kyqjlzyodmxuq@sigill.intra.peff.net","threadId":"46468","inReplyTo":"xmqqo9s8uuth.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-25T20:58:43Z","receivedAt":"2017-07-25T20:58:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 25, 2017 at 01:26:34PM -0700, Junio C Hamano wrote:\n\n> This is not even a limitation but is outside the scope of rerere.\n> Let's understand that first.\n> [...]\n> If we wanted to port the \"merge-fix\" logic, and I do wish it would\n> happen some day, the update belongs to \"git merge\".\n\nLooks like this crossed with my latest email. Overall I agree with you.\n\nI almost said the same thing about scope initially, but I decided it\ndoesn't really matter. From the user's perspective there may be a tool X\nthat replays bits of a previous merge result. And that task can be\nsubdivided into replaying conflict resolution and replaying merge-fixes.\n\nFrom the user's perspective, calling X \"rerere\" would probably be OK[1].\nBut from an implementation perspective (and to keep the existing\nplumbing available and unchanged), it probably makes sense to call it\nsomething else, and have it run both rerere and a new plumbing command\nto do the merge-fix work (or call it nothing, and assume that users will\neither touch the plumbing directly or will use \"git merge\" to trigger\nboth).\n\nSo if you want to shut down immediately the idea that this would be\nbolted onto rerere, I can support that. There are ways of doing it that\nwould make sense to combine with rerere (like the \"tying fixups to\nconflict resolution\" sketch I gave in the other email), but I agree they\nwill end up fundamentally hacky (because of the exact \"you may not even\nhave textual conflicts\" I mentioned).\n\nThe only part I'd disagree with above is that this belongs to git-merge.\nI think it should be its own plumbing tool that merge calls alongside\nrerere. ;)\n\n>    - Then, it looks up the database to find the keys <A, B> where\n>      A is in X but not in Y, and B is not in X but in Y.\n>      These commits are cherry-picked and squashed into the result of\n>      the above.\n\nI think this is the crux of it. I mentioned in my other email that what\nwe really want is some way to say \"this is roughly the same merge\".\nThe Reintegrate script does it with the topic branch name and an\nimplicit \"merging up to an integration branch\".\n\nNot having thought too hard about it yet, this containing relationship\nseems like the right direction. I guess you'd do the lookup by computing\nthe merge-base M of <X,Y> (which we already know anyway), walking M..X\nand looking for any entries which mention those commits (in either A or\nB slots of the entry), and then similarly narrowing it according to\nM..Y.\n\nHrm. That doesn't quite work, though. Because if your <A,B> are the\nmerge, then merging a topic to next will get an \"A\" that is a merge\ncommit from next. But that commit will never end up in master. What's\ncausing the conflict is really some \"A\" that is in the history between\nthe merge base and \"A\" (but we don't know which).\n\nSo you'd almost have to do an intersection of the left side of \"$(git\nmerge-base A B)..A\" with what's in X and Y (with respect to their merge\nbase). Err, maybe vice versa. But the point is that we're looking for\noverlapping set unions, I think, not the presence of particular tips.\n\n> I said A and B in the above are branch names, but in the ideal\n> world, they can be commit object names (possibly in the middle of a\n> branch), as long as we can reliable update the database's keys every\n> time \"git rebase\" etc. rewrites commits.\n\nWhat if instead of commit hashes we used patch ids?\n\nThere's one trick there, which is that merges don't have a well-defined\ncommit id. We could use its actual commit id in that case. That would\nwork OK in practice for a workflow like git.git's, because the merge\ncommits are never rewritten. But it would fall down if people do mixed\nrebases and merges on their topic branches.\n\n> To populate the database, we'd need a reverse.\n> \n>  * When merging branch B into branch A (or the other way around) for\n>    the first time, \"git merge\" would do what it currently does.\n> \n>  * The user then concludes the merge to resolve *ONLY* the textual\n>    conflict, and commit the result.  It is important that no\n>    additional evil merge to correct for semantic conflicts is done\n>    in this step.  Note that if the auto-merge cleanly succeeds, this\n>    step may not even exist.\n> \n>  * Then the user makes another commit to correct for semantic\n>    conflicts (aka \"merge-fix\").\n\nI think it's asking a lot for users to handle the textual conflicts and\nsemantic ones separately. It would be nice if we could tell them apart\nautomatically (and I think we can based on what isn't part of the\nconflict resolution).\n\nThat still ends up with one giant \"fixup\" commit. But I don't know how\nelse you'd do it. I could make several commits, but we still don't know\nhow to attribute them to anything but the mass <A,B> merge. We don't\nknow which commits were responsible for which fixups (and I wouldn't\nwant to ask the user to figure it out), so the best we can do is apply\nthem all.\n\n-Peff\n"},{"id":"325111","messageId":"xmqqeft3u0u5.fsf@gitster.mtv.corp.google.com","threadId":"46468","inReplyTo":"20170725205843.bi6kyqjlzyodmxuq@sigill.intra.peff.net","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-26T07:14:10Z","receivedAt":"2017-07-26T07:14:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> From the user's perspective, calling X \"rerere\" would probably be OK[1].\n> But from an implementation perspective (and to keep the existing\n> plumbing available and unchanged), it probably makes sense to call it\n> something else, and have it run both rerere and a new plumbing command\n> to do the merge-fix work (or call it nothing, and assume that users will\n> either touch the plumbing directly or will use \"git merge\" to trigger\n> both).\n> ...\n> I think it should be its own plumbing tool that merge calls alongside\n> rerere. ;)\n\nAs long as we use the database keyed with <A,B> and take the merge\nbase into account, \"git am\" and \"git cherry-pick\" would not be able\nto use the merge-fix machinery, so in that sense, calling X \"rerere\"\nwould not be OK, but I agree with your general sentiment about the\nUI visible to the end users.  Just like \"rerere\" started with a\nsmall step to avoid automating things too much and then later became\nalmost invisible for normal cases because we managed to automate it\nreasonably well and integrate it into mergy operations, we may be\nable to do the same for merge-fix machinery.  My \"this belongs to\n'merge'\" is primarily coming from it---it might be reusable in other\nmergy operations with some fundamental changes, but I envision that\nthe primarly and only user of that X would initially be 'merge'.\n\n> Not having thought too hard about it yet, this containing relationship\n> seems like the right direction. I guess you'd do the lookup by computing\n> the merge-base M of <X,Y> (which we already know anyway), walking M..X\n> and looking for any entries which mention those commits (in either A or\n> B slots of the entry), and then similarly narrowing it according to\n> M..Y.\n\nYes, every time I look at the Reintegrate script, I try to think of\nan efficient implementation but do not find anything better than the\nleft-right walk over X...Y range, so that is my conclusion after\nhaving thought about it very hard as well ;-)\n\n> What if instead of commit hashes we used patch ids?\n\nNow you may be onto something.  While we aim at the ultimately automated\nideal that would catch the maximal cases, for the earlier 'xyzzy\nturns into frotz' example, the minimal cue to identify one side of\nthe pair that keys the \"change this new instance of xyzzy into\nfrotz, too\" merge-fix is a hunk like this:\n\n    -const char *xyzzy;\n    +const char *frotz;\n\nIt does not matter what other changes also appear in the same\ncommit, and my original \"branch name\" is way too broad, and my\nprevious \"ideal\" which is \"a single problematic commit\" is still\nbroader than necessary.  Well, patch-id identifies the entire change\ncontained in a single commit, so it is still too broad, but if we\ncan narrow it down to a single hunk like that, perhaps we can use it\nas one side of the key.\n\nAnd the other side of the key is naturally a hunk like this:\n\n    +\tprintf(\"%s\\n\", xyzzy);\n\ni.e. a new use of xyzzy appears where it didn't exist before.  And\nwhen we merge two branches, one of which has one half of the key\n(i.e. \"const char *xyzzy;\" turned into \"const char *frotz\"), and the\nother has the other half of the key (i.e. \"printf xyzzy\" is added),\nthen we'd apply a patch that tells us to do this:\n\n    -\tprintf(\"%s\\n\", xyzzy);\n    +\tprintf(\"%s\\n\", frotz);\n\ni.e. that patch would be the value in the database keyed by the pair\nof two previous hunks.\n\n> I think it's asking a lot for users to handle the textual conflicts and\n> semantic ones separately. It would be nice if we could tell them apart\n> automatically (and I think we can based on what isn't part of the\n> conflict resolution).\n\nAfter thinking about this a bit more, I realized that it is possible\nto mechanically sift a human generated resolution that has both\ntextual conflict resolution (which will be handled by \"rerere\") and\nsemantic one (which needs the merge-fix machinery) into two, without\nrequiring the user to make two separate commits.\n\nThe key trick is that \"rerere\" is capable of recording and replaying\nsemantic conflict resolution made to a file that happens to have\ntextual conflict just fine.  Because rerere database records the\npreimage (i.e. with conflict markers) and postimage (i.e. the final\nresolution for the file) as an entire file contents, it can do 3-way\nmerge for parts of the same file that is away from any conflicted\nregion.\n\nIf you tell contrib/rerere-train.sh to learn from \"pu^{/^Merge\nbranch 'bp/fsmonitor' into pu}\", you'll see what I mean.\n\n  $ git show \"pu^{/^Merge branch 'bp/fsmonitor' into pu}\" compat/bswap.h\n\nshows the result of such resolution.  Early part of the combined\ndiff shown by the above command comes from textual conflict\nresolution, but there is a new implementation of inline version of\nget_be64 that has become necessary but did not exist in either of\nthe branches getting merged, which is shown as an evil merge.\n\nSo the way to automatically sift an existing merge into textual\nconflict resolution (i.e. automatable with \"rerere\") and semantic\nevil merge-fix is to first try to recreate the merge and enumerate\nthe paths that get conflicts.  Then resolve these paths by grabbing\nthe resolved result for these paths that get textual conflicts out\nof the original merge.  That gives us the textual part (we can run\n\"git rerere\" at this point to store the resolution away to the\nrerere database---which is essentially what contrib/rerere-train\ndoes).\n\nThere will be difference between the result of the above and the\noriginal merge commit.  The paths that are different are all outside\nthese conflicted paths, and they are what we want in the second\ncommit, i.e. the semantic evil merge-fix, which will be the value\nfor the merge-fix database.\n\nSo that part is easily automatable.  \n\nIt is much harder to come up with a way to index into the merge-fix\ndatabase.  We can \"punt\" and use the same index scheme as the\nReintegrate script (i.e. key it with the name of the branch being\nmerged, which can be read from the original merge) as the initial\napproximation, and that would already be much better than what\nReintegrate script currently does, in that the recording part is\nmuch more automated (in the workflow I use the Reintegrate script,\nthe side that records a merge-fix initially is entirely manual).\n\n\n\n"},{"id":"325114","messageId":"xmqq60eftyfb.fsf@gitster.mtv.corp.google.com","threadId":"46468","inReplyTo":"20170725205843.bi6kyqjlzyodmxuq@sigill.intra.peff.net","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-26T08:06:16Z","receivedAt":"2017-07-26T08:06:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Hrm. That doesn't quite work, though. Because if your <A,B> are the\n> merge, then merging a topic to next will get an \"A\" that is a merge\n> commit from next. But that commit will never end up in master. What's\n> causing the conflict is really some \"A\" that is in the history between\n> the merge base and \"A\" (but we don't know which).\n\nThere may be a misunderstanding.  When I said the key <A,B> is a\npair of branch names, I didn't mean 'A' to be the name of an\nintegration branch (e.g. 'pu') and 'B' to be the name of a topic.\nRather, both 'A' and 'B' are the names of topic branches.  \n\nIOW, instead of having refs/merge-fix/sd/branch-copy that says \"I\nknow when I merge sd/branch-copy to pu or jch, there is a semantic\nconflict with some unnamed topic that is likely to be already in\nthere\", i.e. keying with only a single topic name, the ideal I\npresented would say 'sd/branch-copy and mh/packed-ref-store topics\nare both by themselves OK, but when merged together, the end result\nof textual merge needs to be further fixed up by cherry-picking this\nchange', by keying a change with a pair of topic names,\nsd/branch-copy (which introduces a new method in the ref backend\nvtable) and mh/packed-ref-store (which adds a new ref backend).  The\nlatter does not know the need for the new method, and the former\ndoes not know the need to implement its new method in a new backend,\nso a merge needs a trivial implementation of the new method added to\nthe new backend, which is what refs/merge-fix/sd/branch-copy does.\n\nAnd better yet, instead of A=sd/branch-copy B=mh/packed-ref-store,\nwe could point at the exact commit on each of these branches that\nintroduce the semantic conflict.  I would probably pick these two\n\n  A=52d59cc6 (\"branch: add a --copy (-c) option to go with --move (-m)\", 2017-06-18)\n  B=67be7c5a (\"packed-backend: new module for handling packed references\", 2017-06-23)\n\nso when we are on commit X that has A but not B, and are trying to\nmerge branch Y that has B but not A, we want the merge-fix to kick\nin.  Walking \"rev-list --left-right X...Y\" and noticing A and B in\nthe output would be a way to notice it.\n\n\n[footnote]\n\n*1* https://github.com/gitster/git/ should mirror these refs in the\n    refs/merge-fix/ hierarchymentioned in the body of this article.\n\n\n"},{"id":"325208","messageId":"xmqqfudhmw5x.fsf@gitster.mtv.corp.google.com","threadId":"46468","inReplyTo":"xmqqeft3u0u5.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] Git rerere and non-conflicting changes during conflict resolution","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-27T21:01:30Z","receivedAt":"2017-07-27T21:01:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Jeff King <peff@peff.net> writes:\n>\n>> From the user's perspective, calling X \"rerere\" would probably be OK[1].\n>> But from an implementation perspective (and to keep the existing\n>> plumbing available and unchanged), it probably makes sense to call it\n>> something else, and have it run both rerere and a new plumbing command\n>> to do the merge-fix work (or call it nothing, and assume that users will\n>> either touch the plumbing directly or will use \"git merge\" to trigger\n>> both).\n>> ...\n>> I think it should be its own plumbing tool that merge calls alongside\n>> rerere. ;)\n>\n> As long as we use the database keyed with <A,B> and take the merge\n> base into account, \"git am\" and \"git cherry-pick\" would not be able\n> to use the merge-fix machinery, so in that sense, calling X \"rerere\"\n> would not be OK, but I agree with your general sentiment about the\n> UI visible to the end users.\n\nActually, I guess \"cherry-pick\" could use it if we think hard and\nlong enough and come up with an ideal scheme to compute the index\ninto the merge-fix database.\n\nImagine this topology:\n\n       A---o---o---...        topic #1\n      /\n o---o---o---...              mainline\n      \\\n       o---B---o---C---...    topic #2\n\nwhere topic #1 renames 'xyzzy' to 'frotz' at commit A, and topic #2\nadds a new mention of 'xyzzy' in file F at commit B and another in\nfile E at commit C.\n\nIn the ideal world, we would have two merge-fix database entries,\none that turns 'xyzzy' in file F to 'frotz' that is keyed by the\npair of commits <A,B>, and the other that does the same in file E\nthat is keyed by <A,C>.  When merging the topic #1 and the topic #2\ntogether, or when merging the topic #2 to a mainline that already\nhas merged the topic #1, the merge-fix machinery notices that one\nside has A but not B nor C, and the other side has B and C but not\nA, and finds these two merge-fixes and applies on top of the textual\nmerge.\n\nIf we are cherry-picking C to something that already has A, then, we\nshould be able to notice that the history that receives the cherry-pick\nhas A but not C, and C, which is being picked, does not have A, and\ndecide that merge-fix <A,C> is relevant.\n\nIf we do this purely with commit object name, it will still not work\nif we cherry-pick A to mainline and then we cherry-pick C.  The\nmainline may hae change from A but does not have the exact commit A.\n\nWhich brings us back to your earlier idea to use something like\npatch-id to identify these individual changes.  I am not sure how we\ncan structure the merge-fix database so that we can efficiently find\nwhich \"changes\" are already on a branch.\n"}]}