{"thread":{"id":"47656","subject":"git merge-tree: bug report and some feature requests","startedAt":"2018-01-21T03:00:52Z","lastAt":"2018-01-24T19:39:36Z","messageCount":7,"participants":["Josh Bleecher Snyder","Ævar Arnfjörð Bjarmason","Elijah Newren","Edward Thomson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"337015","messageId":"CAFAcib-2fxiVxtVWcbvafY3-Br7Y70HMiHFZoT0VfK6JU0Dp9A@mail.gmail.com","threadId":"47656","inReplyTo":null,"subject":"git merge-tree: bug report and some feature requests","fromName":"Josh Bleecher Snyder","fromEmail":"josharian@gmail.com","sentAt":"2018-01-21T03:00:06Z","receivedAt":"2018-01-21T03:00:52Z","isPatch":false,"sender":{"key":"josharian@gmail.com","avatar":"https://gravatar.com/avatar/df58f649b62f4d66e2e635810212cb54403228630fabfa080eb5847a6f4cc01b?d=mp&s=160"},"body":"Hi, all.\n\nI'm experimenting with some new porcelain for interactive rebase. One\ngoal is to leave the work tree untouched for most operations. It looks\nto me like 'git merge-tree' may be the right plumbing command for\ndoing the merge part of the pick work of the todo list, one commit at\na time. If I'm wrong about this, I'd love pointers; what follows may\nstill be interesting anyway.\n\nI've encountered some bumps with 'git merge-tree'. A bug report and\nsome feature requests follow. Apologies for the long email.\n\n\n1. Bug\n\nWhen a binary file containing NUL is added on only one side, the\nresulting patch is malformed. Reproduction script:\n\nmkdir test\ncd test\ngit init .\ntouch shared\ngit add shared\ngit commit -m \"base\"\ngit checkout -b left\necho \"left\" > x\nprintf '\\1\\0\\1\\n' > binary\ngit add x binary\ngit commit -m \"left\"\ngit checkout master\ngit checkout -b right\necho \"right\" > x\ngit add x\ngit commit -m \"right\"\ngit merge-tree master right left\ngit merge-tree master right left | xxd\ncd ..\nrm -rf test\n\nThe merge-tree results I get with 2.15.1 are:\n\nadded in remote\n  their  100644 ddc50ce55647db1421b18aa33417442e29f63d2f binary\n@@ -0,0 +1 @@\n+added in both\n  our    100644 c376d892e8b105bd712d06ec5162b5f31ce949c3 x\n  their  100644 45cf141ba67d59203f02a54f03162f3fcef57830 x\n@@ -1 +1,5 @@\n+<<<<<<< .our\n right\n+=======\n+left\n+>>>>>>> .their\n\n00000000: 6164 6465 6420 696e 2072 656d 6f74 650a  added in remote.\n00000010: 2020 7468 6569 7220 2031 3030 3634 3420    their  100644\n00000020: 6464 6335 3063 6535 3536 3437 6462 3134  ddc50ce55647db14\n00000030: 3231 6231 3861 6133 3334 3137 3434 3265  21b18aa33417442e\n00000040: 3239 6636 3364 3266 2062 696e 6172 790a  29f63d2f binary.\n00000050: 4040 202d 302c 3020 2b31 2040 400a 2b01  @@ -0,0 +1 @@.+.\n00000060: 6164 6465 6420 696e 2062 6f74 680a 2020  added in both.\n00000070: 6f75 7220 2020 2031 3030 3634 3420 6333  our    100644 c3\n00000080: 3736 6438 3932 6538 6231 3035 6264 3731  76d892e8b105bd71\n00000090: 3264 3036 6563 3531 3632 6235 6633 3163  2d06ec5162b5f31c\n000000a0: 6539 3439 6333 2078 0a20 2074 6865 6972  e949c3 x.  their\n000000b0: 2020 3130 3036 3434 2034 3563 6631 3431    100644 45cf141\n000000c0: 6261 3637 6435 3932 3033 6630 3261 3534  ba67d59203f02a54\n000000d0: 6630 3331 3632 6633 6663 6566 3537 3833  f03162f3fcef5783\n000000e0: 3020 780a 4040 202d 3120 2b31 2c35 2040  0 x.@@ -1 +1,5 @\n000000f0: 400a 2b3c 3c3c 3c3c 3c3c 202e 6f75 720a  @.+<<<<<<< .our.\n00000100: 2072 6967 6874 0a2b 3d3d 3d3d 3d3d 3d0a   right.+=======.\n00000110: 2b6c 6566 740a 2b3e 3e3e 3e3e 3e3e 202e  +left.+>>>>>>> .\n00000120: 7468 6569 720a                           their.\n\nNote that the \"added in both\" explanation appears to be part of the\ndiff for binary. The diff line should be '\\1\\0\\1\\n', but it is only\n'\\1', obviously suggesting a C string operation gone awry.\n\nI haven't checked whether regular 'git diff' operations contain a\nsimilar bug. (The NUL would have to be pretty far into the file, to\nconfuse the binary file detection heuristic, but that is possible.)\n\nI don't see any particularly good work-arounds. Looking for all\npossible explanations as a trigger to detect a malformed patch runs\ninto false positives with the explanation \"merged\", which occurs in\nregular code.\n\n\n2. Feature suggestion\n\nRelated to the bug, may I suggest a flag to omit unnecessary patches?\nFor \"added in remote\" and \"deleted in remote\", I don't actually need\nthe patch--I can grab the blob contents from the SHA myself if needed.\nThese cases need special handling anyway (to create/delete the file),\nso the (often large) patch doesn't add much anyway. This would provide\na workaround for the bug.\n\n\n3. Feature suggestion\n\nThere's no direct indication of whether any given file's merge\nsucceeded. Currently I sniff for merge conflicts by looking for\n\"+<<<<<<< .our\", which feels like an ugly kludge. Could we provide an\nexplicit indicator? (And maybe also one for binary vs text\nprocessing?)\n\nNote that binary file merge conflicts don't generate patches with\nthree-way merge markers but instead say \"warning: Cannot merge binary\nfiles: binary (.our vs. .their)\". Looking for this case even further\ncomplicates the output parser.\n\n\n4. API suggestion\n\nHere's what I really want 'git merge-tree' to output. :)\n\nIf the merge had no conflicts, write the resulting tree to the object\ndatabase and give me its sha. I can always diff that tree against\nbranch1's tree if I want to see what has changed.\n\nIf the merge had conflicts, write the \"as merged as possible\" tree to\nthe object database and give me its sha, and then also give me the\nthree-way merge diff output for all conflicts, as a regular patch\nagainst that tree, using full path names and shas. (Alternatively,\nmaybe better, give me a second sha for a tree containing all the\nthree-way merge diff patches applied, which I can diff against the\nfirst tree to find the conflict patches.)\n\nI'm not sure what to do about binary merge conflicts, since they\naren't representable with three-way markers. Maybe just list them,\nwith the two conflicting blob ids?\n\n\n\nThanks very much,\n-josh\n"},{"id":"337022","messageId":"87607viq6r.fsf@evledraar.gmail.com","threadId":"47656","inReplyTo":"CAFAcib-2fxiVxtVWcbvafY3-Br7Y70HMiHFZoT0VfK6JU0Dp9A@mail.gmail.com","subject":"Re: git merge-tree: bug report and some feature requests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-01-21T08:14:36Z","receivedAt":"2018-01-21T08:14:43Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jan 21 2018, Josh Bleecher Snyder jotted:\n\n> 3. Feature suggestion\n>\n> There's no direct indication of whether any given file's merge\n> succeeded. Currently I sniff for merge conflicts by looking for\n> \"+<<<<<<< .our\", which feels like an ugly kludge. Could we provide an\n> explicit indicator? (And maybe also one for binary vs text\n> processing?)\n>\n> Note that binary file merge conflicts don't generate patches with\n> three-way merge markers but instead say \"warning: Cannot merge binary\n> files: binary (.our vs. .their)\". Looking for this case even further\n> complicates the output parser.\n\nI thought I had a way to do this, but looking back in my logs I find\nthat I was just using:\n\n    git merge-tree $(git merge-base A B) A B | grep -e '^\\+======='; echo $?\n\nI.e. a variation of what you're doing, which as you note won't work for\nbinary files.\n"},{"id":"337094","messageId":"CABPp-BGYs9jo16OZR8NsL-eO9LwEttBxBspvp1-_JjuD2oBYbA@mail.gmail.com","threadId":"47656","inReplyTo":"CAFAcib-2fxiVxtVWcbvafY3-Br7Y70HMiHFZoT0VfK6JU0Dp9A@mail.gmail.com","subject":"Re: git merge-tree: bug report and some feature requests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2018-01-22T17:58:04Z","receivedAt":"2018-01-22T17:58:10Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Jan 20, 2018 at 7:00 PM, Josh Bleecher Snyder\n<josharian@gmail.com> wrote:\n> Hi, all.\n>\n> I'm experimenting with some new porcelain for interactive rebase. One\n> goal is to leave the work tree untouched for most operations. It looks\n> to me like 'git merge-tree' may be the right plumbing command for\n> doing the merge part of the pick work of the todo list, one commit at\n> a time. If I'm wrong about this, I'd love pointers; what follows may\n> still be interesting anyway.\n\nI don't have a concrete alternative (yet?) but here are some pointers\nto two alternate merge-without-touching-working-tree possibilities, if\nyour current route doesn't pan out as well as you like:\n\nI posted some patches last year to make merge-recursive.c be able to\ndo merges without touching the working tree.  Adding a few flags would\nthen enable it for any of 'merge', 'cherry-pick', 'am', or\n'rebase'...though for unsuccessful merges, there's a clear question of\nwhat/how conflicts should be reported to the user.  That probably\ndepends a fair amount on the precise use-case.\n\nAlthough that series was placed on the backburner due to the immediate\ndriver of the feature going away, I'm still interested in such a\nchange, though I think it would fall out as a nice side effect of\nimplementing Junio's proposed ideal-world-merge-recursive rewrite[1].\nI have started looking into that[2], but no guarantees about how\nquickly I'll find time to finish or even whether I will.\n\n[1] https://public-inbox.org/git/xmqqd147kpdm.fsf@gitster.mtv.corp.google.com\n[2] https://github.com/newren/git/blob/ort/ort-cover-letter contains\noverview of ideas and notes to myself about what I was hoping to\naccomplish; currently it doesn't even compile or do anything\n\n> 4. API suggestion\n>\n> Here's what I really want 'git merge-tree' to output. :)\n...\n> If the merge had conflicts, write the \"as merged as possible\" tree to\n\nYou'd need to define \"as merged as possible\" more carefully, because I\nthought you meant a tree containing all the three-way merge conflict\nmarkers and such being present in the \"resolved\" file, but from your\nparenthetical note below it appears you think that is a different tree\nthat would also be useful to diff against the first one.  That leaves\nme wondering what the first tree is. (Is it just the tree where for\neach path, if that path had no conflicts associated with it then it's\nthe merge-resolved-file, and otherwise it's the file contents from the\nmerge-base?).\n\nBoth of these trees are actually rather non-trivial to define.  The\nwording above isn't actually sufficient, because content conflicts\naren't the only kind of conflict.  More on that below.\n\nThere is already a bunch of code in merge-recursive.c to create a\nforcibly-merged-accepting-conflict-markers-in-the-resolution and\nrecord it as a tree (this is used for creating virtual merge bases in\nthe recursive case, namely when there isn't a single merge-base for\nthe two branches you are merging).  It might be reusable for what you\nwant here, but it's not immediately clear whether all the things it\ndoes are appropriate; someone would have to consider the non-content\n(path-based) conflicts carefully.\n\n> the object database and give me its sha, and then also give me the\n> three-way merge diff output for all conflicts, as a regular patch\n> against that tree, using full path names and shas. (Alternatively,\n> maybe better, give me a second sha for a tree containing all the\n> three-way merge diff patches applied, which I can diff against the\n> first tree to find the conflict patches.)\n\nAs far as I can tell, you're assuming that it's possible with two\ntrees that are crafted \"just right\", that you can tell where the merge\nconflicts are, with binary files being your only difficulty.  Content\nconflicts aren't the only type that exist; there are also path-based\nconflicts.  These type of conflicts also make it difficult to know how\nthe two trees you are requesting should even be created.\n\nFor example, if there is a modify/delete conflict, how can that be\ndetermined from just two trees?  If the first tree has the base\nversion of the file, then the second tree either has a file at the\nsame position or it doesn't.  Neither case looks like a conflict, but\nthe original merge had one.  You need more information.  The exact\nsame thing can be said for rename/delete conflicts.\n\nSimilarly, rename/add (one side renames an existing file to some new\npath (say, \"new_path\"), and the other adds a brand new file at\n\"new_path), or rename/rename(2to1) (each side renames a different file\nto the same location), won't be detectable just by diffing two trees.\nThese are often handled by moving both files to some other location,\nso there's no way to record in a tree that there was a conflict.\n\nrename/rename(1to2) is similar, but instead of two different original\nfiles being renamed to the same thing, this is one file being renamed\ndifferently on different sides of history.\n\nI know that several of the examples above involved rename detection,\nwhich git-merge-trees won't even do, but that means you're even more\nlikely to face the modify/delete conflict cases.  And our list still\nisn't done, either:\n\nDirectory/file conflicts (one side puts a directory of the same name\nthat the other side adds as a file) will also cause problems.\n\nFinally, directory rename detection (currently in pu under review)\nadds a few \"implicit dir rename\" conflict types (renames of multiple\ndirectories would cause multiple files to be renamed to the same\nlocation, or an existing file/dir being in the way of one or more\npath(s) getting implicitly renamed).  This means that the number of\ntypes of non-textual conflicts might also grow in the future so it may\nbe unwise to try to special case existing exceptions with a bag of\nclever workarounds.\n\n\nHope that helps,\nElijah\n"},{"id":"337147","messageId":"CAFAcib9FGfTe8C7TaOY91kzhUvxXtpx5Y8JMFkqxhRhLJeytxg@mail.gmail.com","threadId":"47656","inReplyTo":"CABPp-BGYs9jo16OZR8NsL-eO9LwEttBxBspvp1-_JjuD2oBYbA@mail.gmail.com","subject":"Re: git merge-tree: bug report and some feature requests","fromName":"Josh Bleecher Snyder","fromEmail":"josharian@gmail.com","sentAt":"2018-01-23T07:08:52Z","receivedAt":"2018-01-23T07:09:39Z","isPatch":false,"sender":{"key":"josharian@gmail.com","avatar":"https://gravatar.com/avatar/df58f649b62f4d66e2e635810212cb54403228630fabfa080eb5847a6f4cc01b?d=mp&s=160"},"body":">> I'm experimenting with some new porcelain for interactive rebase. One\n>> goal is to leave the work tree untouched for most operations. It looks\n>> to me like 'git merge-tree' may be the right plumbing command for\n>> doing the merge part of the pick work of the todo list, one commit at\n>> a time. If I'm wrong about this, I'd love pointers; what follows may\n>> still be interesting anyway.\n>\n> I don't have a concrete alternative (yet?) but here are some pointers\n> to two alternate merge-without-touching-working-tree possibilities, if\n> your current route doesn't pan out as well as you like:\n>\n> I posted some patches last year to make merge-recursive.c be able to\n> do merges without touching the working tree.  Adding a few flags would\n> then enable it for any of 'merge', 'cherry-pick', 'am', or\n> 'rebase'...though for unsuccessful merges, there's a clear question of\n> what/how conflicts should be reported to the user.  That probably\n> depends a fair amount on the precise use-case.\n>\n> Although that series was placed on the backburner due to the immediate\n> driver of the feature going away, I'm still interested in such a\n> change, though I think it would fall out as a nice side effect of\n> implementing Junio's proposed ideal-world-merge-recursive rewrite[1].\n> I have started looking into that[2], but no guarantees about how\n> quickly I'll find time to finish or even whether I will.\n>\n> [1] https://public-inbox.org/git/xmqqd147kpdm.fsf@gitster.mtv.corp.google.com\n> [2] https://github.com/newren/git/blob/ort/ort-cover-letter contains\n> overview of ideas and notes to myself about what I was hoping to\n> accomplish; currently it doesn't even compile or do anything\n\nThanks for the pointer. That does seem promising.\n\nAnd yes, I see now that serialization of conflicts is decidedly\nchallenging. More on that below.\n\n\n>> 4. API suggestion\n>>\n>> Here's what I really want 'git merge-tree' to output. :)\n> ...\n>> If the merge had conflicts, write the \"as merged as possible\" tree to\n>\n> You'd need to define \"as merged as possible\" more carefully, because I\n> thought you meant a tree containing all the three-way merge conflict\n> markers and such being present in the \"resolved\" file, but from your\n> parenthetical note below it appears you think that is a different tree\n> that would also be useful to diff against the first one.  That leaves\n> me wondering what the first tree is. (Is it just the tree where for\n> each path, if that path had no conflicts associated with it then it's\n> the merge-resolved-file, and otherwise it's the file contents from the\n> merge-base?).\n\nFWIW, the parenthetical suggestion was indeed what I had in mind. But\nnon-content conflicts appear to make that a non-starter. Or at least\nwoefully incomplete.\n\n\n> Both of these trees are actually rather non-trivial to define.  The\n> wording above isn't actually sufficient, because content conflicts\n> aren't the only kind of conflict.  More on that below.\n>\n> There is already a bunch of code in merge-recursive.c to create a\n> forcibly-merged-accepting-conflict-markers-in-the-resolution and\n> record it as a tree (this is used for creating virtual merge bases in\n> the recursive case, namely when there isn't a single merge-base for\n> the two branches you are merging).  It might be reusable for what you\n> want here, but it's not immediately clear whether all the things it\n> does are appropriate; someone would have to consider the non-content\n> (path-based) conflicts carefully.\n\nAck. I assume this is also the code that generates the existing 'git\nmerge-tree' patches, which includes conflict markers.\n\n\n>> the object database and give me its sha, and then also give me the\n>> three-way merge diff output for all conflicts, as a regular patch\n>> against that tree, using full path names and shas. (Alternatively,\n>> maybe better, give me a second sha for a tree containing all the\n>> three-way merge diff patches applied, which I can diff against the\n>> first tree to find the conflict patches.)\n>\n> As far as I can tell, you're assuming that it's possible with two\n> trees that are crafted \"just right\", that you can tell where the merge\n> conflicts are, with binary files being your only difficulty.  Content\n> conflicts aren't the only type that exist; there are also path-based\n> conflicts.  These type of conflicts also make it difficult to know how\n> the two trees you are requesting should even be created.\n>\n> For example, if there is a modify/delete conflict, how can that be\n> determined from just two trees?  If the first tree has the base\n> version of the file, then the second tree either has a file at the\n> same position or it doesn't.  Neither case looks like a conflict, but\n> the original merge had one.  You need more information.  The exact\n> same thing can be said for rename/delete conflicts.\n>\n> Similarly, rename/add (one side renames an existing file to some new\n> path (say, \"new_path\"), and the other adds a brand new file at\n> \"new_path), or rename/rename(2to1) (each side renames a different file\n> to the same location), won't be detectable just by diffing two trees.\n> These are often handled by moving both files to some other location,\n> so there's no way to record in a tree that there was a conflict.\n>\n> rename/rename(1to2) is similar, but instead of two different original\n> files being renamed to the same thing, this is one file being renamed\n> differently on different sides of history.\n>\n> I know that several of the examples above involved rename detection,\n> which git-merge-trees won't even do, but that means you're even more\n> likely to face the modify/delete conflict cases.  And our list still\n> isn't done, either:\n>\n> Directory/file conflicts (one side puts a directory of the same name\n> that the other side adds as a file) will also cause problems.\n\nMe: If the world were simple, we could build it this simple way!\n\nYou: The world isn't simple.\n\nMe: ...drat. Thanks.\n\n\nI've been looking at libgit2's handling of this. It appears the\nclosest analog is:\n\n* Call git_merge_trees:\nhttps://libgit2.github.com/libgit2/#HEAD/group/merge/git_merge_trees\n* Call git_index_conflict_iterator_new on the resulting *git_index:\nhttps://libgit2.github.com/libgit2/#HEAD/group/index/git_index_conflict_iterator_new\n* Use git_index_conflict_next to inspect a conflict:\nhttps://libgit2.github.com/libgit2/#HEAD/group/index/git_index_conflict_next\n* The conflict provides three git_index_entrys:\nhttps://libgit2.github.com/libgit2/#HEAD/type/git_index_entry\n\nLooking over your list above, at a minimum, libgit2 might not have a\nparticularly good way to represent submodule/file or\nsubmodule/directory conflicts, because is-a-submodule is defined\nexternal to a git_index_entry.\n\n(Incidentally, looking at\nhttps://github.com/git/git/blob/master/Documentation/technical/index-format.txt,\nit appears that there are also possibly symlink/gitlink X\nfile/dir/submodule conflicts? Ugh.)\n\nlibgit2 gets to avoid the bother of serialization and deserialization,\nbut it seems the problem of conflict categorization still remains. See\nmy next comments.\n\n\n\n> Finally, directory rename detection (currently in pu under review)\n> adds a few \"implicit dir rename\" conflict types (renames of multiple\n> directories would cause multiple files to be renamed to the same\n> location, or an existing file/dir being in the way of one or more\n> path(s) getting implicitly renamed).  This means that the number of\n> types of non-textual conflicts might also grow in the future so it may\n> be unwise to try to special case existing exceptions with a bag of\n> clever workarounds.\n\nThere's tension here.\n\nCataloging or special-casing all possible conflict types does seem\nunwise because of the sheer number of kinds of conflicts.\n\nBut the alternative appears to be punting entirely, as libgit2 does,\nand merely providing something akin to three index entries. This which\nleaves it unclear what exactly the conflict was, at which point any\nuser (read: porcelain developer) will end up having to recreate some\nmerge logic to figure out what went wrong. And if merge-tree starts\ndoing rename detection, the user might then have to emulate that as\nwell. Given that, you may as well catalog the many kinds of conflicts\nand report which one occurred. And if there were a list of all\npossible kinds of conflicts, it'd help the user write correct code,\nbecause they're not going to naively fail to consider--as I did--the\nmany kinds of non-content-based conflicts.\n\n\nI admit, I'm somewhat puzzled about where to go from here, both from\ngit/libgit2's perspective and the perspective of my immediate needs.\nFor myself, I probably have the luxury of bailing on anything but the\nsimplest, content-based conflicts and sending the (end) user back to\nregular interactive rebase, if I can find a way to implement that\nreliably and without inordinate difficulty. The first case I checked\ndoesn't look good: For dir/file merge conflicts, 'git merge-tree'\nmerely reports some \"added in remote\" and \"added in local\"\nexplanations, meaning I'd have to add path conflict detection myself.\n\nI'll continue to ponder. Thanks for the enlightening email.\n\n-josh\n\n\nP.S. Is it expected/known that 'git merge --abort' of a\nmerge-in-progress involving a dir/file conflict generates a mildly\nincomprehensible error in addition to aborting the merge?\n\n$ git merge --abort\nerror: 'df' appears as both a file and as a directory\nerror: df: cannot drop to stage #0\n"},{"id":"337155","messageId":"CA+WKDT3ZsObXc2ENV3gUXbMZU=McBJdiEpvk7y2OM0t+PrqZtQ@mail.gmail.com","threadId":"47656","inReplyTo":"CAFAcib9FGfTe8C7TaOY91kzhUvxXtpx5Y8JMFkqxhRhLJeytxg@mail.gmail.com","subject":"Re: git merge-tree: bug report and some feature requests","fromName":"Edward Thomson","fromEmail":"ethomson@edwardthomson.com","sentAt":"2018-01-23T11:52:06Z","receivedAt":"2018-01-23T11:52:14Z","isPatch":false,"sender":{"key":"ethomson@edwardthomson.com","avatar":"https://avatars.githubusercontent.com/u/1130014?v=4"},"body":"On Tue, Jan 23, 2018 at 7:08 AM, Josh Bleecher Snyder\n<josharian@gmail.com> wrote:\n> Looking over your list above, at a minimum, libgit2 might not have a\n> particularly good way to represent submodule/file or\n> submodule/directory conflicts, because is-a-submodule is defined\n> external to a git_index_entry.\n\nlibgit2 should detect submodule/file and submodule/directory\nconflicts.\n\nWhile, yes, some metadata about the submodule is located outside the\nindex, you can look at the mode to determine that this _is_ a\nsubmodule.  You should be able to reliably detect submodule/file\nconflicts because one will be a regular or executable file (mode\n0100644 or 0100755), while the other entry at the same path will\nbe a gitlink (mode 0160000).\n\nSimilarly, submodule/directory conflict detection works just like\nregular file/directory conflict detection.  If some path `foo` is\na submodule (or a regular file) then some path `foo/bar` existing\nin the other side of the merge causes a conflict.\n\n> Cataloging or special-casing all possible conflict types does seem\n> unwise because of the sheer number of kinds of conflicts.\n>\n> But the alternative appears to be punting entirely, as libgit2 does,\n> and merely providing something akin to three index entries.\n\nIndeed, when I added merge to libgit2, we put the higher-level conflict\nanalysis into application code because there was not much interest in it\nat the time.  I've been meaning to add this to `git_status` in libgit2,\nbut it's not been a high priority.\n\n> This which\n> leaves it unclear what exactly the conflict was, at which point any\n> user (read: porcelain developer) will end up having to recreate some\n> merge logic to figure out what went wrong. And if merge-tree starts\n> doing rename detection, the user might then have to emulate that as\n> well.\n\nThat's not a good idea.  Trying to figure out what merge did would be\npainful at best, and likely impossible, since a rename conflict is\nrecorded in the main index without any way to piece it together.  eg:\n\n100644 deadbeefdeadbeefdeadbeefdeadbeefdeadbeef 1       bar.c\n100644 cafec4fecafec4fecafec4fecafec4fecafec4fe 2       bar.c\n100644 c4cc188a892898e13927dc4a02e7f68814b874b2 1       foo.c\n100644 71f5af150b25e3aaaad2d67ff46759311401036f 2       foo.c\n100644 351cfbdd55d656edd2c5c995aae3caafb9ec11fa 3       rename1.c\n100644 e407c7d138fb457674c3b114fcf47748169ab0c5 3       rename2.c\n\nThis is the main index that results when bar.c has a rename/edit\nconflict, and foo.c also has a rename/edit conflict.  One was renamed\nto rename1.c and the other to rename2.c.  Trying to determine which is\nwhich _after the fact_ would be regrettable.  Especially since rename\ndetection is not static - you would need to know the thresholds that\nwere configured at the time merge was performed and try to replay\nthe rename detection with that.\n\nlibgit2 records a `NAME` section in the index that pairs the rename\ndetection decisions that it performed so that you can analyze them\nand display them after the fact.\n\n-ed\n"},{"id":"337281","messageId":"CAFAcib_MysZPZOJ5xTWrh8nLk1Tn3Vi4LbscxgBO-+02S9Wvkw@mail.gmail.com","threadId":"47656","inReplyTo":"CA+WKDT3ZsObXc2ENV3gUXbMZU=McBJdiEpvk7y2OM0t+PrqZtQ@mail.gmail.com","subject":"Re: git merge-tree: bug report and some feature requests","fromName":"Josh Bleecher Snyder","fromEmail":"josharian@gmail.com","sentAt":"2018-01-24T18:46:44Z","receivedAt":"2018-01-24T18:47:33Z","isPatch":false,"sender":{"key":"josharian@gmail.com","avatar":"https://gravatar.com/avatar/df58f649b62f4d66e2e635810212cb54403228630fabfa080eb5847a6f4cc01b?d=mp&s=160"},"body":"Thanks, Ed. I think I'll pursue the libgit2 route; sounds promising.\n\n\n>> But the alternative appears to be punting entirely, as libgit2 does,\n>> and merely providing something akin to three index entries.\n>\n> Indeed, when I added merge to libgit2, we put the higher-level conflict\n> analysis into application code because there was not much interest in it\n> at the time.  I've been meaning to add this to `git_status` in libgit2,\n> but it's not been a high priority.\n\nIs your conflict analysis application code public? I might be game to\ndo some of the legwork to get it into libgit2's git_status (although\nI'm probably not the right person to do the API design). At a minimum,\nit would be helpful as a reference, as I'm probably about to recreate\nsome subset of it myself.\n\n\n-josh\n"},{"id":"337288","messageId":"CABPp-BGLt4FEMZYpy5YtimrDDi7MHstfciucUPrnO5HzcOuBTA@mail.gmail.com","threadId":"47656","inReplyTo":"CA+WKDT3ZsObXc2ENV3gUXbMZU=McBJdiEpvk7y2OM0t+PrqZtQ@mail.gmail.com","subject":"Re: git merge-tree: bug report and some feature requests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2018-01-24T19:39:28Z","receivedAt":"2018-01-24T19:39:36Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 23, 2018 at 3:52 AM, Edward Thomson\n<ethomson@edwardthomson.com> wrote:\n> Indeed, when I added merge to libgit2, we put the higher-level conflict\n> analysis into application code because there was not much interest in it\n> at the time.  I've been meaning to add this to `git_status` in libgit2,\n> but it's not been a high priority.\n\nThat sounds pretty cool.  I'm actually dealing with a similar problem\nin git itself with my replacement merge strategy; since I try to\nperform the merge in-core, and then only try to update the index and\nworking copy if there'd be no loss of information (no overwriting of\ndirty or untracked content), there's a delay between when a conflict\nis detected and when I can report it, meaning that I have to store\ninformation about the conflict somehow, and then report later.  So\nthat means I have to go through the trouble of storing this\ninformation internally, which naturally raises the question of whether\nI want to provide this information to the user in some fashion other\nthan just simple \"CONFLICT: ...\" messages printed out to the user.\n\n>> This which\n>> leaves it unclear what exactly the conflict was, at which point any\n>> user (read: porcelain developer) will end up having to recreate some\n>> merge logic to figure out what went wrong. And if merge-tree starts\n>> doing rename detection, the user might then have to emulate that as\n>> well.\n>\n> That's not a good idea.  Trying to figure out what merge did would be\n> painful at best, and likely impossible, since a rename conflict is\n> recorded in the main index without any way to piece it together.  eg:\n>\n> 100644 deadbeefdeadbeefdeadbeefdeadbeefdeadbeef 1       bar.c\n> 100644 cafec4fecafec4fecafec4fecafec4fecafec4fe 2       bar.c\n> 100644 c4cc188a892898e13927dc4a02e7f68814b874b2 1       foo.c\n> 100644 71f5af150b25e3aaaad2d67ff46759311401036f 2       foo.c\n> 100644 351cfbdd55d656edd2c5c995aae3caafb9ec11fa 3       rename1.c\n> 100644 e407c7d138fb457674c3b114fcf47748169ab0c5 3       rename2.c\n>\n> This is the main index that results when bar.c has a rename/edit\n> conflict, and foo.c also has a rename/edit conflict.  One was renamed\n> to rename1.c and the other to rename2.c.  Trying to determine which is\n> which _after the fact_ would be regrettable.  Especially since rename\n> detection is not static - you would need to know the thresholds that\n> were configured at the time merge was performed and try to replay\n> the rename detection with that.\n>\n> libgit2 records a `NAME` section in the index that pairs the rename\n> detection decisions that it performed so that you can analyze them\n> and display them after the fact.\n\nOoh, I like that idea.  Could we do the same in git itself?  Having\nnot messed with the index format at all, what commands do users use to\nview this supplementary info?\n\nAlso, if you have this NAME section, you could make other commands\nsmarter.  For example, if you had a normal content conflict on some\nfile without rename detection being involved, you'd expect the index\nto look something like\n\n100644 deadbeefdeadbeefdeadbeefdeadbeefdeadbeef 1       file.c\n100644 cafec4fecafec4fecafec4fecafec4fecafec4fe 2       file.c\n100644 ba5eba11ba5eba11ba5eba11ba5eba11ba5eba11 3       file.c\n\nand then when you do a 'git add file.c' you expect all the higher\nstages to go away, i.e.\n\n100644 5ca1ab1e5ca1ab1e5ca1ab1e5ca1ab1e5ca1ab1e 0       file.c\n\nWith your NAME section trick, could one expect 'git add rename1.c' to\ndelete not only the higher stage for rename1.c but also the two for\nwhichever of bar.c or foo.c is relevant?\n\nElijah\n"}]}