{"thread":{"id":"57015","subject":"Potential bug with octopus merges and symlinks","startedAt":"2021-12-02T03:05:08Z","lastAt":"2021-12-06T18:13:38Z","messageCount":4,"participants":["Michael McClimon","Elijah Newren","Martin von Zweigbergk","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"442844","messageId":"20211202030458.GA48278@newk","threadId":"57015","inReplyTo":null,"subject":"Potential bug with octopus merges and symlinks","fromName":"Michael McClimon","fromEmail":"michael@mcclimon.org","sentAt":"2021-12-02T03:04:58Z","receivedAt":"2021-12-02T03:05:08Z","isPatch":false,"sender":{"key":"michael@mcclimon.org","avatar":"https://avatars.githubusercontent.com/u/1698723?v=4"},"body":"I am having a problem with octopus merges when symbolic links are involved.\nI'm not completely convinced this is a bug in git, but I'm also not convinced\nthat it's _not_. (If it's not, my apologies; I often find it very hard to\ntrack down the answers to complicated git questions on the internet.)\n\nThere is a minimal reproducer available at\nhttps://github.com/mmcclimon/git-merge-problem-demo. Fetch all the branches\nthere. The main branch contains a directory (dir1) with a single file\n(file.txt), plus a symlink (dir2), which links to dir1. branch1 replaces this\nsymlink with a copy of the files that were linked to. (This was accomplished\nwith: rm dir2; cp -r dir1 dir2.) branch2 and branch3 do not touch this\ndirectory at all.\n\nMerging these three branches fails:\n\n$ git merge branch1 branch2 branch3\nFast-forwarding to: branch1\nTrying simple merge with branch2\nSimple merge did not work, trying automatic merge.\nTrying simple merge with branch3\nerror: Entry 'dir2/file.txt' not uptodate. Cannot merge.\nMerge with strategy octopus failed.\n\nThe order here matters! Here is every permutation (1 here is the symlink\nchange) to git merge; only the first two fail, all the others work.\n\n1 2 3   FAIL\n1 3 2   FAIL\n2 1 3   PASS\n2 3 1   PASS\n3 1 2   PASS\n3 2 1   PASS\n1 2     PASS\n2 1     PASS\n2 3     PASS\n3 2     PASS\n1 3     PASS\n3 1     PASS\n\nI tracked this down as best I could (I am not a C programmer), by adding \n`set -x` to my copy of git-octopus-merge.sh. I determined that the line is\nfailing is toward the very bottom of the main loop there: \n    git read-tree -u -m --aggressive  $common $MRT $SHA1\n\nI had a read of the index-format.txt and then the git read-tree docs, and I\nconfess that I don't _totally_ follow the latter. But, I wonder if maybe\n--aggressive is perhaps not aggressive enough if the parts of the index that\nstore symbolic link data aren't taken into account, when it's changed between\ntwo versions and the blobs involved aren't actually different.\n\nI recognize that replacing a symlink with real files under the same name is a\nweird thing to do, but this is a pared-down example of a thing that happens\noccasionally at $work, where we regularly octopus-merge 15+ heads into\ndevelopment branches. I would also be happy to know, if this isn't a bug, if\nthere's some other workaround! The only way I know to fix this case is to\nmerge the symlink change, then rebase all in-flight branches against it. That\nis often extremely time-consuming and tedious, though, and requires a human to\nbe involved (all the other merging is done by CI.)\n\nThanks!\nMichael\n\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n[System Info]\ngit version:\ngit version 2.34.1\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64\ncompiler info: clang: 13.0.0 (clang-1300.0.29.3)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/zsh\n\n\n[Enabled Hooks]\n\n\n-- \nMichael McClimon\nmichael@mcclimon.org\n\n"},{"id":"443079","messageId":"CABPp-BErrT9pHMz-G+E1fQTPueaySFWXf69UQ+yJ_H1RTU9LNQ@mail.gmail.com","threadId":"57015","inReplyTo":"20211202030458.GA48278@newk","subject":"Re: Potential bug with octopus merges and symlinks","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-04T17:34:15Z","receivedAt":"2021-12-04T17:34:30Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Dec 4, 2021 at 5:47 AM Michael McClimon <michael@mcclimon.org> wrote:\n>\n> I am having a problem with octopus merges when symbolic links are involved.\n> I'm not completely convinced this is a bug in git, but I'm also not convinced\n> that it's _not_. (If it's not, my apologies; I often find it very hard to\n> track down the answers to complicated git questions on the internet.)\n>\n> There is a minimal reproducer available at\n> https://github.com/mmcclimon/git-merge-problem-demo. Fetch all the branches\n> there. The main branch contains a directory (dir1) with a single file\n> (file.txt), plus a symlink (dir2), which links to dir1. branch1 replaces this\n> symlink with a copy of the files that were linked to. (This was accomplished\n> with: rm dir2; cp -r dir1 dir2.) branch2 and branch3 do not touch this\n> directory at all.\n>\n> Merging these three branches fails:\n>\n> $ git merge branch1 branch2 branch3\n> Fast-forwarding to: branch1\n> Trying simple merge with branch2\n> Simple merge did not work, trying automatic merge.\n> Trying simple merge with branch3\n> error: Entry 'dir2/file.txt' not uptodate. Cannot merge.\n> Merge with strategy octopus failed.\n>\n> The order here matters! Here is every permutation (1 here is the symlink\n> change) to git merge; only the first two fail, all the others work.\n>\n> 1 2 3   FAIL\n> 1 3 2   FAIL\n> 2 1 3   PASS\n> 2 3 1   PASS\n> 3 1 2   PASS\n> 3 2 1   PASS\n> 1 2     PASS\n> 2 1     PASS\n> 2 3     PASS\n> 3 2     PASS\n> 1 3     PASS\n> 3 1     PASS\n>\n> I tracked this down as best I could (I am not a C programmer), by adding\n> `set -x` to my copy of git-octopus-merge.sh. I determined that the line is\n> failing is toward the very bottom of the main loop there:\n>     git read-tree -u -m --aggressive  $common $MRT $SHA1\n>\n> I had a read of the index-format.txt and then the git read-tree docs, and I\n> confess that I don't _totally_ follow the latter. But, I wonder if maybe\n> --aggressive is perhaps not aggressive enough if the parts of the index that\n> store symbolic link data aren't taken into account, when it's changed between\n> two versions and the blobs involved aren't actually different.\n>\n> I recognize that replacing a symlink with real files under the same name is a\n> weird thing to do, but this is a pared-down example of a thing that happens\n> occasionally at $work, where we regularly octopus-merge 15+ heads into\n> development branches. I would also be happy to know, if this isn't a bug, if\n> there's some other workaround! The only way I know to fix this case is to\n> merge the symlink change, then rebase all in-flight branches against it. That\n> is often extremely time-consuming and tedious, though, and requires a human to\n> be involved (all the other merging is done by CI.)\n\nYeah, I'm not too surprised that there are cases like this that trip up\nread-tree and octopus.  There might be a fix in unpack-trees.c, but\nit'd take some digging to verify if that's true or it's just a design\nlimitation.\n\nHowever, I can give you a workaround, based on my nebulous ideas for\nmaking an octopus-based-on-ort strategy (oort?).  Basically, just do a\nsequence of individual merges with ort (which since v2.34 is the default\nnon-octopus merge strategy), then take the resulting tree and recraft it\ninto an octopus merge.  So, something like this:\n\n  git merge --no-ff --no-edit branch1\n  git merge --no-ff --no-edit branch2\n  git merge --no-ff --no-edit branch3\n\n  NEW_TREE=$(git write-tree)\n\n  NEW_COMMIT=$(git commit-tree $NEW_TREE \\\n                   -p HEAD~3 -p HEAD~2^2 -p HEAD~1^2 -p HEAD^2 \\\n                   -m \"Merge branches 'branch1', 'branch2', and 'branch3'\")\n\n  git reset $NEW_COMMIT\n\nWhich at least works with the git-merge-problem-demo repository example\nyou provided.\n\nNote that the reason for the flags above are:\n  --no-ff: ensure merge commits are created, so our -p flags to\n           commit-tree are correct\n  --no-edit: we'll later replace all the commit messages anyway\n\nAlso, if you're using an older version of git, the recursive strategy\nwould be fine too for the steps above; the main advantage of ort would\nbe that the intermediate steps could be done in memory if this were\nimplemented as a real builtin merge strategy rather than just a sequence\nof shell commands.  The in-memory advantage might be good enough reason\nto take this nebulous idea of mine and create the 'oort' strategy at\nsome point in the future, but for now the sequence of shell commands\nshould give you what you need.\n\n\nHope that helps,\nElijah\n"},{"id":"443171","messageId":"CANiSa6jVp5EV73NxchFhHfZxxetdybhZ6r2qXE8O22e8rgpgVg@mail.gmail.com","threadId":"57015","inReplyTo":"20211202030458.GA48278@newk","subject":"Re: Potential bug with octopus merges and symlinks","fromName":"Martin von Zweigbergk","fromEmail":"martinvonz@gmail.com","sentAt":"2021-12-06T16:53:12Z","receivedAt":"2021-12-06T17:04:21Z","isPatch":false,"sender":{"key":"martinvonz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/891642?v=4"},"body":"On Sat, Dec 4, 2021 at 5:20 AM Michael McClimon <michael@mcclimon.org> wrote:\n>\n> There is a minimal reproducer available at\n> https://github.com/mmcclimon/git-merge-problem-demo. Fetch all the branches\n> there. The main branch contains a directory (dir1) with a single file\n> (file.txt), plus a symlink (dir2), which links to dir1. branch1 replaces this\n> symlink with a copy of the files that were linked to. (This was accomplished\n> with: rm dir2; cp -r dir1 dir2.) branch2 and branch3 do not touch this\n> directory at all.\n>\n> Merging these three branches fails:\n>\n> $ git merge branch1 branch2 branch3\n> Fast-forwarding to: branch1\n> Trying simple merge with branch2\n> Simple merge did not work, trying automatic merge.\n> Trying simple merge with branch3\n> error: Entry 'dir2/file.txt' not uptodate. Cannot merge.\n> Merge with strategy octopus failed.\n>\n> The order here matters! Here is every permutation (1 here is the symlink\n> change) to git merge; only the first two fail, all the others work.\n>\n> 1 2 3   FAIL\n> 1 3 2   FAIL\n> 2 1 3   PASS\n> 2 3 1   PASS\n> 3 1 2   PASS\n> 3 2 1   PASS\n> 1 2     PASS\n> 2 1     PASS\n> 2 3     PASS\n> 3 2     PASS\n> 1 3     PASS\n> 3 1     PASS\n\nI'm not a Git contributor (since ~10 years ago) but I was curious and\ntried your repro myself. Thanks for the very simple instructions.\nMaybe it's useful to see what the error is when you use the octopus\nstrategy to merge only two commits:\n\n```\n$ git checkout origin/branch1\n$ git merge -s octopus origin/branch2\nTrying really trivial in-index merge...\nerror: Merge requires file-level merging\nNope.\nMerge with strategy octopus failed.\n```\n\nThat \"Merge requires file-level merging\" is surprising but maybe the\n\"really trivial\" is the explanation :) My *guess* is the answer is\nthat the octopus strategy is really old and should be rewritten using\nthe new \"ort\" strategy (or maybe be integrated into it). I'm afraid I\ncan't think of a workaround for you (other than trying different\npermutations until it works).\n"},{"id":"443185","messageId":"xmqqa6hdha9d.fsf@gitster.g","threadId":"57015","inReplyTo":"20211202030458.GA48278@newk","subject":"Re: Potential bug with octopus merges and symlinks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-06T18:13:34Z","receivedAt":"2021-12-06T18:13:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael McClimon <michael@mcclimon.org> writes:\n\n> The order here matters! Here is every permutation (1 here is the symlink\n> change) to git merge; only the first two fail, all the others work.\n\nI am not an active Octopus contributor, but was the original\ninventor.  And it is not surprising that the order matters, because\nI designed it that way ;-)\n\nIn the original design, octopus was to be used only to create the\nsimplest forms of merges without any conflicts, because octopus\nmerges make it almost impossible to examine the result of the merge\nwith comparison between one of the parent and the result (you have\nto remember that there was no \"diff --cc\" invented yet, let alone\n\"diff --remerge-diff\", back then to help us).  Aborting the merge at\nthe first conflict when merging the remote heads one by one is a way\nto make sure that the result is a simple \"combine unrelated work\ninto one\" merge.  This does make the order matter somewhat.  When\nmore than one remote heads touch the same path, the order in which\nthey are merged into base does make a difference.\n\nTo be more strict for the goal of limiting octopus to the simplest\nforms of merges without any conflicts, we should have insisted that\nthe set of paths these remote heads want to modify should not\noverlap and that would have made the order totally not to matter,\nbut \"abort at the first conflict\" was easier to implement and\npractical.\n\nAnd later, before the command got part of the git suite, the rule\nwas further loosened to allow a conflicting merge when merging the\nlast remote head and let the user hand-resolve.  This makes the\norder matter even more.\n\n"}]}