{"thread":{"id":"55481","subject":"Unexpected conflict during cherry-pick after moving submodule","startedAt":"2021-04-12T14:56:00Z","lastAt":"2021-04-12T17:04:54Z","messageCount":5,"participants":["Tamas Peregi","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"421699","messageId":"CAKJ37DfivxL-2Sy0qa+M_1Pw0c9-CWsCJO5=VqP1UOB5zTSP_Q@mail.gmail.com","threadId":"55481","inReplyTo":null,"subject":"Unexpected conflict during cherry-pick after moving submodule","fromName":"Tamas Peregi","fromEmail":"tamas.peregi@formlabs.com","sentAt":"2021-04-12T14:55:21Z","receivedAt":"2021-04-12T14:56:00Z","isPatch":false,"sender":{"key":"tamas.peregi@formlabs.com","avatar":null},"body":"Hi all,\n\nI ran into something that seems like a bug to me. I'm fairly new to\ngit, especially git submodules, so it's possible this is expected\nbehaviour, but if it is, please explain where I went wrong.\n\nThanks in advance: Tamás\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n========================================================================\n\n- I modified a file on one branch\n- Then switched to another\n- Moved a submodule to another directory\n- Tried to cherry-pick my modification from the other branch\n\nThis script can be used to reproduce my steps (any submodule works, I\nchose gsl-lite because I like it):\n    git init\n\n    echo hello > hello.txt\n    git add hello.txt\n    git commit -m \"Add hello.txt\"\n\n    git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl\n    git commit -a -m \"Add submodule\"\n\n    git branch world\n    git branch move\n\n    git checkout world\n    echo world >> hello.txt\n    git commit -a -m \"Expand hello.txt\"\n\n    git checkout move\n    mkdir thirdparty\n    git mv vendor/gsl thirdparty/gsl\n    git commit -a -m \"Move submodule\"\n\n    git cherry-pick world\n\n\nWhat did you expect to happen? (Expected behavior)\n==================================================\n\nI expected the cherry-pick to happen without conflicts.\n\n\nWhat happened instead? (Actual behavior)\n========================================\nI ran into a conflict during the cherry-pick:\n    Adding as thirdparty/gsl~HEAD instead\n    error: could not apply 516d94f... Expand hello.txt\n    hint: after resolving the conflicts, mark the corrected paths\n    hint: with 'git add <paths>' or 'git rm <paths>'\n    hint: and commit the result with 'git commit'\n\n\nWhat's different between what you expected and what actually happened?\n======================================================================\n\nAs hello.txt has nothing to do with the submodule, I assumed the\ncherry-pick would go through smoothly.\n\n\nAnything else you want to add:\n==============================\n\n- \"git add thirdparty/gsl && git cherry-pick --continue\" resolves the\nissue as expected.\n- If I change which commit in the submodule the superproject points\nto, the problem disappears, but I don't necessarily want to do that.\n- I'm using \"Git for Windows\", but the issue was reproduced by my\nMac-based colleagues too with the exact same results.\n\n[System Info]\ngit version:\ngit version 2.31.1.windows.1\ncpu: x86_64\nbuilt from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb\nsizeof-long: 4\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nuname: Windows 10.0 19042\ncompiler info: gnuc: 10.2\nlibc info: no libc information available\n$SHELL (typically, interactive shell): C:\\Program Files\\Git\\usr\\bin\\bash.exe\n"},{"id":"421700","messageId":"CABPp-BEOnrnQKov1YDdHXkb6CCNxudE9bUdntOeq3k0wOODc1A@mail.gmail.com","threadId":"55481","inReplyTo":"CAKJ37DfivxL-2Sy0qa+M_1Pw0c9-CWsCJO5=VqP1UOB5zTSP_Q@mail.gmail.com","subject":"Re: Unexpected conflict during cherry-pick after moving submodule","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-12T15:29:44Z","receivedAt":"2021-04-12T15:29:57Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Mon, Apr 12, 2021 at 7:59 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:\n>\n> Hi all,\n>\n> I ran into something that seems like a bug to me. I'm fairly new to\n> git, especially git submodules, so it's possible this is expected\n> behaviour, but if it is, please explain where I went wrong.\n>\n> Thanks in advance: Tamás\n>\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> ========================================================================\n>\n> - I modified a file on one branch\n> - Then switched to another\n> - Moved a submodule to another directory\n> - Tried to cherry-pick my modification from the other branch\n>\n> This script can be used to reproduce my steps (any submodule works, I\n> chose gsl-lite because I like it):\n>     git init\n>\n>     echo hello > hello.txt\n>     git add hello.txt\n>     git commit -m \"Add hello.txt\"\n>\n>     git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl\n>     git commit -a -m \"Add submodule\"\n>\n>     git branch world\n>     git branch move\n>\n>     git checkout world\n>     echo world >> hello.txt\n>     git commit -a -m \"Expand hello.txt\"\n>\n>     git checkout move\n>     mkdir thirdparty\n>     git mv vendor/gsl thirdparty/gsl\n>     git commit -a -m \"Move submodule\"\n>\n>     git cherry-pick world\n>\n>\n> What did you expect to happen? (Expected behavior)\n> ==================================================\n>\n> I expected the cherry-pick to happen without conflicts.\n>\n>\n> What happened instead? (Actual behavior)\n> ========================================\n> I ran into a conflict during the cherry-pick:\n>     Adding as thirdparty/gsl~HEAD instead\n>     error: could not apply 516d94f... Expand hello.txt\n>     hint: after resolving the conflicts, mark the corrected paths\n>     hint: with 'git add <paths>' or 'git rm <paths>'\n>     hint: and commit the result with 'git commit'\n>\n>\n> What's different between what you expected and what actually happened?\n> ======================================================================\n>\n> As hello.txt has nothing to do with the submodule, I assumed the\n> cherry-pick would go through smoothly.\n>\n>\n> Anything else you want to add:\n> ==============================\n>\n> - \"git add thirdparty/gsl && git cherry-pick --continue\" resolves the\n> issue as expected.\n> - If I change which commit in the submodule the superproject points\n> to, the problem disappears, but I don't necessarily want to do that.\n> - I'm using \"Git for Windows\", but the issue was reproduced by my\n> Mac-based colleagues too with the exact same results.\n>\n> [System Info]\n> git version:\n> git version 2.31.1.windows.1\n> cpu: x86_64\n> built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb\n> sizeof-long: 4\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> feature: fsmonitor--daemon\n> uname: Windows 10.0 19042\n> compiler info: gnuc: 10.2\n> libc info: no libc information available\n> $SHELL (typically, interactive shell): C:\\Program Files\\Git\\usr\\bin\\bash.exe\n\nThanks for the report.\n\nWe can change the final command to\n    git cherry-pick --strategy recursive world\n\nto be more explicit, though that doesn't change things and still\ntriggers the bug.  However, if we change the above to\n    git cherry-pick --strategy ort world\n\nthen that fixes the bug, including even in git-2.31.1.  The version of\nthe 'ort' merge backend in 2.31.1 isn't ready for general usage (it\ncan't handle file renormalization, or subtree merges, or\nsparse-checkouts with conflicts in files outside of the requested\nsparsity paths, and also has a platform specific bug where it'll die\non corner case rename/rename conflicts due to assumptions about libc\nqsort() stability), but the version in 2.32 will be.  And 'ort' is\ngoing to replace 'recursive' once the project is comfortable enough\nwith the new backend.\n\nAll this is to say that the fix is on the way, even if it doesn't help\nyou quite yet.\n"},{"id":"421704","messageId":"CAKJ37Dc5-rOeN_=vq5P3=z=_1bqGFRuFhuWde6cS0vGg3mkLnQ@mail.gmail.com","threadId":"55481","inReplyTo":"CABPp-BEOnrnQKov1YDdHXkb6CCNxudE9bUdntOeq3k0wOODc1A@mail.gmail.com","subject":"Re: Unexpected conflict during cherry-pick after moving submodule","fromName":"Tamas Peregi","fromEmail":"tamas.peregi@formlabs.com","sentAt":"2021-04-12T16:07:57Z","receivedAt":"2021-04-12T16:08:39Z","isPatch":false,"sender":{"key":"tamas.peregi@formlabs.com","avatar":null},"body":"Hi Elijah,\n\nthanks for the quick reply and the useful information about the ort strategy!\n\nDo I understand correctly that the problem is in the recursive\nstrategy, i.e. inside the cherry-pick step, not the submodule\nmovement? That sounds a bit unfortunate in my case, as I'm the one\nmoving the submodule (then merging it back to master), and others in\nmy company want to cherry-pick over it, so I'm breaking their\nworkflows if I go ahead with moving. (Unless I tell all of them to use\nthe experimental ort strategy instead, which may introduce its own set\nof problems.) Is there any way of moving the submodule that doesn't\nbreak cherry-picking with the default (recursive) strategy? I'm\nwilling to do some extra work to spare confusion by others down the\nline.\n\nAlso: up until now, I thought cherry-pick simply exports the source\ncommit as a patch, then applies it to the target commit, but\n\"recursive\" is a merge strategy, correct? So is cherry-pick doing\nsomething vastly more complex than I thought, involving merging in the\nbackground?\n\nThanks in advance: Tamás\n\n\nOn Mon, Apr 12, 2021 at 5:29 PM Elijah Newren <newren@gmail.com> wrote:\n>\n> Hi,\n>\n> On Mon, Apr 12, 2021 at 7:59 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:\n> >\n> > Hi all,\n> >\n> > I ran into something that seems like a bug to me. I'm fairly new to\n> > git, especially git submodules, so it's possible this is expected\n> > behaviour, but if it is, please explain where I went wrong.\n> >\n> > Thanks in advance: Tamás\n> >\n> > What did you do before the bug happened? (Steps to reproduce your issue)\n> > ========================================================================\n> >\n> > - I modified a file on one branch\n> > - Then switched to another\n> > - Moved a submodule to another directory\n> > - Tried to cherry-pick my modification from the other branch\n> >\n> > This script can be used to reproduce my steps (any submodule works, I\n> > chose gsl-lite because I like it):\n> >     git init\n> >\n> >     echo hello > hello.txt\n> >     git add hello.txt\n> >     git commit -m \"Add hello.txt\"\n> >\n> >     git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl\n> >     git commit -a -m \"Add submodule\"\n> >\n> >     git branch world\n> >     git branch move\n> >\n> >     git checkout world\n> >     echo world >> hello.txt\n> >     git commit -a -m \"Expand hello.txt\"\n> >\n> >     git checkout move\n> >     mkdir thirdparty\n> >     git mv vendor/gsl thirdparty/gsl\n> >     git commit -a -m \"Move submodule\"\n> >\n> >     git cherry-pick world\n> >\n> >\n> > What did you expect to happen? (Expected behavior)\n> > ==================================================\n> >\n> > I expected the cherry-pick to happen without conflicts.\n> >\n> >\n> > What happened instead? (Actual behavior)\n> > ========================================\n> > I ran into a conflict during the cherry-pick:\n> >     Adding as thirdparty/gsl~HEAD instead\n> >     error: could not apply 516d94f... Expand hello.txt\n> >     hint: after resolving the conflicts, mark the corrected paths\n> >     hint: with 'git add <paths>' or 'git rm <paths>'\n> >     hint: and commit the result with 'git commit'\n> >\n> >\n> > What's different between what you expected and what actually happened?\n> > ======================================================================\n> >\n> > As hello.txt has nothing to do with the submodule, I assumed the\n> > cherry-pick would go through smoothly.\n> >\n> >\n> > Anything else you want to add:\n> > ==============================\n> >\n> > - \"git add thirdparty/gsl && git cherry-pick --continue\" resolves the\n> > issue as expected.\n> > - If I change which commit in the submodule the superproject points\n> > to, the problem disappears, but I don't necessarily want to do that.\n> > - I'm using \"Git for Windows\", but the issue was reproduced by my\n> > Mac-based colleagues too with the exact same results.\n> >\n> > [System Info]\n> > git version:\n> > git version 2.31.1.windows.1\n> > cpu: x86_64\n> > built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb\n> > sizeof-long: 4\n> > sizeof-size_t: 8\n> > shell-path: /bin/sh\n> > feature: fsmonitor--daemon\n> > uname: Windows 10.0 19042\n> > compiler info: gnuc: 10.2\n> > libc info: no libc information available\n> > $SHELL (typically, interactive shell): C:\\Program Files\\Git\\usr\\bin\\bash.exe\n>\n> Thanks for the report.\n>\n> We can change the final command to\n>     git cherry-pick --strategy recursive world\n>\n> to be more explicit, though that doesn't change things and still\n> triggers the bug.  However, if we change the above to\n>     git cherry-pick --strategy ort world\n>\n> then that fixes the bug, including even in git-2.31.1.  The version of\n> the 'ort' merge backend in 2.31.1 isn't ready for general usage (it\n> can't handle file renormalization, or subtree merges, or\n> sparse-checkouts with conflicts in files outside of the requested\n> sparsity paths, and also has a platform specific bug where it'll die\n> on corner case rename/rename conflicts due to assumptions about libc\n> qsort() stability), but the version in 2.32 will be.  And 'ort' is\n> going to replace 'recursive' once the project is comfortable enough\n> with the new backend.\n>\n> All this is to say that the fix is on the way, even if it doesn't help\n> you quite yet.\n"},{"id":"421709","messageId":"CABPp-BHASo5CFejED8CFbB1zszAaQW1Q8yitUdqW-wi8eFuvWg@mail.gmail.com","threadId":"55481","inReplyTo":"CAKJ37Dc5-rOeN_=vq5P3=z=_1bqGFRuFhuWde6cS0vGg3mkLnQ@mail.gmail.com","subject":"Re: Unexpected conflict during cherry-pick after moving submodule","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-12T16:53:03Z","receivedAt":"2021-04-12T16:57:33Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Tamas,\n\nOn Mon, Apr 12, 2021 at 9:08 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:\n>\n> Hi Elijah,\n>\n> thanks for the quick reply and the useful information about the ort strategy!\n>\n> Do I understand correctly that the problem is in the recursive\n> strategy, i.e. inside the cherry-pick step, not the submodule\n> movement? That sounds a bit unfortunate in my case, as I'm the one\n> moving the submodule (then merging it back to master), and others in\n> my company want to cherry-pick over it, so I'm breaking their\n> workflows if I go ahead with moving. (Unless I tell all of them to use\n> the experimental ort strategy instead, which may introduce its own set\n> of problems.) Is there any way of moving the submodule that doesn't\n> break cherry-picking with the default (recursive) strategy? I'm\n> willing to do some extra work to spare confusion by others down the\n> line.\n\nUnfortunately, the recursive merge machinery had a variety of issues\nwith submodules (see e.g.\nhttps://git.kernel.org/pub/scm/git/git.git/commit/?id=aa2faac03ad646873ebac2b230581d1d26dd1b99)\nand yes, the merge machinery is intrinsic to cherry-pick's\nfunctionality.  I don't have a good workaround for you, short of \"wait\nuntil git-2.32 is released\".\n\n> Also: up until now, I thought cherry-pick simply exports the source\n> commit as a patch, then applies it to the target commit, but\n> \"recursive\" is a merge strategy, correct? So is cherry-pick doing\n> something vastly more complex than I thought, involving merging in the\n> background?\n\nRight, cherry-pick makes use of the merge machinery using a particular\nspecially crafted merge while recording it as a regular commit.\nInterestingly, the rebase command has multiple backends, one of which\nused the merge machinery and another that behaved as you thought by\ncreating and applying patches.  While the use of an appropriate\nspecial merge (as done by cherry-pick and rebase) is roughly\nsemantically equivalent to what creating and applying patches\nprovides, the create-and-apply-patches procedure actually discards\nrelevant information and results in some shortcomings that simply\ncannot be fixed.  As such, we switched the default rebase backend from\n\"apply\" to \"merge\" as of git-2.26.  If you're curious, read up on the\n\"BEHAVIORAL DIFFERENCES\" section of the git-rebase(1) manual,\nespecially the \"Context\" section.\n\nHope that helps,\nElijah\n"},{"id":"421711","messageId":"CAKJ37DeVyQOfQAN-k9MvUT4TY1HBSU6+hJuLBauGAW_7i=ojTg@mail.gmail.com","threadId":"55481","inReplyTo":"CABPp-BHASo5CFejED8CFbB1zszAaQW1Q8yitUdqW-wi8eFuvWg@mail.gmail.com","subject":"Re: Unexpected conflict during cherry-pick after moving submodule","fromName":"Tamas Peregi","fromEmail":"tamas.peregi@formlabs.com","sentAt":"2021-04-12T17:02:51Z","receivedAt":"2021-04-12T17:04:54Z","isPatch":false,"sender":{"key":"tamas.peregi@formlabs.com","avatar":null},"body":"Hi Elijah,\n\nthanks for the expanded reply, I think it cleared everything up.\n\nRegards,\nTamás\n\nOn Mon, Apr 12, 2021 at 6:53 PM Elijah Newren <newren@gmail.com> wrote:\n>\n> Hi Tamas,\n>\n> On Mon, Apr 12, 2021 at 9:08 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:\n> >\n> > Hi Elijah,\n> >\n> > thanks for the quick reply and the useful information about the ort strategy!\n> >\n> > Do I understand correctly that the problem is in the recursive\n> > strategy, i.e. inside the cherry-pick step, not the submodule\n> > movement? That sounds a bit unfortunate in my case, as I'm the one\n> > moving the submodule (then merging it back to master), and others in\n> > my company want to cherry-pick over it, so I'm breaking their\n> > workflows if I go ahead with moving. (Unless I tell all of them to use\n> > the experimental ort strategy instead, which may introduce its own set\n> > of problems.) Is there any way of moving the submodule that doesn't\n> > break cherry-picking with the default (recursive) strategy? I'm\n> > willing to do some extra work to spare confusion by others down the\n> > line.\n>\n> Unfortunately, the recursive merge machinery had a variety of issues\n> with submodules (see e.g.\n> https://git.kernel.org/pub/scm/git/git.git/commit/?id=aa2faac03ad646873ebac2b230581d1d26dd1b99)\n> and yes, the merge machinery is intrinsic to cherry-pick's\n> functionality.  I don't have a good workaround for you, short of \"wait\n> until git-2.32 is released\".\n>\n> > Also: up until now, I thought cherry-pick simply exports the source\n> > commit as a patch, then applies it to the target commit, but\n> > \"recursive\" is a merge strategy, correct? So is cherry-pick doing\n> > something vastly more complex than I thought, involving merging in the\n> > background?\n>\n> Right, cherry-pick makes use of the merge machinery using a particular\n> specially crafted merge while recording it as a regular commit.\n> Interestingly, the rebase command has multiple backends, one of which\n> used the merge machinery and another that behaved as you thought by\n> creating and applying patches.  While the use of an appropriate\n> special merge (as done by cherry-pick and rebase) is roughly\n> semantically equivalent to what creating and applying patches\n> provides, the create-and-apply-patches procedure actually discards\n> relevant information and results in some shortcomings that simply\n> cannot be fixed.  As such, we switched the default rebase backend from\n> \"apply\" to \"merge\" as of git-2.26.  If you're curious, read up on the\n> \"BEHAVIORAL DIFFERENCES\" section of the git-rebase(1) manual,\n> especially the \"Context\" section.\n>\n> Hope that helps,\n> Elijah\n"}]}