Unexpected conflict during cherry-pick after moving submodule

>
> Hi all,
>
> I ran into something that seems like a bug to me. I'm fairly new to
> git, especially git submodules, so it's possible this is expected
> behaviour, but if it is, please explain where I went wrong.
>
> Thanks in advance: Tamás
>
> What did you do before the bug happened? (Steps to reproduce your issue)
> ========================================================================
>
> - I modified a file on one branch
> - Then switched to another
> - Moved a submodule to another directory
> - Tried to cherry-pick my modification from the other branch
>
> This script can be used to reproduce my steps (any submodule works, I
> chose gsl-lite because I like it):
>     git init
>
>     echo hello > hello.txt
>     git add hello.txt
>     git commit -m "Add hello.txt"
>
>     git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl
>     git commit -a -m "Add submodule"
>
>     git branch world
>     git branch move
>
>     git checkout world
>     echo world >> hello.txt
>     git commit -a -m "Expand hello.txt"
>
>     git checkout move
>     mkdir thirdparty
>     git mv vendor/gsl thirdparty/gsl
>     git commit -a -m "Move submodule"
>
>     git cherry-pick world
>
>
> What did you expect to happen? (Expected behavior)
> ==================================================
>
> I expected the cherry-pick to happen without conflicts.
>
>
> What happened instead? (Actual behavior)
> ========================================
> I ran into a conflict during the cherry-pick:
>     Adding as thirdparty/gsl~HEAD instead
>     error: could not apply 516d94f... Expand hello.txt
>     hint: after resolving the conflicts, mark the corrected paths
>     hint: with 'git add <paths>' or 'git rm <paths>'
>     hint: and commit the result with 'git commit'
>
>
> What's different between what you expected and what actually happened?
> ======================================================================
>
> As hello.txt has nothing to do with the submodule, I assumed the
> cherry-pick would go through smoothly.
>
>
> Anything else you want to add:
> ==============================
>
> - "git add thirdparty/gsl && git cherry-pick --continue" resolves the
> issue as expected.
> - If I change which commit in the submodule the superproject points
> to, the problem disappears, but I don't necessarily want to do that.
> - I'm using "Git for Windows", but the issue was reproduced by my
> Mac-based colleagues too with the exact same results.
>
> [System Info]
> git version:
> git version 2.31.1.windows.1
> cpu: x86_64
> built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb
> sizeof-long: 4
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Windows 10.0 19042
> compiler info: gnuc: 10.2
> libc info: no libc information available
> $SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe

>
> Hi,
>
> On Mon, Apr 12, 2021 at 7:59 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:
> >
> > Hi all,
> >
> > I ran into something that seems like a bug to me. I'm fairly new to
> > git, especially git submodules, so it's possible this is expected
> > behaviour, but if it is, please explain where I went wrong.
> >
> > Thanks in advance: Tamás
> >
> > What did you do before the bug happened? (Steps to reproduce your issue)
> > ========================================================================
> >
> > - I modified a file on one branch
> > - Then switched to another
> > - Moved a submodule to another directory
> > - Tried to cherry-pick my modification from the other branch
> >
> > This script can be used to reproduce my steps (any submodule works, I
> > chose gsl-lite because I like it):
> >     git init
> >
> >     echo hello > hello.txt
> >     git add hello.txt
> >     git commit -m "Add hello.txt"
> >
> >     git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl
> >     git commit -a -m "Add submodule"
> >
> >     git branch world
> >     git branch move
> >
> >     git checkout world
> >     echo world >> hello.txt
> >     git commit -a -m "Expand hello.txt"
> >
> >     git checkout move
> >     mkdir thirdparty
> >     git mv vendor/gsl thirdparty/gsl
> >     git commit -a -m "Move submodule"
> >
> >     git cherry-pick world
> >
> >
> > What did you expect to happen? (Expected behavior)
> > ==================================================
> >
> > I expected the cherry-pick to happen without conflicts.
> >
> >
> > What happened instead? (Actual behavior)
> > ========================================
> > I ran into a conflict during the cherry-pick:
> >     Adding as thirdparty/gsl~HEAD instead
> >     error: could not apply 516d94f... Expand hello.txt
> >     hint: after resolving the conflicts, mark the corrected paths
> >     hint: with 'git add <paths>' or 'git rm <paths>'
> >     hint: and commit the result with 'git commit'
> >
> >
> > What's different between what you expected and what actually happened?
> > ======================================================================
> >
> > As hello.txt has nothing to do with the submodule, I assumed the
> > cherry-pick would go through smoothly.
> >
> >
> > Anything else you want to add:
> > ==============================
> >
> > - "git add thirdparty/gsl && git cherry-pick --continue" resolves the
> > issue as expected.
> > - If I change which commit in the submodule the superproject points
> > to, the problem disappears, but I don't necessarily want to do that.
> > - I'm using "Git for Windows", but the issue was reproduced by my
> > Mac-based colleagues too with the exact same results.
> >
> > [System Info]
> > git version:
> > git version 2.31.1.windows.1
> > cpu: x86_64
> > built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb
> > sizeof-long: 4
> > sizeof-size_t: 8
> > shell-path: /bin/sh
> > feature: fsmonitor--daemon
> > uname: Windows 10.0 19042
> > compiler info: gnuc: 10.2
> > libc info: no libc information available
> > $SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe
>
> Thanks for the report.
>
> We can change the final command to
>     git cherry-pick --strategy recursive world
>
> to be more explicit, though that doesn't change things and still
> triggers the bug.  However, if we change the above to
>     git cherry-pick --strategy ort world
>
> then that fixes the bug, including even in git-2.31.1.  The version of
> the 'ort' merge backend in 2.31.1 isn't ready for general usage (it
> can't handle file renormalization, or subtree merges, or
> sparse-checkouts with conflicts in files outside of the requested
> sparsity paths, and also has a platform specific bug where it'll die
> on corner case rename/rename conflicts due to assumptions about libc
> qsort() stability), but the version in 2.32 will be.  And 'ort' is
> going to replace 'recursive' once the project is comfortable enough
> with the new backend.
>
> All this is to say that the fix is on the way, even if it doesn't help
> you quite yet.

>
> Hi Elijah,
>
> thanks for the quick reply and the useful information about the ort strategy!
>
> Do I understand correctly that the problem is in the recursive
> strategy, i.e. inside the cherry-pick step, not the submodule
> movement? That sounds a bit unfortunate in my case, as I'm the one
> moving the submodule (then merging it back to master), and others in
> my company want to cherry-pick over it, so I'm breaking their
> workflows if I go ahead with moving. (Unless I tell all of them to use
> the experimental ort strategy instead, which may introduce its own set
> of problems.) Is there any way of moving the submodule that doesn't
> break cherry-picking with the default (recursive) strategy? I'm
> willing to do some extra work to spare confusion by others down the
> line.

> Also: up until now, I thought cherry-pick simply exports the source
> commit as a patch, then applies it to the target commit, but
> "recursive" is a merge strategy, correct? So is cherry-pick doing
> something vastly more complex than I thought, involving merging in the
> background?

>
> Hi Tamas,
>
> On Mon, Apr 12, 2021 at 9:08 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:
> >
> > Hi Elijah,
> >
> > thanks for the quick reply and the useful information about the ort strategy!
> >
> > Do I understand correctly that the problem is in the recursive
> > strategy, i.e. inside the cherry-pick step, not the submodule
> > movement? That sounds a bit unfortunate in my case, as I'm the one
> > moving the submodule (then merging it back to master), and others in
> > my company want to cherry-pick over it, so I'm breaking their
> > workflows if I go ahead with moving. (Unless I tell all of them to use
> > the experimental ort strategy instead, which may introduce its own set
> > of problems.) Is there any way of moving the submodule that doesn't
> > break cherry-picking with the default (recursive) strategy? I'm
> > willing to do some extra work to spare confusion by others down the
> > line.
>
> Unfortunately, the recursive merge machinery had a variety of issues
> with submodules (see e.g.
> https://git.kernel.org/pub/scm/git/git.git/commit/?id=aa2faac03ad646873ebac2b230581d1d26dd1b99)
> and yes, the merge machinery is intrinsic to cherry-pick's
> functionality.  I don't have a good workaround for you, short of "wait
> until git-2.32 is released".
>
> > Also: up until now, I thought cherry-pick simply exports the source
> > commit as a patch, then applies it to the target commit, but
> > "recursive" is a merge strategy, correct? So is cherry-pick doing
> > something vastly more complex than I thought, involving merging in the
> > background?
>
> Right, cherry-pick makes use of the merge machinery using a particular
> specially crafted merge while recording it as a regular commit.
> Interestingly, the rebase command has multiple backends, one of which
> used the merge machinery and another that behaved as you thought by
> creating and applying patches.  While the use of an appropriate
> special merge (as done by cherry-pick and rebase) is roughly
> semantically equivalent to what creating and applying patches
> provides, the create-and-apply-patches procedure actually discards
> relevant information and results in some shortcomings that simply
> cannot be fixed.  As such, we switched the default rebase backend from
> "apply" to "merge" as of git-2.26.  If you're curious, read up on the
> "BEHAVIORAL DIFFERENCES" section of the git-rebase(1) manual,
> especially the "Context" section.
>
> Hope that helps,
> Elijah