'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

In the FreeBSD ports repository I resurrected the directory (that was removed a long time ago) with the command:

 > $ git checkout {hash}~1 -- math/polymake

I made local changes to this directory and called 'git add math/polymake'.

Then 'git pull' complained:

 > $ git pull
 > error: Your local changes to the following files would be overwritten 
by merge:
 >   math/polymake/Makefile math/polymake/distinfo 
math/polymake/files/patch-Makefile 
math/polymake/files/patch-support_install.pl math/polymake/pkg-descr 
math/polymake/pkg-plist

No incoming changes affect math/polymake. Nobody has created this directory simultaneously with me. There is no intersection with incoming changes.

Why does 'git pull' complain then?

Yuri

On Tue, Aug 24, 2021 at 09:41:49AM -0700, Yuri wrote:

It's hard to say without seeing a full example. The merge machinery is supposed to handle this situation (and indeed, in a simple reproduction example it does). So presumably it thinks the other side may have touched those files for some reason. Perhaps it's confused by rename detection?

Is it possible to give us a more complete example, including:

  - a url for the repository
  - the commit at HEAD when you ran "git checkout"
  - the {hash} commit from which you rescued the files
  - the state of the remote branch (i.e., what we attempted to merge
    with "git pull")

?

-Peff

On Tue, Aug 24, 2021 at 6:09 PM Jeff King <peff@peff.net> wrote:

The `git checkout {hash}~1 -- math/polymake` is enough to highlight that Yuri doesn't just have local changes (which the merge machinery should allow if the incoming changes don't touch the same files), but local *staged* changes. As per the merge manpage:

""" To avoid recording unrelated changes in the merge commit, git pull and git merge will also abort if there are any changes registered in the index relative to the HEAD commit. """

While this particular example could theoretically be handled by the merge machinery without requiring the index match HEAD, at least with the new ort backend (it'd be way too complex with the recursive backend, see below), in practice I'm so sick of index != HEAD bugs that I have zero motivation to even consider it and I might even NAK patches from others who attempted it. For why, let me quote from the commit message of 9822175d2b ("Ensure index matches head before invoking merge machinery, round N", 2019-08-17), which also references another commit that I'm tempted to also quote:

""" Ensure index matches head before invoking merge machinery, round N

This is the bug that just won't die; there always seems to be another form of it somewhere. See the commit message of 55f39cf7551b ("merge: fix misleading pre-merge check documentation", 2018-06-30) for a more detailed explanation), but in short:

<quick summary>

builtin/merge.c contains this important requirement for merge strategies:

    ...the index must be in sync with the head commit.  The strategies are
    responsible to ensure this.

This condition is important to enforce because there are two likely failure cases when the index isn't in sync with the head commit:

  * we silently throw away changes the user had staged before the merge

  * we accidentally (and silently) include changes in the merge that
    were not part of either of the branches/trees being merged

Discarding users' work and mis-merging are both bad outcomes, especially
when done silently, so naturally this rule was stated sternly -- but,
unfortunately totally ignored in practice unless and until actual bugs
were found.  But, fear not: the bugs from this were fixed in commit
  ee6566e8d70d ("[PATCH] Rewrite read-tree", 2005-09-05)
through a rewrite of read-tree (again, commit 55f39cf7551b has a more
detailed explanation of how this affected merge).  And it was fixed
again in commit
  160252f81626 ("git-merge-ours: make sure our index matches HEAD", 2005-11-03)
...and it was fixed again in commit
  3ec62ad9ffba ("merge-octopus: abort if index does not match HEAD", 2016-04-09)
...and again in commit
  65170c07d466 ("merge-recursive: avoid incorporating uncommitted
changes in a merge", 2017-12-21)
...and again in commit
  eddd1a411d93 ("merge-recursive: enforce rule that index matches head
before merging", 2018-06-30)

...with multiple testcases added to the testsuite that could be enumerated in even more commits.

Then, finally, in a patch in the same series as the last fix above, the documentation about this requirement was fixed in commit 55f39cf7551b ("merge: fix misleading pre-merge check documentation", 2018-06-30), and we all lived happily ever after...

</quick summary>

Unfortunately, "ever after" apparently denotes a limited time and it expired today. """

and then the commit message goes on to explain multiple other bugs found and fixed due to not strictly enforcing that the index needs to match HEAD before starting a merge.

On Wed, Aug 25, 2021 at 08:42:09AM -0700, Elijah Newren wrote:

Doh, you're right. I did a simple reproduction test, but in my example, the merge was a fast-forward, which we do allow. Picking a branch more carefully using:

  git for-each-ref --no-contains HEAD --no-merged HEAD refs/remotes/origin

shows the error. Sorry for the confusion.

-Peff

Elijah Newren <newren@gmail.com> writes:

While I do not mind seeing a patch that loosens the condition ONLY when the merge will cleanly auto-resolve without end-user interaction, when any paths conflict and require editing by the end-user, it is pretty much essential to require that the index matches HEAD to keep "git merge" usable.

This is because the final step to conclude such an "automated procedure cannot cleanly resolve, so the end user helps resolving with the editor and mark the resolved paths with 'git add' or 'git rm'" session will become very error prone if we did not have the requirement. Not just the user MUST remember not to use "commit -a" or "git add" a path that was already dirty in the working tree before the merge started (which is the consequence of the current requirement, which allows local changes to the unrelated working tree files), they must MUST remember to somehow EXCLUDE the changes already registered for unrelted paths from the concluded merge.

On Wed, Aug 25, 2021 at 02:19:15PM -0700, Junio C Hamano wrote:

Good point.

In theory we could reject the merge only after finding that there were conflicts. That lets the happy path continue even with unrelated changes (just like a fast-forward does). I suspect we'd need to change the interface to the merge-backends, though (to say "do the merge, and bail on conflicts, but _don't_ write out conflict markers or touch any other state in that case").

(I am just thinking out loud, though. My personal opinion is that if you have a bunch of staged changes and want to do any non-trivial merging, you should considering committing or stashing those changes).

-Peff

On Thu, Aug 26, 2021 at 6:05 PM Jeff King <peff@peff.net> wrote:

We actually wouldn't need to change the interface. One of the reasons there were so many annoying bugs with index != HEAD, was that builtin/merge.c stated that the merge backends were responsible for enforcing that condition. So every single merge backend had to be individually fixed -- ours, octopus, resolve, recursive. And then some of those, like recursive, had multiple affected code paths, and each one had to be fixed, and I didn't catch them all the first time.

That doesn't mean I'm in favor of the change, just pointing out that one backend could decide to do this on its own even if the other backends didn't support it.

Agreed.

Though perhaps I should have pointed out to Yuri that he wouldn't have
run into these problems if he would have used
    git restore --source={hash}~1 -- math/polymake
instead of
    git checkout {hash}~1 -- math/polymake

(since the former defaults to updating just the working directory, while the later updates both the index and the working directory.)

On 24/08/21 23.41, Yuri wrote:

Have you committed or stashed such changes?

On 8/24/21 10:43 PM, Bagas Sanjaya wrote:

No, they were uncommitted and unstashed.

Yuri