git pull --recurse-submodules wipes uncommitted submodule changes without warning

What did you do before the bug happened? (Steps to reproduce your issue)

The repository has a submodule bar. In the current checkout, bar has new commits, but this is not commited:

----------------------------

$ git status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-
forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   bar (new commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	git-bugreport-2022-05-31-2142.txt

no changes added to commit (use "git add" and/or "git commit -a")

----------------------------

There is one commit to pull. It affects a file in the superproject, but not the submodule.

The issue arises with:

$ git pull --recurse-submodule=on-demand --no-rebase

What did you expect to happen? (Expected behavior)

The file affected by the pulled commit is modified, but the submodule is left untouched. The output of git status should look like:

----------------------------

$ git status On branch master Your branch up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   bar (new commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	git-bugreport-2022-05-31-2142.txt

no changes added to commit (use "git add" and/or "git commit -a")

----------------------------

What happened instead? (Actual behavior)

--------------------------

$  git pull --recurse-submodule=on-demand --no-rebase
Updating 67627dd..80f5c51
Fast-forward
 foo | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Submodule path 'bar': checked out
'ef3c0711fedca48b0b43aadfd01b7bb94b519a13'

$ git status On branch master Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	git-bugreport-2022-05-31-2142.txt

nothing added to commit but untracked files present (use "git add" to track)

-----------------------

my uncommitted changes to the submodule have been wiped, and the commit of the pulled revision was checked out.

What's different between what you expected and what actually happened?

When there are uncommitted changes to a file foo in a repo, and that I `git pull` a commit that does not affect this file foo, git pull does not wipe my changes to `foo`. I expect the same for submodules. If there is a conflict, I am told about it and nothing is wiped either.

Anything else you want to add:

git pull is able to detect conflicts on submodules, so why not non- conflicts on submodules?

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

[System Info] git version: git version 2.36.0 cpu: x86_64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /nix/store/0d3wgx8x6dxdb2cpnq105z23hah07z7l-bash-5.1- p16/bin/bash uname: Linux 5.15.39 #1-NixOS SMP Thu May 12 10:30:34 UTC 2022 x86_64 compiler info: gnuc: 11.3 libc info: glibc: 2.34 $SHELL (typically, interactive shell): /run/current-system/sw/bin/zsh

[Enabled Hooks]

Thanks for the report!

Guillaume Girol <symphorien@xlumurb.eu> writes:

This is very clear, thank you.

So basically, because you made a commit in your submodule but did not commit it back to the superproject, you have 'uncommitted changes'. When doing "git pull --recurse-submodules", you expect your 'uncommitted changes' to be left alone, but instead, your submodule has now checked out an older version.

i.e. your starting state and desired end state is:

  superproject                submodule 
  B (origin/master)           D (HEAD)
  | \                         |
  |  \                        |
  |   ---------               |
  |            \              |
  A (master) ---------------- C

Where you had submodule commit D checked out and it stays checked out.

However your actual end state is:

  superproject                submodule 
  B (origin/master)           D
  | \                         |
  |  \                        |
  |   ---------               |
  |            \              |
  A (master) ---------------- C (HEAD)

where you've checked out submodule commit C, and submodule commit D has been 'abandoned'.

"checkout --recurse-submodules" is known to abandon commits in a similar way; the same thing can happen if you had done "git fetch && git checkout --recurse-submodules origin/master". This is because we only check for uncommitted changes in the _submodule_, even though the submodule itself is not committed to the superproject.

But oddly enough, "pull --recurse-submodules" actually abandons the commits in a _different_ way. After fetching (recursively) and merging (non-recursively), "git pull" updates the submodules using "git submodule update --recursive --checkout". This makes each submodule do a "git checkout <version in merged superproject commit>" (in the example above, this is commit C). This obviously doesn't actually merge anything, and presumably we only do this because we want to update the submodule working tree at the end of "git pull" (see t/t5572-pull-submodule.sh for examples of how this works).

One way we _might_ be able fix this is to teach "git submodule update" to notice if the submodule has changed and to abort the checkout if so (maybe the option could be named --no-abandon?)

IIUC, part of the problem is that we do a very basic job of merging submodules:

- we check to if the two superproject commits point to the same
  submodule commit
- if they are the same, there is no conflict
- if they are different, there is a merge conflict

In your case, the submodule is not changed (both superproject commits point to submodule commit C), so there is no merge conflict.

I don't see why we couldn't tighten up our submodule conflict detection though. This is like a fast-forward/two-way merge, but when we do a two-way merge, we also check the index to make sure we don't abandon any local changes. We could do the same for submodules to make sure we don't abandon any new submodule commits.

What's nice about this idea is that we might be able to reuse this two-way merge logic to teach "git checkout" not to abandon submodule commits.