{"thread":{"id":"57936","subject":"git pull --recurse-submodules wipes uncommitted submodule changes without warning","startedAt":"2022-05-31T20:25:53Z","lastAt":"2022-06-01T00:14:10Z","messageCount":2,"participants":["Guillaume Girol","Glen Choo"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"456423","messageId":"eb2d8ce45e556afd81216746f190bc5c883cdb81.camel@xlumurb.eu","threadId":"57936","inReplyTo":null,"subject":"git pull --recurse-submodules wipes uncommitted submodule changes without warning","fromName":"Guillaume Girol","fromEmail":"symphorien@xlumurb.eu","sentAt":"2022-05-31T20:18:54Z","receivedAt":"2022-05-31T20:25:53Z","isPatch":false,"sender":{"key":"symphorien@xlumurb.eu","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your\nissue)\n\nThe repository has a submodule bar.\nIn the current checkout, bar has new commits, but this is not commited:\n\n----------------------------\n\n$ git status\nOn branch master\nYour branch is behind 'origin/master' by 1 commit, and can be fast-\nforwarded.\n  (use \"git pull\" to update your local branch)\n\nChanges not staged for commit:\n  (use \"git add <file>...\" to update what will be committed)\n  (use \"git restore <file>...\" to discard changes in working directory)\n\tmodified:   bar (new commits)\n\nUntracked files:\n  (use \"git add <file>...\" to include in what will be committed)\n\tgit-bugreport-2022-05-31-2142.txt\n\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n\n----------------------------\n\nThere is one commit to pull. It affects a file in the superproject, but\nnot the submodule.\n\nThe issue arises with:\n\n$ git pull --recurse-submodule=on-demand --no-rebase\n\nWhat did you expect to happen? (Expected behavior)\n\nThe file affected by the pulled commit is modified, but the submodule\nis left untouched.\nThe output of git status should look like:\n\n----------------------------\n\n$ git status\nOn branch master\nYour branch up to date with 'origin/master'.\n\nChanges not staged for commit:\n  (use \"git add <file>...\" to update what will be committed)\n  (use \"git restore <file>...\" to discard changes in working directory)\n\tmodified:   bar (new commits)\n\nUntracked files:\n  (use \"git add <file>...\" to include in what will be committed)\n\tgit-bugreport-2022-05-31-2142.txt\n\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n\n----------------------------\n\nWhat happened instead? (Actual behavior)\n\n--------------------------\n\n$  git pull --recurse-submodule=on-demand --no-rebase\nUpdating 67627dd..80f5c51\nFast-forward\n foo | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\nSubmodule path 'bar': checked out\n'ef3c0711fedca48b0b43aadfd01b7bb94b519a13'\n\n$  git status\nOn branch master\nYour branch is up to date with 'origin/master'.\n\nUntracked files:\n  (use \"git add <file>...\" to include in what will be committed)\n\tgit-bugreport-2022-05-31-2142.txt\n\nnothing added to commit but untracked files present (use \"git add\" to\ntrack)\n\n-----------------------\n\nmy uncommitted changes to the submodule have been wiped, and the commit\nof the pulled revision was checked out.\n\nWhat's different between what you expected and what actually happened?\n\nWhen there are uncommitted changes to a file foo in a repo, and that I\n`git pull` a commit that does not affect this file foo, git pull does\nnot wipe my changes to `foo`. I expect the same for submodules. If\nthere is a conflict, I am told about it and nothing is wiped either.\n\n\nAnything else you want to add:\n\ngit pull is able to detect conflicts on submodules, so why not non-\nconflicts on submodules?\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.36.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /nix/store/0d3wgx8x6dxdb2cpnq105z23hah07z7l-bash-5.1-\np16/bin/bash\nuname: Linux 5.15.39 #1-NixOS SMP Thu May 12 10:30:34 UTC 2022 x86_64\ncompiler info: gnuc: 11.3\nlibc info: glibc: 2.34\n$SHELL (typically, interactive shell): /run/current-system/sw/bin/zsh\n\n\n[Enabled Hooks]\n\n"},{"id":"456428","messageId":"kl6lleuhdyv1.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57936","inReplyTo":"eb2d8ce45e556afd81216746f190bc5c883cdb81.camel@xlumurb.eu","subject":"Re: git pull --recurse-submodules wipes uncommitted submodule changes without warning","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-06-01T00:13:54Z","receivedAt":"2022-06-01T00:14:10Z","isPatch":false,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\nThanks for the report!\n\nGuillaume Girol <symphorien@xlumurb.eu> writes:\n\n> What did you do before the bug happened? (Steps to reproduce your\n> issue)\n>\n> The repository has a submodule bar.\n> In the current checkout, bar has new commits, but this is not commited:\n>\n> ----------------------------\n>\n> $ git status\n> On branch master\n> Your branch is behind 'origin/master' by 1 commit, and can be fast-\n> forwarded.\n>   (use \"git pull\" to update your local branch)\n>\n> Changes not staged for commit:\n>   (use \"git add <file>...\" to update what will be committed)\n>   (use \"git restore <file>...\" to discard changes in working directory)\n> \tmodified:   bar (new commits)\n>\n> Untracked files:\n>   (use \"git add <file>...\" to include in what will be committed)\n> \tgit-bugreport-2022-05-31-2142.txt\n>\n> no changes added to commit (use \"git add\" and/or \"git commit -a\")\n>\n> ----------------------------\n>\n> There is one commit to pull. It affects a file in the superproject, but\n> not the submodule.\n>\n> The issue arises with:\n>\n> $ git pull --recurse-submodule=on-demand --no-rebase\n>\n> What did you expect to happen? (Expected behavior)\n>\n> The file affected by the pulled commit is modified, but the submodule\n> is left untouched.\n> The output of git status should look like:\n>\n> ----------------------------\n>\n> $ git status\n> On branch master\n> Your branch up to date with 'origin/master'.\n>\n> Changes not staged for commit:\n>   (use \"git add <file>...\" to update what will be committed)\n>   (use \"git restore <file>...\" to discard changes in working directory)\n> \tmodified:   bar (new commits)\n>\n> Untracked files:\n>   (use \"git add <file>...\" to include in what will be committed)\n> \tgit-bugreport-2022-05-31-2142.txt\n>\n> no changes added to commit (use \"git add\" and/or \"git commit -a\")\n>\n> ----------------------------\n>\n> What happened instead? (Actual behavior)\n>\n> --------------------------\n>\n> $  git pull --recurse-submodule=on-demand --no-rebase\n> Updating 67627dd..80f5c51\n> Fast-forward\n>  foo | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n> Submodule path 'bar': checked out\n> 'ef3c0711fedca48b0b43aadfd01b7bb94b519a13'\n>\n> $  git status\n> On branch master\n> Your branch is up to date with 'origin/master'.\n>\n> Untracked files:\n>   (use \"git add <file>...\" to include in what will be committed)\n> \tgit-bugreport-2022-05-31-2142.txt\n>\n> nothing added to commit but untracked files present (use \"git add\" to\n> track)\n>\n> -----------------------\n>\n> my uncommitted changes to the submodule have been wiped, and the commit\n> of the pulled revision was checked out.\n\nThis is very clear, thank you.\n\nSo basically, because you made a commit in your submodule but did not\ncommit it back to the superproject, you have 'uncommitted changes'. When\ndoing \"git pull --recurse-submodules\", you expect your 'uncommitted\nchanges' to be left alone, but instead, your submodule has now checked\nout an older version.\n\ni.e. your starting state and desired end state is:\n\n  superproject                submodule \n  B (origin/master)           D (HEAD)\n  | \\                         |\n  |  \\                        |\n  |   ---------               |\n  |            \\              |\n  A (master) ---------------- C\n\nWhere you had submodule commit D checked out and it stays checked out.\n\nHowever your actual end state is:\n\n  superproject                submodule \n  B (origin/master)           D\n  | \\                         |\n  |  \\                        |\n  |   ---------               |\n  |            \\              |\n  A (master) ---------------- C (HEAD)\n\nwhere you've checked out submodule commit C, and submodule commit D has\nbeen 'abandoned'.\n\n\"checkout --recurse-submodules\" is known to abandon commits in a similar\nway; the same thing can happen if you had done \"git fetch && git\ncheckout --recurse-submodules origin/master\". This is because we only\ncheck for uncommitted changes in the _submodule_, even though the\nsubmodule itself is not committed to the superproject.\n\nBut oddly enough, \"pull --recurse-submodules\" actually abandons the\ncommits in a _different_ way. After fetching (recursively) and merging\n(non-recursively), \"git pull\" updates the submodules using \"git\nsubmodule update --recursive --checkout\". This makes each submodule do a\n\"git checkout <version in merged superproject commit>\" (in the example\nabove, this is commit C). This obviously doesn't actually merge\nanything, and presumably we only do this because we want to update the\nsubmodule working tree at the end of \"git pull\" (see\nt/t5572-pull-submodule.sh for examples of how this works).\n\nOne way we _might_ be able fix this is to teach \"git submodule update\"\nto notice if the submodule has changed and to abort the checkout if so\n(maybe the option could be named --no-abandon?)\n\n> Anything else you want to add:\n>\n> git pull is able to detect conflicts on submodules, so why not non-\n> conflicts on submodules?\n\nIIUC, part of the problem is that we do a very basic job of merging\nsubmodules:\n\n- we check to if the two superproject commits point to the same\n  submodule commit\n- if they are the same, there is no conflict\n- if they are different, there is a merge conflict\n\nIn your case, the submodule is not changed (both superproject commits\npoint to submodule commit C), so there is no merge conflict.\n\nI don't see why we couldn't tighten up our submodule conflict detection\nthough. This is like a fast-forward/two-way merge, but when we do a\ntwo-way merge, we also check the index to make sure we don't abandon any\nlocal changes. We could do the same for submodules to make sure we don't\nabandon any new submodule commits.\n\nWhat's nice about this idea is that we might be able to reuse this\ntwo-way merge logic to teach \"git checkout\" not to abandon submodule\ncommits.\n"}]}