{"thread":{"id":"57316","subject":"git worktree, submodule and force checkout/switch","startedAt":"2022-01-26T16:43:27Z","lastAt":"2022-01-27T01:41:00Z","messageCount":3,"participants":["Federico Kircheis","Glen Choo","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"446974","messageId":"2a50d67e-364b-5927-801b-00f38179e904@gmail.com","threadId":"57316","inReplyTo":null,"subject":"git worktree, submodule and force checkout/switch","fromName":"Federico Kircheis","fromEmail":"federico.kircheis@gmail.com","sentAt":"2022-01-26T16:37:52Z","receivedAt":"2022-01-26T16:43:27Z","isPatch":false,"sender":{"key":"federico.kircheis@gmail.com","avatar":null},"body":"Hello to everyone,\n\nI would like to report what I believe is a bug, or at least an \ninconsistent behavior when using submodules and worktrees.\n\n\nConsider following test-case\n\n----\n# create 2 repositories with one commit\nmkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\")\nmkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\")\n\n# add submodule, a couple of branches, and a worktree\ncd repo1\ngit submodule add ../repo2 && git commit -m \"add submodule\"\ngit switch -c branch1\ngit switch -c branch2\ngit worktree add ../repo1.w --detach\n\n\n# test switch in the worktree\ncd ../repo1.w\n# git switch works\ngit switch --recurse-submodule branch1\ngit switch --recurse-submodule master\n#git submodule update # (1)\ncat .git\ncat .gitmodules\ncat repo2/.git # (2)\ngit switch --force branch1 # (3)error if no submodule update\n----\n\n\nNotice that if one forgets to git submodule update (1) before git switch \n--force branch1, even when using --recurse-submodule, there is no \nsubmodule, as repo1.w/repo2/ is empty (2).\n\nIt is confusing/unexpected that git switch --force fails and creates a \nrepo1.w/repo2/.git file pointing to the wrong location.\n\n\nAs comparison, when cloning a repository and forgetting to do \"git \nsubmodule update\", then \"git switch --force branch1\" works as expected:\n\n\n\n----\n# create 2 repositories with one commit\nmkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\")\nmkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\")\n\ncd repo1\ngit submodule add ../repo2 && git commit -m \"add submodule\"\ngit switch -c branch1\ngit switch -c branch2\n\ncd ..\ngit clone repo1 repo1.c\ncd repo1.c\ngit switch branch2\ngit switch --force branch1 # works, event without git submodule update\n----\n\n\n\n\nNotice:\nIn both cases \"git switch\" and \"git checkout\" behave the same.\nAlso the parameter \"--recurse-submodule\" does not change anything.\n\n\nBest\n\nFederico\n\n\nPS: I'm not subscribed to the mailing list (yet), so please keep me in CC.\n"},{"id":"447005","messageId":"kl6lwnimyxbq.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57316","inReplyTo":"2a50d67e-364b-5927-801b-00f38179e904@gmail.com","subject":"Re: git worktree, submodule and force checkout/switch","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-01-26T21:57:29Z","receivedAt":"2022-01-26T21:57:35Z","isPatch":false,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Federico Kircheis <federico.kircheis@gmail.com> writes:\n\n> Hello to everyone,\n>\n> I would like to report what I believe is a bug, or at least an \n> inconsistent behavior when using submodules and worktrees.\n>\n>\n> Consider following test-case\n>\n> ----\n> # create 2 repositories with one commit\n> mkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\")\n> mkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\")\n>\n> # add submodule, a couple of branches, and a worktree\n> cd repo1\n> git submodule add ../repo2 && git commit -m \"add submodule\"\n> git switch -c branch1\n> git switch -c branch2\n> git worktree add ../repo1.w --detach\n>\n>\n> # test switch in the worktree\n> cd ../repo1.w\n> # git switch works\n> git switch --recurse-submodule branch1\n> git switch --recurse-submodule master\n> #git submodule update # (1)\n> cat .git\n> cat .gitmodules\n> cat repo2/.git # (2)\n> git switch --force branch1 # (3)error if no submodule update\n> ----\n>\n>\n> Notice that if one forgets to git submodule update (1) before git switch \n> --force branch1, even when using --recurse-submodule, there is no \n> submodule, as repo1.w/repo2/ is empty (2).\n>\n> It is confusing/unexpected that git switch --force fails and creates a \n> repo1.w/repo2/.git file pointing to the wrong location.\n>\n>\n> As comparison, when cloning a repository and forgetting to do \"git \n> submodule update\", then \"git switch --force branch1\" works as expected:\n>\n>\n>\n> ----\n> # create 2 repositories with one commit\n> mkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\")\n> mkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\")\n>\n> cd repo1\n> git submodule add ../repo2 && git commit -m \"add submodule\"\n> git switch -c branch1\n> git switch -c branch2\n>\n> cd ..\n> git clone repo1 repo1.c\n> cd repo1.c\n> git switch branch2\n> git switch --force branch1 # works, event without git submodule update\n> ----\n>\n>\n>\n>\n> Notice:\n> In both cases \"git switch\" and \"git checkout\" behave the same.\n> Also the parameter \"--recurse-submodule\" does not change anything.\n>\n>\n> Best\n>\n> Federico\n>\n>\n> PS: I'm not subscribed to the mailing list (yet), so please keep me in CC.\n\nThanks for the report! This is very cosmetically similar to another bug\nthat Elijah (cc-ed) sent a patch for [1]. However, in my testing, the\ntest case doesn't even pass in v2.34.0, which doesn't have en/keep-cwd\n(which is the branch that introduced the motivation for [1]).\n\nSo despite the unfortunate timing, I am inclined to think that this is a\nlongstanding issue with worktrees and submodules (and not a recent\nregression in 2.35). I'm sending this out quickly just to say that it's\nnot a regression, and I might take a closer look when I have time.\n\nThis is the test case I used:\n\n  test_expect_success 'bugreport' '\n    # create 2 repositories with one commit\n    mkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\") &&\n    mkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\") &&\n\n    # add submodule, a couple of branches, and a worktree\n    cd repo1 &&\n    git submodule add ../repo2 && git commit -m \"add submodule\" &&\n    git switch -c branch1 &&\n    git switch -c branch2 &&\n    git worktree add ../repo1.w --detach &&\n\n    # test switch in the worktree\n    cd ../repo1.w &&\n    # git switch works\n    git switch --recurse-submodules branch1 &&\n    git switch --recurse-submodules main &&\n    cat .git &&\n    cat .gitmodules &&\n    cat repo2/.git &&\n    git switch --force branch1\n  '\n\n[1] https://lore.kernel.org/git/pull.1205.git.git.1643161426138.gitgitgadget@gmail.com\n"},{"id":"447024","messageId":"CABPp-BE6ZBjVv8f8af+M=yN2c50=dyefULfsAjqswych9BrJFg@mail.gmail.com","threadId":"57316","inReplyTo":"kl6lwnimyxbq.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: git worktree, submodule and force checkout/switch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-27T01:40:45Z","receivedAt":"2022-01-27T01:41:00Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jan 26, 2022 at 1:57 PM Glen Choo <chooglen@google.com> wrote:\n>\n> Federico Kircheis <federico.kircheis@gmail.com> writes:\n>\n> > Hello to everyone,\n> >\n> > I would like to report what I believe is a bug, or at least an\n> > inconsistent behavior when using submodules and worktrees.\n> >\n> >\n> > Consider following test-case\n> >\n> > ----\n> > # create 2 repositories with one commit\n> > mkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\")\n> > mkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\")\n> >\n> > # add submodule, a couple of branches, and a worktree\n> > cd repo1\n> > git submodule add ../repo2 && git commit -m \"add submodule\"\n> > git switch -c branch1\n> > git switch -c branch2\n> > git worktree add ../repo1.w --detach\n> >\n> >\n> > # test switch in the worktree\n> > cd ../repo1.w\n> > # git switch works\n> > git switch --recurse-submodule branch1\n> > git switch --recurse-submodule master\n> > #git submodule update # (1)\n> > cat .git\n> > cat .gitmodules\n> > cat repo2/.git # (2)\n> > git switch --force branch1 # (3)error if no submodule update\n> > ----\n> >\n> >\n> > Notice that if one forgets to git submodule update (1) before git switch\n> > --force branch1, even when using --recurse-submodule, there is no\n> > submodule, as repo1.w/repo2/ is empty (2).\n> >\n> > It is confusing/unexpected that git switch --force fails and creates a\n> > repo1.w/repo2/.git file pointing to the wrong location.\n> >\n> >\n> > As comparison, when cloning a repository and forgetting to do \"git\n> > submodule update\", then \"git switch --force branch1\" works as expected:\n> >\n> >\n> >\n> > ----\n> > # create 2 repositories with one commit\n> > mkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\")\n> > mkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\")\n> >\n> > cd repo1\n> > git submodule add ../repo2 && git commit -m \"add submodule\"\n> > git switch -c branch1\n> > git switch -c branch2\n> >\n> > cd ..\n> > git clone repo1 repo1.c\n> > cd repo1.c\n> > git switch branch2\n> > git switch --force branch1 # works, event without git submodule update\n> > ----\n> >\n> >\n> >\n> >\n> > Notice:\n> > In both cases \"git switch\" and \"git checkout\" behave the same.\n> > Also the parameter \"--recurse-submodule\" does not change anything.\n> >\n> >\n> > Best\n> >\n> > Federico\n> >\n> >\n> > PS: I'm not subscribed to the mailing list (yet), so please keep me in CC.\n>\n> Thanks for the report! This is very cosmetically similar to another bug\n> that Elijah (cc-ed) sent a patch for [1]. However, in my testing, the\n> test case doesn't even pass in v2.34.0, which doesn't have en/keep-cwd\n> (which is the branch that introduced the motivation for [1]).\n\nThe extent of the similarity to me seems to be that both testcases use\ngit-worktree.  So yeah, I think only cosmetically similar.\n\nen/keep-cwd was very specifically about running commands from some\nsubdirectory, not the toplevel.  Here, the command being run is from\nthe toplevel, so that series couldn't really affect anything here.\n\nThe previous bug also was about GIT_DIR being implicitly set and then\nforking subcommands; but the issue being observed here seems to be\nwith git-switch, which I don't think forks out to subprocesses.\n\nSo, I agree that this looks like some older issue.\n\nHowever, I played with your testcase a little bit...\n\n> So despite the unfortunate timing, I am inclined to think that this is a\n> longstanding issue with worktrees and submodules (and not a recent\n> regression in 2.35). I'm sending this out quickly just to say that it's\n> not a regression, and I might take a closer look when I have time.\n>\n> This is the test case I used:\n>\n>   test_expect_success 'bugreport' '\n>     # create 2 repositories with one commit\n>     mkdir repo1 && (cd repo1 && git init && git commit --allow-empty -m \"repo1\") &&\n>     mkdir repo2 && (cd repo2 && git init && git commit --allow-empty -m \"repo2\") &&\n>\n>     # add submodule, a couple of branches, and a worktree\n>     cd repo1 &&\n>     git submodule add ../repo2 && git commit -m \"add submodule\" &&\n>     git switch -c branch1 &&\n>     git switch -c branch2 &&\n>     git worktree add ../repo1.w --detach &&\n>\n>     # test switch in the worktree\n>     cd ../repo1.w &&\n>     # git switch works\n>     git switch --recurse-submodules branch1 &&\n\nSo, the 'checkout' subsection of the 'update' command in git submodule\nhas this interesting tidbit:\n\n\"\"\"\nIf --force is specified, the submodule will be checked out\n(using git checkout --force), even if the commit specified in\nthe index of the containing repository already matches the\ncommit checked out in the submodule.\n\"\"\"\n\nThe test repository in question has the same submodule commit on both\nbranches, so I think it's hitting some short-circuiting logic to avoid\ndoing anything based on that.  Given that hunch, I added a \"-f\" (short\nfor --force) to this switch command, and then saw the following\noutput:\n\n    ++ git switch --recurse-submodules -f branch1\n    fatal: not a git repository:\n../../repo1/.git/worktrees/repo1.w/modules/repo2\n    fatal: could not reset submodule index\n\nand indeed, you'll note that repo1/.git/modules/repo2/ looks like a\ngit repository, but there is no\nrepo1/.git/worktrees/repo1.w/modules/repo2 directory; in fact, the\nmodules directory is missing.\n\nIf you insert a\n     ln -s ../../modules .git/worktrees/repo1.w\nright after the 'git worktree add...' command and keep the -f I added\nabove, then the whole test runs to completion.\n\nSo, I think the relevant bits are potentially interesting:\n  * git submodule appears to look for the submodule repository\ninformation in $(git rev-parse --git-dir)/modules/$MODULE_NAME, but it\nactually exists in $(git rev-parse\n--git-common-dir)/modules/$MODULE_NAME.  For primary worktrees those\nare the same, but not for added worktrees.  Some code probably needs\nto be updated somewhere to look in the right location.\n  * There are config values (submodule.repo2.url and\nsubmodule.repo2.active) that were set from repo1.  I think you want\nthese same settings shared with other worktrees, so this probably\nmakes sense.  So you probably don't have to worry about\nworktree-specific config (see the extensions.worktreeConfig\ndocumentation) in the way that sparse-checkout does.\n  * 'git worktree add' does a checkout, but doesn't have a\n--recurse-submodules setting.  So it's not bothering to check them\nout, but it kind of looks like it is active and checked out due to the\nshared config.  It might make sense to add a --recurse-submodules\noption to 'git worktree add' to initialize these better.\n  * If your testcase setup would have had different submodule commits\nfor the different branches, you might have found this easier and not\nneeded the -f.\n\n>     git switch --recurse-submodules main &&\n>     cat .git &&\n>     cat .gitmodules &&\n>     cat repo2/.git &&\n>     git switch --force branch1\n>   '\n>\n> [1] https://lore.kernel.org/git/pull.1205.git.git.1643161426138.gitgitgadget@gmail.com\n\nAnyway, I won't be looking at submodule issues anytime soon, but I\nhope those pointers help someone else who wants to get this running.\n"}]}