{"thread":{"id":"52807","subject":"Nested submodule checkout","startedAt":"2020-02-14T22:42:48Z","lastAt":"2020-03-03T22:44:40Z","messageCount":13,"participants":["Damien Robert","Philippe Blain","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"391803","messageId":"20200214224242.knmzkwx7ls4sote7@doriath","threadId":"52807","inReplyTo":null,"subject":"Nested submodule checkout","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-02-14T22:42:42Z","receivedAt":"2020-02-14T22:42:48Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"Dear git developers,\n\nI stumbled into this situation from which it was a bit painful to recover:\n\n-> test script:\n\n    mkdir ploum\n    cd ploum\n    git init\n    echo 'foo' > foo\n    git add foo\n    git commit -m foo\n    git branch nosubmodules\n\n    mkdir plam\n    cd plam\n    git init\n    echo 'bar' > bar\n    git add bar\n    git commit -m bar\n\n    mkdir plim\n    cd plim\n    git init\n    echo 'baz' > baz\n    git add baz\n    git commit -m baz\n\n    cd ..\n    git submodule add ./plim\n    git commit -am 'Add submodule plim'\n\n    cd ..\n    git submodule add ./plam\n    git commit -am 'Add submodule plam'\n\n    git checkout nosubmodules\n    git checkout --recurse-submodules master\n\n-> The result is as follow:\n\nInitialized empty Git repository in /data/dams/var/tmp/ploum/.git/\n[master (root-commit) ec7c09a] foo\n 1 file changed, 1 insertion(+)\n create mode 100644 foo\nBranch 'nosubmodules' set up to track local branch 'master'.\nInitialized empty Git repository in /data/dams/var/tmp/ploum/plam/.git/\n[master (root-commit) 35e6696] bar\n 1 file changed, 1 insertion(+)\n create mode 100644 bar\nInitialized empty Git repository in /data/dams/var/tmp/ploum/plam/plim/.git/\n[master (root-commit) b4712c1] baz\n 1 file changed, 1 insertion(+)\n create mode 100644 baz\nAdding existing repo at 'plim' to the index\n[master 989c11d] Add submodule plim\n 2 files changed, 4 insertions(+)\n create mode 100644 .gitmodules\n create mode 160000 plim\nAdding existing repo at 'plam' to the index\n[master 5b34041] Add submodule plam\n 2 files changed, 4 insertions(+)\n create mode 100644 .gitmodules\n create mode 160000 plam\nMigrating git directory of 'plam' from\n'/data/dams/var/tmp/ploum/plam/.git' to\n'/data/dams/var/tmp/ploum/.git/modules/plam'\nMigrating git directory of 'plam/plim' from\n'/data/dams/var/tmp/ploum/plam/plim/.git' to\n'/data/dams/var/tmp/ploum/.git/modules/plam/modules/plim'\nSwitched to branch 'nosubmodules'\nYour branch is behind 'master' by 1 commit, and can be fast-forwarded.\n  (use \"git pull\" to update your local branch)\nfatal: exec '--super-prefix=plam/plim/': cd to 'plim' failed: No such file or directory\nerror: Submodule 'plim' could not be updated.\nerror: Submodule 'plam/plim' cannot checkout new HEAD.\nerror: Submodule 'plam' could not be updated.\nM\tplam\nSwitched to branch 'master'\n\nAs you can see, the nested plim submodules could not be recreated since the\nfolder does not exists yet in the 'nosubmodules' branch. This makes the\n'plam' submodule update fails, and in the following state\n\nUnstaged changes after reset:\nD\t.gitmodules\nD\tbar\nD\tplim\n\n-> To recover\n\nIn the folder plam, do a `git reset` followed by a `git reset --hard`\n(`git reset --hard` directly does not work:\nfatal: exec '--super-prefix=plim/': cd to 'plim' failed: No such file or directory)\n\nIndeed the first reset, which puts .gitmodules back in the index, is what\nallows to do the `git submodule update` implied by `git reset --hard`.\nSince (from what I understand) the path is only read from .gitmodules and\nnot from the .git/config (where there are only the submodules name which\nare distinct from the path), this explain the failures observed.\n\nNote that I wasn't able to reproduce in this small examples, but when\ntrying to repair I also add some strange errors of the form\n'.git is not a git directory' (where .git was a pseudo symlink\ngitdir: ../.git/modules/plam).\n\n-> Question\n\nMy usage is probably non standard (I have quite a lot of nested\nsubmodules), so I had a hard time to recover from this checkout. Is there a\nbetter way? Would it be possible to make nested submodules checkout of this\nform work out of the box?\n\nThanks!\nDamien Robert\n"},{"id":"391887","messageId":"0123F1ED-C421-4C1F-896B-E54C9D345A34@gmail.com","threadId":"52807","inReplyTo":"20200214224242.knmzkwx7ls4sote7@doriath","subject":"Re: Nested submodule checkout","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2020-02-17T04:51:43Z","receivedAt":"2020-02-17T04:51:54Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi Damien,\n\nI reported the same bug to the list back in September [1], and I’m glad to say I just finished (today!) a patch series [2] that fixes this bug.\n\n> Le 14 févr. 2020 à 17:42, Damien Robert <damien.olivier.robert@gmail.com> a écrit :\n> \n> Dear git developers,\n> \n> I stumbled into this situation from which it was a bit painful to recover:\n> \n> -> test script:\n> \n>    mkdir ploum\n>    cd ploum\n>    git init\n>    echo 'foo' > foo\n>    git add foo\n>    git commit -m foo\n>    git branch nosubmodules\n> \n>    mkdir plam\n>    cd plam\n>    git init\n>    echo 'bar' > bar\n>    git add bar\n>    git commit -m bar\n> \n>    mkdir plim\n>    cd plim\n>    git init\n>    echo 'baz' > baz\n>    git add baz\n>    git commit -m baz\n> \n>    cd ..\n>    git submodule add ./plim\n>    git commit -am 'Add submodule plim'\n> \n>    cd ..\n>    git submodule add ./plam\n>    git commit -am 'Add submodule plam'\n> \n>    git checkout nosubmodules\n>    git checkout --recurse-submodules master\n> \n> -> The result is as follow:\n> \n> Initialized empty Git repository in /data/dams/var/tmp/ploum/.git/\n> [master (root-commit) ec7c09a] foo\n> 1 file changed, 1 insertion(+)\n> create mode 100644 foo\n> Branch 'nosubmodules' set up to track local branch 'master'.\n> Initialized empty Git repository in /data/dams/var/tmp/ploum/plam/.git/\n> [master (root-commit) 35e6696] bar\n> 1 file changed, 1 insertion(+)\n> create mode 100644 bar\n> Initialized empty Git repository in /data/dams/var/tmp/ploum/plam/plim/.git/\n> [master (root-commit) b4712c1] baz\n> 1 file changed, 1 insertion(+)\n> create mode 100644 baz\n> Adding existing repo at 'plim' to the index\n> [master 989c11d] Add submodule plim\n> 2 files changed, 4 insertions(+)\n> create mode 100644 .gitmodules\n> create mode 160000 plim\n> Adding existing repo at 'plam' to the index\n> [master 5b34041] Add submodule plam\n> 2 files changed, 4 insertions(+)\n> create mode 100644 .gitmodules\n> create mode 160000 plam\n\nHere you just did ` git commit -am 'Add submodule plam’` so the next command according to your reproducer above would be `git checkout nosubmodules`\n\n> Migrating git directory of 'plam' from\n> '/data/dams/var/tmp/ploum/plam/.git' to\n> '/data/dams/var/tmp/ploum/.git/modules/plam'\n> Migrating git directory of 'plam/plim' from\n> '/data/dams/var/tmp/ploum/plam/plim/.git' to\n> '/data/dams/var/tmp/ploum/.git/modules/plam/modules/plim'\n> Switched to branch 'nosubmodules'\n> Your branch is behind 'master' by 1 commit, and can be fast-forwarded.\n>  (use \"git pull\" to update your local branch)\n\nHere, git is migrating the git directories of both submodules to the git directory of the superproject (ploum). This tells me you probably have the `submodule.recurse` config set somewhere, as this is the behaviour I get I if I do `git checkout --recurse-submodules nosubmodules`.\nIf I just do `git checkout nosubmodules`, I get \n    \n    $ git checkout nosubmodules \n    warning: unable to rmdir 'plam': Directory not empty\n    Switched to branch 'nosubmodules'\n    Your branch is behind 'master' by 1 commit, and can be fast-forwarded.\n      (use \"git pull\" to update your local branch)\n\nand then doing `git checkout --recurse-submodules master` actually works. \n\n> fatal: exec '--super-prefix=plam/plim/': cd to 'plim' failed: No such file or directory\n> error: Submodule 'plim' could not be updated.\n> error: Submodule 'plam/plim' cannot checkout new HEAD.\n> error: Submodule 'plam' could not be updated.\n> M\tplam\n> Switched to branch 'master'\n> \n> As you can see, the nested plim submodules could not be recreated since the\n> folder does not exists yet in the 'nosubmodules' branch.\n\nThat’s the cause of the bug: in fact git tries to change directory into ‘plim’ before the ‘plim’ directory is created in the filesystem.\n\n> This makes the\n> 'plam' submodule update fails, and in the following state\n> \n> Unstaged changes after reset:\n> D\t.gitmodules\n> D\tbar\n> D\tplim\n\nAt this point, if you go into ‘plam’ and do `git ls-files -s` (to list the content of the index), you will see that the index is empty (the checkout died before the index could be populated.) \n\n> -> To recover\n> \n> In the folder plam, do a `git reset` followed by a `git reset --hard`\n> (`git reset --hard` directly does not work:\n> fatal: exec '--super-prefix=plim/': cd to 'plim' failed: No such file or directory)\n\nThat’s another hint that you have `submodule.recurse` set. I don’t get this error doing `git reset --hard`, but I get it doing `git reset --hard --recurse-submodules` (or `git reset --hard --r`, which works and is quicker to type!). `git reset` populates the index, so now `git ls-files -s` would now show the correct content of ‘plam’.\n\n> Indeed the first reset, which puts .gitmodules back in the index, is what\n> allows to do the `git submodule update` implied by `git reset --hard`.\n\nIn fact, `git reset --hard` does not spawn `git submodule update`, it calls functions in unpack-trees.c (that actually spawn `git read-tree —recurse-submodules`) to update the submodules recursively. \n\n> Note that I wasn't able to reproduce in this small examples, but when\n> trying to repair I also add some strange errors of the form\n> '.git is not a git directory' (where .git was a pseudo symlink\n> gitdir: ../.git/modules/plam).\n> \n> -> Question\n> \n> My usage is probably non standard (I have quite a lot of nested\n> submodules), so I had a hard time to recover from this checkout. Is there a\n> better way? Would it be possible to make nested submodules checkout of this\n> form work out of the box?\n\nIt is supposed to work out of the box when using `--recurse-submodules` or the `submodule.recurse` config. Although, it’s always possible to run into some bugs. \nOne thing you can do to get out of tricky situations is to temporarily deactivate the config (`git -c submodule.recurse=0 <command>`). For example, after the failed `git checkout --recurse-submodules master` above, issuing\n\n    git -c submodule.recurse=0 submodule update --recursive --force\n\nwould have correctly checked out the submodules. I have a git alias ‘no-rs’ (for no recurse-submodules) that I use in these situations:\n\n    git config --global alias.no-rs ‘-c submodule.recurse=0’\n\nThen the `submodule update` call above could be shortened to \n\n    git no-rs submodule update --recursive --force\n\nNote that using the `submodule.recurse` config also applies to internal calls to git commands (issued by other git commands), so using adding `--no-recurse-submodules` to the command line might not be enough to completely turn off the effect of that config, hence this handy alias.\n\n> \n> Thanks!\n> Damien Robert\n> \n\nCheers,\nPhilippe.\n\n[1] https://lore.kernel.org/git/7437BB59-4605-48EC-B05E-E2BDB2D9DABC@gmail.com/\n[2] https://github.com/gitgitgadget/git/pull/555"},{"id":"391993","messageId":"20200218170808.dh4s65b475yqs3oz@mithrim","threadId":"52807","inReplyTo":"0123F1ED-C421-4C1F-896B-E54C9D345A34@gmail.com","subject":"Re: Nested submodule checkout","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-02-18T17:08:08Z","receivedAt":"2020-02-18T17:08:13Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"Hi Philippe,\n\nFrom Philippe Blain, Sun 16 Feb 2020 at 23:51:43 (-0500) :\n> I reported the same bug to the list back in September [1]\n\nMeta question: is there an easy way I could have found your bug report?\n\n> and I’m glad to say I just finished (today!) a patch series [2] that fixes this bug.\n\nGreat! Thanks for the patches!\n\n> Here you just did ` git commit -am 'Add submodule plam’` so the next command according to your reproducer above would be `git checkout nosubmodules`\n> \n> > Migrating git directory of 'plam' from\n> > '/data/dams/var/tmp/ploum/plam/.git' to\n> > '/data/dams/var/tmp/ploum/.git/modules/plam'\n> > Migrating git directory of 'plam/plim' from\n> > '/data/dams/var/tmp/ploum/plam/plim/.git' to\n> > '/data/dams/var/tmp/ploum/.git/modules/plam/modules/plim'\n> > Switched to branch 'nosubmodules'\n> > Your branch is behind 'master' by 1 commit, and can be fast-forwarded.\n> >  (use \"git pull\" to update your local branch)\n> \n> Here, git is migrating the git directories of both submodules to the git directory of the superproject (ploum). This tells me you probably have the `submodule.recurse` config set somewhere, as this is the behaviour I get I if I do `git checkout --recurse-submodules nosubmodules`.\n\nYes, you are of course right. I should have specified it in my bug report,\nI tried to specify the setting explicitely in the last checkout but as you\nnoticed I forgot to specify it in all other commands, sorry about the\nnoise.\n\n> That’s another hint that you have `submodule.recurse` set. I don’t get this error doing `git reset --hard`, but I get it doing `git reset --hard --recurse-submodules` (or `git reset --hard --r`, which works and is quicker to type!). `git reset` populates the index, so now `git ls-files -s` would now show the correct content of ‘plam’.\n\nOh, I did not know git expand unambiguous long options!\n\nBy the way the fact that `git reset` also support `--recurse-submodules` is not\nspecified in the man page. (It is in the help text thought).\n\nAnd it would be nice if the documentation of submodule.recurse in\ngit-config specify the list of all affected commands, rather than just \"all\ncommands that have a --recurse-submodules options\".\n\n(I could send a patch for this if there is interest)\n\n> > Note that I wasn't able to reproduce in this small examples, but when\n> > trying to repair I also add some strange errors of the form\n> > '.git is not a git directory' (where .git was a pseudo symlink\n> > gitdir: ../.git/modules/plam).\n\n-> This is explained by your Patch 5/6\n\n> would have correctly checked out the submodules. I have a git alias ‘no-rs’ (for no recurse-submodules) that I use in these situations:\n>     git config --global alias.no-rs ‘-c submodule.recurse=0’\n\nCan you use alias to define option settings (rather than commands followed\nby options) without using the 'f() {}' trick?\n\nUsing your alias, I get\nfatal: empty alias for no-rs\n\n> Then the `submodule update` call above could be shortened to \n>     git no-rs submodule update --recursive --force\n\n> Note that using the `submodule.recurse` config also applies to internal calls to git commands (issued by other git commands), so using adding `--no-recurse-submodules` to the command line might not be enough to completely turn off the effect of that config, hence this handy alias.\n\nOhh, good to know!\n\n-- \nDamien Robert\nhttp://www.normalesup.org/~robert/pro\n"},{"id":"392047","messageId":"14BE3511-17A8-48D8-B0C4-923745E5462A@gmail.com","threadId":"52807","inReplyTo":"20200218170808.dh4s65b475yqs3oz@mithrim","subject":"Re: Nested submodule checkout","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2020-02-19T04:19:20Z","receivedAt":"2020-02-19T04:19:28Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi Damien,\n\n> Le 18 févr. 2020 à 12:08, Damien Robert <damien.olivier.robert@gmail.com> a écrit :\n> \n> Hi Philippe,\n> \n> From Philippe Blain, Sun 16 Feb 2020 at 23:51:43 (-0500) :\n>> I reported the same bug to the list back in September [1]\n> \n> Meta question: is there an easy way I could have found your bug report?\n\nThat’s a good question. The search capabilities of public-inbox (the software running at lore.kernel.org) are explained at \nhttps://lore.kernel.org/git/_/text/help/\n\nIn this case, my bug report comes out in the following searches:\nb:(nested submodule checkout)\ns:(bug* submodule* checkout)\n\nthe first one can be refined with\nb:(nested submodule checkout) NOT s:(cooking) NOT s:(ANNOUNCE) NOT s:(PATCH*)\n\n> \n>> That’s another hint that you have `submodule.recurse` set. I don’t get this error doing `git reset --hard`, but I get it doing `git reset --hard --recurse-submodules` (or `git reset --hard --r`, which works and is quicker to type!). `git reset` populates the index, so now `git ls-files -s` would now show the correct content of ‘plam’.\n> \n> Oh, I did not know git expand unambiguous long options!\n\nYes, it’s not widely known but can be useful! It’s documented in gitcli(7) (`man gitcli`) or online at\nhttps://git-scm.com/docs/gitcli#_abbreviating_long_options\n\n> By the way the fact that `git reset` also support `--recurse-submodules` is not\n> specified in the man page. (It is in the help text thought).\n\nYes, this is on my radar, I’d like to get some time to clean this up sometimes soon. It’s also missing for `git restore`.\n\n> And it would be nice if the documentation of submodule.recurse in\n> git-config specify the list of all affected commands, rather than just \"all\n> commands that have a --recurse-submodules options\".\n> \n> (I could send a patch for this if there is interest)\n\nThat’s a very good idea in my opinion, especially since the search functionality on git-scm.com is presently limited to 10 hits from the manpages so it fails to find all occurrences of `--recurse-submodules` [1].\n\n[1] https://github.com/git/git-scm.com/issues/1374\n\n>> would have correctly checked out the submodules. I have a git alias ‘no-rs’ (for no recurse-submodules) that I use in these situations:\n>>    git config --global alias.no-rs ‘-c submodule.recurse=0’\n> \n> Can you use alias to define option settings (rather than commands followed\n> by options) without using the 'f() {}' trick?\n> \n> Using your alias, I get\n> fatal: empty alias for no-rs\n\nThe alias specifies an option directly to the git \"main\" executable instead of a specific Git command (in this case setting a particular configuration variable for the duration of the command that will follow). This alias works for me, at least on Git 2.25. The git-config man page mentions aliases can start with an option to git [2]. If it does not work in an older version of Git then you can use \n\n    git config --global alias.no-rs ‘!git -c submodule.recurse=0'\n\nwhich should do the trick.\n\n[2] https://git-scm.com/docs/git-config#Documentation/git-config.txt-alias\n\n>> Note that using the `submodule.recurse` config also applies to internal calls to git commands (issued by other git commands), so using adding `--no-recurse-submodules` to the command line might not be enough to completely turn off the effect of that config, hence this handy alias.\n> \n> Ohh, good to know!\n\nIf you do go ahead with a patch listing the commands affected by submodule.recurse in git-config, then I think this should also be mentioned. \n\nCheers,\nPhilippe.\n"},{"id":"392545","messageId":"20200226172338.unembhjhog36sqj7@mithrim","threadId":"52807","inReplyTo":"0123F1ED-C421-4C1F-896B-E54C9D345A34@gmail.com","subject":"Nested submodule status bug","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-02-26T17:23:38Z","receivedAt":"2020-02-26T17:23:46Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"Dear git developers,\n\nstill about nested submodules, I stumbled upon the following very strange\nbehaviour.\n\n1) Make a baz submodule inside a bar submodule inside foo\n  mkdir foo; cd foo; git init\n  mkdir bar; cd bar; git init\n  mkdir baz; cd baz; git init\n  touch 1\n  git add 1\n  git commit -m 'first baz commit'\n  cd ..\n  git submodule add ./baz\n  git commit -m \"Add baz submodule\"\n  cd ..\n  git submodule add ./bar\n  git commit -m \"Add bar submodule\\n\"\n  git submodule absorbgitdirs\n  cd bar\n  touch 2\n\n2) Now add the following alias:\n  subs=\"! git -C baz status\"\n\n3) And run it\n  git subs\n\nTo get:\n~~~\nOn branch master\nUntracked files:\n  (use \"git add <file>...\" to include in what will be committed)\n\t../2\n\nnothing added to commit but untracked files present (use \"git add\" to track)\n~~~\n\n\nWhile when not launched as an alias, `git -C baz status` give the correct\nresult:\nOn branch master\nnothing to commit, working tree clean\n\n\nNote: I get the same bug when I use\n  subs=\"! sh -c 'cd baz; git status'\"\nThe bug is only there if absorbgitdirs is used. Also I have to be in an\nintermediate submodule.\n\nBest regards,\nDamien\n"},{"id":"392604","messageId":"20200227100557.ydouc4n3jphzbits@feanor","threadId":"52807","inReplyTo":"20200226172338.unembhjhog36sqj7@mithrim","subject":"Re: Nested submodule status bug","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-02-27T10:05:57Z","receivedAt":"2020-02-27T10:06:04Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"From Damien Robert, Wed 26 Feb 2020 at 18:23:41 (+0100) :\n> 2) Now add the following alias:\n>   subs=\"! git -C baz status\"\n> \n> 3) And run it\n>   git subs\n> \n> To get:\n> ~~~\n> On branch master\n> Untracked files:\n>   (use \"git add <file>...\" to include in what will be committed)\n> \t../2\n> \n> nothing added to commit but untracked files present (use \"git add\" to track)\n> ~~~\n\nOk, so using GIT_TRACE and GIT_TRACE_STATUS and doing some debugging I\nunderstand the cause of the bug. When using the alias,\nthe GIT_DIR environment is set to the bar module:\n    GIT_DIR=/tmp/test/foo/.git/modules/bar\nand is exported.\n\nThat's explain why in `baz` the status report the status of the `bar`\ndirectory.\n"},{"id":"392605","messageId":"20200227104330.hp7zf2suquxsf6zw@feanor","threadId":"52807","inReplyTo":"20200227100557.ydouc4n3jphzbits@feanor","subject":"Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-02-27T10:43:30Z","receivedAt":"2020-02-27T10:43:37Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"> Ok, so using GIT_TRACE and GIT_TRACE_STATUS and doing some debugging I\n> understand the cause of the bug. When using the alias,\n> the GIT_DIR environment is set to the bar module:\n>     GIT_DIR=/tmp/test/foo/.git/modules/bar\n> and is exported.\n> \n> That's explain why in `baz` the status report the status of the `bar`\n> directory.\n\nAnd looking at setup.c:760 it appears that `set_git_dir` (which set up among\nother things the GIT_DIR env) is called because we are in a worktree.\n\nMore precisely: we have discovered a git dir (a git file), so we call\nsetup_discovered_git_dir, which itself calls setup_explicit_git_dir because\nwe are in a work tree environment. And set_up_explicit_git_dir calls\nset_git_dir.\n\n\nSo I don't know what's the proper way to fix this. Keep track when we add\nourselves the GIT_DIR rather than when it is already provided and remove it\nbefore executing a shell alias? Don't run the full configuration setup when\nwe are executing a shell alias anyway?\n\nI am adding Nguyễn Thái Ngọc Duy to the discussion (I hope this is the\nproper etiquette).\n\nBest regards,\nDamien\n"},{"id":"392619","messageId":"20200227155057.7idpa447ixo6sf6j@feanor","threadId":"52807","inReplyTo":"20200227104330.hp7zf2suquxsf6zw@feanor","subject":"GIT_DIR in aliases [Re: Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-02-27T15:50:57Z","receivedAt":"2020-02-27T15:51:04Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"From Damien Robert, Thu 27 Feb 2020 at 11:43:30 (+0100) :\n> And looking at setup.c:760 it appears that `set_git_dir` (which set up among\n> other things the GIT_DIR env) is called because we are in a worktree.\n> \n> More precisely: we have discovered a git dir (a git file), so we call\n> setup_discovered_git_dir, which itself calls setup_explicit_git_dir because\n> we are in a work tree environment. And set_up_explicit_git_dir calls\n> set_git_dir.\n> \n> So I don't know what's the proper way to fix this. Keep track when we add\n> ourselves the GIT_DIR rather than when it is already provided and remove it\n> before executing a shell alias? Don't run the full configuration setup when\n> we are executing a shell alias anyway?\n\nSorry about the spam. Thinking about it some more, this has nothing to do\nwith submodules or worktrees, but about how we expect shell aliases to\nwork.\n\nIn Documentation/config/alias, it is documented that the alias will run a\nthe top level and GIT_PREFIX will be set.\n\nIn git.c#handle_alias, it is explicitly stated that shell alias call\nsetup_git_directory_gently because:\n/* Aliases expect GIT_PREFIX, GIT_DIR etc to be set */\n\nSo one might argue that the behaviour I observed is not a bug, but it is\nstill surprising for me (as a user), and maybe this could be stated more\nclearly in the docs?\n\nFurthermore there is a question of consistency. GIT_DIR will not always be set\nbefore running a shell alias. Looking at `setup_discovered_git_dir`, it will\nbe set if we are in a bare dir, or core.worktree / WORK_TREE is set, or if\nwe have a gitfile.\n\nThe annoying side effect is that I cannot use as an alias a command that\niterate over submodules and run git commands inside them, because in this\nalias GIT_DIR will be set sometimes, and sometimes not (a quick fix would be to\nunset GIT_DIR in my alias).\n\nSo what would be the best way to handle this?\nThanks,\nDamien\n"},{"id":"392683","messageId":"20200228190218.GC1408759@coredump.intra.peff.net","threadId":"52807","inReplyTo":"20200227155057.7idpa447ixo6sf6j@feanor","subject":"Re: GIT_DIR in aliases [Re: Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-02-28T19:02:18Z","receivedAt":"2020-02-28T19:02:21Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 27, 2020 at 04:50:57PM +0100, Damien Robert wrote:\n\n> So one might argue that the behaviour I observed is not a bug, but it is\n> still surprising for me (as a user), and maybe this could be stated more\n> clearly in the docs?\n> \n> Furthermore there is a question of consistency. GIT_DIR will not always be set\n> before running a shell alias. Looking at `setup_discovered_git_dir`, it will\n> be set if we are in a bare dir, or core.worktree / WORK_TREE is set, or if\n> we have a gitfile.\n\nWe were discussing the same issue recently with regards to hooks. See:\n\n  https://lore.kernel.org/git/20200130102933.GE840531@coredump.intra.peff.net/\n\nand the responses. I think we could do better, but at the cost of\nbreaking a relatively obscure git-clone feature.\n\n> The annoying side effect is that I cannot use as an alias a command that\n> iterate over submodules and run git commands inside them, because in this\n> alias GIT_DIR will be set sometimes, and sometimes not (a quick fix would be to\n> unset GIT_DIR in my alias).\n\nYes, the recommended thing is to make sure GIT_DIR is unset if you're\ngoing to chdir around and expect auto-discovery of the repository to\nwork.\n\nNote there are other variables you might want to unset, too, if you're\nswitching repositories. Doing:\n\n  unset $(git rev-parse --local-env-vars)\n\nwould cover the full list.\n\n-Peff\n"},{"id":"392687","messageId":"xmqq4kvaea2l.fsf@gitster-ct.c.googlers.com","threadId":"52807","inReplyTo":"20200228190218.GC1408759@coredump.intra.peff.net","subject":"Re: GIT_DIR in aliases [Re: Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-28T20:22:26Z","receivedAt":"2020-02-28T20:22:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Yes, the recommended thing is to make sure GIT_DIR is unset if you're\n> going to chdir around and expect auto-discovery of the repository to\n> work.\n>\n> Note there are other variables you might want to unset, too, if you're\n> switching repositories. Doing:\n>\n>   unset $(git rev-parse --local-env-vars)\n>\n> would cover the full list.\n\nThanks for mentioning --local-env-vars; I was going to ask you to\nmention GIT_WORK_TREE anytime you mention unsetting GIT_DIR, but you\ndid it much better ;-).\n\n"},{"id":"392705","messageId":"6D4FEE68-EF3C-491E-A7C1-3454B88977C3@gmail.com","threadId":"52807","inReplyTo":"20200227155057.7idpa447ixo6sf6j@feanor","subject":"Re: GIT_DIR in aliases [Re: Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2020-02-29T01:42:52Z","receivedAt":"2020-02-29T01:43:01Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi Damien,\n\n> Le 27 févr. 2020 à 10:50, Damien Robert <damien.olivier.robert@gmail.com> a écrit :\n> \n> So one might argue that the behaviour I observed is not a bug, but it is\n> still surprising for me (as a user), and maybe this could be stated more\n> clearly in the docs?\n\nIt sure is surprising, and not super intuitive.\n\n> \n> Furthermore there is a question of consistency. GIT_DIR will not always be set\n> before running a shell alias. Looking at `setup_discovered_git_dir`, it will\n> be set if we are in a bare dir, or core.worktree / WORK_TREE is set, or if\n> we have a gitfile.\n> \n> The annoying side effect is that I cannot use as an alias a command that\n> iterate over submodules and run git commands inside them, because in this\n> alias GIT_DIR will be set sometimes, and sometimes not (a quick fix would be to\n> unset GIT_DIR in my alias).\n\nIf that’s what your are trying to do, ‘git submodule foreach’ [1] seems like a better way to go. \nFor example I have an alias ‘st-sub’ that calls git status in the superproject and then recursively in the submodules:\n\n    $ cat ~/.gitconfig | grep st-sub\n          st-sub = !git status \"$@\" && git sub foreach --recursive git status \"$@\"\n\n[1] https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-foreach--recursiveltcommandgt\n\n> \n> So what would be the best way to handle this?\n> Thanks,\n> Damien\n\nCheers,\nPhilippe."},{"id":"392818","messageId":"20200303215650.vpsfgcjrsv42dqob@doriath","threadId":"52807","inReplyTo":"6D4FEE68-EF3C-491E-A7C1-3454B88977C3@gmail.com","subject":"Re: GIT_DIR in aliases [Re: Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-03-03T21:56:50Z","receivedAt":"2020-03-03T21:56:56Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"> If that’s what your are trying to do, ‘git submodule foreach’ [1] seems like a better way to go. \n> For example I have an alias ‘st-sub’ that calls git status in the superproject and then recursively in the submodules:\n> \n>     $ cat ~/.gitconfig | grep st-sub\n>           st-sub = !git status \"$@\" && git sub foreach --recursive git status \"$@\"\n\nI have almost the same aliases :)\n\ts = status --short --branch\n\tsubfor=\"submodule foreach --recursive\"\n\tsubs=\"!f() { git subfor git s \\\"$@\\\";}; f\"\n\trecs=\"!f() { git s \\\"$@\\\"; git subs \\\"$@\\\";}; f\"\n\nBut in practice I often want to run commands only in *changed* submodules.\n\nFor instance with the above 'git subs' yields a lot of non informative\nmessages like:\n  Entering 'non/changed/submodule'\n  ## master...origin/master\n(and git foreach --quiet) removes the first line but not the second.\n\nThis is also useful for an alias like 'recursive_commit' where I only want\nto commit changed submodules, to not get a lot of messages like\n     nothing to commit, working tree clean.\n\nSo I wrote a script that parses the output of 'git status --porcelain=v2'\nand run a command only in changed submodules (where I can specify if\nchanged means 'C', 'M' or 'U'). Except that I did not think about unseting\nGIT_DIR and stumbled into this bug (I am sorry I went a bit on a wild chase\nin this mailing list but I had already spent quite some time in trying to\nfind a minimal exemple, I first thought that my program had a parsing bug).\n\nNow that I think about it, I probably also could do that with 'git\nsubmodule foreach', by testing 'git diff --no-ext-diff [--cached] --quiet'\nfirst before running the subcommand. This would give me the 'M' submodules,\nbut it is not clear to me how I could test for 'C' changed submodules with\nthis method.\n\n-- \nDamien Robert\nhttp://www.normalesup.org/~robert/pro\n"},{"id":"392827","messageId":"20200303224430.c64dmvzr4c72f2tv@doriath","threadId":"52807","inReplyTo":"20200228190218.GC1408759@coredump.intra.peff.net","subject":"Re: GIT_DIR in aliases [Re: Spurious GIT_DIR set when in a worktree [was Re: Nested submodule status bug]","fromName":"Damien Robert","fromEmail":"damien.olivier.robert@gmail.com","sentAt":"2020-03-03T22:44:30Z","receivedAt":"2020-03-03T22:44:40Z","isPatch":false,"sender":{"key":"damien.olivier.robert@gmail.com","avatar":null},"body":"From Jeff King, Fri 28 Feb 2020 at 14:02:18 (-0500) :\n> > Furthermore there is a question of consistency. GIT_DIR will not always be set\n> > before running a shell alias. Looking at `setup_discovered_git_dir`, it will\n> > be set if we are in a bare dir, or core.worktree / WORK_TREE is set, or if\n> > we have a gitfile.\n\n> We were discussing the same issue recently with regards to hooks. See:\n>   https://lore.kernel.org/git/20200130102933.GE840531@coredump.intra.peff.net/\n> and the responses. I think we could do better, but at the cost of\n> breaking a relatively obscure git-clone feature.\n\nYes I found this discussion afterwards. This is related but a bit\ndifferent, the problem you discuss is that GIT_WORK_TREE is not always set\neven when GIT_DIR is. The problem I have is that GIT_DIR is not always set; it\ndepends on the phase of the moon (are we in a worktree? ...), which makes\nthings that forget to unset it half broken and hard to debug.\n\nI would argue that for scripts and hooks we should always set both. This is\neasy to do (at least for scripts, I haven't looked at hooks) without\ntouching at setup_discovered_git_dir, so not affecting git clone.\n\nIn fact I would even argue that in setup_discovered_git_dir we should\nalways set both, and fix the breakages that appear, like the one you\nmention for git clone. For instance I wonder if the recent report about\n'git gui' setting GIT_DIR is not due to this.\n\n> Note there are other variables you might want to unset, too, if you're\n> switching repositories. Doing:\n>   unset $(git rev-parse --local-env-vars)\n> would cover the full list.\n\nThanks for the hint; I had forgotten about local-env-vars.\n"}]}