sudmodule.<name>.recurse ignored

> THE CONFIGURATION OF SUBMODULES
>        Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):
>
>        •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
>            flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
>            take enums, such as fetch and push, where you can specify how submodules are affected.
>
>        •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
>            as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.
>
>            For example an effect from the submodule’s .gitignore file would be observed when you run git status
>            --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
>            status in the submodule while paying attention to the .gitignore file of the submodule.
>
>            The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
>            superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
>            in the submodule as usual in the $GIT_DIR/config file.
>
>        •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
>            SUBMODULES" section below).
>
>            If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
>            the submodule from is configured here for example.
>
>        •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
>            collection of repositories for the mapping that is required between a submodule’s name and its path.
>
>            This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
>            submodule’s Git directory can be located.
>
>            If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
>            the last fallback to specify where to obtain the submodule from.
>
>
> However, when we are talking about the recurse attribute, it is
> not taken from .gitmodules - only command line and .git/config seem
> to be consulted.
> Is this a bug or a feature?

> 
> On Fri, Oct 07 2022, Michael S. Tsirkin wrote:
> 
> > THE CONFIGURATION OF SUBMODULES
> >        Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):
> >
> >        •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
> >            flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
> >            take enums, such as fetch and push, where you can specify how submodules are affected.
> >
> >        •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
> >            as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.
> >
> >            For example an effect from the submodule’s .gitignore file would be observed when you run git status
> >            --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
> >            status in the submodule while paying attention to the .gitignore file of the submodule.
> >
> >            The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
> >            superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
> >            in the submodule as usual in the $GIT_DIR/config file.
> >
> >        •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
> >            SUBMODULES" section below).
> >
> >            If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
> >            the submodule from is configured here for example.
> >
> >        •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
> >            collection of repositories for the mapping that is required between a submodule’s name and its path.
> >
> >            This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
> >            submodule’s Git directory can be located.
> >
> >            If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
> >            the last fallback to specify where to obtain the submodule from.
> >
> >
> > However, when we are talking about the recurse attribute, it is
> > not taken from .gitmodules - only command line and .git/config seem
> > to be consulted.
> > Is this a bug or a feature?
> 
> It's a (security) feature. We have had disussions[1] about how to safely
> read in-repo config, but we are very far away from that becoming a
> reality (if it ever happens).
> 
> If we read this from the .gitmodules the repo could change how you
> interact with your submodules.
> 
> 1. https://lore.kernel.org/git/YzXwZQbM69eNJfm7@nand.local/

> On Fri, Oct 07, 2022 at 01:09:39PM +0200, Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Fri, Oct 07 2022, Michael S. Tsirkin wrote:
>> 
>> > THE CONFIGURATION OF SUBMODULES
>> >        Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):
>> >
>> >        •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
>> >            flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
>> >            take enums, such as fetch and push, where you can specify how submodules are affected.
>> >
>> >        •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
>> >            as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.
>> >
>> >            For example an effect from the submodule’s .gitignore file would be observed when you run git status
>> >            --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
>> >            status in the submodule while paying attention to the .gitignore file of the submodule.
>> >
>> >            The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
>> >            superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
>> >            in the submodule as usual in the $GIT_DIR/config file.
>> >
>> >        •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
>> >            SUBMODULES" section below).
>> >
>> >            If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
>> >            the submodule from is configured here for example.
>> >
>> >        •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
>> >            collection of repositories for the mapping that is required between a submodule’s name and its path.
>> >
>> >            This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
>> >            submodule’s Git directory can be located.
>> >
>> >            If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
>> >            the last fallback to specify where to obtain the submodule from.
>> >
>> >
>> > However, when we are talking about the recurse attribute, it is
>> > not taken from .gitmodules - only command line and .git/config seem
>> > to be consulted.
>> > Is this a bug or a feature?
>> 
>> It's a (security) feature. We have had disussions[1] about how to safely
>> read in-repo config, but we are very far away from that becoming a
>> reality (if it ever happens).
>> 
>> If we read this from the .gitmodules the repo could change how you
>> interact with your submodules.
>> 
>> 1. https://lore.kernel.org/git/YzXwZQbM69eNJfm7@nand.local/
>
> Hmm. Well we have a bunch of attributes there. Why this one? What did I miss?

> - where to get the submodule from during a fetch/clone, e.g.
>   `submodule.<name>.url`, `submodule.<name>.branch`
> - information needed for submodules to work correctly, e.g.
>   `submodule.<name>.path`
> - default config that gets populated in .git/config via `git submodule
>   init`, e.g. `submodule.<name>.update`,
>   `submodule.<name>.fetchRecurseSubmodules`
>
> With respect to config, the general stance is that .gitmodules is meant
> to provide defaults for .git/config and _not_ to be read directly (see
> Ævar's answer).

> Glen Choo <chooglen@google.com> writes:
> 
> > - where to get the submodule from during a fetch/clone, e.g.
> >   `submodule.<name>.url`, `submodule.<name>.branch`
> > - information needed for submodules to work correctly, e.g.
> >   `submodule.<name>.path`
> > - default config that gets populated in .git/config via `git submodule
> >   init`, e.g. `submodule.<name>.update`,
> >   `submodule.<name>.fetchRecurseSubmodules`
> >
> > With respect to config, the general stance is that .gitmodules is meant
> > to provide defaults for .git/config and _not_ to be read directly (see
> > Ævar's answer).
> 
> I think that is a sensible way to think about this.
> 
> It is more like "suggestion made by the upstream", to be taken after
> examining it, just like you should type "make" after examining the
> Makefile.  At least enough to trust its contents ;-)
> 
> > There are some very limited cases where config is read
> > directly from .gitmodules, but that approach is very frowned upon and
> > it's unlikely that we'll add more of them in the future.
> 
> Yup, we should be careful and see if we can get rid of existing
> uses.
> 
> Thanks.

> Fundamentally the problem we encounter regularly is this:
> 	qemu is superproject, ui/keycodemapdb is subproject.
>
>
> 	I have a change on master setting the submodule commit:
>
> 	qemu$ git show master | grep +Sub
> 	+Subproject commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
>
> 	and check it out:
>
> 	qemu$ git submodule update --init ui/keycodemapdb
> 	Submodule 'ui/keycodemapdb' (https://gitlab.com/qemu-project/keycodemapdb.git) registered for path 'ui/keycodemapdb'
> 	Submodule path 'ui/keycodemapdb': checked out '7381b9bfadd31c4c9e9a10b5bb5032f9189d4352'
>
>
> 	In another branch I have a different commit:
>
> 	qemu$ git show sub-foo  | grep +Sub
> 	+Subproject commit 57ba70da5312170883a3d622cd2aa3fd0e2ec7ae
>
>
> 	Now I switch branches and nothing happens, the submodule
> 	is marked as dirty:
>
> 	qemu$ git checkout sub-foo
> 	M       ui/keycodemapdb
> 	Switched to branch 'sub-foo'
>
> 	qemu$ (cd ui/keycodemapdb && git show | head -1)
> 	commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
>
>
> it is now very easy to generate incorrect code:
> - git commit -a  will commit the change to submodule
> - any build will use a mix of super and subproject that
>   is completely untested
>
>
> As a result people are basically saying avoid using submodules
> but I am wondering whether git can be tweaked to do the
> right thing by default.
>

> "Michael S. Tsirkin" <mst@redhat.com> writes:
> 
> > Fundamentally the problem we encounter regularly is this:
> > 	qemu is superproject, ui/keycodemapdb is subproject.
> >
> >
> > 	I have a change on master setting the submodule commit:
> >
> > 	qemu$ git show master | grep +Sub
> > 	+Subproject commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
> >
> > 	and check it out:
> >
> > 	qemu$ git submodule update --init ui/keycodemapdb
> > 	Submodule 'ui/keycodemapdb' (https://gitlab.com/qemu-project/keycodemapdb.git) registered for path 'ui/keycodemapdb'
> > 	Submodule path 'ui/keycodemapdb': checked out '7381b9bfadd31c4c9e9a10b5bb5032f9189d4352'
> >
> >
> > 	In another branch I have a different commit:
> >
> > 	qemu$ git show sub-foo  | grep +Sub
> > 	+Subproject commit 57ba70da5312170883a3d622cd2aa3fd0e2ec7ae
> >
> >
> > 	Now I switch branches and nothing happens, the submodule
> > 	is marked as dirty:
> >
> > 	qemu$ git checkout sub-foo
> > 	M       ui/keycodemapdb
> > 	Switched to branch 'sub-foo'
> >
> > 	qemu$ (cd ui/keycodemapdb && git show | head -1)
> > 	commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
> >
> >
> > it is now very easy to generate incorrect code:
> > - git commit -a  will commit the change to submodule
> > - any build will use a mix of super and subproject that
> >   is completely untested
> >
> >
> > As a result people are basically saying avoid using submodules
> > but I am wondering whether git can be tweaked to do the
> > right thing by default.
> >
> 
> It sounds like you want submodule.recurse [1] :)

> With that enabled, your
> "git checkout" should behave like "git checkout --recurse-submodules",
> which should make ui/keycodemapdb check out the correct commit as long
> as the submodule commit is present locally. If is is _not_ present
> locally, you will have to run "git submodule update".
> 
> Unfortunately, you typically won't know whether the commit is present
> before running the command. This is yet another of those things that are
> painful with submodules.
> 
> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse
> 
> > I understand we can work around it by asking everyone to
> > create a correct config, but can't we make it DTRT by default
> > to reduce friction?
> 
> You might be interested in the proposed 'new' Submodule UX [2]; one of
> the goals is to make manual submodule management via "git submodule"
> unnecessary.
> 
> As a part of that, you should be able to set "submodule.recurse = true"
> and have high confidence that all necessary submodules and submodule
> commits are present.

> Work on that is still ongoing, but this situation
> should have improved as of [3].
> 
> [2] https://lore.kernel.org/git/YHofmWcIAidkvJiD@google.com/
> [3] https://lore.kernel.org/git/20220308001433.94995-1-chooglen@google.com/
> 
> >
> >
> > -- 
> > MST