Re: [RFD/PATCH] submodule doc: describe where we can configure them

From: Jonathan Nieder <jrnieder@gmail.com>
Date: May 3, 2016, 23:56 UTC
Message-ID: <20160503235652.GA395@google.com>
In-Reply-To: <1462317985-640-1-git-send-email-sbeller@google.com>

Stefan Beller wrote:

> This is similar to the gitignore document, but doesn't mirror
> the current situation. It is rather meant to start a discussion for
> the right approach for mirroring repositories with submodules.

Ooh.

[...]

Show 21 quoted lines

> --- a/Documentation/git-submodule.txt
> +++ b/Documentation/git-submodule.txt
> @@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories
>  that come from both projects can be cloned and checked out as a whole
>  if you choose to go that route.
>  
> +Submodule operations can be configured using the following mechanisms
> +(from highest to lowest precedence):
> +
> + * the command line for those commands that support taking submodule specs.
> +
> + * the configuration file `$GIT_DIR/config`.
> +
> + * the configuration file `config` found in the `refs/submodule/config` branch.
> +   This can be used to overwrite the upstream configuration in the `.gitmodules`
> +   file without changing the history of the project.
> +   Useful options here are overwriting the base, where relative URLs apply to,
> +   when mirroring only parts of the larger collection of submodules.
> +
> + * the `.gitmodules` file inside the repository. A project usually includes this
> +   file to suggest defaults for the upstream collection of repositories.

(This documentation probably belongs in gitmodules(5) --- then, git-submodule(1) could focus on command-line usage and point there for configuration information.)

There are two aspects of this to be separated: what governs the behavior of commands running locally, and where we get information about submodules from a remote repository.

Local commands -------------- The original submodule design was that local commands rely on information from .git/config, and that information gets copied there from .gitmodules when a submodule is initialized. That way, a local user can specify their preferred mirror or other options using some straightforward 'git config' commands.

As a side effect, the settings in .git/config tell git which submodules to pay attention to (which submodules were initialized).

When .gitmodules changes, the settings in .git/config are left alone, since the end user *might* have manually set something up and we don't want to trample on it.

This design is somewhat problematic for a few reasons:

- When I want to stop paying attention to a particular submodule and
  start paying attention to it again later, all my local settings are
  gone.

- When upstream adds a new submodule, I have to do the same manual
  work to change the options for that new submodule.

- When upstream changes submodule options (perhaps to fix a URL
  typo), I don't get those updates.

A fix is to use settings from .git/config when present and fall back to .gitmodules when not. I believe the submodule code has been slowly moving in that direction for new features. Perhaps we can do so for existing features (like submodule.*.url) too.

An alternative would have been to introduce a .git/info/submodules file that overrides settings from .gitmodules, analagous to .git/info/excludes overriding .gitignore and .git/info/attributes overriding .gitattributes. We are already using .git/config for this so that doesn't seem necessary.

Remote repositories ------------------- The .gitmodules file has some odd properties as a place to put configuration:

- it is versioned.  There is no way to change URLs in an old version
  of .gitmodules retroactively when a URL has changed.

- it is controlled by whoever writes history.  There is no way for me
  to change the URLs in my mirror of https://gerrit.googlesource.com/gerrit
  to match my mirror's different filesystem layout without producing
  my own history that diverges from the commits I am mirroring.

When the URLs in .gitmodules are relative URLs, this means that if I mirror a superproject, I have to mirror all its submodules, too, with the same layout. It's not so easy for me to publish my copy of the parent project and the one subproject I made changes in --- I have to mirror everything. In particular, this means I can't mirror https://gerrit.googlesource.com/gerrit to github.

When the URLs in .gitmodules are absolute URLs, this means that if I mirror a superproject, I cannot ask people consuming my mirror to use my mirrors of child projects, too. I cannot publish my copy of the parent project and the one subproject I made changes in and expect people to be able to "git clone --recurse-submodules" the result successfully.

It is as though refs were stored in a .gitrefs file, with all the attendant disadvantages, instead of being a separate component of the repository that a particular repository owner can manipulate without changing history.

To fix this, we could allow additional .gitmodules settings to be put in another ref (perhaps something like "refs/repository/config" to allow sharing additional repository-specific configuration in other files within the same tree --- e.g., branch descriptions). The semantics:

* If there is a gitmodules file in refs/repository/config in the
  repository I clone, then the submodule settings from it are stored
  locally somewhere that overrides .gitmodules.  Perhaps
  .git/info/<remotename>/gitmodules?

* Later fetches from the remote would also update this gitmodules
  file.

* Settings from this gitmodules file can be overridden locally
  using 'git config' until an explicit "git submodule sync" to
  override the local configuration.

What do you think?

If two different remotes provide conflicting values for a setting in their gitmodules files, git would error out and ask the user to intervene with a tie-breaking "git config" setting.

Thanks, Jonathan