Hi,

Would a contribution to add mirror repositories as alternate submodule sources be considered for inclusion? Some projects have mirror repositories on other hosting services, and may have bandwidth limits on their primary hosting service. Being able to indicate mirror repositories for where to check for updates and sources for submodules when doing `git clone --recurse-submodules https://my.repo ` or `git submodule update --init --recursive` would be helpful when there is a timeout.

Regards, Benson

Benson Muite <benson_muite@emailplus.org> writes:

I do not see why such a "oh, the repository at $URL1 seems to be down, but we know $URL2 serves the equivalent information, so let's go there instead" feature has to be limited to submodule use case.

So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.

Hi,

On 6/4/26 10:09 AM, Junio C Hamano wrote:

This is relevant to the Debian use case: we run a git server that archives git trees for Debian packages, and ideally the objects on this server should be identical to what you get from upstream projects.

This is a big problem for archiving projects that use submodules, because we cannot alter the reference URLs.

Cloning from our server will, depending on what upstream uses, either a relative URL (which will go to our server, but we have little control over what the name part of the repository base URL is going to be), or an absolute URL that instructs clients to pull from another place, which conflicts with our goal to have a self-contained archive.

The idea posited earlier, to have a "repository identity" that remains the same across forks and clones, is somewhat appealing, but the best idea I can come up with is generating some kind of repository UUID, and adding a symlink -- not a great design because it pollutes outside the repo:

     $ mkdir myproject
     $ cd myproject
     $ git init
     $ ls -l ..
     lrwxrwxrwx 1 simon simon   9 Jun  4 14:05 
12345678-9abc-def0-1234-56789abcdef0.git -> myproject
     drwxrwxr-x 2 simon simon  40 Jun  4 14:04 myproject

On the other hand, this can be used to construct a stable relative submodule URL.

Making the symlinks optional would require keeping a list of local clones and their UUIDs, and resolving them.

I don't like that design, but as I said it's the best idea I have for now.

I also fully expect that Debian's servers will be used by a lot of people outside the project as soon as it becomes a convenient fallback, in the same way people are pulling .orig.tar.gz archives from Debian mirrors, so we need to make it easy to set up a mirror, to allow this to scale.

    Simon

On Thu, Jun 04, 2026 at 02:11:38PM +0900, Simon Richter wrote:

Here's a thought experiment. What if you put the UUID into a URL, like:

  repoid://123456789.git

Then your in-repo .gitconfig would point to that repo id and be consistent. Of course you need some way to tell Git how to retrieve repoid:// URLs. You could do so with a custom remote helper (git-remote-repoid), but presumably that helper is eventually going to end up going over one of the normal Git protocols.

So we just need to tell Git how to resolve repo id URLs into concrete URLs. And indeed, we have url.*.insteadOf to do rewriting already. So for example, you can add a submodule but convert it into a uuid like this:

  $ git submodule add https://github.com/git/git.git
  $ git config -f .gitmodules submodule.git.url
  https://github.com/git/git.git
  $ git config -f .gitmodules submodule.git.url repoid://123456789.git
  $ git commit -am 'add submodule with magic repoid'

Now if somebody else comes along and clones it naively, the repo uuid is not useful to git by itself:

  $ git clone --recurse-submodules repo
  Submodule 'git' (repoid://123456789.git) registered for path 'git'
  Cloning into '/home/peff/tmp/repo/git'...
  fatal: transport 'repoid' not allowed
  fatal: clone of 'repoid://123456789.git' into submodule path '/home/peff/tmp/repo/git' failed

But imagine that "somehow" they have learned that 123456789.git can be found at some URL. You can do this:

  git -c url.https://github.com/git/git.git.insteadOf=repoid://123456789.git \
      clone --recurse-submodules repo.git

which would clone from the original URL. Or you could even imagine that they have a cache of repositories named by uuid, and then:

  git -c url.https://my/cache/.insteadOf=repoid:// ...

would rewrite all repoid://'s automatically.

The use of "-c" here is mostly for illustration. It is a per-command config, so when you later try to update the submodule, you'd run into the same problem. Probably you'd want to stuff your mapping into on-disk config (either ~/.gitconfig, or if you have a lot of them, perhaps some file included from there).

It would be nice if you could use "git clone -c" (note "-c" as an option to "clone", not to "git") to set a permanent per-repo config variable. But sadly the URL rewriting happens in the submodule repository, not the parent. So it has to be a per-user setting.

Now, all of that said, do we still need uuids at all? If the canonical submodule name is https://github.com/git/git.git, then anybody can just rewrite that locally in the same way using url.*.insteadOf config. And I think this is a pretty standard way of using submodules. E.g., you might rewrite https:// into ssh:// if you prefer that protocol. Or point to a local server if it's faster for you.

Which makes me wonder if I am missing something about the original request that started this thread. But it sounds to me like it is just asking for the existing URL-rewriting feature.

-Peff

Hi,

On 6/4/26 3:16 PM, Jeff King wrote:

Yes, that's the idea, except I would want to use a relative URL, like

     ../123456789.git

This could solve the "naive cloning" problem, because it creates an expectation that the submodules can be found on the same server, or in a nearby path.

I'm aware that this is *also* bad for decentralization, because it makes it easier to use one of the big forges where the repositories for often-used submodules are are already likely to be present, but it plays into our use case, where we want to share the repositories for often-used subprojects.

Yes, but we'd then need a mechanism for a server to indicate "for cloning, you should use these 'insteadOf' settings, which is a massive can of worms from a security standpoint.

I also don't think these canonical URLs can ever be stable if they refer to infrastructure that is not under the control of the maintainer -- it would tie the project identity to the hosting provider, and increase the inertia to overcome for moves (such as the current exodus from github and gitlab towards codeberg).

The original mail has a similar problem as we do in Debian, and as my employer has: CI jobs should exclusively talk to in-house infrastructure, because continuously cloning repositories for each build is bad for the environment.

The common goal is that a naive clone should get submodules from a local server, ideally without us having to write some tool to make an initial checkout, enumerate submodules, create insteadOf settings, clone first layer of submodules, enumerate second layer, ...

    Simon

Junio C Hamano <gitster@pobox.com> writes:

Thanks for the feedback. This was motivated by problems when trying to recursively clone, but a more general solution is also fine.

Simon Richter <Simon.Richter@hogyros.de> writes:

For submodules, the metadata consists of the url of the repository to clone from. One could have a list of absolute URLs. The default would be to assume that the URLs are tried in order, and if a URL times out, the next one would be tried. One may want to change the default ordering as a user setting, or do a ping test to get obtain content from the closest repository.

As an example, for linphone-desktop, the first part of the .gitmodules file contains:

[submodule "linphone-sdk"]
path = external/linphone-sdk
	url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
[submodule "external/google/gn"]

This could be updated to

[submodule "linphone-sdk"]
path = external/linphone-sdk
	url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
        url = https://github.com/BelledonneCommunications/linphone-sdk.git
[submodule "external/google/gn"]
        

Jeff King <peff@peff.net> writes:

The problem is that one might have multiple repositories, submodules may themselves have submodules. Typically a primary development organization will have its own host, but may also have mirrors on other services which maybe more convenient for others to use. A recursive clone could give upto 20 repositories not all of which are maintained by the same organization. URL-rewriting each of them can be inefficient, especially when the upstream maintains the mirror repositories and can indicate that in the source repositories.

Junio C Hamano <gitster@pobox.com> writes:

Thanks for the feedback. This was motivated by problems when trying to recursively clone, but a more general solution is also fine.

Simon Richter <Simon.Richter@hogyros.de> writes:

For submodules, the metadata consists of the url of the repository to clone from. One could have a list of absolute URLs. The default would be to assume that the URLs are tried in order, and if a URL times out, the next one would be tried. One may want to change the default ordering as a user setting, or do a ping test to get obtain content from the closest repository.

As an example, for linphone-desktop, the first part of the .gitmodules file contains:

[submodule "linphone-sdk"]
path = external/linphone-sdk
         url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
[submodule "external/google/gn"]

This could be updated to

[submodule "linphone-sdk"]
path = external/linphone-sdk
         url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
         url = https://github.com/BelledonneCommunications/linphone-sdk.git
[submodule "external/google/gn"]

On Fri Jun 5, 2026 at 12:47 AM EDT, Benson Muite wrote:

Another idea is for the client to attempt in a random order as a kind of load balancing.

Relevant to Junio's earlier comment about aiming for a general solution:

It looks like the remote URL configuration already supports multiple URLs per a single entry. It's just that git only considers the first in the list for fetching, and will push to all.

Your proposed change to .gitmodules may be used to initialize the list of URLs for a cloned submodule's (single) initial remote? At this point, any kind of mirror management logic could generically operate on each remote's URL list.

From git-config(1):
    remote.<name>.url
        The  URL of a remote repository. See git-fetch(1) or git-push(1).
        A configured remote can have multiple URLs; in this case the first
        is used for fetching, and all are used for pushing (assuming no
        remote.<name>.pushurl is defined).  Setting this key to the empty
        string clears the list of urls, allowing you to override earlier
        config.

From gitmodules(5):
    submodule.<name>.url
        Defines a URL from which the submodule repository can be cloned.
        This may be either an absolute URL ready to be passed to
        git-clone(1) or (if it begins with ./ or ../) a location relative
        to the superproject’s origin repository.

Note that the submodule format seems to explicitly support only a single value right now. Of course, I am assuming that the implementation matches the documentation.

Hi,

On 6/5/26 2:05 PM, Benson Muite wrote:

That is precisely what precludes mirroring: if I clone and republish a repository, people can clone from that repository, but will still fetch submodules from the URLs listed in the .gitmodules file.

If that is a relative URL, then all is (mostly) well: they will also ask my mirror server for the submodule, and all I have to do is make it available.

If it is an absolute URL, then I need a side channel to communicate to the client "you can also get this repository from me." This could, for example, generate an insteadOf config, but that would be a horrible hack that becomes unmanageable pretty quickly (updates? security implications?)

Hence this thread: is there a way to represent submodules so that their identity is independent from the hosting location -- and this ties into the other thread from last week, giving projects a stable identity that follows them through clones (or, if someone is using a forge, forks).

The download location for a project is project metadata that lives outside the project view of time, but it is expressed as (versioned) data in git, in the .gitmodules file, so if hosting for a project changes, projects referring to them must either rewrite all of their history, accept that old versions will no longer be buildable because they contain a broken link, or expect people/CI to manually generate insteadOf entries.

So the problem here is that we are treating metadata as data.

    Simon

On Thu, Jun 04, 2026 at 06:27:31PM +0900, Simon Richter wrote:

I see. I forgot that we allowed relative submodule URLs.

From your description I was assuming the cloner had to always specify insteadOf (which they find out about "somehow").

If they're not, then your choice of canonical URL is effectively trading off some cases for others. In the scenario you care about, you assume that the submodules are hosted relative to the superproject, so clients can usually get what they need without further config. The server operator and the superproject repo coordinate on the names.

But in many decentralized cases, there's no URL or administrative relationship between the superproject and the submodules. They might happen to be on the same server, but even that falls down if the superproject is mirrored elsewhere. So using some canonical name which works in practice _now_ is usually the best we can do.

You shouldn't need to do the recursive enumeration if you set up the inteadOf ahead of time. You don't know which insteadOf settings you'll want, but you can feed the whole possible mapping. How you get that mapping is unspecified, but if you are mirroring the submodules already on your local infrastructure, then whatever process does that can also output the mapping.

Just to be clear, I'm not trying to dismiss what you're going for. I'm looking at this from the lens of Git developers: how do existing Git features fit into this space, and which features are missing that might assist in a generalized way.

-Peff