{"thread":{"id":"65723","subject":"Mirror repositories for submodules","startedAt":"2026-06-01T06:11:32Z","lastAt":"2026-06-08T23:41:17Z","messageCount":13,"participants":["Benson Muite","Junio C Hamano","Simon Richter","Jeff King","Matt Hunter"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"544344","messageId":"875x42vlgv.fsf@emailplus.org","threadId":"65723","inReplyTo":null,"subject":"Mirror repositories for submodules","fromName":"Benson Muite","fromEmail":"benson_muite@emailplus.org","sentAt":"2026-06-01T06:11:28Z","receivedAt":"2026-06-01T06:11:32Z","isPatch":false,"body":"Hi,\n\nWould a contribution to add mirror repositories as alternate submodule\nsources be considered for inclusion?  Some projects have mirror\nrepositories on other hosting services, and may have bandwidth limits on\ntheir primary hosting service.  Being able to indicate mirror\nrepositories for where to check for updates and sources for submodules\nwhen doing `git clone --recurse-submodules https://my.repo ` or `git\nsubmodule update --init --recursive` would be helpful when there is a\ntimeout.\n\nRegards,\nBenson\n"},{"id":"544651","messageId":"xmqqcxy7qfgk.fsf@gitster.g","threadId":"65723","inReplyTo":"875x42vlgv.fsf@emailplus.org","subject":"Re: Mirror repositories for submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-06-04T01:09:15Z","receivedAt":"2026-06-04T01:09:17Z","isPatch":false,"body":"Benson Muite <benson_muite@emailplus.org> writes:\n\n> Would a contribution to add mirror repositories as alternate submodule\n> sources be considered for inclusion?  Some projects have mirror\n> repositories on other hosting services, and may have bandwidth limits on\n> their primary hosting service.  Being able to indicate mirror\n> repositories for where to check for updates and sources for submodules\n> when doing `git clone --recurse-submodules https://my.repo ` or `git\n> submodule update --init --recursive` would be helpful when there is a\n> timeout.\n\nI do not see why such a \"oh, the repository at $URL1 seems to be\ndown, but we know $URL2 serves the equivalent information, so let's\ngo there instead\" feature has to be limited to submodule use case.\n\nSo, no, I do not think a contribution to add mirror repositories as\nalternate submodule sources should be considered for inclusion, as\nit artificially limits usefulness of the feature.  A feature to add\nmirror repositories as alternate sources might be worth considering,\nthough.\n"},{"id":"544658","messageId":"d64e7f31-4e00-478c-ab31-b671242865fb@hogyros.de","threadId":"65723","inReplyTo":"xmqqcxy7qfgk.fsf@gitster.g","subject":"Re: Mirror repositories for submodules","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2026-06-04T05:11:38Z","receivedAt":"2026-06-04T05:11:45Z","isPatch":false,"body":"Hi,\n\nOn 6/4/26 10:09 AM, Junio C Hamano wrote:\n\n> So, no, I do not think a contribution to add mirror repositories as\n> alternate submodule sources should be considered for inclusion, as\n> it artificially limits usefulness of the feature.  A feature to add\n> mirror repositories as alternate sources might be worth considering,\n> though.\n\nThis is relevant to the Debian use case: we run a git server that \narchives git trees for Debian packages, and ideally the objects on this \nserver should be identical to what you get from upstream projects.\n\nThis is a big problem for archiving projects that use submodules, \nbecause we cannot alter the reference URLs.\n\nCloning from our server will, depending on what upstream uses, either a \nrelative URL (which will go to our server, but we have little control \nover what the name part of the repository base URL is going to be), or \nan absolute URL that instructs clients to pull from another place, which \nconflicts with our goal to have a self-contained archive.\n\nThe idea posited earlier, to have a \"repository identity\" that remains \nthe same across forks and clones, is somewhat appealing, but the best \nidea I can come up with is generating some kind of repository UUID, and \nadding a symlink -- not a great design because it pollutes outside the repo:\n\n     $ mkdir myproject\n     $ cd myproject\n     $ git init\n     $ ls -l ..\n     lrwxrwxrwx 1 simon simon   9 Jun  4 14:05 \n12345678-9abc-def0-1234-56789abcdef0.git -> myproject\n     drwxrwxr-x 2 simon simon  40 Jun  4 14:04 myproject\n\nOn the other hand, this can be used to construct a stable relative \nsubmodule URL.\n\nMaking the symlinks optional would require keeping a list of local \nclones and their UUIDs, and resolving them.\n\nI don't like that design, but as I said it's the best idea I have for now.\n\nI also fully expect that Debian's servers will be used by a lot of \npeople outside the project as soon as it becomes a convenient fallback, \nin the same way people are pulling .orig.tar.gz archives from Debian \nmirrors, so we need to make it easy to set up a mirror, to allow this to \nscale.\n\n    Simon\n"},{"id":"544666","messageId":"20260604061605.GA3194609@coredump.intra.peff.net","threadId":"65723","inReplyTo":"d64e7f31-4e00-478c-ab31-b671242865fb@hogyros.de","subject":"Re: Mirror repositories for submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-06-04T06:16:05Z","receivedAt":"2026-06-04T06:22:47Z","isPatch":false,"body":"On Thu, Jun 04, 2026 at 02:11:38PM +0900, Simon Richter wrote:\n\n> Cloning from our server will, depending on what upstream uses, either a\n> relative URL (which will go to our server, but we have little control over\n> what the name part of the repository base URL is going to be), or an\n> absolute URL that instructs clients to pull from another place, which\n> conflicts with our goal to have a self-contained archive.\n> \n> The idea posited earlier, to have a \"repository identity\" that remains the\n> same across forks and clones, is somewhat appealing, but the best idea I can\n> come up with is generating some kind of repository UUID, and adding a\n> symlink -- not a great design because it pollutes outside the repo:\n> \n>     $ mkdir myproject\n>     $ cd myproject\n>     $ git init\n>     $ ls -l ..\n>     lrwxrwxrwx 1 simon simon   9 Jun  4 14:05\n> 12345678-9abc-def0-1234-56789abcdef0.git -> myproject\n>     drwxrwxr-x 2 simon simon  40 Jun  4 14:04 myproject\n> \n> On the other hand, this can be used to construct a stable relative submodule\n> URL.\n\nHere's a thought experiment. What if you put the UUID into a URL, like:\n\n  repoid://123456789.git\n\nThen your in-repo .gitconfig would point to that repo id and be\nconsistent. Of course you need some way to tell Git how to retrieve\nrepoid:// URLs. You could do so with a custom remote helper\n(git-remote-repoid), but presumably that helper is eventually going to\nend up going over one of the normal Git protocols.\n\nSo we just need to tell Git how to resolve repo id URLs into concrete\nURLs. And indeed, we have url.*.insteadOf to do rewriting already. So\nfor example, you can add a submodule but convert it into a uuid like\nthis:\n\n  $ git submodule add https://github.com/git/git.git\n  $ git config -f .gitmodules submodule.git.url\n  https://github.com/git/git.git\n  $ git config -f .gitmodules submodule.git.url repoid://123456789.git\n  $ git commit -am 'add submodule with magic repoid'\n\nNow if somebody else comes along and clones it naively, the repo uuid is\nnot useful to git by itself:\n\n  $ git clone --recurse-submodules repo\n  Submodule 'git' (repoid://123456789.git) registered for path 'git'\n  Cloning into '/home/peff/tmp/repo/git'...\n  fatal: transport 'repoid' not allowed\n  fatal: clone of 'repoid://123456789.git' into submodule path '/home/peff/tmp/repo/git' failed\n\nBut imagine that \"somehow\" they have learned that 123456789.git can be\nfound at some URL. You can do this:\n\n  git -c url.https://github.com/git/git.git.insteadOf=repoid://123456789.git \\\n      clone --recurse-submodules repo.git\n\nwhich would clone from the original URL. Or you could even imagine that\nthey have a cache of repositories named by uuid, and then:\n\n  git -c url.https://my/cache/.insteadOf=repoid:// ...\n\nwould rewrite all repoid://'s automatically.\n\nThe use of \"-c\" here is mostly for illustration. It is a per-command\nconfig, so when you later try to update the submodule, you'd run into\nthe same problem. Probably you'd want to stuff your mapping into on-disk\nconfig (either ~/.gitconfig, or if you have a lot of them, perhaps some\nfile included from there).\n\nIt would be nice if you could use \"git clone -c\" (note \"-c\" as an option\nto \"clone\", not to \"git\") to set a permanent per-repo config variable.\nBut sadly the URL rewriting happens in the submodule repository, not the\nparent. So it has to be a per-user setting.\n\n\nNow, all of that said, do we still need uuids at all? If the canonical\nsubmodule name is https://github.com/git/git.git, then anybody can just\nrewrite that locally in the same way using url.*.insteadOf config. And I\nthink this is a pretty standard way of using submodules. E.g., you might\nrewrite https:// into ssh:// if you prefer that protocol. Or point to a\nlocal server if it's faster for you.\n\nWhich makes me wonder if I am missing something about the original\nrequest that started this thread. But it sounds to me like it is just\nasking for the existing URL-rewriting feature.\n\n-Peff\n"},{"id":"544704","messageId":"fa075b7a-96f6-4fd9-ae94-30ddf323f759@hogyros.de","threadId":"65723","inReplyTo":"20260604061605.GA3194609@coredump.intra.peff.net","subject":"Re: Mirror repositories for submodules","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2026-06-04T09:27:31Z","receivedAt":"2026-06-04T09:27:43Z","isPatch":false,"body":"Hi,\n\nOn 6/4/26 3:16 PM, Jeff King wrote:\n\n> Here's a thought experiment. What if you put the UUID into a URL, like:\n>    repoid://123456789.git\n\nYes, that's the idea, except I would want to use a relative URL, like\n\n     ../123456789.git\n\nThis could solve the \"naive cloning\" problem, because it creates an \nexpectation that the submodules can be found on the same server, or in a \nnearby path.\n\nI'm aware that this is *also* bad for decentralization, because it makes \nit easier to use one of the big forges where the repositories for \noften-used submodules are are already likely to be present, but it plays \ninto our use case, where we want to share the repositories for \noften-used subprojects.\n\n> Now, all of that said, do we still need uuids at all? If the canonical\n> submodule name is https://github.com/git/git.git, then anybody can just\n> rewrite that locally in the same way using url.*.insteadOf config.\n\nYes, but we'd then need a mechanism for a server to indicate \"for \ncloning, you should use these 'insteadOf' settings, which is a massive \ncan of worms from a security standpoint.\n\nI also don't think these canonical URLs can ever be stable if they refer \nto infrastructure that is not under the control of the maintainer -- it \nwould tie the project identity to the hosting provider, and increase the \ninertia to overcome for moves (such as the current exodus from github \nand gitlab towards codeberg).\n\n> Which makes me wonder if I am missing something about the original\n> request that started this thread. But it sounds to me like it is just\n> asking for the existing URL-rewriting feature.\n\nThe original mail has a similar problem as we do in Debian, and as my \nemployer has: CI jobs should exclusively talk to in-house \ninfrastructure, because continuously cloning repositories for each build \nis bad for the environment.\n\nThe common goal is that a naive clone should get submodules from a local \nserver, ideally without us having to write some tool to make an initial \ncheckout, enumerate submodules, create insteadOf settings, clone first \nlayer of submodules, enumerate second layer, ...\n\n    Simon\n"},{"id":"544769","messageId":"87se71r4ax.fsf@emailplus.org","threadId":"65723","inReplyTo":"xmqqcxy7qfgk.fsf@gitster.g","subject":"Re: Mirror repositories for submodules","fromName":"Benson Muite","fromEmail":"benson_muite@emailplus.org","sentAt":"2026-06-05T04:37:10Z","receivedAt":"2026-06-05T04:37:14Z","isPatch":false,"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Benson Muite <benson_muite@emailplus.org> writes:\n>\n>> Would a contribution to add mirror repositories as alternate submodule\n>> sources be considered for inclusion?  Some projects have mirror\n>> repositories on other hosting services, and may have bandwidth limits on\n>> their primary hosting service.  Being able to indicate mirror\n>> repositories for where to check for updates and sources for submodules\n>> when doing `git clone --recurse-submodules https://my.repo ` or `git\n>> submodule update --init --recursive` would be helpful when there is a\n>> timeout.\n>\n> I do not see why such a \"oh, the repository at $URL1 seems to be\n> down, but we know $URL2 serves the equivalent information, so let's\n> go there instead\" feature has to be limited to submodule use case.\n>\n> So, no, I do not think a contribution to add mirror repositories as\n> alternate submodule sources should be considered for inclusion, as\n> it artificially limits usefulness of the feature.  A feature to add\n> mirror repositories as alternate sources might be worth considering,\n> though.\n\nThanks for the feedback. This was motivated by problems when trying to\nrecursively clone, but a more general solution is also fine.\n"},{"id":"544770","messageId":"87pl25r3tz.fsf@emailplus.org","threadId":"65723","inReplyTo":"d64e7f31-4e00-478c-ab31-b671242865fb@hogyros.de","subject":"Re: Mirror repositories for submodules","fromName":"Benson Muite","fromEmail":"benson_muite@emailplus.org","sentAt":"2026-06-05T04:47:20Z","receivedAt":"2026-06-05T04:47:26Z","isPatch":false,"body":"Simon Richter <Simon.Richter@hogyros.de> writes:\n\n> Hi,\n>\n> On 6/4/26 10:09 AM, Junio C Hamano wrote:\n>\n>> So, no, I do not think a contribution to add mirror repositories as\n>> alternate submodule sources should be considered for inclusion, as\n>> it artificially limits usefulness of the feature.  A feature to add\n>> mirror repositories as alternate sources might be worth considering,\n>> though.\n>\n> This is relevant to the Debian use case: we run a git server that \n> archives git trees for Debian packages, and ideally the objects on this \n> server should be identical to what you get from upstream projects.\n>\n> This is a big problem for archiving projects that use submodules, \n> because we cannot alter the reference URLs.\n>\n> Cloning from our server will, depending on what upstream uses, either a \n> relative URL (which will go to our server, but we have little control \n> over what the name part of the repository base URL is going to be), or \n> an absolute URL that instructs clients to pull from another place, which \n> conflicts with our goal to have a self-contained archive.\n>\n> The idea posited earlier, to have a \"repository identity\" that remains \n> the same across forks and clones, is somewhat appealing, but the best \n> idea I can come up with is generating some kind of repository UUID, and \n> adding a symlink -- not a great design because it pollutes outside the repo:\n>\n>      $ mkdir myproject\n>      $ cd myproject\n>      $ git init\n>      $ ls -l ..\n>      lrwxrwxrwx 1 simon simon   9 Jun  4 14:05 \n> 12345678-9abc-def0-1234-56789abcdef0.git -> myproject\n>      drwxrwxr-x 2 simon simon  40 Jun  4 14:04 myproject\n>\n> On the other hand, this can be used to construct a stable relative \n> submodule URL.\n>\n> Making the symlinks optional would require keeping a list of local \n> clones and their UUIDs, and resolving them.\n>\n> I don't like that design, but as I said it's the best idea I have for now.\n>\n> I also fully expect that Debian's servers will be used by a lot of \n> people outside the project as soon as it becomes a convenient fallback, \n> in the same way people are pulling .orig.tar.gz archives from Debian \n> mirrors, so we need to make it easy to set up a mirror, to allow this to \n> scale.\n>\n\nFor submodules, the metadata consists of the url of the repository to\nclone from.  One could have a list of absolute URLs.  The default would\nbe to assume that the URLs are tried in order, and if a URL times out,\nthe next one would be tried.  One may want to change the default\nordering as a user setting, or do a ping test to get obtain content from\nthe closest repository.\n\nAs an example, for linphone-desktop, the first part of the .gitmodules\nfile contains:\n\n[submodule \"linphone-sdk\"]\npath = external/linphone-sdk\n\turl = https://gitlab.linphone.org/BC/public/linphone-sdk.git\n[submodule \"external/google/gn\"]\n\nThis could be updated to\n\n[submodule \"linphone-sdk\"]\npath = external/linphone-sdk\n\turl = https://gitlab.linphone.org/BC/public/linphone-sdk.git\n        url = https://github.com/BelledonneCommunications/linphone-sdk.git\n[submodule \"external/google/gn\"]\n        \n\n"},{"id":"544771","messageId":"87mrx9r3hh.fsf@emailplus.org","threadId":"65723","inReplyTo":"20260604061605.GA3194609@coredump.intra.peff.net","subject":"Re: Mirror repositories for submodules","fromName":"Benson Muite","fromEmail":"benson_muite@emailplus.org","sentAt":"2026-06-05T04:54:50Z","receivedAt":"2026-06-05T04:54:55Z","isPatch":false,"body":"Jeff King <peff@peff.net> writes:\n\n> On Thu, Jun 04, 2026 at 02:11:38PM +0900, Simon Richter wrote:\n>\n>> Cloning from our server will, depending on what upstream uses, either a\n>> relative URL (which will go to our server, but we have little control over\n>> what the name part of the repository base URL is going to be), or an\n>> absolute URL that instructs clients to pull from another place, which\n>> conflicts with our goal to have a self-contained archive.\n>> \n>> The idea posited earlier, to have a \"repository identity\" that remains the\n>> same across forks and clones, is somewhat appealing, but the best idea I can\n>> come up with is generating some kind of repository UUID, and adding a\n>> symlink -- not a great design because it pollutes outside the repo:\n>> \n>>     $ mkdir myproject\n>>     $ cd myproject\n>>     $ git init\n>>     $ ls -l ..\n>>     lrwxrwxrwx 1 simon simon   9 Jun  4 14:05\n>> 12345678-9abc-def0-1234-56789abcdef0.git -> myproject\n>>     drwxrwxr-x 2 simon simon  40 Jun  4 14:04 myproject\n>> \n>> On the other hand, this can be used to construct a stable relative submodule\n>> URL.\n>\n> Here's a thought experiment. What if you put the UUID into a URL, like:\n>\n>   repoid://123456789.git\n>\n> Then your in-repo .gitconfig would point to that repo id and be\n> consistent. Of course you need some way to tell Git how to retrieve\n> repoid:// URLs. You could do so with a custom remote helper\n> (git-remote-repoid), but presumably that helper is eventually going to\n> end up going over one of the normal Git protocols.\n>\n> So we just need to tell Git how to resolve repo id URLs into concrete\n> URLs. And indeed, we have url.*.insteadOf to do rewriting already. So\n> for example, you can add a submodule but convert it into a uuid like\n> this:\n>\n>   $ git submodule add https://github.com/git/git.git\n>   $ git config -f .gitmodules submodule.git.url\n>   https://github.com/git/git.git\n>   $ git config -f .gitmodules submodule.git.url repoid://123456789.git\n>   $ git commit -am 'add submodule with magic repoid'\n>\n> Now if somebody else comes along and clones it naively, the repo uuid is\n> not useful to git by itself:\n>\n>   $ git clone --recurse-submodules repo\n>   Submodule 'git' (repoid://123456789.git) registered for path 'git'\n>   Cloning into '/home/peff/tmp/repo/git'...\n>   fatal: transport 'repoid' not allowed\n>   fatal: clone of 'repoid://123456789.git' into submodule path '/home/peff/tmp/repo/git' failed\n>\n> But imagine that \"somehow\" they have learned that 123456789.git can be\n> found at some URL. You can do this:\n>\n>   git -c url.https://github.com/git/git.git.insteadOf=repoid://123456789.git \\\n>       clone --recurse-submodules repo.git\n>\n> which would clone from the original URL. Or you could even imagine that\n> they have a cache of repositories named by uuid, and then:\n>\n>   git -c url.https://my/cache/.insteadOf=repoid:// ...\n>\n> would rewrite all repoid://'s automatically.\n>\n> The use of \"-c\" here is mostly for illustration. It is a per-command\n> config, so when you later try to update the submodule, you'd run into\n> the same problem. Probably you'd want to stuff your mapping into on-disk\n> config (either ~/.gitconfig, or if you have a lot of them, perhaps some\n> file included from there).\n>\n> It would be nice if you could use \"git clone -c\" (note \"-c\" as an option\n> to \"clone\", not to \"git\") to set a permanent per-repo config variable.\n> But sadly the URL rewriting happens in the submodule repository, not the\n> parent. So it has to be a per-user setting.\n>\n>\n> Now, all of that said, do we still need uuids at all? If the canonical\n> submodule name is https://github.com/git/git.git, then anybody can just\n> rewrite that locally in the same way using url.*.insteadOf config. And I\n> think this is a pretty standard way of using submodules. E.g., you might\n> rewrite https:// into ssh:// if you prefer that protocol. Or point to a\n> local server if it's faster for you.\n>\n> Which makes me wonder if I am missing something about the original\n> request that started this thread. But it sounds to me like it is just\n> asking for the existing URL-rewriting feature.\n>\n\nThe  problem is that one might have multiple repositories, submodules\nmay themselves have submodules.  Typically a primary development\norganization will have its own host, but may also have mirrors on other\nservices which maybe more convenient for others to use.  A recursive\nclone could give upto 20 repositories not all of which are maintained by\nthe same organization.  URL-rewriting each of them can be inefficient,\nespecially when the upstream maintains the mirror repositories and can\nindicate that in the source repositories.\n\n\n> -Peff\n"},{"id":"544772","messageId":"87jysdr3cq.fsf@emailplus.org","threadId":"65723","inReplyTo":"xmqqcxy7qfgk.fsf@gitster.g","subject":"Re: Mirror repositories for submodules","fromName":"Benson Muite","fromEmail":"benson_muite@emailplus.org","sentAt":"2026-06-05T04:57:41Z","receivedAt":"2026-06-05T04:57:45Z","isPatch":false,"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Benson Muite <benson_muite@emailplus.org> writes:\n>\n>> Would a contribution to add mirror repositories as alternate submodule\n>> sources be considered for inclusion?  Some projects have mirror\n>> repositories on other hosting services, and may have bandwidth limits on\n>> their primary hosting service.  Being able to indicate mirror\n>> repositories for where to check for updates and sources for submodules\n>> when doing `git clone --recurse-submodules https://my.repo ` or `git\n>> submodule update --init --recursive` would be helpful when there is a\n>> timeout.\n>\n> I do not see why such a \"oh, the repository at $URL1 seems to be\n> down, but we know $URL2 serves the equivalent information, so let's\n> go there instead\" feature has to be limited to submodule use case.\n>\n> So, no, I do not think a contribution to add mirror repositories as\n> alternate submodule sources should be considered for inclusion, as\n> it artificially limits usefulness of the feature.  A feature to add\n> mirror repositories as alternate sources might be worth considering,\n> though.\n\nThanks for the feedback. This was motivated by problems when trying to\nrecursively clone, but a more general solution is also fine.\n"},{"id":"544773","messageId":"87h5nhr2zp.fsf@emailplus.org","threadId":"65723","inReplyTo":"d64e7f31-4e00-478c-ab31-b671242865fb@hogyros.de","subject":"Re: Mirror repositories for submodules","fromName":"Benson Muite","fromEmail":"benson_muite@emailplus.org","sentAt":"2026-06-05T05:05:30Z","receivedAt":"2026-06-05T05:05:35Z","isPatch":false,"body":"Simon Richter <Simon.Richter@hogyros.de> writes:\n\n> Hi,\n>\n> On 6/4/26 10:09 AM, Junio C Hamano wrote:\n>\n>> So, no, I do not think a contribution to add mirror repositories as\n>> alternate submodule sources should be considered for inclusion, as\n>> it artificially limits usefulness of the feature.  A feature to add\n>> mirror repositories as alternate sources might be worth considering,\n>> though.\n>\n> This is relevant to the Debian use case: we run a git server that \n> archives git trees for Debian packages, and ideally the objects on this \n> server should be identical to what you get from upstream projects.\n>\n> This is a big problem for archiving projects that use submodules, \n> because we cannot alter the reference URLs.\n>\n> Cloning from our server will, depending on what upstream uses, either a \n> relative URL (which will go to our server, but we have little control \n> over what the name part of the repository base URL is going to be), or \n> an absolute URL that instructs clients to pull from another place, which \n> conflicts with our goal to have a self-contained archive.\n>\n> The idea posited earlier, to have a \"repository identity\" that remains \n> the same across forks and clones, is somewhat appealing, but the best \n> idea I can come up with is generating some kind of repository UUID, and \n> adding a symlink -- not a great design because it pollutes outside the repo:\n>\n>      $ mkdir myproject\n>      $ cd myproject\n>      $ git init\n>      $ ls -l ..\n>      lrwxrwxrwx 1 simon simon   9 Jun  4 14:05 \n> 12345678-9abc-def0-1234-56789abcdef0.git -> myproject\n>      drwxrwxr-x 2 simon simon  40 Jun  4 14:04 myproject\n>\n> On the other hand, this can be used to construct a stable relative \n> submodule URL.\n>\n> Making the symlinks optional would require keeping a list of local \n> clones and their UUIDs, and resolving them.\n>\n> I don't like that design, but as I said it's the best idea I have for now.\n>\n\nFor submodules, the metadata consists of the url of the repository to\nclone from.  One could have a list of absolute URLs.  The default would\nbe to assume that the URLs are tried in order, and if a URL times out,\nthe next one would be tried.  One may want to change the default\nordering as a user setting, or do a ping test to get obtain content from\nthe closest repository.\n\nAs an example, for linphone-desktop, the first part of the .gitmodules\nfile contains:\n\n[submodule \"linphone-sdk\"]\npath = external/linphone-sdk\n         url = https://gitlab.linphone.org/BC/public/linphone-sdk.git\n[submodule \"external/google/gn\"]\n\nThis could be updated to\n\n[submodule \"linphone-sdk\"]\npath = external/linphone-sdk\n         url = https://gitlab.linphone.org/BC/public/linphone-sdk.git\n         url = https://github.com/BelledonneCommunications/linphone-sdk.git\n[submodule \"external/google/gn\"]\n\n> I also fully expect that Debian's servers will be used by a lot of \n> people outside the project as soon as it becomes a convenient fallback, \n> in the same way people are pulling .orig.tar.gz archives from Debian \n> mirrors, so we need to make it easy to set up a mirror, to allow this to \n> scale.\n>\n>     Simon\n"},{"id":"544776","messageId":"DJ10HH5HZF0E.3000F6WUQ1M08@lfurio.us","threadId":"65723","inReplyTo":"87pl25r3tz.fsf@emailplus.org","subject":"Re: Mirror repositories for submodules","fromName":"Matt Hunter","fromEmail":"m@lfurio.us","sentAt":"2026-06-05T09:34:49Z","receivedAt":"2026-06-05T09:34:56Z","isPatch":false,"body":"On Fri Jun 5, 2026 at 12:47 AM EDT, Benson Muite wrote:\n>\n> For submodules, the metadata consists of the url of the repository to\n> clone from.  One could have a list of absolute URLs.  The default would\n> be to assume that the URLs are tried in order, and if a URL times out,\n> the next one would be tried.  One may want to change the default\n> ordering as a user setting, or do a ping test to get obtain content from\n> the closest repository.\n\nAnother idea is for the client to attempt in a random order as a kind of\nload balancing.\n\n>\n> As an example, for linphone-desktop, the first part of the .gitmodules\n> file contains:\n>\n> [submodule \"linphone-sdk\"]\n> path = external/linphone-sdk\n> \turl = https://gitlab.linphone.org/BC/public/linphone-sdk.git\n> [submodule \"external/google/gn\"]\n>\n> This could be updated to\n>\n> [submodule \"linphone-sdk\"]\n> path = external/linphone-sdk\n> \turl = https://gitlab.linphone.org/BC/public/linphone-sdk.git\n>         url = https://github.com/BelledonneCommunications/linphone-sdk.git\n> [submodule \"external/google/gn\"]\n\nRelevant to Junio's earlier comment about aiming for a general solution:\n\nIt looks like the remote URL configuration already supports multiple\nURLs per a single entry.  It's just that git only considers the first in\nthe list for fetching, and will push to all.\n\nYour proposed change to .gitmodules may be used to initialize the list\nof URLs for a cloned submodule's (single) initial remote?  At this\npoint, any kind of mirror management logic could generically operate on\neach remote's URL list.\n\nFrom git-config(1):\n    remote.<name>.url\n        The  URL of a remote repository. See git-fetch(1) or git-push(1).\n        A configured remote can have multiple URLs; in this case the first\n        is used for fetching, and all are used for pushing (assuming no\n        remote.<name>.pushurl is defined).  Setting this key to the empty\n        string clears the list of urls, allowing you to override earlier\n        config.\n\nFrom gitmodules(5):\n    submodule.<name>.url\n        Defines a URL from which the submodule repository can be cloned.\n        This may be either an absolute URL ready to be passed to\n        git-clone(1) or (if it begins with ./ or ../) a location relative\n        to the superproject’s origin repository.\n\nNote that the submodule format seems to explicitly support only a single\nvalue right now.  Of course, I am assuming that the implementation\nmatches the documentation.\n"},{"id":"544780","messageId":"01d627c2-3625-4de4-978e-6fa4f63bcaeb@hogyros.de","threadId":"65723","inReplyTo":"87h5nhr2zp.fsf@emailplus.org","subject":"Re: Mirror repositories for submodules","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2026-06-05T12:10:52Z","receivedAt":"2026-06-05T12:11:02Z","isPatch":false,"body":"Hi,\n\nOn 6/5/26 2:05 PM, Benson Muite wrote:\n\n> Simon Richter <Simon.Richter@hogyros.de> writes:\n\n>> On the other hand, this can be used to construct a stable relative\n>> submodule URL.\n\n> For submodules, the metadata consists of the url of the repository to\n> clone from.\n\nThat is precisely what precludes mirroring: if I clone and republish a \nrepository, people can clone from that repository, but will still fetch \nsubmodules from the URLs listed in the .gitmodules file.\n\nIf that is a relative URL, then all is (mostly) well: they will also ask \nmy mirror server for the submodule, and all I have to do is make it \navailable.\n\nIf it is an absolute URL, then I need a side channel to communicate to \nthe client \"you can also get this repository from me.\" This could, for \nexample, generate an insteadOf config, but that would be a horrible hack \nthat becomes unmanageable pretty quickly (updates? security implications?)\n\nHence this thread: is there a way to represent submodules so that their \nidentity is independent from the hosting location -- and this ties into \nthe other thread from last week, giving projects a stable identity that \nfollows them through clones (or, if someone is using a forge, forks).\n\nThe download location for a project is project metadata that lives \noutside the project view of time, but it is expressed as (versioned) \ndata in git, in the .gitmodules file, so if hosting for a project \nchanges, projects referring to them must either rewrite all of their \nhistory, accept that old versions will no longer be buildable because \nthey contain a broken link, or expect people/CI to manually generate \ninsteadOf entries.\n\nSo the problem here is that we are treating metadata as data.\n\n    Simon\n"},{"id":"544982","messageId":"20260608234116.GA358144@coredump.intra.peff.net","threadId":"65723","inReplyTo":"fa075b7a-96f6-4fd9-ae94-30ddf323f759@hogyros.de","subject":"Re: Mirror repositories for submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-06-08T23:41:16Z","receivedAt":"2026-06-08T23:41:17Z","isPatch":false,"body":"On Thu, Jun 04, 2026 at 06:27:31PM +0900, Simon Richter wrote:\n\n> Hi,\n> \n> On 6/4/26 3:16 PM, Jeff King wrote:\n> \n> > Here's a thought experiment. What if you put the UUID into a URL, like:\n> >    repoid://123456789.git\n> \n> Yes, that's the idea, except I would want to use a relative URL, like\n> \n>     ../123456789.git\n> \n> This could solve the \"naive cloning\" problem, because it creates an\n> expectation that the submodules can be found on the same server, or in a\n> nearby path.\n\nI see. I forgot that we allowed relative submodule URLs.\n\n> > Now, all of that said, do we still need uuids at all? If the canonical\n> > submodule name is https://github.com/git/git.git, then anybody can just\n> > rewrite that locally in the same way using url.*.insteadOf config.\n> \n> Yes, but we'd then need a mechanism for a server to indicate \"for cloning,\n> you should use these 'insteadOf' settings, which is a massive can of worms\n> from a security standpoint.\n> \n> I also don't think these canonical URLs can ever be stable if they refer to\n> infrastructure that is not under the control of the maintainer -- it would\n> tie the project identity to the hosting provider, and increase the inertia\n> to overcome for moves (such as the current exodus from github and gitlab\n> towards codeberg).\n\nFrom your description I was assuming the cloner had to always specify\ninsteadOf (which they find out about \"somehow\").\n\nIf they're not, then your choice of canonical URL is effectively trading\noff some cases for others. In the scenario you care about, you assume\nthat the submodules are hosted relative to the superproject, so clients\ncan usually get what they need without further config. The server\noperator and the superproject repo coordinate on the names.\n\nBut in many decentralized cases, there's no URL or administrative\nrelationship between the superproject and the submodules. They might\nhappen to be on the same server, but even that falls down if the\nsuperproject is mirrored elsewhere. So using some canonical name which\nworks in practice _now_ is usually the best we can do.\n\n> The common goal is that a naive clone should get submodules from a local\n> server, ideally without us having to write some tool to make an initial\n> checkout, enumerate submodules, create insteadOf settings, clone first layer\n> of submodules, enumerate second layer, ...\n\nYou shouldn't need to do the recursive enumeration if you set up the\ninteadOf ahead of time. You don't know which insteadOf settings you'll\nwant, but you can feed the whole possible mapping. How you get that\nmapping is unspecified, but if you are mirroring the submodules already\non your local infrastructure, then whatever process does that can also\noutput the mapping.\n\n\nJust to be clear, I'm not trying to dismiss what you're going for. I'm\nlooking at this from the lens of Git developers: how do existing Git\nfeatures fit into this space, and which features are missing that might\nassist in a generalized way.\n\n-Peff\n"}]}