threads / rfc / 40591

[RFC] URL rewrite in .gitmodules

Subject: [RFC] URL rewrite in .gitmodules

## tl;dr

8 messages between Oct 19, 2015 and Nov 15, 2015.

replies: 7people: 4as markdown or json

Lars Schneider· Oct 19, 2015, 19:28 UTC · lore
Hi,
I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the "outside of company" repos. I know I can rewrite Git URLs with the git config "url.<base>.insteadOf" option. However, git configs are client specific. I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the
  very same hashes as defined by the "upstream" Open Source repos.
Is there yet another way in Git to change URLs of Submodules in the way I want it?
If not, what do you think about a patch that adds a "url" section similar to the one in git config to a .gitmodules file?
Example:
----------
[submodule "git"]
	path = git
        url=git://github.com/larsxschneider/git.git
[url "mycompany.com"]
        insteadOf = outside.com
----------

Thanks, Lars

Stefan Beller· Oct 19, 2015, 22:07 UTC · re: Lars Schneider · lore

Re: [RFC] URL rewrite in .gitmodules

On Mon, Oct 19, 2015 at 12:28 PM, Lars Schneider <larsxschneider@gmail.com> wrote:

> Hi,
>
> I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the "outside of company" repos. I know I can rewrite Git URLs with the git config "url.<base>.insteadOf" option. However, git configs are client specific.

I feel like this is working as intended. You only want to improve your one client (say the buildbot) to not goto the open source site, while the developer may do want to fetch from external sources ("Hey shiny new code!";)

> I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the very same hashes as defined by the "upstream" Open Source repos.

You could carry a patch on top of the tip of the first submodule re-pointing the nested submodule. This requires good workflows available to deal with submodules though. (Fetch and merge or rebase, git submodule update should be able to do that?)

Show 5 quoted lines
>
> Is there yet another way in Git to change URLs of Submodules in the way I want it?
>
> If not, what do you think about a patch that adds a "url" section similar to the one in git config to a .gitmodules file?
>

So we have different kinds of git configs. within one repository, in the home director (global to the one machine), maybe you would want to have one "global" config on a network share, such that every box in your company reads that "company-wide" global config and acts upon that?

Show 8 quoted lines
> Example:
> ----------
> [submodule "git"]
>         path = git
>         url=git://github.com/larsxschneider/git.git
>
> [url "mycompany.com"]
>         insteadOf = outside.com

Wouldn't that be better put into say a global git config instead of repeating it for every submodule?

In case of the nested submodule you would need to carry the last lines as an extra patch anyway if this was done in the .gitmodules files? Or do you expect this to be applied recursively (i.e. nested submodules all the way down also substitute outside.com)

> ----------
>
Am I missing your point?
Show 5 quoted lines
> Thanks,
> Lars--
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lars Schneider· Oct 25, 2015, 14:43 UTC · re: Stefan Beller · lore

Re: [RFC] URL rewrite in .gitmodules

On 20 Oct 2015, at 00:07, Stefan Beller <sbeller@google.com> wrote:
Show 10 quoted lines
> On Mon, Oct 19, 2015 at 12:28 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>> 
>> I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the "outside of company" repos. I know I can rewrite Git URLs with the git config "url.<base>.insteadOf" option. However, git configs are client specific.
> 
> I feel like this is working as intended. You only want to improve your
> one client (say the buildbot) to not goto the open source site, while
> the developer may do want to fetch from external sources ("Hey shiny
> new code!";)
Well, that's a good argument. However, our developers have usually no write access to these repos. If they want to push a commit then they need to fork the open source repo and change the submodule URL in the parent repo. I fear that this kind of process might overwhelm them and/or troubles them (changing Git submodules URLs has a few pitfalls). As a result they might be less inclined to make a contribution or - even worse - they copy the code in the parent repo, don't use Submodules and make no contribution at all.
Show 7 quoted lines
> 
>> I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the very same hashes as defined by the "upstream" Open Source repos.
> 
> You could carry a patch on top of the tip of the first submodule
> re-pointing the nested submodule. This requires good workflows
> available to deal with submodules though. (Fetch and merge or rebase,
> git submodule update should be able to do that?)
True. However, we have many Git beginners and I fear that this workflow would overwhelm them.
Show 11 quoted lines
>> 
>> Is there yet another way in Git to change URLs of Submodules in the way I want it?
>> 
>> If not, what do you think about a patch that adds a "url" section similar to the one in git config to a .gitmodules file?
>> 
> 
> So we have different kinds of git configs. within one repository, in
> the home director (global to the one machine),
> maybe you would want to have one "global" config on a network share,
> such that every box in your company
> reads that "company-wide" global config and acts upon that?
That could actually work. The only downside I see is that the devs need to intentionally update their "company" git config. We have +4000 engineers and therefore I want to establish processes that are as easy and fault-tolerant as possible.
Show 12 quoted lines
> 
>> Example:
>> ----------
>> [submodule "git"]
>>        path = git
>>        url=git://github.com/larsxschneider/git.git
>> 
>> [url "mycompany.com"]
>>        insteadOf = outside.com
> 
> Wouldn't that be better put into say a global git config instead of
> repeating it for every submodule?
See answer above. The git config setup could be an obstacle.
Show 6 quoted lines
> 
> In case of the nested submodule you would need to carry the last lines
> as an extra patch anyway
> if this was done in the .gitmodules files? Or do you expect this to be
> applied recursively (i.e. nested
> submodules all the way down also substitute outside.com)
Yes, my intention was to apply these recursively.
> Am I missing your point?
I don't think so :-)

Thanks, Lars

Junio C Hamano· Oct 20, 2015, 17:33 UTC · re: Lars Schneider · lore

Re: [RFC] URL rewrite in .gitmodules

Lars Schneider <larsxschneider@gmail.com> writes:
Show 12 quoted lines
> If not, what do you think about a patch that adds a "url" section
> similar to the one in git config to a .gitmodules file?
>
> Example:
> ----------
> [submodule "git"]
> 	path = git
>         url=git://github.com/larsxschneider/git.git
>
> [url "mycompany.com"]
>         insteadOf = outside.com
> ----------

It is unclear to me if you are adding the last two (or three, counting the blank before) lines to your company's private fork of the opensource project, but if that is the case, then that would defeat your earlier desire:

> ... I also would prefer not to do this as I want to use the
> very same hashes as defined by the "upstream" ...
wouldn't it?

I do not think this topic is specific to use of submodules. If you want to encourage your engineers to fetch from nearby mirrors you maintain, you would want a forest of url.mine.insteadof=theirs for the external repositories that matter to you specified by everybody's $HOME/.gitconfig, and one way to do so would be to have them use the configuration inclusion. An item in your engineer orientation material could tell them to add

	[include]
		path = /usr/local/etc/git/mycompany.urlrewrite
when they set up their "[user] name/email" in there.

And you can update /usr/local/etc/git/mycompany.urlrewrite as needed.

Lars Schneider· Oct 25, 2015, 15:12 UTC · re: Junio C Hamano · lore

Re: [RFC] URL rewrite in .gitmodules

On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
Show 24 quoted lines
> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>> If not, what do you think about a patch that adds a "url" section
>> similar to the one in git config to a .gitmodules file?
>> 
>> Example:
>> ----------
>> [submodule "git"]
>> 	path = git
>>        url=git://github.com/larsxschneider/git.git
>> 
>> [url "mycompany.com"]
>>        insteadOf = outside.com
>> ----------
> 
> It is unclear to me if you are adding the last two (or three,
> counting the blank before) lines to your company's private fork of
> the opensource project, but if that is the case, then that would
> defeat your earlier desire:
> 
>> ... I also would prefer not to do this as I want to use the
>> very same hashes as defined by the "upstream" ...
> 
> wouldn't it?
The last three lines are added to my companies closed source Git repo. In this example the company repo references git://github.com/larsxschneider/git.git as submodule. This submodule in turn references another submodule with a URL "outside.com". This is the URL I want to rewrite. Do you think this could be useful to others as well?
Show 15 quoted lines
> I do not think this topic is specific to use of submodules.  If you
> want to encourage your engineers to fetch from nearby mirrors you
> maintain, you would want a forest of url.mine.insteadof=theirs for
> the external repositories that matter to you specified by
> everybody's $HOME/.gitconfig, and one way to do so would be to have
> them use the configuration inclusion.  An item in your engineer
> orientation material could tell them to add
> 
> 	[include]
> 		path = /usr/local/etc/git/mycompany.urlrewrite
> 
> when they set up their "[user] name/email" in there.
> 
> And you can update /usr/local/etc/git/mycompany.urlrewrite as
> needed.
Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.

Thanks, Lars

Stefan Beller· Oct 26, 2015, 16:34 UTC · re: Lars Schneider · lore

Re: [RFC] URL rewrite in .gitmodules

On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:

Show 46 quoted lines
>
> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
>
>> Lars Schneider <larsxschneider@gmail.com> writes:
>>
>>> If not, what do you think about a patch that adds a "url" section
>>> similar to the one in git config to a .gitmodules file?
>>>
>>> Example:
>>> ----------
>>> [submodule "git"]
>>>      path = git
>>>        url=git://github.com/larsxschneider/git.git
>>>
>>> [url "mycompany.com"]
>>>        insteadOf = outside.com
>>> ----------
>>
>> It is unclear to me if you are adding the last two (or three,
>> counting the blank before) lines to your company's private fork of
>> the opensource project, but if that is the case, then that would
>> defeat your earlier desire:
>>
>>> ... I also would prefer not to do this as I want to use the
>>> very same hashes as defined by the "upstream" ...
>>
>> wouldn't it?
> The last three lines are added to my companies closed source Git repo. In this example the company repo references git://github.com/larsxschneider/git.git as submodule. This submodule in turn references another submodule with a URL "outside.com". This is the URL I want to rewrite. Do you think this could be useful to others as well?
>
>
>> I do not think this topic is specific to use of submodules.  If you
>> want to encourage your engineers to fetch from nearby mirrors you
>> maintain, you would want a forest of url.mine.insteadof=theirs for
>> the external repositories that matter to you specified by
>> everybody's $HOME/.gitconfig, and one way to do so would be to have
>> them use the configuration inclusion.  An item in your engineer
>> orientation material could tell them to add
>>
>>       [include]
>>               path = /usr/local/etc/git/mycompany.urlrewrite
>>
>> when they set up their "[user] name/email" in there.
>>
>> And you can update /usr/local/etc/git/mycompany.urlrewrite as
>> needed.
> Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.

How do you distribute new copies of Git to your engineers? Maybe you could ship them a version which has the "include" line already builtin as default? So your distributed copy of Git would not just check the default places for configs, but also some complied in /net/share/mycompany.gitconfig

>
> Thanks,
> Lars
>
Jens Lehmann· Oct 26, 2015, 16:52 UTC · re: Stefan Beller · lore

Re: [RFC] URL rewrite in .gitmodules

Am 26.10.2015 um 17:34 schrieb Stefan Beller:
Show 24 quoted lines
> On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:
>> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
>>> I do not think this topic is specific to use of submodules.  If you
>>> want to encourage your engineers to fetch from nearby mirrors you
>>> maintain, you would want a forest of url.mine.insteadof=theirs for
>>> the external repositories that matter to you specified by
>>> everybody's $HOME/.gitconfig, and one way to do so would be to have
>>> them use the configuration inclusion.  An item in your engineer
>>> orientation material could tell them to add
>>>
>>>        [include]
>>>                path = /usr/local/etc/git/mycompany.urlrewrite
>>>
>>> when they set up their "[user] name/email" in there.
>>>
>>> And you can update /usr/local/etc/git/mycompany.urlrewrite as
>>> needed.
>> Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.
>
> How do you distribute new copies of Git to your engineers?
> Maybe you could ship them a version which has the "include" line
> already builtin as default? So your distributed copy of Git
> would not just check the default places for configs, but also
> some complied in /net/share/mycompany.gitconfig

Which is just what we do at $DAYJOB, that way you can easily distribute all kinds of settings, customizations and hooks company-wide.

Lars Schneider· Nov 15, 2015, 13:16 UTC · re: Jens Lehmann · lore

Re: [RFC] URL rewrite in .gitmodules

On 26 Oct 2015, at 17:52, Jens Lehmann <Jens.Lehmann@web.de> wrote:
Show 29 quoted lines
> Am 26.10.2015 um 17:34 schrieb Stefan Beller:
>> On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:
>>> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
>>>> I do not think this topic is specific to use of submodules.  If you
>>>> want to encourage your engineers to fetch from nearby mirrors you
>>>> maintain, you would want a forest of url.mine.insteadof=theirs for
>>>> the external repositories that matter to you specified by
>>>> everybody's $HOME/.gitconfig, and one way to do so would be to have
>>>> them use the configuration inclusion.  An item in your engineer
>>>> orientation material could tell them to add
>>>> 
>>>>       [include]
>>>>               path = /usr/local/etc/git/mycompany.urlrewrite
>>>> 
>>>> when they set up their "[user] name/email" in there.
>>>> 
>>>> And you can update /usr/local/etc/git/mycompany.urlrewrite as
>>>> needed.
>>> Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.
>> 
>> How do you distribute new copies of Git to your engineers?
>> Maybe you could ship them a version which has the "include" line
>> already builtin as default? So your distributed copy of Git
>> would not just check the default places for configs, but also
>> some complied in /net/share/mycompany.gitconfig
> 
> Which is just what we do at $DAYJOB, that way you can easily
> distribute all kinds of settings, customizations and hooks
> company-wide.
That's a very good idea. I will try to establish this practice, too.

Thanks, Lars

← back to recent threads