{"thread":{"id":"40591","subject":"[RFC] URL rewrite in .gitmodules","startedAt":"2015-10-19T19:28:20Z","lastAt":"2015-11-15T13:16:13Z","messageCount":8,"participants":["Lars Schneider","Stefan Beller","Junio C Hamano","Jens Lehmann"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"271946","messageId":"DCA9FD0F-252B-4CD7-9137-395771CBC62E@gmail.com","threadId":"40591","inReplyTo":null,"subject":"[RFC] URL rewrite in .gitmodules","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-10-19T19:28:20Z","receivedAt":"2015-10-19T19:28:20Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Hi,\n\nI have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the \"outside of company\" repos. I know I can rewrite Git URLs with the git config \"url.<base>.insteadOf\" option. However, git configs are client specific. I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the\n  very same hashes as defined by the \"upstream\" Open Source repos.\n\nIs there yet another way in Git to change URLs of Submodules in the way I want it?\n\nIf not, what do you think about a patch that adds a \"url\" section similar to the one in git config to a .gitmodules file?\n\nExample:\n----------\n[submodule \"git\"]\n\tpath = git\n        url=git://github.com/larsxschneider/git.git\n\n[url \"mycompany.com\"]\n        insteadOf = outside.com\n----------\n\nThanks,\nLars"},{"id":"271951","messageId":"CAGZ79kY4A3nbp006JyMCgR_Oe6uHf-ECVD+6fJ-naa=XynUxRQ@mail.gmail.com","threadId":"40591","inReplyTo":"DCA9FD0F-252B-4CD7-9137-395771CBC62E@gmail.com","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-19T22:07:31Z","receivedAt":"2015-10-19T22:07:31Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 19, 2015 at 12:28 PM, Lars Schneider\n<larsxschneider@gmail.com> wrote:\n> Hi,\n>\n> I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the \"outside of company\" repos. I know I can rewrite Git URLs with the git config \"url.<base>.insteadOf\" option. However, git configs are client specific.\n\nI feel like this is working as intended. You only want to improve your\none client (say the buildbot) to not goto the open source site, while\nthe developer may do want to fetch from external sources (\"Hey shiny\nnew code!\";)\n\n\n> I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the very same hashes as defined by the \"upstream\" Open Source repos.\n\nYou could carry a patch on top of the tip of the first submodule\nre-pointing the nested submodule. This requires good workflows\navailable to deal with submodules though. (Fetch and merge or rebase,\ngit submodule update should be able to do that?)\n\n>\n> Is there yet another way in Git to change URLs of Submodules in the way I want it?\n>\n> If not, what do you think about a patch that adds a \"url\" section similar to the one in git config to a .gitmodules file?\n>\n\nSo we have different kinds of git configs. within one repository, in\nthe home director (global to the one machine),\nmaybe you would want to have one \"global\" config on a network share,\nsuch that every box in your company\nreads that \"company-wide\" global config and acts upon that?\n\n> Example:\n> ----------\n> [submodule \"git\"]\n>         path = git\n>         url=git://github.com/larsxschneider/git.git\n>\n> [url \"mycompany.com\"]\n>         insteadOf = outside.com\n\nWouldn't that be better put into say a global git config instead of\nrepeating it for every submodule?\n\nIn case of the nested submodule you would need to carry the last lines\nas an extra patch anyway\nif this was done in the .gitmodules files? Or do you expect this to be\napplied recursively (i.e. nested\nsubmodules all the way down also substitute outside.com)\n\n\n> ----------\n>\n\nAm I missing your point?\n\n> Thanks,\n> Lars--\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"271983","messageId":"xmqqio618mxc.fsf@gitster.mtv.corp.google.com","threadId":"40591","inReplyTo":"DCA9FD0F-252B-4CD7-9137-395771CBC62E@gmail.com","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-20T17:33:35Z","receivedAt":"2015-10-20T17:33:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider <larsxschneider@gmail.com> writes:\n\n> If not, what do you think about a patch that adds a \"url\" section\n> similar to the one in git config to a .gitmodules file?\n>\n> Example:\n> ----------\n> [submodule \"git\"]\n> \tpath = git\n>         url=git://github.com/larsxschneider/git.git\n>\n> [url \"mycompany.com\"]\n>         insteadOf = outside.com\n> ----------\n\nIt is unclear to me if you are adding the last two (or three,\ncounting the blank before) lines to your company's private fork of\nthe opensource project, but if that is the case, then that would\ndefeat your earlier desire:\n\n> ... I also would prefer not to do this as I want to use the\n> very same hashes as defined by the \"upstream\" ...\n\nwouldn't it?\n\nI do not think this topic is specific to use of submodules.  If you\nwant to encourage your engineers to fetch from nearby mirrors you\nmaintain, you would want a forest of url.mine.insteadof=theirs for\nthe external repositories that matter to you specified by\neverybody's $HOME/.gitconfig, and one way to do so would be to have\nthem use the configuration inclusion.  An item in your engineer\norientation material could tell them to add\n\n\t[include]\n\t\tpath = /usr/local/etc/git/mycompany.urlrewrite\n\nwhen they set up their \"[user] name/email\" in there.\n\nAnd you can update /usr/local/etc/git/mycompany.urlrewrite as\nneeded.\n"},{"id":"272198","messageId":"CD8200EC-CE3A-40B7-A595-3B47F47A263B@gmail.com","threadId":"40591","inReplyTo":"CAGZ79kY4A3nbp006JyMCgR_Oe6uHf-ECVD+6fJ-naa=XynUxRQ@mail.gmail.com","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-10-25T14:43:35Z","receivedAt":"2015-10-25T14:43:35Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 20 Oct 2015, at 00:07, Stefan Beller <sbeller@google.com> wrote:\n\n> On Mon, Oct 19, 2015 at 12:28 PM, Lars Schneider\n> <larsxschneider@gmail.com> wrote:\n>> Hi,\n>> \n>> I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the \"outside of company\" repos. I know I can rewrite Git URLs with the git config \"url.<base>.insteadOf\" option. However, git configs are client specific.\n> \n> I feel like this is working as intended. You only want to improve your\n> one client (say the buildbot) to not goto the open source site, while\n> the developer may do want to fetch from external sources (\"Hey shiny\n> new code!\";)\nWell, that's a good argument. However, our developers have usually no write access to these repos. If they want to push a commit then they need to fork the open source repo and change the submodule URL in the parent repo. I fear that this kind of process might overwhelm them and/or troubles them (changing Git submodules URLs has a few pitfalls). As a result they might be less inclined to make a contribution or - even worse - they copy the code in the parent repo, don't use Submodules and make no contribution at all. \n\n\n> \n>> I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the very same hashes as defined by the \"upstream\" Open Source repos.\n> \n> You could carry a patch on top of the tip of the first submodule\n> re-pointing the nested submodule. This requires good workflows\n> available to deal with submodules though. (Fetch and merge or rebase,\n> git submodule update should be able to do that?)\nTrue. However, we have many Git beginners and I fear that this workflow would overwhelm them.\n\n\n>> \n>> Is there yet another way in Git to change URLs of Submodules in the way I want it?\n>> \n>> If not, what do you think about a patch that adds a \"url\" section similar to the one in git config to a .gitmodules file?\n>> \n> \n> So we have different kinds of git configs. within one repository, in\n> the home director (global to the one machine),\n> maybe you would want to have one \"global\" config on a network share,\n> such that every box in your company\n> reads that \"company-wide\" global config and acts upon that?\nThat could actually work. The only downside I see is that the devs need to intentionally update their \"company\" git config. We have +4000 engineers and therefore I want to establish processes that are as easy and fault-tolerant as possible.  \n\n> \n>> Example:\n>> ----------\n>> [submodule \"git\"]\n>>        path = git\n>>        url=git://github.com/larsxschneider/git.git\n>> \n>> [url \"mycompany.com\"]\n>>        insteadOf = outside.com\n> \n> Wouldn't that be better put into say a global git config instead of\n> repeating it for every submodule?\nSee answer above. The git config setup could be an obstacle.\n\n> \n> In case of the nested submodule you would need to carry the last lines\n> as an extra patch anyway\n> if this was done in the .gitmodules files? Or do you expect this to be\n> applied recursively (i.e. nested\n> submodules all the way down also substitute outside.com)\nYes, my intention was to apply these recursively.\n\n\n> Am I missing your point?\nI don't think so :-)\n\nThanks,\nLars"},{"id":"272200","messageId":"F01EF930-9787-44DD-A2E5-F5FBA029D3E7@gmail.com","threadId":"40591","inReplyTo":"xmqqio618mxc.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-10-25T15:12:23Z","receivedAt":"2015-10-25T15:12:23Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:\n\n> Lars Schneider <larsxschneider@gmail.com> writes:\n> \n>> If not, what do you think about a patch that adds a \"url\" section\n>> similar to the one in git config to a .gitmodules file?\n>> \n>> Example:\n>> ----------\n>> [submodule \"git\"]\n>> \tpath = git\n>>        url=git://github.com/larsxschneider/git.git\n>> \n>> [url \"mycompany.com\"]\n>>        insteadOf = outside.com\n>> ----------\n> \n> It is unclear to me if you are adding the last two (or three,\n> counting the blank before) lines to your company's private fork of\n> the opensource project, but if that is the case, then that would\n> defeat your earlier desire:\n> \n>> ... I also would prefer not to do this as I want to use the\n>> very same hashes as defined by the \"upstream\" ...\n> \n> wouldn't it?\nThe last three lines are added to my companies closed source Git repo. In this example the company repo references git://github.com/larsxschneider/git.git as submodule. This submodule in turn references another submodule with a URL \"outside.com\". This is the URL I want to rewrite. Do you think this could be useful to others as well?\n\n\n> I do not think this topic is specific to use of submodules.  If you\n> want to encourage your engineers to fetch from nearby mirrors you\n> maintain, you would want a forest of url.mine.insteadof=theirs for\n> the external repositories that matter to you specified by\n> everybody's $HOME/.gitconfig, and one way to do so would be to have\n> them use the configuration inclusion.  An item in your engineer\n> orientation material could tell them to add\n> \n> \t[include]\n> \t\tpath = /usr/local/etc/git/mycompany.urlrewrite\n> \n> when they set up their \"[user] name/email\" in there.\n> \n> And you can update /usr/local/etc/git/mycompany.urlrewrite as\n> needed.\nOh nice, I didn't know about \"include\". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.\n\nThanks,\nLars\n"},{"id":"272244","messageId":"CAGZ79kbFDB55mKnw-ONPGBmHfeXZDBKOb=HrjjepiBh3kgcO1g@mail.gmail.com","threadId":"40591","inReplyTo":"F01EF930-9787-44DD-A2E5-F5FBA029D3E7@gmail.com","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-26T16:34:33Z","receivedAt":"2015-10-26T16:34:33Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider\n<larsxschneider@gmail.com> wrote:\n>\n> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:\n>\n>> Lars Schneider <larsxschneider@gmail.com> writes:\n>>\n>>> If not, what do you think about a patch that adds a \"url\" section\n>>> similar to the one in git config to a .gitmodules file?\n>>>\n>>> Example:\n>>> ----------\n>>> [submodule \"git\"]\n>>>      path = git\n>>>        url=git://github.com/larsxschneider/git.git\n>>>\n>>> [url \"mycompany.com\"]\n>>>        insteadOf = outside.com\n>>> ----------\n>>\n>> It is unclear to me if you are adding the last two (or three,\n>> counting the blank before) lines to your company's private fork of\n>> the opensource project, but if that is the case, then that would\n>> defeat your earlier desire:\n>>\n>>> ... I also would prefer not to do this as I want to use the\n>>> very same hashes as defined by the \"upstream\" ...\n>>\n>> wouldn't it?\n> The last three lines are added to my companies closed source Git repo. In this example the company repo references git://github.com/larsxschneider/git.git as submodule. This submodule in turn references another submodule with a URL \"outside.com\". This is the URL I want to rewrite. Do you think this could be useful to others as well?\n>\n>\n>> I do not think this topic is specific to use of submodules.  If you\n>> want to encourage your engineers to fetch from nearby mirrors you\n>> maintain, you would want a forest of url.mine.insteadof=theirs for\n>> the external repositories that matter to you specified by\n>> everybody's $HOME/.gitconfig, and one way to do so would be to have\n>> them use the configuration inclusion.  An item in your engineer\n>> orientation material could tell them to add\n>>\n>>       [include]\n>>               path = /usr/local/etc/git/mycompany.urlrewrite\n>>\n>> when they set up their \"[user] name/email\" in there.\n>>\n>> And you can update /usr/local/etc/git/mycompany.urlrewrite as\n>> needed.\n> Oh nice, I didn't know about \"include\". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.\n\nHow do you distribute new copies of Git to your engineers?\nMaybe you could ship them a version which has the \"include\" line\nalready builtin as default? So your distributed copy of Git\nwould not just check the default places for configs, but also\nsome complied in /net/share/mycompany.gitconfig\n\n>\n> Thanks,\n> Lars\n>\n"},{"id":"272245","messageId":"562E5A31.7050903@web.de","threadId":"40591","inReplyTo":"CAGZ79kbFDB55mKnw-ONPGBmHfeXZDBKOb=HrjjepiBh3kgcO1g@mail.gmail.com","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-10-26T16:52:01Z","receivedAt":"2015-10-26T16:52:01Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 26.10.2015 um 17:34 schrieb Stefan Beller:\n> On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:\n>> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:\n>>> I do not think this topic is specific to use of submodules.  If you\n>>> want to encourage your engineers to fetch from nearby mirrors you\n>>> maintain, you would want a forest of url.mine.insteadof=theirs for\n>>> the external repositories that matter to you specified by\n>>> everybody's $HOME/.gitconfig, and one way to do so would be to have\n>>> them use the configuration inclusion.  An item in your engineer\n>>> orientation material could tell them to add\n>>>\n>>>        [include]\n>>>                path = /usr/local/etc/git/mycompany.urlrewrite\n>>>\n>>> when they set up their \"[user] name/email\" in there.\n>>>\n>>> And you can update /usr/local/etc/git/mycompany.urlrewrite as\n>>> needed.\n>> Oh nice, I didn't know about \"include\". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.\n>\n> How do you distribute new copies of Git to your engineers?\n> Maybe you could ship them a version which has the \"include\" line\n> already builtin as default? So your distributed copy of Git\n> would not just check the default places for configs, but also\n> some complied in /net/share/mycompany.gitconfig\n\nWhich is just what we do at $DAYJOB, that way you can easily\ndistribute all kinds of settings, customizations and hooks\ncompany-wide.\n"},{"id":"273346","messageId":"289C8789-5102-4721-994E-E0B39FD1BFD0@gmail.com","threadId":"40591","inReplyTo":"562E5A31.7050903@web.de","subject":"Re: [RFC] URL rewrite in .gitmodules","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-11-15T13:16:13Z","receivedAt":"2015-11-15T13:16:13Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 26 Oct 2015, at 17:52, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n\n> Am 26.10.2015 um 17:34 schrieb Stefan Beller:\n>> On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:\n>>> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:\n>>>> I do not think this topic is specific to use of submodules.  If you\n>>>> want to encourage your engineers to fetch from nearby mirrors you\n>>>> maintain, you would want a forest of url.mine.insteadof=theirs for\n>>>> the external repositories that matter to you specified by\n>>>> everybody's $HOME/.gitconfig, and one way to do so would be to have\n>>>> them use the configuration inclusion.  An item in your engineer\n>>>> orientation material could tell them to add\n>>>> \n>>>>       [include]\n>>>>               path = /usr/local/etc/git/mycompany.urlrewrite\n>>>> \n>>>> when they set up their \"[user] name/email\" in there.\n>>>> \n>>>> And you can update /usr/local/etc/git/mycompany.urlrewrite as\n>>>> needed.\n>>> Oh nice, I didn't know about \"include\". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.\n>> \n>> How do you distribute new copies of Git to your engineers?\n>> Maybe you could ship them a version which has the \"include\" line\n>> already builtin as default? So your distributed copy of Git\n>> would not just check the default places for configs, but also\n>> some complied in /net/share/mycompany.gitconfig\n> \n> Which is just what we do at $DAYJOB, that way you can easily\n> distribute all kinds of settings, customizations and hooks\n> company-wide.\n\nThat's a very good idea. I will try to establish this practice, too.\n\nThanks,\nLars"}]}