{"thread":{"id":"45176","subject":"url..insteadOf vs. submodules","startedAt":"2017-02-19T21:21:38Z","lastAt":"2017-02-22T19:14:26Z","messageCount":19,"participants":["Toolforger","Jeff King","Stefan Beller","Junio C Hamano","Jon Loeliger"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"312091","messageId":"84fcb0bd-85dc-0142-dd58-47a04eaa7c2b@durchholz.org","threadId":"45176","inReplyTo":null,"subject":"url..insteadOf vs. submodules","fromName":"Toolforger","fromEmail":"toolforger@durchholz.org","sentAt":"2017-02-19T21:12:28Z","receivedAt":"2017-02-19T21:21:38Z","isPatch":false,"sender":{"key":"toolforger@durchholz.org","avatar":"https://gravatar.com/avatar/dc558206553a959ec2b81ae9833a718c1c8453f6bf1817db1f7ccf2d563f5f89?d=mp&s=160"},"body":"Hi all,\n\nI am trying to make url..insteadOf work on the URLs inside \n.gitmodules, but it won't work (applying it to the repo itself works \nfine, to the config setting seems to be fine).\n\nI do not want to modify .gitmodules: It is maintained upstream.\n\nI cannot simply reconfigure submodule..url: the Configure script \n(regularly called during each compile) does\n git submodule sync\n git submodule update --init\nI could tell upstream to change these commands if I can make a good \nargument; for them, it is relevant that they can change the submodule \nURL inside .gitmodule and have it \"just work\" for everybody downstream.\n\nMy own use case is that I want to be able to work with various \nexperimental local clones even if I do not have Internet access.\nI'm all ears if there's a way to do this without using insteadOf.\n\n\nHere are the relevant two lines from the output of \"git config -l\" \n(after \"git submodule init\"):\n\nurl./home/jo/Projekte/perl6/bare-repos.insteadof=https://github.com\nsubmodule.3rdparty/dynasm.url=https://github.com/MoarVM/dynasm.git\n\n\nHere is what \"git submodule update\" does:\n\nCloning into '3rdparty/dyncall'...\nfatal: unable to access 'https://github.com/MoarVM/dyncall.git/': Could \nnot resolve host: github.com\nfatal: clone of 'https://github.com/MoarVM/dyncall.git' into submodule \npath '3rdparty/dyncall' failed\n\n\nAny help appreciated!\n\nRegards,\nJo\n"},{"id":"312130","messageId":"20170220090115.6kfzwl62opj4q7k7@sigill.intra.peff.net","threadId":"45176","inReplyTo":"84fcb0bd-85dc-0142-dd58-47a04eaa7c2b@durchholz.org","subject":"Re: url..insteadOf vs. submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-20T09:01:16Z","receivedAt":"2017-02-20T09:01:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 19, 2017 at 10:12:28PM +0100, Toolforger wrote:\n\n> I am trying to make url..insteadOf work on the URLs inside\n> .gitmodules, but it won't work (applying it to the repo itself works fine,\n> to the config setting seems to be fine).\n\nThe submodule operations happen in their own processes, and do not look\nat the config of the parent repo. Are you setting the config in\n.git/config of the super-project?\n\nI don't know if there plans to make that work, but one workaround is to\nset the config in ~/.gitconfig.\n\n-Peff\n"},{"id":"312166","messageId":"404d109f-e5a7-85a3-e64c-ab1b21c3045d@durchholz.org","threadId":"45176","inReplyTo":"20170220090115.6kfzwl62opj4q7k7@sigill.intra.peff.net","subject":"Re: url..insteadOf vs. submodules","fromName":"Toolforger","fromEmail":"toolforger@durchholz.org","sentAt":"2017-02-20T20:31:40Z","receivedAt":"2017-02-20T20:31:49Z","isPatch":false,"sender":{"key":"toolforger@durchholz.org","avatar":"https://gravatar.com/avatar/dc558206553a959ec2b81ae9833a718c1c8453f6bf1817db1f7ccf2d563f5f89?d=mp&s=160"},"body":"On 20.02.2017 10:01, Jeff King wrote:\n> On Sun, Feb 19, 2017 at 10:12:28PM +0100, Toolforger wrote:\n>\n>> I am trying to make url..insteadOf work on the URLs inside\n>> .gitmodules, but it won't work (applying it to the repo itself works fine,\n>> to the config setting seems to be fine).\n>\n> The submodule operations happen in their own processes, and do not look\n> at the config of the parent repo.\n\nAh, then we have a docbug.\ngit help config has this to say:\n\nurl..insteadOf\n Any URL that starts with this value will be rewritten to start,\n instead, with .\n\nThe \"Any\" here is wrong, it would be \"any except submodule\" (possibly \nother exceptions).\n\n > Are you setting the config in\n> .git/config of the super-project?\n\nExactly.\nMy thinking was that since the submodule URLs are specified in the super \nproject's .gitmodules, that setting should apply.\n\n> I don't know if there plans to make that work,\n\nIt would certainly help me out, though I guess it's going to be too late \nfor my current project :-)\n\n > but one workaround is to set the config in ~/.gitconfig.\n\nNo can do - that's under version control.\nMy personal setup does not belong there I think ;-)\n\nI am currently trying to write a shell script that\n- does git submodule init\n- pulls submodule configuration out of git config -l\n- configures each submodule with insteadOf\nIt fits with my workflow because setting up the repositories is going to \nbe done via script anyway.\nI'm neither a shell nor a git expert, so any advice still appreciated.\n\nRegards,\nJo\n"},{"id":"312169","messageId":"20170220205243.lynnmxouwq7jelld@sigill.intra.peff.net","threadId":"45176","inReplyTo":"404d109f-e5a7-85a3-e64c-ab1b21c3045d@durchholz.org","subject":"Re: url..insteadOf vs. submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-20T20:52:43Z","receivedAt":"2017-02-20T20:52:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 20, 2017 at 09:31:40PM +0100, Toolforger wrote:\n\n> > The submodule operations happen in their own processes, and do not look\n> > at the config of the parent repo.\n> \n> Ah, then we have a docbug.\n> git help config has this to say:\n> \n> url..insteadOf\n> Any URL that starts with this value will be rewritten to start,\n> instead, with .\n> \n> The \"Any\" here is wrong, it would be \"any except submodule\" (possibly other\n> exceptions).\n\nI'm not sure that \"any\" is wrong here. Repository-specific config does\nnot cross repository boundaries. That applies to this config value, and\nto all the others, too (e.g., if you set \"diff.renames\" in the\nsuper-project, it would not have an effect in the submodule).\n\nI think if there is a doc bug, it is that the repo boundary between the\nsubmodule and the super-project is not made more clear.\n\nThat said, I do think it would be a useful feature for the super-project\nto rewrite URLs before handing them off to the submodule. But I do not\nreally work on submodules nor use them myself, so there may be\ncomplications.\n\nI suppose you could argue that failing to rewrite violates the \"any\" in\nthe quoted text. It doesn't say when the rewriting occurs, but it is\nessentially \"when the URL is accessed\". So the super-project feeds the\nraw URL to the submodule `git clone`, which then applies any URL\nrewriting.\n\n> > but one workaround is to set the config in ~/.gitconfig.\n> \n> No can do - that's under version control.\n> My personal setup does not belong there I think ;-)\n\nI'm not sure I understand. You have a project policy to use certain\nURLs. But you, the user, want to override that. Why isn't the\nuser-specific config file the right place to put that?\n\n(I think there _is_ a mismatch, in that the change is specific not just\nto your user, but to the repo. So you would not want to rewrite other\nreferences to the same URL in other repos. But that does not seem to be\nyour objection).\n\n-Peff\n"},{"id":"312186","messageId":"28fb85d4-89cd-1f32-3063-2f48d8b935be@durchholz.org","threadId":"45176","inReplyTo":"20170220205243.lynnmxouwq7jelld@sigill.intra.peff.net","subject":"Re: url..insteadOf vs. submodules","fromName":"Toolforger","fromEmail":"toolforger@durchholz.org","sentAt":"2017-02-21T05:11:51Z","receivedAt":"2017-02-21T05:12:18Z","isPatch":false,"sender":{"key":"toolforger@durchholz.org","avatar":"https://gravatar.com/avatar/dc558206553a959ec2b81ae9833a718c1c8453f6bf1817db1f7ccf2d563f5f89?d=mp&s=160"},"body":"On 20.02.2017 21:52, Jeff King wrote:\n > I think if there is a doc bug, it is that the repo boundary between the\n > submodule and the super-project is not made more clear.\n\nIt's not mentioned anywhere I'm aware of, particularly not on the \ninsteadOf docs.\n\n > That said, I do think it would be a useful feature for the super-project\n > to rewrite URLs before handing them off to the submodule. But I do not\n > really work on submodules nor use them myself, so there may be\n > complications.\n\nAgreed.\n\n > I suppose you could argue that failing to rewrite violates the \"any\" in\n > the quoted text. It doesn't say when the rewriting occurs, but it is\n > essentially \"when the URL is accessed\". So the super-project feeds the\n > raw URL to the submodule `git clone`, which then applies any URL\n > rewriting.\n\n\n\n >>> but one workaround is to set the config in ~/.gitconfig.\n >>\n >> No can do - that's under version control.\n >> My personal setup does not belong there I think ;-)\n >\n > I'm not sure I understand. You have a project policy to use certain\n > URLs. But you, the user, want to override that. Why isn't the\n > user-specific config file the right place to put that?\n\nAh right, I mistook ~/ for \"project root\" instead of \"home dir\".\nSorry for the confusion.\n\n > (I think there _is_ a mismatch, in that the change is specific not just\n > to your user, but to the repo. So you would not want to rewrite other\n > references to the same URL in other repos.\n\nIndeed, and that's actually a problem.\n\nThe setup I'm aiming for is\n github -> local bare repo -> local clones with worktrees\n\nIf I place insteadOf rules in ~/.gitconfig, I will be unable to pull \nfrom github to my local bare repos.\nMmm... I could try to undo the insteadOf configuration from ~/.gitconfig \nin the local bare repos. Not sure whether I have to redirect from the \ngithub URL to itself.\n\nDownside is that I'll have to remember to modify ~/.gitconfig whenever \nthe upstream project changes its dependencies. Or whenever I want to \nreorganize my local project directory structure.\nIt's not totally out of the window, but right now it does not seem very \nattractive to me, and it's certainly not a good solution for everyone.\n\nRegards,\nJo\n"},{"id":"312191","messageId":"20170221070653.65ho2anbp55uzjeu@sigill.intra.peff.net","threadId":"45176","inReplyTo":"28fb85d4-89cd-1f32-3063-2f48d8b935be@durchholz.org","subject":"Re: url..insteadOf vs. submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-21T07:06:54Z","receivedAt":"2017-02-21T07:08:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 21, 2017 at 06:11:51AM +0100, Toolforger wrote:\n\n> > I'm not sure I understand. You have a project policy to use certain\n> > URLs. But you, the user, want to override that. Why isn't the\n> > user-specific config file the right place to put that?\n> \n> Ah right, I mistook ~/ for \"project root\" instead of \"home dir\".\n> Sorry for the confusion.\n\nAh, OK, that makes more sense.\n\n> > (I think there _is_ a mismatch, in that the change is specific not just\n> > to your user, but to the repo. So you would not want to rewrite other\n> > references to the same URL in other repos.\n> \n> Indeed, and that's actually a problem.\n> \n> The setup I'm aiming for is\n> github -> local bare repo -> local clones with worktrees\n> \n> If I place insteadOf rules in ~/.gitconfig, I will be unable to pull from\n> github to my local bare repos.\n> Mmm... I could try to undo the insteadOf configuration from ~/.gitconfig in\n> the local bare repos. Not sure whether I have to redirect from the github\n> URL to itself.\n\nYeah, I think you would probably have to do a redirect-to-self to\noverride the global one.\n\nAt one point we discussed having conditional-config that would kick in\nbased on path-matching. I think it would be another way to do what you\nwant, but there's nothing merged.\n\nI think anything involving ~/.gitconfig is basically a hack, though.\nWhat you really want is for submodules to better support your\nURL-rewriting case, and that's not an unreasonable thing to want.\n\nWe'll see if the submodule folks have any ideas on how to implement\nthat.\n\n-Peff\n"},{"id":"312212","messageId":"CAGZ79kZgMbEZy7hoA+VxsKdKBavt59SmC1c6FpDdgrW2GKMHvQ@mail.gmail.com","threadId":"45176","inReplyTo":"20170221070653.65ho2anbp55uzjeu@sigill.intra.peff.net","subject":"Re: url..insteadOf vs. submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-21T18:19:38Z","receivedAt":"2017-02-21T18:20:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Feb 20, 2017 at 11:06 PM, Jeff King wrote:\n>\n> We'll see if the submodule folks have any ideas on how to implement\n> that.\n>\n\nSo from reading your discussion, the user expectation is to have\n`git submodule {init, update --init, sync}`\nto pay attention to url..insteadOf when setting up the\nsubmodule..URL, such that the modified URL is used for the\ninitial clone of the submodule (and hence any subsequent usage within\nthe submodule).\n\nThat sounds like a good idea to me.\n\nTwo caveates:\n\n* After running `git submodule init`, you change url..insteadOf\n in the superproject. How do we need to word the documentation to\n have users expecting this change doesn't affect submodules?\n (See above Any vs. \"Any except (initialized) submodules\")\n\n* So with the point above the insteadOf config only applies to the\n init/sync process, (i.e. once in time, ideally).\n Is that confusing or actually simplifying the submodule workflow?\n\nThanks,\nStefan\n"},{"id":"312235","messageId":"20170221230029.cs36tjwpsw2opuwp@sigill.intra.peff.net","threadId":"45176","inReplyTo":"CAGZ79kZgMbEZy7hoA+VxsKdKBavt59SmC1c6FpDdgrW2GKMHvQ@mail.gmail.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-21T23:00:29Z","receivedAt":"2017-02-21T23:00:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 21, 2017 at 10:19:38AM -0800, Stefan Beller wrote:\n\n> On Mon, Feb 20, 2017 at 11:06 PM, Jeff King wrote:\n> >\n> > We'll see if the submodule folks have any ideas on how to implement\n> > that.\n> >\n> \n> So from reading your discussion, the user expectation is to have\n> `git submodule {init, update --init, sync}`\n> to pay attention to url..insteadOf when setting up the\n> submodule..URL, such that the modified URL is used for the\n> initial clone of the submodule (and hence any subsequent usage within\n> the submodule).\n\nYeah, that was what I was envisioning.\n\n> Two caveates:\n> \n> * After running `git submodule init`, you change url..insteadOf\n> in the superproject. How do we need to word the documentation to\n> have users expecting this change doesn't affect submodules?\n> (See above Any vs. \"Any except (initialized) submodules\")\n\nGood question.\n\nI guess one answer is that this is the wrong approach entirely, and the\nright one is something like: submodules should understand that they are\npart of a superproject, and respect some whitelisted set of config from\nthe superproject .git/config file.\n\nThe second half is pretty easy to do (use git_config_from_file on the\nsuper-project's $GIT_DIR/config, and pass a callback which filters the\nkeys before passing them along to the real callback).\n\nI'm not sure about the first half (submodules know about their\nsuperproject), though.\n\n> * So with the point above the insteadOf config only applies to the\n> init/sync process, (i.e. once in time, ideally).\n> Is that confusing or actually simplifying the submodule workflow?\n\nNot sure. That's why I asked you. :)\n\nOne other caveat: I'm not sure if we do insteadOf recursively, but it\nmay be surprising to the child \"git clone\" that we've already applied\nthe insteadOf rewriting (especially if the rules are coming from\n~/.gitconfig and may be applied twice).\n\n-Peff\n"},{"id":"312236","messageId":"CAGZ79kby-UhUqci9Mgdhw+wvS5Y39=Q7AmCrWaTMWbcZPNT6Dw@mail.gmail.com","threadId":"45176","inReplyTo":"20170221230029.cs36tjwpsw2opuwp@sigill.intra.peff.net","subject":"Re: url..insteadOf vs. submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-21T23:16:27Z","receivedAt":"2017-02-21T23:16:33Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 21, 2017 at 3:00 PM, Jeff King wrote:\n> On Tue, Feb 21, 2017 at 10:19:38AM -0800, Stefan Beller wrote:\n>\n>> On Mon, Feb 20, 2017 at 11:06 PM, Jeff King wrote:\n>> >\n>> > We'll see if the submodule folks have any ideas on how to implement\n>> > that.\n>> >\n>>\n>> So from reading your discussion, the user expectation is to have\n>> `git submodule {init, update --init, sync}`\n>> to pay attention to url..insteadOf when setting up the\n>> submodule..URL, such that the modified URL is used for the\n>> initial clone of the submodule (and hence any subsequent usage within\n>> the submodule).\n>\n> Yeah, that was what I was envisioning.\n>\n>> Two caveates:\n>>\n>> * After running `git submodule init`, you change url..insteadOf\n>> in the superproject. How do we need to word the documentation to\n>> have users expecting this change doesn't affect submodules?\n>> (See above Any vs. \"Any except (initialized) submodules\")\n>\n> Good question.\n>\n> I guess one answer is that this is the wrong approach entirely, and the\n> right one is something like: submodules should understand that they are\n> part of a superproject, and respect some whitelisted set of config from\n> the superproject .git/config file.\n\nThis would break one of the core assumptions that submodules\nare \"independent\" repos.\n\nThe way of action is a one way street:\n* The superproject is aware of the submodule and when you invoke a\ncommand on the superproject, you may mess around with the submodule,\ne.g. update/remove it; absorb its git directory.\n* The submodule is \"just\" a repository with weird .git link file and a\n respective core.worktree setup. Currently it doesn't know if it is\n guided by a superproject.\n\n\nThough I do not know if this is actually a good assumption.\ne.g. \"[PATCH v2] git-prompt.sh: add submodule indicator\"\nhttps://public-inbox.org/git/1486075892-20676-2-git-send-email-email@benjaminfuchs.de/\nreally had trouble in the first version to nail down how to tell you are in\na submodule, but people want to know that.\n\n>\n> The second half is pretty easy to do (use git_config_from_file on the\n> super-project's $GIT_DIR\n\nThere goes the \"pretty easy\"; currently there is no concept to find out\nthe existence of a super-project.\n\n> /config, and pass a callback which filters the\n> keys before passing them along to the real callback).\n>\n> I'm not sure about the first half (submodules know about their\n> superproject), though.\n\nMaybe we need to change that fundamental assumption.\nSo a more sophisticated way (thinking long term here) would be\nto include the superprojects config file (with exceptions), and that\nconfig file has more priority than e.g. the ~/.gitconfig file, but less\nthan the submodules own $GIT_DIR/config file.\nThen a setting like the url rewriting would be \"inherited\" by the\nsubmodule, with the option to overwrite the default as given by the\nsuperproject.\n\n>\n>> * So with the point above the insteadOf config only applies to the\n>> init/sync process, (i.e. once in time, ideally).\n>> Is that confusing or actually simplifying the submodule workflow?\n>\n> Not sure. That's why I asked you. :)\n\nI think that would be ok. With the idea of inheriting the superprojects\nconfig, we allow for not storing the rewritten url, so the submodule\nhandling is less of a corner case here, and as another advantage the\nrewriting rule is applied in real time, e.g. you can change the superprojects\nrule after the fact and the submodule would automagically make use of it.\n\n>\n> One other caveat: I'm not sure if we do insteadOf recursively, but it\n> may be surprising to the child \"git clone\" that we've already applied\n> the insteadOf rewriting (especially if the rules are coming from\n> ~/.gitconfig and may be applied twice).\n\nWhen a rule is having effect twice the rule sounds broken. (the outcome\nought to be sufficiently different from the original?)\n\n>\n> -Peff\n\nThanks,\nStefan\n"},{"id":"312239","messageId":"xmqqo9xvdsji.fsf@gitster.mtv.corp.google.com","threadId":"45176","inReplyTo":"CAGZ79kby-UhUqci9Mgdhw+wvS5Y39=Q7AmCrWaTMWbcZPNT6Dw@mail.gmail.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-21T23:37:37Z","receivedAt":"2017-02-21T23:37:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> On Tue, Feb 21, 2017 at 3:00 PM, Jeff King wrote:\n> ...\n>> I guess one answer is that this is the wrong approach entirely, and the\n>> right one is something like: submodules should understand that they are\n>> part of a superproject, and respect some whitelisted set of config from\n>> the superproject .git/config file.\n>\n> This would break one of the core assumptions that submodules\n> are \"independent\" repos.\n>\n> The way of action is a one way street:\n> * The superproject is aware of the submodule and when you invoke a\n> command on the superproject, you may mess around with the submodule,\n> e.g. update/remove it; absorb its git directory.\n> * The submodule is \"just\" a repository with weird .git link file and a\n> respective core.worktree setup. Currently it doesn't know if it is\n> guided by a superproject.\n\nWhile that is a good discipline to follow, I think you need to\ndifferenciate the project that is bound as a submodule to a\nsuperproject, and a specific instance of a submodule repository,\ni.e. a clone of such a project.\n\nIt is true that the Linux kernel project should *NEVER* know your\nappliance project only because you happen to use it as a component\nof your appliance that happens to use the kernel as one of its\nsubmodules. But that does not mean your copy of the kernel that\nsits in your recursive checkout of your appliance project should\nnot know anything about your superproject.\n\nThis is true even without any submodules. The Git project itself\ndoes not even care you are Stefan, but you still can and do add\n[user] name = \"Stefan Beller\" to .git/config of your clone of the\nGit project. A clone of the project may want to know more than the\ndata project itself keeps track of to describe the context in which\nthe particular clone is being used. And .git/config is a good place\nto keep such pieces of information.\n\nSo I would think it is entirely reasonable if \"git submodule init\nsub\" that is run in the superproject to initialize \"sub\" writes\nsomething in \"sub/.git\" to tell that \"sub\" is used in the context of\nthat particular toplevel superproject and customize its behavour\naccordingly. Perhaps it may want to add the url.*.insteadOf that is\nuseful for updating the submodule repository when it does \"submodule\ninit\", for example.\n"},{"id":"312240","messageId":"20170221234037.ga44u3birwd5whab@sigill.intra.peff.net","threadId":"45176","inReplyTo":"CAGZ79kby-UhUqci9Mgdhw+wvS5Y39=Q7AmCrWaTMWbcZPNT6Dw@mail.gmail.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-21T23:40:37Z","receivedAt":"2017-02-21T23:40:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 21, 2017 at 03:16:27PM -0800, Stefan Beller wrote:\n\n> > I guess one answer is that this is the wrong approach entirely, and the\n> > right one is something like: submodules should understand that they are\n> > part of a superproject, and respect some whitelisted set of config from\n> > the superproject .git/config file.\n> \n> This would break one of the core assumptions that submodules\n> are \"independent\" repos.\n\nYeah, that was the \"first half\" that I said was hard. :)\n\nYou could rationalize it under the fact that they _are_ independent\nrepos; we're just adding a new config source. Arguably it could be a\nfeature for any repository embedded inside the working tree of another,\nsubmodule or not, to consider the outer repository as a (limited) source\nof config.\n\nBut there are probably a lot of irritating corner cases with the whole\nconcept unless we apply a strict whitelist of keys (e.g., you probably\ndon't want remote.* to be propagated). And as the recent\nGIT_CONFIG_PARAMETERS whitelist showed, that approach ended up confusing\nand annoying.\n\nSo maybe the whole thing is insane, and the right answer is that config\nvalues should go into ~/.gitconfig. And we may need better tools there\nfor limiting that global config to certain parts of the tree (like Duy's\nconditional include thing).\n\n> Though I do not know if this is actually a good assumption.\n> e.g. \"[PATCH v2] git-prompt.sh: add submodule indicator\"\n> https://public-inbox.org/git/1486075892-20676-2-git-send-email-email@benjaminfuchs.de/\n> really had trouble in the first version to nail down how to tell you are in\n> a submodule, but people want to know that.\n\nRight, I think it's an interesting thing to know, but I agree there are\nprobably a lot of corner cases.\n\n> Maybe we need to change that fundamental assumption.\n> So a more sophisticated way (thinking long term here) would be\n> to include the superprojects config file (with exceptions), and that\n> config file has more priority than e.g. the ~/.gitconfig file, but less\n> than the submodules own $GIT_DIR/config file.\n\nYeah, that priority matches what I had been thinking.\n\n> > One other caveat: I'm not sure if we do insteadOf recursively, but it\n> > may be surprising to the child \"git clone\" that we've already applied\n> > the insteadOf rewriting (especially if the rules are coming from\n> > ~/.gitconfig and may be applied twice).\n> \n> When a rule is having effect twice the rule sounds broken. (the outcome\n> ought to be sufficiently different from the original?)\n\nIf you have:\n\n url.bar.insteadOf=foo\n url.baz.insteadOf=bar\n\ndo we convert \"foo\" to \"baz\"? If so, then I think applying the rules\nagain shouldn't matter. But if we don't, and only do a single level,\nthen having the caller rewrite the URL before it hands it to \"git clone\"\nmeans we may end up unexpectedly doing two levels of rewriting.\n\n-Peff\n"},{"id":"312262","messageId":"xmqqk28jdrih.fsf@gitster.mtv.corp.google.com","threadId":"45176","inReplyTo":"xmqqo9xvdsji.fsf@gitster.mtv.corp.google.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-21T23:59:50Z","receivedAt":"2017-02-21T23:59:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> So I would think it is entirely reasonable if \"git submodule init\n> sub\" that is run in the superproject to initialize \"sub\" writes\n> something in \"sub/.git\" to tell that \"sub\" is used in the context of\n> that particular toplevel superproject and customize its behavour\n> accordingly. Perhaps it may want to add the url.*.insteadOf that is\n> useful for updating the submodule repository when it does \"submodule\n> init\", for example.\n\nOf course, \"copying\" is usually not very desirable, as it invites\none of the copies to go stale. An actual implementation may just\nsay \"the name of submodule the superproject uses this as is 'foo'\".\n\nThat way, if such a configuration exists, Git can first do cd-up to\nthe root of the working tree, go one level up, verify that it is in\na worktree of its superproject, verify that the root of the working\ntree it came from was indeed bound to the submodule called 'foo' and\nthen do the selective/filtered \"config-include\" Peff outlined. That\nwould allow superproject to move submodules around (as opposed to\nrecording \"this submodule is used at this/path of the superproject\"\nor \"the superproject of this submodule is at ../../that/path\"), and\ndoes not penalize repositories that are not used as submodules of\nany superproject (because the \"cd-up, up, verify and include\" won't\nbe done for them). As opposed to \"I am used as a submodule\" bit,\nrecording the name the superproject uses to call the submodule would\nalso serve as a sanity check measure.\n\n"},{"id":"312263","messageId":"CAGZ79ka2S=V1x2fSQq+E-yE0Ao36-4tuTvnD6uXpPXJPLFN3JA@mail.gmail.com","threadId":"45176","inReplyTo":"xmqqo9xvdsji.fsf@gitster.mtv.corp.google.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-22T00:07:36Z","receivedAt":"2017-02-22T00:07:42Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 21, 2017 at 3:37 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> On Tue, Feb 21, 2017 at 3:00 PM, Jeff King wrote:\n>> ...\n>>> I guess one answer is that this is the wrong approach entirely, and the\n>>> right one is something like: submodules should understand that they are\n>>> part of a superproject, and respect some whitelisted set of config from\n>>> the superproject .git/config file.\n>>\n>> This would break one of the core assumptions that submodules\n>> are \"independent\" repos.\n>>\n>> The way of action is a one way street:\n>> * The superproject is aware of the submodule and when you invoke a\n>> command on the superproject, you may mess around with the submodule,\n>> e.g. update/remove it; absorb its git directory.\n>> * The submodule is \"just\" a repository with weird .git link file and a\n>> respective core.worktree setup. Currently it doesn't know if it is\n>> guided by a superproject.\n>\n> While that is a good discipline to follow, I think you need to\n> differenciate the project that is bound as a submodule to a\n> superproject, and a specific instance of a submodule repository,\n> i.e. a clone of such a project.\n>\n> It is true that the Linux kernel project should *NEVER* know your\n> appliance project only because you happen to use it as a component\n> of your appliance that happens to use the kernel as one of its\n> submodules. But that does not mean your copy of the kernel that\n> sits in your recursive checkout of your appliance project should\n> not know anything about your superproject.\n\nOh, I see. For this use case as well as the prompt indicator that\nI mentioned in the previous email, the most basic question is\n* Do we have a superproject? [yes/no]\nThe next level of awareness would be\n* Where is the superproject? [ ]\n\nThese questions may not be interesting for a user (they ought to know\nabout that appliance;) ), but rather for scripted usage, which I think\nhints at the lack of a submodule plumbing command.\n\nCurrently we only have git-submodule that is a helper used to somehow\ncope with submodules. It is used by humans directly and it is listed\nunder \"Main porcelain commands\" in our man page.\n\nProbably we'd also do not want to cram this stuff into the already bloated\nrev-parse (that has --show-toplevel, which has nothing to do with\nparsing revs, but as Jeff put it it is the kitchen sink of Git).\n\n>\n> This is true even without any submodules. The Git project itself\n> does not even care you are Stefan, but you still can and do add\n> [user] name = \"Stefan Beller\" to .git/config of your clone of the\n> Git project. A clone of the project may want to know more than the\n> data project itself keeps track of to describe the context in which\n> the particular clone is being used. And .git/config is a good place\n> to keep such pieces of information.\n\nThis analogy is less clear to me than the kernel& appliance.\nWhen applying it to you (user.name=Junio) that has write powers\nover the blessed repository, the project cares a lot about you. ;)\n\n> So I would think it is entirely reasonable if \"git submodule init\n> sub\" that is run in the superproject to initialize \"sub\" writes\n> something in \"sub/.git\" to tell that \"sub\" is used in the context of\n> that particular toplevel superproject and customize its behavour\n> accordingly. Perhaps it may want to add the url.*.insteadOf that is\n> useful for updating the submodule repository when it does \"submodule\n> init\", for example.\n\nDo we want to invent a special value for url.*.insteadOf to mean\n \"look up in superproject, so I don't have to keep\n a copy that may get stale\" ?\n"},{"id":"312264","messageId":"CAGZ79kZRZz8h8cfrzsOPH+YT7QdF9vQ3C3XBZfGA1SaF+1mEzw@mail.gmail.com","threadId":"45176","inReplyTo":"20170221234037.ga44u3birwd5whab@sigill.intra.peff.net","subject":"Re: url..insteadOf vs. submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-22T00:10:03Z","receivedAt":"2017-02-22T00:10:14Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 21, 2017 at 3:40 PM, Jeff King wrote:\n\n>> > One other caveat: I'm not sure if we do insteadOf recursively, but it\n>> > may be surprising to the child \"git clone\" that we've already applied\n>> > the insteadOf rewriting (especially if the rules are coming from\n>> > ~/.gitconfig and may be applied twice).\n>>\n>> When a rule is having effect twice the rule sounds broken. (the outcome\n>> ought to be sufficiently different from the original?)\n>\n> If you have:\n>\n> url.bar.insteadOf=foo\n> url.baz.insteadOf=bar\n>\n> do we convert \"foo\" to \"baz\"? If so, then I think applying the rules\n> again shouldn't matter. But if we don't, and only do a single level,\n> then having the caller rewrite the URL before it hands it to \"git clone\"\n> means we may end up unexpectedly doing two levels of rewriting.\n>\n\nI see. Thanks for the example. So really what we want is to record the\nunencumbered URL (with no rewriting) and then at run time lookup various\nplaces of url.*.insteadOf (which might change with the git version\nthat you use)\n\nThanks,\nStefan\n"},{"id":"312272","messageId":"xmqqbmtvdj7p.fsf@gitster.mtv.corp.google.com","threadId":"45176","inReplyTo":"CAGZ79ka2S=V1x2fSQq+E-yE0Ao36-4tuTvnD6uXpPXJPLFN3JA@mail.gmail.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-22T02:59:06Z","receivedAt":"2017-02-22T02:59:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n>> This is true even without any submodules. The Git project itself\n>> does not even care you are Stefan, but you still can and do add\n>> [user] name = \"Stefan Beller\" to .git/config of your clone of the\n>> Git project. A clone of the project may want to know more than the\n>> data project itself keeps track of to describe the context in which\n>> the particular clone is being used. And .git/config is a good place\n>> to keep such pieces of information.\n>\n> This analogy is less clear to me than the kernel& appliance.\n> When applying it to you (user.name=Junio) that has write powers\n> over the blessed repository, the project cares a lot about you. ;)\n\nThe name that is recorded in the project history \"Stefan Beller\"\nmatters and the project cares about it, when the commit created in\nthat repository is pulled (or exported and imported via the e-mail\nto \"git am\" route). But what name you have configured in your\nrepository's .git/config, or the presense of your particular\nrepository for that matter, is much less significant (and that\napplies to my primary working area as well). The point is that the\nproject and a particular clone of it are entities at conceptually\ndifferent levels.\n\n>> So I would think it is entirely reasonable if \"git submodule init\n>> sub\" that is run in the superproject to initialize \"sub\" writes\n>> something in \"sub/.git\" to tell that \"sub\" is used in the context of\n>> that particular toplevel superproject and customize its behavour\n>> accordingly. Perhaps it may want to add the url.*.insteadOf that is\n>> useful for updating the submodule repository when it does \"submodule\n>> init\", for example.\n>\n> Do we want to invent a special value for url.*.insteadOf to mean\n> \"look up in superproject, so I don't have to keep\n> a copy that may get stale\" ?\n\nMy gut feeling is that we should do the selective/filtered include\nPeff mentioned when a repository is known to be used as a submodule\nof somebody else.\n"},{"id":"312308","messageId":"E1cgXSe-0007jp-QI@mylo.jdl.com","threadId":"45176","inReplyTo":"xmqqbmtvdj7p.fsf@gitster.mtv.corp.google.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Jon Loeliger","fromEmail":"jdl@jdl.com","sentAt":"2017-02-22T14:00:04Z","receivedAt":"2017-02-22T14:35:52Z","isPatch":false,"sender":{"key":"jdl@jdl.com","avatar":"https://gravatar.com/avatar/75ce9a10b151acd2c28ec4ab2136dba7b2ff1634530bd04b155981a749d08a64?d=mp&s=160"},"body":"So, like, Junio C Hamano said:\n> Stefan Beller writes:\n> \n> > Do we want to invent a special value for url.*.insteadOf to mean\n> > \"look up in superproject, so I don't have to keep\n> > a copy that may get stale\" ?\n> \n> My gut feeling is that we should do the selective/filtered include\n> Peff mentioned when a repository is known to be used as a submodule\n> of somebody else.\n\nDoes the management of these submodue-related config values\nbecome easier if, instead of placing them in .config, we\nplace them in a git/.context file?\n\njdl\n\n"},{"id":"312324","messageId":"xmqqh93mcelv.fsf@gitster.mtv.corp.google.com","threadId":"45176","inReplyTo":"E1cgXSe-0007jp-QI@mylo.jdl.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-22T17:36:12Z","receivedAt":"2017-02-22T17:45:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jon Loeliger writes:\n\n> So, like, Junio C Hamano said:\n>> Stefan Beller writes:\n>> \n>> > Do we want to invent a special value for url.*.insteadOf to mean\n>> > \"look up in superproject, so I don't have to keep\n>> > a copy that may get stale\" ?\n>> \n>> My gut feeling is that we should do the selective/filtered include\n>> Peff mentioned when a repository is known to be used as a submodule\n>> of somebody else.\n>\n> Does the management of these submodue-related config values\n> become easier if, instead of placing them in .config, we\n> place them in a git/.context file?\n\nDo you mean that Git users that use submodules adopt a convention\nwhere a separate file in $GIT_DIR of the toplevel superproject holds\npieces of configuration that are meant to be shared between the\nsuperproject and across all its submodules, and the $GIT_DIR/config\nfile in submodules and the superproject all include that shared one\nvia include.path mechanism?\n\nThat may allow us to do without being responsible for sifting of\nconfiguration variables into safe and unsafe bins.\n\nI dunno.\n"},{"id":"312331","messageId":"20170222185711.2kpzeypptg6deytc@sigill.intra.peff.net","threadId":"45176","inReplyTo":"xmqqh93mcelv.fsf@gitster.mtv.corp.google.com","subject":"Re: url..insteadOf vs. submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-22T18:57:11Z","receivedAt":"2017-02-22T18:57:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 22, 2017 at 09:36:12AM -0800, Junio C Hamano wrote:\n\n> >> My gut feeling is that we should do the selective/filtered include\n> >> Peff mentioned when a repository is known to be used as a submodule\n> >> of somebody else.\n> >\n> > Does the management of these submodue-related config values\n> > become easier if, instead of placing them in .config, we\n> > place them in a git/.context file?\n> \n> Do you mean that Git users that use submodules adopt a convention\n> where a separate file in $GIT_DIR of the toplevel superproject holds\n> pieces of configuration that are meant to be shared between the\n> superproject and across all its submodules, and the $GIT_DIR/config\n> file in submodules and the superproject all include that shared one\n> via include.path mechanism?\n> \n> That may allow us to do without being responsible for sifting of\n> configuration variables into safe and unsafe bins.\n> \n> I dunno.\n\nHmm. I certainly like that we punt on having to decide on the \"should\nthis be shared with submodules\" decision. That makes the end result more\nflexible, and we don't have to get into a never-ending stream of\n\"whitelist this config option\" patches.\n\nMy only concern is that it's not as discoverable. In the situation that\nkicked off this thread, somebody put url.X.insteadOf into their\nsuper-project .git/config, expecting it to work in the submodules. That\n_still_ wouldn't work with this proposal. They'd have to:\n\n 1. Put it in .git/context (or whatever we call it)\n\n 2. Maybe add include.path=context in .git/config if they want the\n config shared with the super-project (or this could be automatic?)\n\nI guess it gives _a_ solution, which is more than we have now, but it\ndoesn't feel very ergonomic.\n\n-Peff\n"},{"id":"312332","messageId":"CAGZ79kYrYtxGyEji0BRoPjBhZK25vvOT5JaS_jhj1_vAre17Yw@mail.gmail.com","threadId":"45176","inReplyTo":"20170222185711.2kpzeypptg6deytc@sigill.intra.peff.net","subject":"Re: url..insteadOf vs. submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-22T19:11:12Z","receivedAt":"2017-02-22T19:14:26Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Feb 22, 2017 at 10:57 AM, Jeff King wrote:\n> On Wed, Feb 22, 2017 at 09:36:12AM -0800, Junio C Hamano wrote:\n>\n>> >> My gut feeling is that we should do the selective/filtered include\n>> >> Peff mentioned when a repository is known to be used as a submodule\n>> >> of somebody else.\n>> >\n>> > Does the management of these submodue-related config values\n>> > become easier if, instead of placing them in .config, we\n>> > place them in a git/.context file?\n>>\n>> Do you mean that Git users that use submodules adopt a convention\n>> where a separate file in $GIT_DIR of the toplevel superproject holds\n>> pieces of configuration that are meant to be shared between the\n>> superproject and across all its submodules, and the $GIT_DIR/config\n>> file in submodules and the superproject all include that shared one\n>> via include.path mechanism?\n>>\n>> That may allow us to do without being responsible for sifting of\n>> configuration variables into safe and unsafe bins.\n>>\n>> I dunno.\n>\n> Hmm. I certainly like that we punt on having to decide on the \"should\n> this be shared with submodules\" decision. That makes the end result more\n> flexible, and we don't have to get into a never-ending stream of\n> \"whitelist this config option\" patches.\n>\n> My only concern is that it's not as discoverable. In the situation that\n> kicked off this thread, somebody put url.X.insteadOf into their\n> super-project .git/config, expecting it to work in the submodules. That\n> _still_ wouldn't work with this proposal. They'd have to:\n>\n> 1. Put it in .git/context (or whatever we call it)\n>\n> 2. Maybe add include.path=context in .git/config if they want the\n> config shared with the super-project (or this could be automatic?)\n>\n> I guess it gives _a_ solution, which is more than we have now, but it\n> doesn't feel very ergonomic.\n\nWell, currently \".git/config\" is the one and only blessed way to configure\na single repo and our documentation and user expectations reflect that.\nOnce git-worktree takes off (which has per working tree configuration files)\nit doesn't feel as obscure anymore to have multiple config files.\n\nThe working trees will share the $GIT_COMMON_DIR/config file for\nall working trees and have its own config file at $GIT_DIR/config.worktree\nin its respective git directories. C.f.\nhttps://public-inbox.org/git/20170110112524.12870-2-pclouds@gmail.com/\n\nSo I could imagine that we just introduce another config file\nconfig.submodules which is source'd by the submodules.\nThen the hard part becomes to decide which config value to put\nin which config file. (We'd still be left to guess where to put some initial\nnew configuration value. config or config.submodules. Any update of a\nvalue can just stay in its respective file. And I don't think we'd want\nto invent a config option that tells us which policy we use where to\nput config options. That sounds just scary.)\n"}]}