{"thread":{"id":"44878","subject":"submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","startedAt":"2017-01-13T18:30:58Z","lastAt":"2017-01-25T07:07:45Z","messageCount":8,"participants":["Stefan Beller","Brian J. Davis"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"309379","messageId":"CAGZ79kZRV7x9B6SHRcHjJweHrjURxWKmN-=Wz_aNw2TPwYh_xw@mail.gmail.com","threadId":"44878","inReplyTo":null,"subject":"submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-01-13T18:30:52Z","receivedAt":"2017-01-13T18:30:58Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This question is about networking; the patch you originally replied to\nwas strictly about local operations in the filesystem, which is quite\na difference, so let's discuss it separately.\n\nOn Fri, Jan 13, 2017 at 9:56 AM, Brian J. Davis wrote:\n>\n> In response to a post at:\n>\n> https://groups.google.com/forum/#!topic/git-users/BVLcKHhSUKo\n>\n> I was asked to post here to explain a potential problem with current modules\n> implementation. Apologies if I am posting in the wrong place... so good bad\n> or otherwise here goes:\n>\n> +-------------------------------\n> With:\n>\n> git push origin branchname\n>\n> or\n>\n> git push server2 branchname\n>\n> I can push or pull from multiple servers so no problem as git supports this.\n>\n> Now the problem with use of submodules\n>\n> submodules are listed:\n>\n> submodule.directory_to_\n> checkout/proj1_dir.url=https://git.someserver1/git/proj1_dir\n> \n\nTechnically it is submodule..url instead of\nsubmodule..url. The name is usually the path initially, and once\nyou move the submodule, only the path changes, the name is supposed to\nbe constant and stay the same.\n\n>\n>\n> but if say I want to pull from some server 2 and do a\n>\n> git submodule update --init --recursive\n\nThat is why the \"git submodule init\" exists at all.\n\n git submodule init\n $EDIT .git/config # change submodule..url to server2\n git submodule update # that uses the adapted url and\n # creates the submodule repository.\n\n # From now on the submodule is on its own.\n cd && git config --list\n # prints an \"origin\" remote and a lot more\n\nFor a better explanation, I started a documentation series, see\nhttps://github.com/gitster/git/commit/e2b51b9df618ceeff7c4ec044e20f5ce9a87241e\n\n(It is not finished, but that is supposed to explain this exact pain\npoint in the\nSTATES section, feel free to point out missing things or what is hard\nto understand)\n\n>\n> what I would get is from someserver1 not someserver2 as there is no \"origin\"\n> support for submodules. Is this correct? If so can origin support be added\n> to submodules?\n\nCan you explain more in detail what you mean by origin support?\n\n> +-------------------------------\n>\n> So above was post from google group. Maybe above is enough to explain the\n> problem that can result, but if not here is a discussion as to why this can\n> be confusing resulting to pushing or pulling from the incorrect server.\n>\n> Lets say projects starts at origin on https://server0.org. Project is then\n> brought in house to server https://indhouse.corp by developer A.\n>\n> Developer A then changes the submodule projects to point to indhouse and\n> changes submodules in super project to point to indhouse so everything is\n> great.\n>\n> Then Developer B clones from indhouse makes changes to submodule1 and\n> submodule1 and pushes them to indhouse. Dev A tells Dev B hey thoes changes\n> are great why don't you push to public server0 so every one can get thoes\n> changes. Dev B then git push origin branch_name on submodule1 and push\n> origin on submodule 2 and superproject. And everything is great ... right?\n>\n> Yes by now those who know git know what dev B forgot or more to the point\n> what git does not support in a \"clean\" way. For those who don't enter the\n> life of dev C\n>\n> So dev C clones from server0 and performs a git submodule update --init\n> --recursive. Now Dev C is on the www and can't see indhouse.corp and ...\n> kerpow... Dev B and Dev C just experienced one of the many SW mines on the\n> battlefield.\n>\n> I ask that git devs first see if I am correct with this assessment as I\n> could be wrong... maybe there is a way of doing this... and if not add a\n> feature to git to handle submodules and multiple origins cleanly.... and yes\n> beating dev B with rubber chicken might be a solution though I am looking\n> for a slightly better option.\n\nYes this is a big point that we want to solve eventually.\n\nWhen devA brought it inhouse, what they meant to do was this:\n\"This superproject is actually from server0, but we want to work on it, which\nmay have submodules diverge from server0 eventually. So if a submodule changed\nyou need to get it from the inhouse server, otherwise fall back to the server0\".\n\nThat way developerB can just make changes to some submodules and when\ndevC clones\nthey get the \"correct\" submodule.\n\nA weak attempt to do this is to use *relative* submodule urls. When\nusing relative urls, and then mirroring the supeproject inhouse, then\nGit will look for the submodules as well inhouse, but there\nis no such \"or if not found look at the original superprojects\norigin\", which means, you have to mirror all submodules.\n\nAnd then about upstreaming changes. If you have a single repo (no\nsubmodules), you have to teach people to run \"git remote add remote\nserver0.org && git push upstream ...\", but that you can do for each\nsubmodule. (This is tedious:/ but maybe ok; some submodules are free\nto sent things upstream whereas others are supersecret that you do not\nwant to push upstream ever.)\n\nSo yeah maybe we want to have more power in the superprojects push operation\n\n (in the superproject) $ git push --recurse-submodules \\\n --only-these-submodules=subA,subB \\\n --submodule-to=upstream-as-configered-in-super-project\n\nThis is a lot of words but for explaining that is ok?\n\nThanks,\nStefan\n"},{"id":"309453","messageId":"ebf6c90e-044f-5538-2325-601d002a81fe@gmail.com","threadId":"44878","inReplyTo":"CAGZ79kZRV7x9B6SHRcHjJweHrjURxWKmN-=Wz_aNw2TPwYh_xw@mail.gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Brian J. Davis","fromEmail":"bitminer@gmail.com","sentAt":"2017-01-15T21:02:41Z","receivedAt":"2017-01-15T20:56:11Z","isPatch":true,"sender":{"key":"bitminer@gmail.com","avatar":null},"body":"\nOn 1/13/2017 12:30 PM, Stefan Beller wrote:\n> This question is about networking; the patch you originally replied to\n> was strictly about local operations in the filesystem, which is quite\n> a difference, so let's discuss it separately.\nYep ok look like you reclassified this and opened new topic which I hope \nI am responding to correctly now.\n> On Fri, Jan 13, 2017 at 9:56 AM, Brian J. Davis wrote:\n>> In response to a post at:\n>>\n>> https://groups.google.com/forum/#!topic/git-users/BVLcKHhSUKo\n>>\n>> I was asked to post here to explain a potential problem with current modules\n>> implementation. Apologies if I am posting in the wrong place... so good bad\n>> or otherwise here goes:\n>>\n>> +-------------------------------\n>> With:\n>>\n>> git push origin branchname\n>>\n>> or\n>>\n>> git push server2 branchname\n>>\n>> I can push or pull from multiple servers so no problem as git supports this.\n>>\n>> Now the problem with use of submodules\n>>\n>> submodules are listed:\n>>\n>> submodule.directory_to_\n>> checkout/proj1_dir.url=https://git.someserver1/git/proj1_dir\n>> \n> Technically it is submodule..url instead of\n> submodule..url. The name is usually the path initially, and once\n> you move the submodule, only the path changes, the name is supposed to\n> be constant and stay the same.\nI am not certain what is meant by this. All I know is I can use my \n\"directory_to_checkout\" above to place in tree relative from root the \nproject any where in the tree not already tracked by git. You state \nname instead of path, but it allows path correct? Either that or I have \ngone off reservation with my use of git for years now. Maybe this is a \ndeviation from how it is documented/should work and how it actually \nworks? It works great how I use it.\n>>\n>> but if say I want to pull from some server 2 and do a\n>>\n>> git submodule update --init --recursive\n> That is why the \"git submodule init\" exists at all.\n>\n> git submodule init\n> $EDIT .git/config # change submodule..url to server2\n> git submodule update # that uses the adapted url and\n> # creates the submodule repository.\n>\n> # From now on the submodule is on its own.\n> cd && git config --list\n> # prints an \"origin\" remote and a lot more\n>\n> For a better explanation, I started a documentation series, see\n> https://github.com/gitster/git/commit/e2b51b9df618ceeff7c4ec044e20f5ce9a87241e\n>\n> (It is not finished, but that is supposed to explain this exact pain\n> point in the\n> STATES section, feel free to point out missing things or what is hard\n> to understand)\nI am not sure I got much out of the STATES section regarding my problem.\n>> what I would get is from someserver1 not someserver2 as there is no \"origin\"\n>> support for submodules. Is this correct? If so can origin support be added\n>> to submodules?\n> Can you explain more in detail what you mean by origin support?\nYes so when we do a:\n\ngit push origin master\n\norigin is of course the Remote (Remotes \nhttps://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes)\n\nSo I best use terminology \"Remotes\" support. Git push supports remotes, \nbut git submodules does not. The basic idea is this:\n\nIf Git allowed multiple submodule \n(https://git-scm.com/book/en/v2/Git-Tools-Submodules) Remotes to be \nspecified say as an example:\n\ngit submodule add [remote] [url]\n\ngit submodule add origin https://github.com/chaconinc/DbConnector\ngit submodule add indhouse https://indhouse .corp/chaconinc/DbConnector\n\nWhere:\n\n[submodule \"DbConnector\"]\n path = DbConnector\n url = https://github.com/chaconinc/DbConnector\n\nCould then change to:\n\n[submodule \"DbConnector\"]\n path = DbConnector\n remote.origin.url = https://github.com/chaconinc/DbConnector\n remote.origin.url = https://indhouse .corp/chaconinc/DbConnector\n\n\nThen it should be possible to get:\n\ngit submodules update --init --recursive\n\nTo support\n\ngit submodules update [remote] --init --recursive\n\nAnd thus allow\n\ngit submodules update origin --init --recursive\n\ngit submodules update indhouse --init --recursive\n\n>\n>> +-------------------------------\n>>\n>> So above was post from google group. Maybe above is enough to explain the\n>> problem that can result, but if not here is a discussion as to why this can\n>> be confusing resulting to pushing or pulling from the incorrect server.\n>>\n>> Lets say projects starts at origin on https://server0.org. Project is then\n>> brought in house to server https://indhouse.corp by developer A.\n>>\n>> Developer A then changes the submodule projects to point to indhouse and\n>> changes submodules in super project to point to indhouse so everything is\n>> great.\n>>\n>> Then Developer B clones from indhouse makes changes to submodule1 and\n>> submodule1 and pushes them to indhouse. Dev A tells Dev B hey thoes changes\n>> are great why don't you push to public server0 so every one can get thoes\n>> changes. Dev B then git push origin branch_name on submodule1 and push\n>> origin on submodule 2 and superproject. And everything is great ... right?\n>>\n>> Yes by now those who know git know what dev B forgot or more to the point\n>> what git does not support in a \"clean\" way. For those who don't enter the\n>> life of dev C\n>>\n>> So dev C clones from server0 and performs a git submodule update --init\n>> --recursive. Now Dev C is on the www and can't see indhouse.corp and ...\n>> kerpow... Dev B and Dev C just experienced one of the many SW mines on the\n>> battlefield.\n>>\n>> I ask that git devs first see if I am correct with this assessment as I\n>> could be wrong... maybe there is a way of doing this... and if not add a\n>> feature to git to handle submodules and multiple origins cleanly.... and yes\n>> beating dev B with rubber chicken might be a solution though I am looking\n>> for a slightly better option.\n> Yes this is a big point that we want to solve eventually.\n>\n> When devA brought it inhouse, what they meant to do was this:\n> \"This superproject is actually from server0, but we want to work on it, which\n> may have submodules diverge from server0 eventually. So if a submodule changed\n> you need to get it from the inhouse server, otherwise fall back to the server0\".\n>\n> That way developerB can just make changes to some submodules and when\n> devC clones\n> they get the \"correct\" submodule.\n>\n> A weak attempt to do this is to use *relative* submodule urls. When\n> using relative urls, and then mirroring the supeproject inhouse, then\n> Git will look for the submodules as well inhouse, but there\n> is no such \"or if not found look at the original superprojects\n> origin\", which means, you have to mirror all submodules.\nYes I don't see how *relative* urls are a good solution.\n> And then about upstreaming changes. If you have a single repo (no\n> submodules), you have to teach people to run \"git remote add remote\n> server0.org && git push upstream ...\", but that you can do for each\n> submodule. (This is tedious:/ but maybe ok; some submodules are free\n> to sent things upstream whereas others are supersecret that you do not\n> want to push upstream ever.)\nSo right yes there are ways and means this can fail and not be \nultimately what the developer wants after all we all need to be aware of \nthe repercussion of the commands typed at the terminal. My goal here is \nI notice a process that Git does not seem to handle well for a \ndistributed revision control system and thought it could use some \nimprovements. To your point on \"supersecret\" .... when pushing to \norigin git could always be allowed to notify the developer of multi \nremotes before pushing with a \"freak-out\" flag set by Dev A where git \nwould then ask:\ngit: are you sure [y/(n)] y\ngit: are you sure your sure [y/(n)] Y\ngit :you seem certain, but you might not be aware of the true \nconsequences of your actions... are you sure you are sure that your are \nsure [Y/(N)] Y\ngit: ..... Ok committing... it's your fault if this goes pear shaped \nsideways.\n\n> So yeah maybe we want to have more power in the superprojects push operation\n>\n> (in the superproject) $ git push --recurse-submodules \\\n> --only-these-submodules=subA,subB \\\n> --submodule-to=upstream-as-configered-in-super-project\nSure ... Yes... plus \"Remotes\" support.\n\n git push --recurse-submodules \\\n --only-these-submodules=subA,subB \\\n --submodule-to=upstream-as-configered-in-super-project\n\nAllow:\n\n git push [remote] --recurse-submodules \\\n --only-these-submodules=subA,subB \\\n --submodule-to=upstream-as-configered-in-super-project\n\n\n git push indhouse --recurse-submodules \\\n --only-these-submodules=subA,subB \\\n --submodule-to=upstream-as-configered-in-super-project\nand\n\n git push origin --recurse-submodules \\\n --only-these-submodules=subA,subB \\\n --submodule-to=upstream-as-configered-in-super-project\n\n> This is a lot of words but for explaining that is ok?\n>\n> Thanks,\n> Stefan\nYep got it thanks.\n"},{"id":"309535","messageId":"CAGZ79kaf0BhbnJGhkjT_Ys44y4c4AaxV8U_ydWp4bbFMkGRcsQ@mail.gmail.com","threadId":"44878","inReplyTo":"ebf6c90e-044f-5538-2325-601d002a81fe@gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-01-17T18:43:17Z","receivedAt":"2017-01-17T18:50:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Jan 15, 2017 at 1:02 PM, Brian J. Davis wrote:\n\n>>\n>> Technically it is submodule..url instead of\n>> submodule..url. The name is usually the path initially, and once\n>> you move the submodule, only the path changes, the name is supposed to\n>> be constant and stay the same.\n>\n> I am not certain what is meant by this. All I know is I can use my\n> \"directory_to_checkout\" above to place in tree relative from root the\n> project any where in the tree not already tracked by git. You state name\n> instead of path, but it allows path correct? Either that or I have gone off\n> reservation with my use of git for years now. Maybe this is a deviation from\n> how it is documented/should work and how it actually works? It works great\n> how I use it.\n\nYes name can equal the path (and usually does). This is a minor detail\nthat is only relevant for renaming submodules, so ... maybe let's not\nfocus on it too much. :)\n\n>>>\n>>>\n>>> but if say I want to pull from some server 2 and do a\n>>>\n>>> git submodule update --init --recursive\n>>\n>> That is why the \"git submodule init\" exists at all.\n>>\n>> git submodule init\n>> $EDIT .git/config # change submodule..url to server2\n>> git submodule update # that uses the adapted url and\n>> # creates the submodule repository.\n>>\n>> # From now on the submodule is on its own.\n>> cd && git config --list\n>> # prints an \"origin\" remote and a lot more\n>>\n>> For a better explanation, I started a documentation series, see\n>>\n>> https://github.com/gitster/git/commit/e2b51b9df618ceeff7c4ec044e20f5ce9a87241e\n>>\n>> (It is not finished, but that is supposed to explain this exact pain\n>> point in the\n>> STATES section, feel free to point out missing things or what is hard\n>> to understand)\n>\n> I am not sure I got much out of the STATES section regarding my problem.\n\nYour original problem as far as I understand is this:\n\n You have a project with submodules.\n The submodules are described in the .gitmodules file.\n But the URL is pointing to an undesired location.\n You want to rewrite the URLs before actually cloning the submodules.\n\nAnd to solve this problem we need to perform multiple steps:\n\n # --no is the default, just for clarity here:\n git clone --no-recurse-submodules\n # The submodules are now in the *uninitialized* state\n\n git submodule init\n # the submodules are in the initialized state\n\n git submodule update\n # submodules are populated, i.e. cloned from\n # the configured URLs and put into the working tree at\n # the appropriate path.\n\nBetween the init and the update step you can modify the URLs.\nThese commands are just a repetition from the first email, but the\ngit commands can be viewed as moving from one state to another\nfor submodules; submodules itself can be seen as a state machine\naccording to that proposed documentation. Maybe such a state machine\nmakes it easier to understand for some people.\n\n>>> what I would get is from someserver1 not someserver2 as there is no\n>>> \"origin\"\n>>> support for submodules. Is this correct? If so can origin support be\n>>> added\n>>> to submodules?\n>>\n>> Can you explain more in detail what you mean by origin support?\n>\n> Yes so when we do a:\n>\n> git push origin master\n>\n> origin is of course the Remote (Remotes\n> https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes)\n>\n> So I best use terminology \"Remotes\" support. Git push supports remotes, but\n> git submodules does not. The basic idea is this:\n>\n> If Git allowed multiple submodule\n> (https://git-scm.com/book/en/v2/Git-Tools-Submodules) Remotes to be\n> specified say as an example:\n>\n> git submodule add [remote] [url]\n>\n> git submodule add origin https://github.com/chaconinc/DbConnector\n> git submodule add indhouse https://indhouse .corp/chaconinc/DbConnector\n>\n> Where:\n>\n> [submodule \"DbConnector\"]\n> path = DbConnector\n> url = https://github.com/chaconinc/DbConnector\n>\n> Could then change to:\n>\n> [submodule \"DbConnector\"]\n> path = DbConnector\n> remote.origin.url = https://github.com/chaconinc/DbConnector\n> remote.origin.url = https://indhouse .corp/chaconinc/DbConnector\n\nhere I assume there is a typo and the second remote.origin.url should be\nremote.inhouse.url ?\n\n>\n>\n> Then it should be possible to get:\n>\n> git submodules update --init --recursive\n\nwhich would setup the submodule with both\n\n[remote \"origin\"]\n url = https://github.com/..\n[remote \"inhouse\"]\n url = https://inhouse.corp/..\n\nBut where do we clone it from?\n(Or do we just do a \"git init\" on that submodule and fetch\nfrom both remotes? in which order?)\n\n>\n> To support\n>\n> git submodules update [remote] --init --recursive\n\nThis would just clone/fetch from the specified remote?\nIf implementing this, we may run into a collision with the\nspecified submodules, what if a submodule is at\npath \"origin\" ?\n\nDoes \"git submodule update origin --init --recursive\"\nthen mean to update the single \"origin\" submodule or\nall submodules from their origin remote?\n\n>\n> And thus allow\n>\n> git submodules update origin --init --recursive\n>\n> git submodules update indhouse --init --recursive\n\nunderstood. I like the idea of being able to specify\nmultiple remotes from the superproject..\n"},{"id":"309802","messageId":"e6c5567a-2032-c598-97c8-08518f86b611@gmail.com","threadId":"44878","inReplyTo":"CAGZ79kaf0BhbnJGhkjT_Ys44y4c4AaxV8U_ydWp4bbFMkGRcsQ@mail.gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Brian J. Davis","fromEmail":"bitminer@gmail.com","sentAt":"2017-01-19T23:11:53Z","receivedAt":"2017-01-19T23:12:55Z","isPatch":true,"sender":{"key":"bitminer@gmail.com","avatar":null},"body":"\nOn 1/17/2017 12:43 PM, Stefan Beller wrote:\n> On Sun, Jan 15, 2017 at 1:02 PM, Brian J. Davis wrote:\n>\n>>> Technically it is submodule..url instead of\n>>> submodule..url. The name is usually the path initially, and once\n>>> you move the submodule, only the path changes, the name is supposed to\n>>> be constant and stay the same.\n>> I am not certain what is meant by this. All I know is I can use my\n>> \"directory_to_checkout\" above to place in tree relative from root the\n>> project any where in the tree not already tracked by git. You state name\n>> instead of path, but it allows path correct? Either that or I have gone off\n>> reservation with my use of git for years now. Maybe this is a deviation from\n>> how it is documented/should work and how it actually works? It works great\n>> how I use it.\n> Yes name can equal the path (and usually does). This is a minor detail\n> that is only relevant for renaming submodules, so ... maybe let's not\n> focus on it too much. :)\n>\n>>>>\n>>>> but if say I want to pull from some server 2 and do a\n>>>>\n>>>> git submodule update --init --recursive\n>>> That is why the \"git submodule init\" exists at all.\n>>>\n>>> git submodule init\n>>> $EDIT .git/config # change submodule..url to server2\n>>> git submodule update # that uses the adapted url and\n>>> # creates the submodule repository.\n>>>\n>>> # From now on the submodule is on its own.\n>>> cd && git config --list\n>>> # prints an \"origin\" remote and a lot more\n>>>\n>>> For a better explanation, I started a documentation series, see\n>>>\n>>> https://github.com/gitster/git/commit/e2b51b9df618ceeff7c4ec044e20f5ce9a87241e\n>>>\n>>> (It is not finished, but that is supposed to explain this exact pain\n>>> point in the\n>>> STATES section, feel free to point out missing things or what is hard\n>>> to understand)\n>> I am not sure I got much out of the STATES section regarding my problem.\n> Your original problem as far as I understand is this:\n>\n> You have a project with submodules.\n> The submodules are described in the .gitmodules file.\n> But the URL is pointing to an undesired location.\n> You want to rewrite the URLs before actually cloning the submodules.\n>\n> And to solve this problem we need to perform multiple steps:\n>\n> # --no is the default, just for clarity here:\n> git clone --no-recurse-submodules\n> # The submodules are now in the *uninitialized* state\n>\n> git submodule init\n> # the submodules are in the initialized state\n>\n> git submodule update\n> # submodules are populated, i.e. cloned from\n> # the configured URLs and put into the working tree at\n> # the appropriate path.\n>\n> Between the init and the update step you can modify the URLs.\n> These commands are just a repetition from the first email, but the\n> git commands can be viewed as moving from one state to another\n> for submodules; submodules itself can be seen as a state machine\n> according to that proposed documentation. Maybe such a state machine\n> makes it easier to understand for some people.\n\n\"Between the init and the update step you can modify the URLs.\" Yes I can and have to... wish it was not this way.\n\n>>>> what I would get is from someserver1 not someserver2 as there is no\n>>>> \"origin\"\n>>>> support for submodules. Is this correct? If so can origin support be\n>>>> added\n>>>> to submodules?\n>>> Can you explain more in detail what you mean by origin support?\n>> Yes so when we do a:\n>>\n>> git push origin master\n>>\n>> origin is of course the Remote (Remotes\n>> https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes)\n>>\n>> So I best use terminology \"Remotes\" support. Git push supports remotes, but\n>> git submodules does not. The basic idea is this:\n>>\n>> If Git allowed multiple submodule\n>> (https://git-scm.com/book/en/v2/Git-Tools-Submodules) Remotes to be\n>> specified say as an example:\n>>\n>> git submodule add [remote] [url]\n>>\n>> git submodule add origin https://github.com/chaconinc/DbConnector\n>> git submodule add indhouse https://indhouse .corp/chaconinc/DbConnector\n>>\n>> Where:\n>>\n>> [submodule \"DbConnector\"]\n>> path = DbConnector\n>> url = https://github.com/chaconinc/DbConnector\n>>\n>> Could then change to:\n>>\n>> [submodule \"DbConnector\"]\n>> path = DbConnector\n>> remote.origin.url = https://github.com/chaconinc/DbConnector\n>> remote.origin.url = https://indhouse .corp/chaconinc/DbConnector\n> here I assume there is a typo and the second remote.origin.url should be\n> remote.inhouse.url ?\nyes second should have read remote.inhouse.url:\n\n[submodule \"DbConnector\"]\n path = DbConnector\n remote.origin.url = https://github.com/chaconinc/DbConnector\n remote.inhouse.url = https://indhouse.corp/chaconinc/DbConnector\n\n>>\n>> Then it should be possible to get:\n>>\n>> git submodules update --init --recursive\n> which would setup the submodule with both\n>\n> [remote \"origin\"]\n> url = https://github.com/..\n> [remote \"inhouse\"]\n> url = https://inhouse.corp/..\n>\n> But where do we clone it from?\n> (Or do we just do a \"git init\" on that submodule and fetch\n> from both remotes? in which order?)\norigin by default and inhouse if specified. There is already a implied \ndefault (origin). The idea was not to do both but rather what is \nspecified. Origin and inhouse are just names for remotes. If one wanted \na \"--all-remotes\" could pull from everywhere in the Ether if feature was \nto be implemented.\n>> To support\n>>\n>> git submodules update [remote] --init --recursive\n> This would just clone/fetch from the specified remote?\n> If implementing this, we may run into a collision with the\n> specified submodules, what if a submodule is at\n> path \"origin\" ?\n>\n> Does \"git submodule update origin --init --recursive\"\n> then mean to update the single \"origin\" submodule or\n> all submodules from their origin remote?\nYes. That is what I would think. It does this already by default. It's \nnot as though submodules were/are all that well thought out to begin \nwith in git IMO.\n>> And thus allow\n>>\n>> git submodules update origin --init --recursive\n>>\n>> git submodules update indhouse --init --recursive\n> understood. I like the idea of being able to specify\n> multiple remotes from the superproject..\n\nYes so do I! It *could* be better than the current offering which is \ndefaulted origin.\n\n"},{"id":"309813","messageId":"CAGZ79kaS7zt3DKrRuqzzODc1HHEP-xd-8HBC0JA-HvmqAJOZfw@mail.gmail.com","threadId":"44878","inReplyTo":"e6c5567a-2032-c598-97c8-08518f86b611@gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-01-20T01:22:12Z","receivedAt":"2017-01-20T01:22:39Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> Between the init and the update step you can modify the URLs.\n>> These commands are just a repetition from the first email, but the\n>> git commands can be viewed as moving from one state to another\n>> for submodules; submodules itself can be seen as a state machine\n>> according to that proposed documentation. Maybe such a state machine\n>> makes it easier to understand for some people.\n>\n>\n> \"Between the init and the update step you can modify the URLs.\" Yes I can\n> and have to... wish it was not this way.\n\nSo how would yo u rather want to do it?\nlook at the .gitmodules file beforehand and then run a \"submodule update\" ?\nOr a thing like\n\n git -c url.https://internal.insteadOf git://github.com/ \\\n -c submodule.record-rewritten-urls submodule update\n\n(no need for init there as theoretically there is not\nneed for such an intermediate step)\n\n\n>> [remote \"origin\"]\n>> url = https://github.com/..\n>> [remote \"inhouse\"]\n>> url = https://inhouse.corp/..\n>>\n>> But where do we clone it from?\n>> (Or do we just do a \"git init\" on that submodule and fetch\n>> from both remotes? in which order?)\n>\n> origin by default and inhouse if specified. There is already a implied\n> default (origin). The idea was not to do both but rather what is specified.\n> Origin and inhouse are just names for remotes. If one wanted a\n> \"--all-remotes\" could pull from everywhere in the Ether if feature was to be\n> implemented.\n\nHow is origin implied to be the default?\nShould there be an order (e.g. if you cannot find it at inhouse get it\nfrom github,\nif they are down, get it from kernel.org)\n"},{"id":"309899","messageId":"04fe8035-dbf0-83d2-c465-f746b99ce609@gmail.com","threadId":"44878","inReplyTo":"CAGZ79kaS7zt3DKrRuqzzODc1HHEP-xd-8HBC0JA-HvmqAJOZfw@mail.gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Brian J. Davis","fromEmail":"bitminer@gmail.com","sentAt":"2017-01-21T15:53:59Z","receivedAt":"2017-01-21T15:47:16Z","isPatch":true,"sender":{"key":"bitminer@gmail.com","avatar":null},"body":"\nOn 1/19/2017 7:22 PM, Stefan Beller wrote:\n>>> Between the init and the update step you can modify the URLs.\n>>> These commands are just a repetition from the first email, but the\n>>> git commands can be viewed as moving from one state to another\n>>> for submodules; submodules itself can be seen as a state machine\n>>> according to that proposed documentation. Maybe such a state machine\n>>> makes it easier to understand for some people.\n>>\n>> \"Between the init and the update step you can modify the URLs.\" Yes I can\n>> and have to... wish it was not this way.\n> So how would yo u rather want to do it?\n> look at the .gitmodules file beforehand and then run a \"submodule update\" ?\n> Or a thing like\n>\n> git -c url.https://internal.insteadOf git://github.com/ \\\n> -c submodule.record-rewritten-urls submodule update\n>\n> (no need for init there as theoretically there is not\n> need for such an intermediate step)\n>\n\"Yes please and thank you\" ... both.\n\nMy thought was to simply allow addition to .gitmodules. If I understand \ncorrectly you are proposing, to override these at the command line and \npossibly rewrite them on submodule update, but maybe not save or add to \n.gitmodules. I would then propose both.\n1) allow user to add to .gitmodules for those who do not care that \n\"outsiders\" know the internal dev server\nand\n2) allow to rewrite while not saving to .gitmodules on fresh clone and \nsubmodule update for thoes that do not want ousiders to known internal \ndev server.\nand possibly:\n3) allow at command line to add remote to .gitmodules on submodule \ncommands (note add optoin in -c = pair)\n\n.gitmodules before:\n\n[submodule \"subprojects/wrangler\"]\n path = subprojects/wrangler\n url = git://github.com/\n\nThen your adapted command:\n\ngit -c url.https://internal.insteadOf git://github.com/ \\\n -c submodule.record-rewritten-urls=add,internal --add submodule update\n\nwould produce\n\n[submodule \"subprojects/projname\"]\n path = subprojects/projname\n remote.origin.url = git://github.com/\n remote.internal.url =https://internal.insteadOf\n\nOr similar support.\n\n>>> [remote \"origin\"]\n>>> url = https://github.com/..\n>>> [remote \"inhouse\"]\n>>> url = https://inhouse.corp/..\n>>>\n>>> But where do we clone it from?\n>>> (Or do we just do a \"git init\" on that submodule and fetch\n>>> from both remotes? in which order?)\n>> origin by default and inhouse if specified. There is already a implied\n>> default (origin). The idea was not to do both but rather what is specified.\n>> Origin and inhouse are just names for remotes. If one wanted a\n>> \"--all-remotes\" could pull from everywhere in the Ether if feature was to be\n>> implemented.\n> How is origin implied to be the default?\n> Should there be an order (e.g. if you cannot find it at inhouse get it\n> from github,\n> if they are down, get it from kernel.org)\nAs it is in the Highlander series... \"there can be only one\" (remote). \nSo that is what I mean by origin. The only remote allowed is the \n\"origin\" unless changed by the user... but there can still only be one \n*currently*. Though I see your point as it is not labeled \"origin\". It \nis not labeled at all. Apologies for confusion there.\n\n\n\n\n"},{"id":"310102","messageId":"CAGZ79kZzekMc=pUN3Q2+zmaSNo-BLJ6TxaQPN8ik1B2+ACy9dw@mail.gmail.com","threadId":"44878","inReplyTo":"04fe8035-dbf0-83d2-c465-f746b99ce609@gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-01-24T18:49:36Z","receivedAt":"2017-01-24T18:49:43Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sat, Jan 21, 2017 at 7:53 AM, Brian J. Davis wrote:\n>\n> On 1/19/2017 7:22 PM, Stefan Beller wrote:\n>>>>\n>>>> Between the init and the update step you can modify the URLs.\n>>>> These commands are just a repetition from the first email, but the\n>>>> git commands can be viewed as moving from one state to another\n>>>> for submodules; submodules itself can be seen as a state machine\n>>>> according to that proposed documentation. Maybe such a state machine\n>>>> makes it easier to understand for some people.\n>>>\n>>>\n>>> \"Between the init and the update step you can modify the URLs.\" Yes I\n>>> can\n>>> and have to... wish it was not this way.\n>>\n>> So how would yo u rather want to do it?\n>> look at the .gitmodules file beforehand and then run a \"submodule update\"\n>> ?\n>> Or a thing like\n>>\n>> git -c url.https://internal.insteadOf git://github.com/ \\\n>> -c submodule.record-rewritten-urls submodule update\n>>\n>> (no need for init there as theoretically there is not\n>> need for such an intermediate step)\n>>\n> \"Yes please and thank you\" ... both.\n>\n> My thought was to simply allow addition to .gitmodules. If I understand\n> correctly you are proposing, to override these at the command line and\n> possibly rewrite them on submodule update, but maybe not save or add to\n> .gitmodules. I would then propose both.\n> 1) allow user to add to .gitmodules for those who do not care that\n> \"outsiders\" know the internal dev server\n> and\n> 2) allow to rewrite while not saving to .gitmodules on fresh clone and\n> submodule update for thoes that do not want ousiders to known internal dev\n> server.\n> and possibly:\n> 3) allow at command line to add remote to .gitmodules on submodule commands\n> (note add optoin in -c = pair)\n>\n> .gitmodules before:\n>\n> [submodule \"subprojects/wrangler\"]\n> path = subprojects/wrangler\n> url = git://github.com/\n>\n> Then your adapted command:\n>\n> git -c url.https://internal.insteadOf git://github.com/ \\\n> -c submodule.record-rewritten-urls=add,internal --add submodule\n> update\n>\n> would produce\n>\n> [submodule \"subprojects/projname\"]\n> path = subprojects/projname\n> remote.origin.url = git://github.com/\n> remote.internal.url =https://internal.insteadOf\n>\n> Or similar support.\n\nI think this was avoided until now as it would rewrite/add history.\nSo what if you want ot \"just mirror\" a large project with a lot\nof submodules? You would want to do that without touching\nthe history, hence we'd need to do such a configuration in a separate\nplace. IIRC there was a proposal to have a ref e.g.\n\"refs/submodule/config\", that can overwrite/extend the .gitmodules\nfile with your own configuration. It is a ref, such that mirroring would\nwork, but not part of the main history, such that yoiu can still change it.\n\nI think to get it right we need to enable a workflow that allows easy\n\"multi-step\" mirroring, e.g. A (source of truth) can be mirrored to B,\nwho can overlay the .gitmodules to point to server-B, which then can\nbe mirrored by C, who can have its own serverC.\n\nWhen C forgot to configure all the submodules, it should fall back to\nserverB as that was closest, and if that is unconfigured it should\nfallback to A IMO.\n\n\n\n>\n>>>> [remote \"origin\"]\n>>>> url = https://github.com/..\n>>>> [remote \"inhouse\"]\n>>>> url = https://inhouse.corp/..\n>>>>\n>>>> But where do we clone it from?\n>>>> (Or do we just do a \"git init\" on that submodule and fetch\n>>>> from both remotes? in which order?)\n>>>\n>>> origin by default and inhouse if specified. There is already a implied\n>>> default (origin). The idea was not to do both but rather what is\n>>> specified.\n>>> Origin and inhouse are just names for remotes. If one wanted a\n>>> \"--all-remotes\" could pull from everywhere in the Ether if feature was to\n>>> be\n>>> implemented.\n>>\n>> How is origin implied to be the default?\n>> Should there be an order (e.g. if you cannot find it at inhouse get it\n>> from github,\n>> if they are down, get it from kernel.org)\n>\n> As it is in the Highlander series... \"there can be only one\" (remote). So\n> that is what I mean by origin. The only remote allowed is the \"origin\"\n> unless changed by the user... but there can still only be one *currently*.\n> Though I see your point as it is not labeled \"origin\". It is not labeled at\n> all. Apologies for confusion there.\n\n\"origin\" is just a common name for a remote, like \"master\" is a common name\nfor branches. There is nothing inherently special about these except for\ntheir automatic setup after cloning/initializing a repo.\n\nSo you can delete the master branch (which I did in a project\nthat I am not the authoritative maintainer of; I only have feature\nbranches), and\nthe repository just works fine.\n"},{"id":"310142","messageId":"fe03b88b-b3b2-0906-3912-2e123a4d8af9@gmail.com","threadId":"44878","inReplyTo":"CAGZ79kZzekMc=pUN3Q2+zmaSNo-BLJ6TxaQPN8ik1B2+ACy9dw@mail.gmail.com","subject":"Re: submodule network operations [WAS: Re: [RFC/PATCH 0/4] working tree operations: support superprefix]","fromName":"Brian J. Davis","fromEmail":"bitminer@gmail.com","sentAt":"2017-01-25T07:14:34Z","receivedAt":"2017-01-25T07:07:45Z","isPatch":true,"sender":{"key":"bitminer@gmail.com","avatar":null},"body":"\n\nOn 1/24/2017 12:49 PM, Stefan Beller wrote:\n> On Sat, Jan 21, 2017 at 7:53 AM, Brian J. Davis wrote:\n>> On 1/19/2017 7:22 PM, Stefan Beller wrote:\n>>>>> Between the init and the update step you can modify the URLs.\n>>>>> These commands are just a repetition from the first email, but the\n>>>>> git commands can be viewed as moving from one state to another\n>>>>> for submodules; submodules itself can be seen as a state machine\n>>>>> according to that proposed documentation. Maybe such a state machine\n>>>>> makes it easier to understand for some people.\n>>>>\n>>>> \"Between the init and the update step you can modify the URLs.\" Yes I\n>>>> can\n>>>> and have to... wish it was not this way.\n>>> So how would yo u rather want to do it?\n>>> look at the .gitmodules file beforehand and then run a \"submodule update\"\n>>> ?\n>>> Or a thing like\n>>>\n>>> git -c url.https://internal.insteadOf git://github.com/ \\\n>>> -c submodule.record-rewritten-urls submodule update\n>>>\n>>> (no need for init there as theoretically there is not\n>>> need for such an intermediate step)\n>>>\n>> \"Yes please and thank you\" ... both.\n>>\n>> My thought was to simply allow addition to .gitmodules. If I understand\n>> correctly you are proposing, to override these at the command line and\n>> possibly rewrite them on submodule update, but maybe not save or add to\n>> .gitmodules. I would then propose both.\n>> 1) allow user to add to .gitmodules for those who do not care that\n>> \"outsiders\" know the internal dev server\n>> and\n>> 2) allow to rewrite while not saving to .gitmodules on fresh clone and\n>> submodule update for thoes that do not want ousiders to known internal dev\n>> server.\n>> and possibly:\n>> 3) allow at command line to add remote to .gitmodules on submodule commands\n>> (note add optoin in -c = pair)\n>>\n>> .gitmodules before:\n>>\n>> [submodule \"subprojects/wrangler\"]\n>> path = subprojects/wrangler\n>> url = git://github.com/\n>>\n>> Then your adapted command:\n>>\n>> git -c url.https://internal.insteadOf git://github.com/ \\\n>> -c submodule.record-rewritten-urls=add,internal --add submodule\n>> update\n>>\n>> would produce\n>>\n>> [submodule \"subprojects/projname\"]\n>> path = subprojects/projname\n>> remote.origin.url = git://github.com/\n>> remote.internal.url =https://internal.insteadOf\n>>\n>> Or similar support.\n> I think this was avoided until now as it would rewrite/add history.\n> So what if you want ot \"just mirror\" a large project with a lot\n> of submodules? You would want to do that without touching\n> the history, hence we'd need to do such a configuration in a separate\n> place. IIRC there was a proposal to have a ref e.g.\n> \"refs/submodule/config\", that can overwrite/extend the .gitmodules\n> file with your own configuration. It is a ref, such that mirroring would\n> work, but not part of the main history, such that yoiu can still change it.\nHow would this handle the hashes for the submodules tracked in the super \nproject. This gets to my earlier statement that git submodules is not \nreally that well thought out. When/if /refs/submodule/config \noverwrites/extends .gitmoduels what does git checkout for a hash code \n(version) for the subproject? Maybe there are some simple ways of doing \nthis, but it really seems to me that git submodules needs an overhaul to \nsupport multiple remotes. Would the refs/submodule/config also include \nthe versions hashes for the submodule for both remotes? When switching \nbetween remotes would the git submodule update pull the correct hash \nversion for the subproject and pull the correct version of the subproject?\n> I think to get it right we need to enable a workflow that allows easy\n> \"multi-step\" mirroring, e.g. A (source of truth) can be mirrored to B,\n> who can overlay the .gitmodules to point to server-B, which then can\n> be mirrored by C, who can have its own serverC.\n>\n> When C forgot to configure all the submodules, it should fall back to\n> serverB as that was closest, and if that is unconfigured it should\n> fallback to A IMO.\n>\n>\n>\n>>>>> [remote \"origin\"]\n>>>>> url = https://github.com/..\n>>>>> [remote \"inhouse\"]\n>>>>> url = https://inhouse.corp/..\n>>>>>\n>>>>> But where do we clone it from?\n>>>>> (Or do we just do a \"git init\" on that submodule and fetch\n>>>>> from both remotes? in which order?)\n>>>> origin by default and inhouse if specified. There is already a implied\n>>>> default (origin). The idea was not to do both but rather what is\n>>>> specified.\n>>>> Origin and inhouse are just names for remotes. If one wanted a\n>>>> \"--all-remotes\" could pull from everywhere in the Ether if feature was to\n>>>> be\n>>>> implemented.\n>>> How is origin implied to be the default?\n>>> Should there be an order (e.g. if you cannot find it at inhouse get it\n>>> from github,\n>>> if they are down, get it from kernel.org)\n>> As it is in the Highlander series... \"there can be only one\" (remote). So\n>> that is what I mean by origin. The only remote allowed is the \"origin\"\n>> unless changed by the user... but there can still only be one *currently*.\n>> Though I see your point as it is not labeled \"origin\". It is not labeled at\n>> all. Apologies for confusion there.\n> \"origin\" is just a common name for a remote, like \"master\" is a common name\n> for branches. There is nothing inherently special about these except for\n> their automatic setup after cloning/initializing a repo.\nOrigin is the remote from where all was derived and yes it can certainly \nbe renamed or deleted (provided another remote name exists) in git. \nSadly \"master\" is a term used in git which always implies another term. \nI thought term \"master\" was long since killed in the computer world with \nthe USB 1.1 to 2.0 debate, ... but here it still lives in git. What is \nwrong with the term trunk? Trees don't have masters... they have \ntrunks... but i digress on the term \"master\". I will try and get my git \nterms right even if IMO they are at times the wrong terms.\n> So you can delete the master branch (which I did in a project\n> that I am not the authoritative maintainer of; I only have feature\n> branches), and\n> the repository just works fine.\n\nI too have deleted the \"master\" branches in previous projects.\n\n"}]}