{"thread":{"id":"42309","subject":"Improve initial setup of submodules","startedAt":"2016-05-13T17:32:38Z","lastAt":"2016-05-13T18:13:28Z","messageCount":2,"participants":["Jonas Bernoulli","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"286470","messageId":"87posplv7t.fsf@bernoul.li","threadId":"42309","inReplyTo":null,"subject":"Improve initial setup of submodules","fromName":"Jonas Bernoulli","fromEmail":"jonas@bernoul.li","sentAt":"2016-05-13T17:32:38Z","receivedAt":"2016-05-13T17:32:38Z","isPatch":false,"sender":{"key":"jonas@bernoul.li","avatar":"https://gravatar.com/avatar/67252b2d3a6aacebe15776a6c0cb26c78b475dedc8410caa8765f1f4953222aa?d=mp&s=160"},"body":"Hello,\n\nCurrently \"git submodule update\" appears to be optimized for updating\nexisting submodules - which isn't surprising given its name. However it\nis also used for the \"initial setup\" of submodules, i.e. right after\nrunning \"git submodule init\" or with the \"--init\" argument.\n\nBelow you find a list of features I wish existed during the initial\nsetup. It might make sense to add an additional subcommand named\ne.g. \"bootstrap\" or \"setup\" to accomplish this, but maybe a\n\"--bootstrap\" argument might also suffice.\n\nThe goal is for a submodule to be \"ready for use\" after running \"git\nsubmodule init\" and \"git submodule bootstrap\"/\"git submodule update\n--bootstrap\".\n\n* Add additional remotes.\n\n \"submodule update\" can only add a single remote, but a submodule might\n require additional remotes, i.e. \"origin\" and \"my-fork\". It's likely\n that \"my-fork\" contains commits that are not available from \"origin\"\n and that such a commit is tracked as the HEAD of the submodule.\n\n When that is the case and if \"submodule..url\" points to the\n upstream repository, then \"submodule update \" would fail because\n the commit is not available.\n\n To deal with that the user would have to set \"url\" to the url of the\n fork repository. But then \"submodule update\" would name that remote\n \"origin\". And the user would then have to rename that to \"my-fork\"\n and also add the real \"origin\".\n\n It would be nice if it were possible to configure additional remotes\n in .gitmodules or elsewhere and if \"submodule bootstrap/update\" could\n be told to setup these remotes before attempting to checkout the\n tracked commit.\n\n Since Git doesn't support nested config sections I am unsure what\n would be the best way to store this information in \".gitmodules\". A\n hacky implementation could use something like this:\n\n [submodule \"example\"]\n url = git://example.com/upstream.git\n remote = my-fork git@example.me:my-fork.git\n remote = third-party git://example.com/with-patches.git\n\n Of course Git usually doesn't use a single variable to define multiple\n values, so that's not really an option except while experimenting.\n\n Another alternative might be to use separate files located in a\n directory \".gitmodule.d\", e.g. the remotes of the module\n \".gitmodule.d/example\" could contain:\n\n [remote \"origin\"]\n url = git://example.com/upstream.git\n [remote \"my-fork\"]\n url = git@example.me:my-fork.git\n [remote \"third-party\"]\n url = git://example.com/with-patches.git\n\n* Attach HEAD.\n\n I think it makes sense for \"submodule update\" to default to using\n \"checkout\" as \"submodule..update\" method, since it is the safest\n option which doesn't do any harm.\n\n It's also nice that there are some alternatives, \"rebase\", \"merge\" and\n \"none\", and even support for custom commands. (By the way \"ff-only\"\n and \"ff-only-else-checkout\" would be nice to have.)\n\n However neither \"rebase\" nor \"merge\" are suitable during bootstrap.\n\n During \"submodule bootstrap\" (but not \"submodule update\") a \"rewind\"\n variant might make sense. During bootstrap, after cloning but before\n checkout/merge/update, we might have history like this:\n\n A---B---C origin/master, master -> origin/master, HEAD\n ^\n |\n commit recorded for submodule\n\n The \"rewind\" update variant (actually boostrap variant) would then do\n a \"git reset --hard B\", resulting in:\n\n A---B---C origin/master\n ^\n |\n master > origin/master, HEAD, commit recorded for submodule\n\n Of course the situation might be more complicated:\n\n A---B---C origin/master, master -> origin/master, HEAD\n \\\n D---E---F my-fork/master\n ^\n |\n commit recorded for submodule\n\n Here \"rewind\" would fail because \"master\" cannot be rewound to E. In\n some cases that might be the right thing to do, but another\n alternative would be a \"reset\" variant, which does a \"git reset\n --hard\" even if the recorded commit isn't an ancestor of the checked\n out branch.\n \n A---B---C origin/master, HEAD\n \\\n D---E---F my-fork/master\n ^\n |\n master -> origin/master, HEAD, commit recorded for submodule\n\n Other, smarter, variants could be implemented, but the key point I am\n trying to make is that I would like to use diffent \"update methods\"\n during \"submodule bootstrap\" and \"submodule update\", and that \"rewind\"\n and \"reset\" should be available during \"boostrap\" by default.\n\n* Preserve additional information.\n\n If a mechanism such as the \".gitmodules.d/\" file I suggested\n above were used to add additional remotes, then that could be used to\n distribute other information from \".git/modules//config\". This\n could also be used by the various \"bootstrap\" variants.\n\n As I have shown above \".gitmodules.d/example\" could for example\n contain this:\n\n [remote \"origin\"]\n url = git://example.com/upstream.git\n [remote \"my-fork\"]\n url = git@example.me:my-fork.git\n [remote \"third-party\"]\n url = git://example.com/with-patches.git\n\n Here \"origin.url\" would be redundant because the same value is already\n set in \"submodule.example.url\". However this could be used to allow\n setting an alternative remote name:\n\n (in .gitmodules)\n\n [submodule \"example\"]\n remote = upstream\n\n (in .gitmodules.d/example)\n\n [remote \"upstream\"]\n url = git://example.com/upstream.git\n\n But that's just a minor detail. What I really would like to be able\n to do is this:\n \n (in .gitmodules)\n\n [submodule \"example\"]\n remote = origin\n bootstrap = rewind-either\n\n (in .gitmodules.d/example)\n\n [remote \"origin\"]\n url = git://example.com/upstream.git\n [remote \"my-fork\"]\n url = git@example.me:my-fork.git\n [branch \"master\"]\n remote = origin\n merge = refs/heads/master\n pushRemote = my-fork\n\n With such a setup the second example from above makes more sense:\n\n A---B---C origin/master, HEAD\n \\\n D---E---F my-fork/master\n ^\n |\n master -> origin/master, HEAD, commit recorded for submodule\n\n And a new bootstrap variant \"rewind-either\" could make sense: if the\n commit recorded for the submodule is reachable from either the\n upstream branch *or* from \"branch..pushRemote\", then rewind,\n else checkout a detached HEAD.\n\n Additionally the settings from \".gitmodules.d/\" should be copied\n to \".git/modules//config\", just like those from \".gitmodules\"\n are copied to \".git/config\":\n\n (in .git/modules/example/config)\n\n [remote \"origin\"]\n url = git://example.com/upstream.git\n fetch = +refs/heads/*:refs/remotes/origin/*\n [remote \"my-fork\"]\n url = git@example.me:my-fork.git\n fetch = +refs/heads/*:refs/remotes/my-fork/*\n [branch \"master\"]\n remote = origin\n merge = refs/heads/master\n pushRemote = my-fork\n\n Most of these variables are \"copied\" by adding the remotes, but\n \"branch.master.pushRemote\" would have to be explictily set.\n\n It should also be possible to checkout another branch by setting\n \"submodule..branch\", and to automatically create additional\n branches by adding additional \"branch\" sections to\n \".gitmodules.d/\".\n\n In addition to \"branch..pushRemote\" it would be nice if\n \"remote.pushDefault\" could be set in \".gitmodules.d/\".\n\nSo what do you think? Is there a change that such features could be\nadded to Git (using the suggested mechanisms or something else)?\n\n Best regards,\n Jonas Bernoulli\n \n \n \n"},{"id":"286475","messageId":"CAGZ79kaJy40DPC4wzyvzPr2myn3SStyaqeLoATMmWXXUnbp8Fw@mail.gmail.com","threadId":"42309","inReplyTo":"87posplv7t.fsf@bernoul.li","subject":"Re: Improve initial setup of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-05-13T18:13:28Z","receivedAt":"2016-05-13T18:13:28Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"+cc Heiko and Jens as they are submodule experts.\n\nOn Fri, May 13, 2016 at 10:32 AM, Jonas Bernoulli wrote:\n> Hello,\n>\n> Currently \"git submodule update\" appears to be optimized for updating\n> existing submodules - which isn't surprising given its name. However it\n> is also used for the \"initial setup\" of submodules, i.e. right after\n> running \"git submodule init\" or with the \"--init\" argument.\n>\n> Below you find a list of features I wish existed during the initial\n> setup. It might make sense to add an additional subcommand named\n> e.g. \"bootstrap\" or \"setup\" to accomplish this, but maybe a\n> \"--bootstrap\" argument might also suffice.\n\nWould the \"--bootstrap\" be an argument to \"submodule init\" ?\nSo \"submodule update --init\" is the same as \"submodule init --bootstrap\" ?\nHow about \"submmodule init --update\" ?\n\n>\n> The goal is for a submodule to be \"ready for use\" after running \"git\n> submodule init\" and \"git submodule bootstrap\"/\"git submodule update\n> --bootstrap\".\n\nIdeally you'd only use the submodule command for specifics on submodules.\ne.g. add a new submodule, configure a submodule update strategy or such.\nThis is similar to the \"remote command\", which ought to be used only for remote\nconfiguration etc. Historically there is \"remote update\", but nowadays the\nsuggested way would rather be \"git fetch/pull\"\n\n\"git clone\" has \"--recurse-submodules\", which does the bootstrapping for you.\n(i.e. run submodule update --init).\n\nCurrently there are 3 states for submodules:\n (1) not initialized\n (2) initialized but not checked out\n (3) checked out\n\nTo go from (1) to (2) you'd use \"submodule init\"\nfor going (2) to (3) you'd use \"submodule update\"\nGoing from (3') to (3) you'd also use \"submodule update\".\n(e.g. checking out the latest HEAD)\n\nTo go back from (3) to (2), you would use \"rm -rf submodule\" (no\ngit command available). Heiko recently proposed another command\nto go from either (3) to (2) as \"deinit --light\" IIUC[1].\n\nTo go back from either (3) or (2) to (1) you can use \"submodule deinit [-f]\".\n\nThe state (2) is a bit confusing I would think as it's somewhere in the middle\nof \"have\" or \"have not\", so an easy command to go from (1) to (3) would\nbe nice as then (2) becomes less exposed to users.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/293478/focus=294009\n\n\n>\n> * Add additional remotes.\n>\n> \"submodule update\" can only add a single remote, but a submodule might\n> require additional remotes, i.e. \"origin\" and \"my-fork\". It's likely\n> that \"my-fork\" contains commits that are not available from \"origin\"\n> and that such a commit is tracked as the HEAD of the submodule.\n>\n> When that is the case and if \"submodule..url\" points to the\n> upstream repository, then \"submodule update \" would fail because\n> the commit is not available.\n>\n> To deal with that the user would have to set \"url\" to the url of the\n> fork repository. But then \"submodule update\" would name that remote\n> \"origin\". And the user would then have to rename that to \"my-fork\"\n> and also add the real \"origin\".\n\nSo rather fetching from multiple sources until we find the right commit,\nbut pushing to only one?\n\n>\n> It would be nice if it were possible to configure additional remotes\n> in .gitmodules or elsewhere and if \"submodule bootstrap/update\" could\n> be told to setup these remotes before attempting to checkout the\n> tracked commit.\n>\n> Since Git doesn't support nested config sections I am unsure what\n> would be the best way to store this information in \".gitmodules\". A\n> hacky implementation could use something like this:\n>\n> [submodule \"example\"]\n> url = git://example.com/upstream.git\n> remote = my-fork git@example.me:my-fork.git\n> remote = third-party git://example.com/with-patches.git\n>\n> Of course Git usually doesn't use a single variable to define multiple\n> values, so that's not really an option except while experimenting.\n\nThis is not hacky I would think. It's the same as git push urls work[2]\n\n[2] http://stackoverflow.com/a/14290145\n\n>\n> Another alternative might be to use separate files located in a\n> directory \".gitmodule.d\", e.g. the remotes of the module\n> \".gitmodule.d/example\" could contain:\n>\n> [remote \"origin\"]\n> url = git://example.com/upstream.git\n> [remote \"my-fork\"]\n> url = git@example.me:my-fork.git\n> [remote \"third-party\"]\n> url = git://example.com/with-patches.git\n\nWe already have so many .git{stuff} things (the .git directory,\n.gitignore, .gitattributes, .gitmodules}, so I would suggest not\nadding more of that.\n\n>\n> * Attach HEAD.\n>\n> I think it makes sense for \"submodule update\" to default to using\n> \"checkout\" as \"submodule..update\" method, since it is the safest\n> option which doesn't do any harm.\n\nIf the submodule is in detached HEAD state and you checkout another\ncommit, you may loose commits?\n\n>\n> It's also nice that there are some alternatives, \"rebase\", \"merge\" and\n> \"none\", and even support for custom commands. (By the way \"ff-only\"\n> and \"ff-only-else-checkout\" would be nice to have.)\n>\n> However neither \"rebase\" nor \"merge\" are suitable during bootstrap.\n>\n> During \"submodule bootstrap\" (but not \"submodule update\") a \"rewind\"\n> variant might make sense. During bootstrap, after cloning but before\n> checkout/merge/update, we might have history like this:\n>\n> A---B---C origin/master, master -> origin/master, HEAD\n> ^\n> |\n> commit recorded for submodule\n>\n> The \"rewind\" update variant (actually boostrap variant) would then do\n> a \"git reset --hard B\", resulting in:\n>\n> A---B---C origin/master\n> ^\n> |\n> master > origin/master, HEAD, commit recorded for submodule\n>\n> Of course the situation might be more complicated:\n>\n> A---B---C origin/master, master -> origin/master, HEAD\n> \\\n> D---E---F my-fork/master\n> ^\n> |\n> commit recorded for submodule\n>\n> Here \"rewind\" would fail because \"master\" cannot be rewound to E. In\n> some cases that might be the right thing to do, but another\n> alternative would be a \"reset\" variant, which does a \"git reset\n> --hard\" even if the recorded commit isn't an ancestor of the checked\n> out branch.\n>\n> A---B---C origin/master, HEAD\n> \\\n> D---E---F my-fork/master\n> ^\n> |\n> master -> origin/master, HEAD, commit recorded for submodule\n>\n> Other, smarter, variants could be implemented, but the key point I am\n> trying to make is that I would like to use diffent \"update methods\"\n> during \"submodule bootstrap\" and \"submodule update\", and that \"rewind\"\n> and \"reset\" should be available during \"boostrap\" by default.\n>\n> * Preserve additional information.\n>\n> If a mechanism such as the \".gitmodules.d/\" file I suggested\n> above were used to add additional remotes, then that could be used to\n> distribute other information from \".git/modules//config\". This\n> could also be used by the various \"bootstrap\" variants.\n>\n> As I have shown above \".gitmodules.d/example\" could for example\n> contain this:\n>\n> [remote \"origin\"]\n> url = git://example.com/upstream.git\n> [remote \"my-fork\"]\n> url = git@example.me:my-fork.git\n> [remote \"third-party\"]\n> url = git://example.com/with-patches.git\n>\n> Here \"origin.url\" would be redundant because the same value is already\n> set in \"submodule.example.url\". However this could be used to allow\n> setting an alternative remote name:\n>\n> (in .gitmodules)\n>\n> [submodule \"example\"]\n> remote = upstream\n>\n> (in .gitmodules.d/example)\n>\n> [remote \"upstream\"]\n> url = git://example.com/upstream.git\n>\n> But that's just a minor detail. What I really would like to be able\n> to do is this:\n>\n> (in .gitmodules)\n>\n> [submodule \"example\"]\n> remote = origin\n> bootstrap = rewind-either\n>\n> (in .gitmodules.d/example)\n>\n> [remote \"origin\"]\n> url = git://example.com/upstream.git\n> [remote \"my-fork\"]\n> url = git@example.me:my-fork.git\n> [branch \"master\"]\n> remote = origin\n> merge = refs/heads/master\n> pushRemote = my-fork\n\nboth of .gitmodules and .gitmodules.d/example are part of the tree?\n\n\n>\n> With such a setup the second example from above makes more sense:\n>\n> A---B---C origin/master, HEAD\n> \\\n> D---E---F my-fork/master\n> ^\n> |\n> master -> origin/master, HEAD, commit recorded for submodule\n>\n> And a new bootstrap variant \"rewind-either\" could make sense: if the\n> commit recorded for the submodule is reachable from either the\n> upstream branch *or* from \"branch..pushRemote\", then rewind,\n> else checkout a detached HEAD.\n>\n> Additionally the settings from \".gitmodules.d/\" should be copied\n> to \".git/modules//config\", just like those from \".gitmodules\"\n> are copied to \".git/config\":\n>\n> (in .git/modules/example/config)\n>\n> [remote \"origin\"]\n> url = git://example.com/upstream.git\n> fetch = +refs/heads/*:refs/remotes/origin/*\n> [remote \"my-fork\"]\n> url = git@example.me:my-fork.git\n> fetch = +refs/heads/*:refs/remotes/my-fork/*\n> [branch \"master\"]\n> remote = origin\n> merge = refs/heads/master\n> pushRemote = my-fork\n>\n> Most of these variables are \"copied\" by adding the remotes, but\n> \"branch.master.pushRemote\" would have to be explictily set.\n>\n> It should also be possible to checkout another branch by setting\n> \"submodule..branch\", and to automatically create additional\n> branches by adding additional \"branch\" sections to\n> \".gitmodules.d/\".\n>\n> In addition to \"branch..pushRemote\" it would be nice if\n> \"remote.pushDefault\" could be set in \".gitmodules.d/\".\n>\n> So what do you think? Is there a change that such features could be\n> added to Git (using the suggested mechanisms or something else)?\n\nThe selling point is not obvious to me, yet. (Why is that better,\nwhich feature makes the big difference?)\n\nThanks for starting a discussion!\n\nStefan\n\n>\n> Best regards,\n> Jonas Bernoulli\n>\n>\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"}]}