{"thread":{"id":"42214","subject":"[RFD/PATCH] submodule doc: describe where we can configure them","startedAt":"2016-05-03T23:26:25Z","lastAt":"2016-05-11T16:54:00Z","messageCount":13,"participants":["Stefan Beller","Jonathan Nieder","Heiko Voigt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"285400","messageId":"1462317985-640-1-git-send-email-sbeller@google.com","threadId":"42214","inReplyTo":null,"subject":"[RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-05-03T23:26:25Z","receivedAt":"2016-05-03T23:26:25Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is similar to the gitignore document, but doesn't mirror\nthe current situation. It is rather meant to start a discussion for\nthe right approach for mirroring repositories with submodules.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n\n Jonathan, is this something you had in mind?\n\n Documentation/git-submodule.txt | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex 13adebf..b5559e5 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories\n that come from both projects can be cloned and checked out as a whole\n if you choose to go that route.\n \n+Submodule operations can be configured using the following mechanisms\n+(from highest to lowest precedence):\n+\n+ * the command line for those commands that support taking submodule specs.\n+\n+ * the configuration file `$GIT_DIR/config`.\n+\n+ * the configuration file `config` found in the `refs/submodule/config` branch.\n+   This can be used to overwrite the upstream configuration in the `.gitmodules`\n+   file without changing the history of the project.\n+   Useful options here are overwriting the base, where relative URLs apply to,\n+   when mirroring only parts of the larger collection of submodules.\n+\n+ * the `.gitmodules` file inside the repository. A project usually includes this\n+   file to suggest defaults for the upstream collection of repositories.\n+\n COMMANDS\n --------\n add::\n-- \n2.8.0.rc4.10.geb92688.dirty\n"},{"id":"285403","messageId":"20160503235652.GA395@google.com","threadId":"42214","inReplyTo":"1462317985-640-1-git-send-email-sbeller@google.com","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-05-03T23:56:53Z","receivedAt":"2016-05-03T23:56:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stefan Beller wrote:\n\n> This is similar to the gitignore document, but doesn't mirror\n> the current situation. It is rather meant to start a discussion for\n> the right approach for mirroring repositories with submodules.\n\nOoh.\n\n[...]\n> --- a/Documentation/git-submodule.txt\n> +++ b/Documentation/git-submodule.txt\n> @@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories\n>  that come from both projects can be cloned and checked out as a whole\n>  if you choose to go that route.\n>  \n> +Submodule operations can be configured using the following mechanisms\n> +(from highest to lowest precedence):\n> +\n> + * the command line for those commands that support taking submodule specs.\n> +\n> + * the configuration file `$GIT_DIR/config`.\n> +\n> + * the configuration file `config` found in the `refs/submodule/config` branch.\n> +   This can be used to overwrite the upstream configuration in the `.gitmodules`\n> +   file without changing the history of the project.\n> +   Useful options here are overwriting the base, where relative URLs apply to,\n> +   when mirroring only parts of the larger collection of submodules.\n> +\n> + * the `.gitmodules` file inside the repository. A project usually includes this\n> +   file to suggest defaults for the upstream collection of repositories.\n\n(This documentation probably belongs in gitmodules(5) --- then,\ngit-submodule(1) could focus on command-line usage and point there for\nconfiguration information.)\n\nThere are two aspects of this to be separated: what governs the behavior\nof commands running locally, and where we get information about\nsubmodules from a remote repository.\n\nLocal commands\n--------------\nThe original submodule design was that local commands rely on\ninformation from .git/config, and that information gets copied there\nfrom .gitmodules when a submodule is initialized.  That way, a local\nuser can specify their preferred mirror or other options using some\nstraightforward 'git config' commands.\n\nAs a side effect, the settings in .git/config tell git which submodules\nto pay attention to (which submodules were initialized).\n\nWhen .gitmodules changes, the settings in .git/config are left alone,\nsince the end user *might* have manually set something up and we don't\nwant to trample on it.\n\nThis design is somewhat problematic for a few reasons:\n\n- When I want to stop paying attention to a particular submodule and\n  start paying attention to it again later, all my local settings are\n  gone.\n\n- When upstream adds a new submodule, I have to do the same manual\n  work to change the options for that new submodule.\n\n- When upstream changes submodule options (perhaps to fix a URL\n  typo), I don't get those updates.\n\nA fix is to use settings from .git/config when present and fall back\nto .gitmodules when not.  I believe the submodule code has been slowly\nmoving in that direction for new features.  Perhaps we can do so for\nexisting features (like submodule.*.url) too.\n\nAn alternative would have been to introduce a .git/info/submodules\nfile that overrides settings from .gitmodules, analagous to\n.git/info/excludes overriding .gitignore and .git/info/attributes\noverriding .gitattributes.  We are already using .git/config for\nthis so that doesn't seem necessary.\n\nRemote repositories\n-------------------\nThe .gitmodules file has some odd properties as a place to put\nconfiguration:\n\n- it is versioned.  There is no way to change URLs in an old version\n  of .gitmodules retroactively when a URL has changed.\n\n- it is controlled by whoever writes history.  There is no way for me\n  to change the URLs in my mirror of https://gerrit.googlesource.com/gerrit\n  to match my mirror's different filesystem layout without producing\n  my own history that diverges from the commits I am mirroring.\n\nWhen the URLs in .gitmodules are relative URLs, this means that if\nI mirror a superproject, I have to mirror all its submodules, too,\nwith the same layout.  It's not so easy for me to publish my copy\nof the parent project and the one subproject I made changes in --- I\nhave to mirror everything.  In particular, this means I can't mirror\nhttps://gerrit.googlesource.com/gerrit to github.\n\nWhen the URLs in .gitmodules are absolute URLs, this means that if\nI mirror a superproject, I cannot ask people consuming my mirror to\nuse my mirrors of child projects, too.  I cannot publish my copy of\nthe parent project and the one subproject I made changes in and\nexpect people to be able to \"git clone --recurse-submodules\" the\nresult successfully.\n\nIt is as though refs were stored in a .gitrefs file, with all the\nattendant disadvantages, instead of being a separate component of\nthe repository that a particular repository owner can manipulate\nwithout changing history.\n\nTo fix this, we could allow additional .gitmodules settings to be put\nin another ref (perhaps something like \"refs/repository/config\" to allow\nsharing additional repository-specific configuration in other files\nwithin the same tree --- e.g., branch descriptions).  The semantics:\n\n* If there is a gitmodules file in refs/repository/config in the\n  repository I clone, then the submodule settings from it are stored\n  locally somewhere that overrides .gitmodules.  Perhaps\n  .git/info/<remotename>/gitmodules?\n\n* Later fetches from the remote would also update this gitmodules\n  file.\n\n* Settings from this gitmodules file can be overridden locally\n  using 'git config' until an explicit \"git submodule sync\" to\n  override the local configuration.\n\nWhat do you think?\n\nIf two different remotes provide conflicting values for a setting\nin their gitmodules files, git would error out and ask the user\nto intervene with a tie-breaking \"git config\" setting.\n\nThanks,\nJonathan\n"},{"id":"285406","messageId":"CAGZ79kb4GNZi93JpsPKUb7ScmePqDC1Xp+WZTGD7hfPUn9tWaQ@mail.gmail.com","threadId":"42214","inReplyTo":"20160503235652.GA395@google.com","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-05-04T00:59:58Z","receivedAt":"2016-05-04T00:59:58Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> Stefan Beller wrote:\n>\n>> This is similar to the gitignore document, but doesn't mirror\n>> the current situation. It is rather meant to start a discussion for\n>> the right approach for mirroring repositories with submodules.\n>\n> Ooh.\n\nThanks for writing such a detailed answer. :)\n\n>\n> [...]\n>> --- a/Documentation/git-submodule.txt\n>> +++ b/Documentation/git-submodule.txt\n>> @@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories\n>>  that come from both projects can be cloned and checked out as a whole\n>>  if you choose to go that route.\n>>\n>> +Submodule operations can be configured using the following mechanisms\n>> +(from highest to lowest precedence):\n>> +\n>> + * the command line for those commands that support taking submodule specs.\n>> +\n>> + * the configuration file `$GIT_DIR/config`.\n>> +\n>> + * the configuration file `config` found in the `refs/submodule/config` branch.\n>> +   This can be used to overwrite the upstream configuration in the `.gitmodules`\n>> +   file without changing the history of the project.\n>> +   Useful options here are overwriting the base, where relative URLs apply to,\n>> +   when mirroring only parts of the larger collection of submodules.\n>> +\n>> + * the `.gitmodules` file inside the repository. A project usually includes this\n>> +   file to suggest defaults for the upstream collection of repositories.\n>\n> (This documentation probably belongs in gitmodules(5) --- then,\n> git-submodule(1) could focus on command-line usage and point there for\n> configuration information.)\n\nThat makes sense!\n\n>\n> There are two aspects of this to be separated: what governs the behavior\n> of commands running locally, and where we get information about\n> submodules from a remote repository.\n\nAfter reading the first time, this seems to also contain \"historical context\".\n\n>\n> Local commands\n> --------------\n> The original submodule design was that local commands rely on\n> information from .git/config, and that information gets copied there\n> from .gitmodules when a submodule is initialized.  That way, a local\n> user can specify their preferred mirror or other options using some\n> straightforward 'git config' commands.\n>\n> As a side effect, the settings in .git/config tell git which submodules\n> to pay attention to (which submodules were initialized).\n>\n> When .gitmodules changes, the settings in .git/config are left alone,\n> since the end user *might* have manually set something up and we don't\n> want to trample on it.\n>\n> This design is somewhat problematic for a few reasons:\n>\n> - When I want to stop paying attention to a particular submodule and\n>   start paying attention to it again later, all my local settings are\n>   gone.\n>\n> - When upstream adds a new submodule, I have to do the same manual\n>   work to change the options for that new submodule.\n>\n> - When upstream changes submodule options (perhaps to fix a URL\n>   typo), I don't get those updates.\n>\n> A fix is to use settings from .git/config when present and fall back\n> to .gitmodules when not.  I believe the submodule code has been slowly\n> moving in that direction for new features.  Perhaps we can do so for\n> existing features (like submodule.*.url) too.\n>\n> An alternative would have been to introduce a .git/info/submodules\n> file that overrides settings from .gitmodules, analagous to\n> .git/info/excludes overriding .gitignore and .git/info/attributes\n> overriding .gitattributes.  We are already using .git/config for\n> this so that doesn't seem necessary.\n\nI don't know if it is a worthwhile goal nevertheless to move\nthe information about submodules to .git/info/submodules eventually\nas that brings consistency across different features of Git?\n\n>\n> Remote repositories\n> -------------------\n> The .gitmodules file has some odd properties as a place to put\n> configuration:\n>\n> - it is versioned.  There is no way to change URLs in an old version\n>   of .gitmodules retroactively when a URL has changed.\n\nI would not call it odd for having one versioned place. Consider your\nbuild process is updated and the new build process produces new\nintermediate files. You would add these files to the .gitignore file\neventually, but when building old revisions with the new build chain\nyou'd be surprised by all those untracked files being displayed.\nOr another example: Recently in git.git some test helper files were moved.\nBy checking out an older version of git you see a lot of test-* files\nin your worktree although they were ignored at another revision.\n\nThat paragraph got longer than expected, but I just wanted to say that\nbeing versioned can be either good or bad.\n\n>\n> - it is controlled by whoever writes history.  There is no way for me\n>   to change the URLs in my mirror of https://gerrit.googlesource.com/gerrit\n>   to match my mirror's different filesystem layout without producing\n>   my own history that diverges from the commits I am mirroring.\n\nTo come up with an analogy to ignored files:\nIf I use a project and use a different build system, I may see untracked\nfiles as they are not ignored by the .gitignore file.\n\nThen I have a way of ignoring them nevertheless in .git/info/excludes.\nSharing this information beyond this repository is hard though, but\nthat wasn't seen as a feature yet?\n\n>\n> When the URLs in .gitmodules are relative URLs, this means that if\n> I mirror a superproject, I have to mirror all its submodules, too,\n> with the same layout.  It's not so easy for me to publish my copy\n> of the parent project and the one subproject I made changes in --- I\n> have to mirror everything.  In particular, this means I can't mirror\n> https://gerrit.googlesource.com/gerrit to github.\n\nbecause the way repository URLs work are different for these 2 hosts.\ngooglesource.com allows to have URLs that are nested in another level\ne.g. Gerrit references \"../plugins/download-commands\", such that\nremote URL becomes https://gerrit.googlesource.com/plugins/download-commands\n\nAt Github we cannot create another level of nesting as their naming follows the\nowner/name scheme.\n\n>\n> When the URLs in .gitmodules are absolute URLs, this means that if\n> I mirror a superproject, I cannot ask people consuming my mirror to\n> use my mirrors of child projects, too.  I cannot publish my copy of\n> the parent project and the one subproject I made changes in and\n> expect people to be able to \"git clone --recurse-submodules\" the\n> result successfully.\n\n\n>\n> It is as though refs were stored in a .gitrefs file, with all the\n> attendant disadvantages, instead of being a separate component of\n> the repository that a particular repository owner can manipulate\n> without changing history.\n>\n> To fix this, we could allow additional .gitmodules settings to be put\n> in another ref (perhaps something like \"refs/repository/config\" to allow\n> sharing additional repository-specific configuration in other files\n> within the same tree --- e.g., branch descriptions).  The semantics:\n>\n> * If there is a gitmodules file in refs/repository/config in the\n>   repository I clone, then the submodule settings from it are stored\n>   locally somewhere that overrides .gitmodules.  Perhaps\n>   .git/info/<remotename>/gitmodules?\n>\n> * Later fetches from the remote would also update this gitmodules\n>   file.\n>\n> * Settings from this gitmodules file can be overridden locally\n>   using 'git config' until an explicit \"git submodule sync\" to\n>   override the local configuration.\n>\n> What do you think?\n>\n> If two different remotes provide conflicting values for a setting\n> in their gitmodules files, git would error out and ask the user\n> to intervene with a tie-breaking \"git config\" setting.\n\nLet's look at an example with C mirroring from B, who mirrors from A.\n\nThe user who clones the superproject from C may want to obtain submodules\nfrom either C or B or A. All this can be configured in\nthe refs/repository/config value of C, but in case it is not configured in C,\nit may fall back to the same branch from B. When and how would B get\nthat branch?\n\nThanks for writing out this detailed brain dump :)\nStefan\n\n>\n> Thanks,\n> Jonathan\n"},{"id":"285473","messageId":"20160504150143.GA8575@sandbox","threadId":"42214","inReplyTo":"CAGZ79kb4GNZi93JpsPKUb7ScmePqDC1Xp+WZTGD7hfPUn9tWaQ@mail.gmail.com","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-05-04T15:01:44Z","receivedAt":"2016-05-04T15:01:44Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Tue, May 03, 2016 at 05:59:58PM -0700, Stefan Beller wrote:\n> On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> > Stefan Beller wrote:\n> >\n> >> This is similar to the gitignore document, but doesn't mirror\n> >> the current situation. It is rather meant to start a discussion for\n> >> the right approach for mirroring repositories with submodules.\n> >\n> > Ooh.\n> \n> Thanks for writing such a detailed answer. :)\n\nBTW, here is a pointer to the discussion (and what I wrote down) about\nthis from back in 2014:\n\nhttps://github.com/jlehmann/git-submod-enhancements/wiki/Ideas#special-ref-overriding-gitmodules-values\n\n> > To fix this, we could allow additional .gitmodules settings to be put\n> > in another ref (perhaps something like \"refs/repository/config\" to allow\n> > sharing additional repository-specific configuration in other files\n> > within the same tree --- e.g., branch descriptions).  The semantics:\n> >\n> > * If there is a gitmodules file in refs/repository/config in the\n> >   repository I clone, then the submodule settings from it are stored\n> >   locally somewhere that overrides .gitmodules.  Perhaps\n> >   .git/info/<remotename>/gitmodules?\n> >\n> > * Later fetches from the remote would also update this gitmodules\n> >   file.\n> >\n> > * Settings from this gitmodules file can be overridden locally\n> >   using 'git config' until an explicit \"git submodule sync\" to\n> >   override the local configuration.\n> >\n> > What do you think?\n> >\n> > If two different remotes provide conflicting values for a setting\n> > in their gitmodules files, git would error out and ask the user\n> > to intervene with a tie-breaking \"git config\" setting.\n> \n> Let's look at an example with C mirroring from B, who mirrors from A.\n> \n> The user who clones the superproject from C may want to obtain submodules\n> from either C or B or A. All this can be configured in\n> the refs/repository/config value of C, but in case it is not configured in C,\n> it may fall back to the same branch from B. When and how would B get\n> that branch?\n\nI think B has to setup that branch on its own when it starts to mirror\nA and uses different submodule urls or other configs.\n\nJonathan you suggested to copy the content from a remote to\n.git/info/<remotename>/gitmodules locally. How would one get it to the\nremote side? It seems to me as if we would need to implement additional\ninfrastructure to do this. Would it not be simpler if we just kept it on\na ref on the local side as well? We already have the infrastructure to\nread those values from a ref. We only would need to add something to\nwrite them. Then a simple push, which could be aliased in form of a\ngit-submodule subcommand, suffices to get the values to the remote.\n\nThat also solves issues when people clone from their working copy.\n\nI would like to think a little bit further about the conflict situation\nwhen two remotes are providing values. Configuring this looks to me like\na nightmare for users. Maybe there is some sort of elegant solution?\nE.g. like we use the values from remote A during a fetch from A, the\nones from B during a fetch from B and no values from a special ref in\ncase there is no remote operation involved. Since the main goal is to\nsupport forking of submodules isn't there always a remote operation\ninvolved?\n\nMy suggested scheme above does not solve the currently quite typical use\ncase where you might 'git fetch' without submodules first and then do\nthe submodule fetches during a 'git submodule update'. On the other hand\nin the 'ideal future world' where submodules behave like \"normal files\" the\nfetch will be done during the superproject fetch so in that case we\ncould solve such conflicts.\n\nThe main thing which we could keep in mind is that we only allow certain\nvalues in such special refs. E.g. only the ones needed to support the\nfork workflow. BTW, do we actually need to change other values than the\nURL? Addtionally we ignore other values that are more related to the\noverall project structure. E.g. like submodule.<name>.ignore.\n\nOk after writing this it really feels like special casing a lot. I would\nnot really call it elegant. At the same time limiting these special refs\nto one special use case (forking) might help us to keep the user\ninterface[1] simpler and conflict free in the long run. Not sure. What\ndo you think?\n\nCheers Heiko\n\n[1] Which is not the simplest already.\n"},{"id":"285512","messageId":"xmqqh9edft1p.fsf@gitster.mtv.corp.google.com","threadId":"42214","inReplyTo":"1462317985-640-1-git-send-email-sbeller@google.com","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-04T20:48:50Z","receivedAt":"2016-05-04T20:48:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> This is similar to the gitignore document, but doesn't mirror\n> the current situation. It is rather meant to start a discussion for\n> the right approach for mirroring repositories with submodules.\n>\n> Signed-off-by: Stefan Beller <sbeller@google.com>\n> ---\n>\n>  Jonathan, is this something you had in mind?\n>\n>  Documentation/git-submodule.txt | 16 ++++++++++++++++\n>  1 file changed, 16 insertions(+)\n>\n> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\n> index 13adebf..b5559e5 100644\n> --- a/Documentation/git-submodule.txt\n> +++ b/Documentation/git-submodule.txt\n> @@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories\n>  that come from both projects can be cloned and checked out as a whole\n>  if you choose to go that route.\n>  \n> +Submodule operations can be configured using the following mechanisms\n> +(from highest to lowest precedence):\n> +\n> + * the command line for those commands that support taking submodule specs.\n\nSorry, but have we introduced <submodule spec> as a Git lingo?  What\ndoes it mean?\n\n> +\n> + * the configuration file `$GIT_DIR/config`.\n> +\n> + * the configuration file `config` found in the `refs/submodule/config` branch.\n> +   This can be used to overwrite the upstream configuration in the `.gitmodules`\n> +   file without changing the history of the project.\n> +   Useful options here are overwriting the base, where relative URLs apply to,\n> +   when mirroring only parts of the larger collection of submodules.\n\nThis smells like something server side people may come up with; how\nwould an end user with a usual \"repository with working tree\" layout\nfutz with this thing?  Can it even be checked out, or would we have\na UI similar to \"notes\"?\n\n> + * the `.gitmodules` file inside the repository. A project usually includes this\n> +   file to suggest defaults for the upstream collection of repositories.\n> +\n>  COMMANDS\n>  --------\n>  add::\n"},{"id":"285513","messageId":"CAGZ79kZs5zLAUdi53D=pTyFZF__73ZbW4o28C2hOuTV7_NfaDQ@mail.gmail.com","threadId":"42214","inReplyTo":"20160504150143.GA8575@sandbox","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-05-04T20:50:24Z","receivedAt":"2016-05-04T20:50:24Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 4, 2016 at 8:01 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:\n> On Tue, May 03, 2016 at 05:59:58PM -0700, Stefan Beller wrote:\n>> On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n>> > Stefan Beller wrote:\n>> >\n>> >> This is similar to the gitignore document, but doesn't mirror\n>> >> the current situation. It is rather meant to start a discussion for\n>> >> the right approach for mirroring repositories with submodules.\n>> >\n>> > Ooh.\n>>\n>> Thanks for writing such a detailed answer. :)\n>\n> BTW, here is a pointer to the discussion (and what I wrote down) about\n> this from back in 2014:\n>\n> https://github.com/jlehmann/git-submod-enhancements/wiki/Ideas#special-ref-overriding-gitmodules-values\n\nThanks for pointing at the prior discussion!\nAlthough not much happened since then (code wise)?\n\n>\n>> > To fix this, we could allow additional .gitmodules settings to be put\n>> > in another ref (perhaps something like \"refs/repository/config\" to allow\n>> > sharing additional repository-specific configuration in other files\n>> > within the same tree --- e.g., branch descriptions).  The semantics:\n>> >\n>> > * If there is a gitmodules file in refs/repository/config in the\n>> >   repository I clone, then the submodule settings from it are stored\n>> >   locally somewhere that overrides .gitmodules.  Perhaps\n>> >   .git/info/<remotename>/gitmodules?\n>> >\n>> > * Later fetches from the remote would also update this gitmodules\n>> >   file.\n>> >\n>> > * Settings from this gitmodules file can be overridden locally\n>> >   using 'git config' until an explicit \"git submodule sync\" to\n>> >   override the local configuration.\n>> >\n>> > What do you think?\n>> >\n>> > If two different remotes provide conflicting values for a setting\n>> > in their gitmodules files, git would error out and ask the user\n>> > to intervene with a tie-breaking \"git config\" setting.\n>>\n>> Let's look at an example with C mirroring from B, who mirrors from A.\n>>\n>> The user who clones the superproject from C may want to obtain submodules\n>> from either C or B or A. All this can be configured in\n>> the refs/repository/config value of C, but in case it is not configured in C,\n>> it may fall back to the same branch from B. When and how would B get\n>> that branch?\n>\n> I think B has to setup that branch on its own when it starts to mirror\n> A and uses different submodule urls or other configs.\n>\n> Jonathan you suggested to copy the content from a remote to\n> .git/info/<remotename>/gitmodules locally. How would one get it to the\n> remote side? It seems to me as if we would need to implement additional\n> infrastructure to do this. Would it not be simpler if we just kept it on\n> a ref on the local side as well? We already have the infrastructure to\n> read those values from a ref. We only would need to add something to\n> write them. Then a simple push, which could be aliased in form of a\n> git-submodule subcommand, suffices to get the values to the remote.\n\nThat is good idea!\n\n>\n> That also solves issues when people clone from their working copy.\n>\n> I would like to think a little bit further about the conflict situation\n> when two remotes are providing values. Configuring this looks to me like\n> a nightmare for users. Maybe there is some sort of elegant solution?\n> E.g. like we use the values from remote A during a fetch from A, the\n> ones from B during a fetch from B and no values from a special ref in\n> case there is no remote operation involved. Since the main goal is to\n> support forking of submodules isn't there always a remote operation\n> involved?\n\nHere is what I imagine\nWhen B mirrors from A, B sets up this special ref for its repository,\ne.g. refs/meta/submodule-B and have a symbolic ref pointing at that.\n(e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,\nwhich has a worktree which contains a .gitmodules files which\nsets up\n  \"submodule.baz.url = http://B/baz\"\n  \"submodule.relativeBase = http://A\"\n\nThat way anyone cloning from B would get\nthe superproject and the submodule baz from B while the\nrest of the submodules are found at A.\n\nWhen C mirrors from A, they add another branch  refs/meta/submodule-C,\nwhich can either be a fork of refs/meta/submodule-B with some changes on\ntop of it or it can add a reference to refs/meta/submodule-B, i.e. the\nconfiguration\nwould be:\n\n  \"submodule.baseConfig = refs/meta/submodule-B\"\n  \"submodule.foo.url = ssh://C/foo\"\n\nand SUBMODULE_CONFIG would point at refs/meta/submodule-C.\n\nWhen cloning from C, the user would get\n\n * the superproject from C\n * submodule foo from C\n * submodule baz from B\n * all other submodules from A\n\nBy the inheriting property of the branch of B there are no conflicting values.\nC could just overwrite submodule.baseConfig for example.\n\n>\n> My suggested scheme above does not solve the currently quite typical use\n> case where you might 'git fetch' without submodules first and then do\n> the submodule fetches during a 'git submodule update'. On the other hand\n> in the 'ideal future world' where submodules behave like \"normal files\" the\n> fetch will be done during the superproject fetch so in that case we\n> could solve such conflicts.\n>\n> The main thing which we could keep in mind is that we only allow certain\n> values in such special refs. E.g. only the ones needed to support the\n> fork workflow. BTW, do we actually need to change other values than the\n> URL? Addtionally we ignore other values that are more related to the\n> overall project structure. E.g. like submodule.<name>.ignore.\n\nMaybe we want to have a dedicated protocol field, eventually.\nA,B,C may have different standards on what they use by default.\ne.g. Use ssh at kernel.org, but http in a corporate mirror, because http is\nthe only protocol not blocked by firewall. So I could imagine that a\ncomplete mirror of submodules with relative URLs wants to only replace\nssh by http.\n\n>\n> Ok after writing this it really feels like special casing a lot. I would\n> not really call it elegant. At the same time limiting these special refs\n> to one special use case (forking) might help us to keep the user\n> interface[1] simpler and conflict free in the long run. Not sure. What\n> do you think?\n\n\n\n>\n> Cheers Heiko\n>\n> [1] Which is not the simplest already.\n"},{"id":"285514","messageId":"xmqqd1p1frw4.fsf@gitster.mtv.corp.google.com","threadId":"42214","inReplyTo":"20160503235652.GA395@google.com","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-04T21:13:47Z","receivedAt":"2016-05-04T21:13:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> This design is somewhat problematic for a few reasons:\n>\n> - When I want to stop paying attention to a particular submodule and\n>   start paying attention to it again later, all my local settings are\n>   gone.\n\nTrue; \"[submodule \"foo\"] enabled = no\" may also be a way to fix it\nwithout throwing the whole with bathwater, though.\n\n> - When upstream adds a new submodule, I have to do the same manual\n>   work to change the options for that new submodule.\n\nBecause a new module is not automatically \"init\"ed by default?\n\nIsn't \"config only\" vs \"config with gitmodules fallback\" orthogonal\nto that issue?\n\n> - When upstream changes submodule options (perhaps to fix a URL\n>   typo), I don't get those updates.\n\nTrue.\n\n> A fix is to use settings from .git/config when present and fall back\n> to .gitmodules when not.  \n\nHow would that fix the \"upstream updated\"?\n\nI think an alternative suggested in an ancient time had a more\nelaborate scheme:\n\n * Use .git/config as the authoritative source, but record\n   sufficient information to detect the case and cope with it when\n   entry in .gitmodules changes (details below).\n\n * When seeing a new .gitmodules entry, either by \"git pull\" or even\n   \"git checkout other-branch\", copy that to .git/config (just like\n   what \"git submodule init\" does).  It would be a policy decision\n   to automatically enabling it or not.  If the policy is \"no\n   autoinit\", then \"module.<name>.inited = no\" may also have to be\n   added to .git/config at this point.\n\n   Record contents of the entry in .gitmodules to the corresponding\n   .git/config entry as \"seen\".\n\n * When the entry in .gitmodules for a submodule known to\n   .git/config is different from what has been \"seen\", offer the\n   user a chance to update corresponding .git/config entry, and\n   append to the \"seen\" set of variants in .gitmodules so that the\n   user will not be bugged with \"we see .gitmodules entry for module\n   <foo> is different from anything you have ever seen; do you want\n   to make corresponding changes to the module entry in your\n   .git/config\" again.\n\nwhich would handle all of the above, and without using anything from\n.gitmodules before the user has a chance to vet it.\n"},{"id":"285901","messageId":"20160508215401.GA59077@book.hvoigt.net","threadId":"42214","inReplyTo":"CAGZ79kZs5zLAUdi53D=pTyFZF__73ZbW4o28C2hOuTV7_NfaDQ@mail.gmail.com","subject":"Re: Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-05-08T21:54:08Z","receivedAt":"2016-05-08T21:54:08Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"Hi,\n\nOn Wed, May 04, 2016 at 01:50:24PM -0700, Stefan Beller wrote:\n> On Wed, May 4, 2016 at 8:01 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:\n> > On Tue, May 03, 2016 at 05:59:58PM -0700, Stefan Beller wrote:\n> >> On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> >> > Stefan Beller wrote:\n> >> >\n> >> >> This is similar to the gitignore document, but doesn't mirror\n> >> >> the current situation. It is rather meant to start a discussion for\n> >> >> the right approach for mirroring repositories with submodules.\n> >> >\n> >> > Ooh.\n> >>\n> >> Thanks for writing such a detailed answer. :)\n> >\n> > BTW, here is a pointer to the discussion (and what I wrote down) about\n> > this from back in 2014:\n> >\n> > https://github.com/jlehmann/git-submod-enhancements/wiki/Ideas#special-ref-overriding-gitmodules-values\n> \n> Thanks for pointing at the prior discussion!\n> Although not much happened since then (code wise)?\n\nYes, IIRC nothing happened code wise. It went so far that a rough\nconsensus was made but nobody actually stepped in to scratch that itch.\n\n> > Jonathan you suggested to copy the content from a remote to\n> > .git/info/<remotename>/gitmodules locally. How would one get it to the\n> > remote side? It seems to me as if we would need to implement additional\n> > infrastructure to do this. Would it not be simpler if we just kept it on\n> > a ref on the local side as well? We already have the infrastructure to\n> > read those values from a ref. We only would need to add something to\n> > write them. Then a simple push, which could be aliased in form of a\n> > git-submodule subcommand, suffices to get the values to the remote.\n> \n> That is good idea!\n\nThanks.\n\n> > That also solves issues when people clone from their working copy.\n> >\n> > I would like to think a little bit further about the conflict situation\n> > when two remotes are providing values. Configuring this looks to me like\n> > a nightmare for users. Maybe there is some sort of elegant solution?\n> > E.g. like we use the values from remote A during a fetch from A, the\n> > ones from B during a fetch from B and no values from a special ref in\n> > case there is no remote operation involved. Since the main goal is to\n> > support forking of submodules isn't there always a remote operation\n> > involved?\n> \n> Here is what I imagine\n> When B mirrors from A, B sets up this special ref for its repository,\n> e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.\n> (e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,\n> which has a worktree which contains a .gitmodules files which\n> sets up\n>   \"submodule.baz.url = http://B/baz\"\n>   \"submodule.relativeBase = http://A\"\n> \n> That way anyone cloning from B would get\n> the superproject and the submodule baz from B while the\n> rest of the submodules are found at A.\n\nThis sounds sensible. But my imagination of a conflict was in a\ndifferent way. E.g. project A has a submodule B. And now A has a remote\n1 where you publish and maybe another remote 2 where someone else (a\ncolleague?) publishes. Which configuration do you use? Here the two\nremotes are independent instead of subsequent forks. In this case my\nsolution would be to use the configuration branch from 1 for B when\ninteracting with 1. I do not have completely checked whether we always\nhave a remote at hand for such a resolution.\n\n> When C mirrors from A, they add another branch  refs/meta/submodule-C,\n> which can either be a fork of refs/meta/submodule-B with some changes on\n> top of it or it can add a reference to refs/meta/submodule-B, i.e. the\n> configuration\n> would be:\n> \n>   \"submodule.baseConfig = refs/meta/submodule-B\"\n>   \"submodule.foo.url = ssh://C/foo\"\n> \n> and SUBMODULE_CONFIG would point at refs/meta/submodule-C.\n> \n> When cloning from C, the user would get\n> \n>  * the superproject from C\n>  * submodule foo from C\n>  * submodule baz from B\n>  * all other submodules from A\n> \n> By the inheriting property of the branch of B there are no conflicting values.\n> C could just overwrite submodule.baseConfig for example.\n\nSo that means in the default case we create a chain of all previous\nforks embedded in repository database. I am not saying that this is\nnecessarily a bad thing but I feel that it is a new property which we\nshould think about. It helps because users will get updated values from\nsources that are in the chain. On the other hand it adds a lot of\ndependencies which are point of failures in case a remote disappears. I\nam undecided on this. I would prefer if we could let people play with it\na little (maybe on pu?) and then decide if there are practical pitfalls\nwith this.\n\n> > My suggested scheme above does not solve the currently quite typical use\n> > case where you might 'git fetch' without submodules first and then do\n> > the submodule fetches during a 'git submodule update'. On the other hand\n> > in the 'ideal future world' where submodules behave like \"normal files\" the\n> > fetch will be done during the superproject fetch so in that case we\n> > could solve such conflicts.\n> >\n> > The main thing which we could keep in mind is that we only allow certain\n> > values in such special refs. E.g. only the ones needed to support the\n> > fork workflow. BTW, do we actually need to change other values than the\n> > URL? Addtionally we ignore other values that are more related to the\n> > overall project structure. E.g. like submodule.<name>.ignore.\n> \n> Maybe we want to have a dedicated protocol field, eventually.\n> A,B,C may have different standards on what they use by default.\n> e.g. Use ssh at kernel.org, but http in a corporate mirror, because http is\n> the only protocol not blocked by firewall. So I could imagine that a\n> complete mirror of submodules with relative URLs wants to only replace\n> ssh by http.\n\nBy this you mean 'submodule.relativeBase' that you introduced above\nright? Or something similar. These values I would still consider them\nURL'ish. But my question was more geared towards this direction: Are\nthere other values than the ones used to assemble the URL that make\nsense to share?\n\nE.g.: Someone might want to fork a repository and might want to change\nthe default set of submodules that are populated with 'git submodule\nupdate --init'. Is this something we should allow via these special refs\nor is this actually changing the project structure and should also be\nreflected in project history? IMO the latter is the case.\n\nOnly things like the technical organisation (like the place where a\nrepository can be found) justify to be outside of the repository IMO.\n\nA repository without submodules does have one collection of remote\nrepository urls. To me adding proper fork support seems be the switch\nfrom one collection for one repository to many collections for many\nrepositories. Since this one collection is already outside of the\nsuperproject it makes sense to do the same for the submodules. So my\nquestion reformulated could be: Are there more values we currently keep\ninside the repository for submodules that actually belong outside? A\ngood indication could be that they are already outside in the\nsuperproject.\n\nI did not find any flaw in these statements yet, but maybe I am\noversimplifying?\n\nCheers Heiko\n"},{"id":"285902","messageId":"20160508220127.GB59077@book.hvoigt.net","threadId":"42214","inReplyTo":"xmqqd1p1frw4.fsf@gitster.mtv.corp.google.com","subject":"Re: Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-05-08T22:01:28Z","receivedAt":"2016-05-08T22:01:28Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"Hi,\n\nOn Wed, May 04, 2016 at 02:13:47PM -0700, Junio C Hamano wrote:\n> Jonathan Nieder <jrnieder@gmail.com> writes:\n> \n> > This design is somewhat problematic for a few reasons:\n> >\n> > - When I want to stop paying attention to a particular submodule and\n> >   start paying attention to it again later, all my local settings are\n> >   gone.\n> \n> True; \"[submodule \"foo\"] enabled = no\" may also be a way to fix it\n> without throwing the whole with bathwater, though.\n\nIMO we already have this. With\n\n\tgit config submodule.<name>.update none\n\trm -rf <path>\n\tmkdir <path>\n\nWe remove a submodule from the working copy and disable any updates with\n'git submodule update ...'. Maybe we should add this 'light' operation\nas an option to 'git submodule deinit' in the long run?\n\n> > - When upstream adds a new submodule, I have to do the same manual\n> >   work to change the options for that new submodule.\n> \n> Because a new module is not automatically \"init\"ed by default?\n> \n> Isn't \"config only\" vs \"config with gitmodules fallback\" orthogonal\n> to that issue?\n\nWhat do you mean with \"orthogonal to that issue\"? AFAICS a gitmodule\nfallback does not have that issue. Actually I would see it more like:\n.gitmodule is the default and .git/config a possibility to override.\nWhen viewing it like this and using .gitmodule directly is the default\na user does not have any issues when upstream changes submodule\nconfigurations.\n\nOr are we talking about subsequent forks from upstreams? Like C forked\nfrom B and B from A... Then forget what I said.\n\n> > - When upstream changes submodule options (perhaps to fix a URL\n> >   typo), I don't get those updates.\n> \n> True.\n\nI would say it depends on what is your default view. See above.\n\n> > A fix is to use settings from .git/config when present and fall back\n> > to .gitmodules when not.  \n> \n> How would that fix the \"upstream updated\"?\n\nWhen .gitmodules is the default source \"upstream updated\" is\nautomatically read.\n\n> I think an alternative suggested in an ancient time had a more\n> elaborate scheme:\n> \n>  * Use .git/config as the authoritative source, but record\n>    sufficient information to detect the case and cope with it when\n>    entry in .gitmodules changes (details below).\n> \n>  * When seeing a new .gitmodules entry, either by \"git pull\" or even\n>    \"git checkout other-branch\", copy that to .git/config (just like\n>    what \"git submodule init\" does).  It would be a policy decision\n>    to automatically enabling it or not.  If the policy is \"no\n>    autoinit\", then \"module.<name>.inited = no\" may also have to be\n>    added to .git/config at this point.\n> \n>    Record contents of the entry in .gitmodules to the corresponding\n>    .git/config entry as \"seen\".\n> \n>  * When the entry in .gitmodules for a submodule known to\n>    .git/config is different from what has been \"seen\", offer the\n>    user a chance to update corresponding .git/config entry, and\n>    append to the \"seen\" set of variants in .gitmodules so that the\n>    user will not be bugged with \"we see .gitmodules entry for module\n>    <foo> is different from anything you have ever seen; do you want\n>    to make corresponding changes to the module entry in your\n>    .git/config\" again.\n> \n> which would handle all of the above, and without using anything from\n> .gitmodules before the user has a chance to vet it.\n\nI can see that for some users it might be important not to pull every\nsubmodule that upstream decides they should have. On the other hand: Is\nit really a decision a user can/should make during a pull or a checkout.\nI would be annoyed by it, since it interrupts me from the thing I really\nwant to do and would mostly just choose some default (like always yes or\nalways no) depending on what is important to me (e.g. faster checkout or\ncomplete repository). So IMO it is more sensible if we just give the\nuser some default to configure and then use that instead of asking\nquestions in a situation where the user is not ready to answer them.\n\nAnd when the user has his defaults we can actually try to deduct such\ndecisions directly from .gitmodules and do not need to store anything in\n.git/config as long as the user goes with the defaults.\n\nCheers Heiko\n"},{"id":"285931","messageId":"xmqqlh3jxkyn.fsf@gitster.mtv.corp.google.com","threadId":"42214","inReplyTo":"20160508220127.GB59077@book.hvoigt.net","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-09T16:19:44Z","receivedAt":"2016-05-09T16:19:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Heiko Voigt <hvoigt@hvoigt.net> writes:\n\n>> > - When upstream adds a new submodule, I have to do the same manual\n>> >   work to change the options for that new submodule.\n>> \n>> Because a new module is not automatically \"init\"ed by default?\n>> \n>> Isn't \"config only\" vs \"config with gitmodules fallback\" orthogonal\n>> to that issue?\n>\n> What do you mean with \"orthogonal to that issue\"? AFAICS a gitmodule\n> fallback does not have that issue.\n>\n> Actually I would see it more like:\n> .gitmodule is the default and .git/config a possibility to override.\n\nThe way I read Jonathan's \"I have to do the same manual...\" above is:\n\n  Back when I cloned, the upstream had one submodule A.  I didn't like\n  some aspect of the configuration for that submodule so I did a\n  customization in [submodule \"A\"] section of .git/config for it.\n\n  Now the upstream added another submodule B.  I want a tweak similar\n  to what I did to A applied to this one, but that would mean I need\n  to edit the entry in .git/config copied by \"init\" from .gitmodules.\n\nI do not see how difference between \".git/config is the only source\nof truth\" or \".git/config overrides what is in .gitmodules\" would\nmatter to the above scenario.\n"},{"id":"285946","messageId":"CAGZ79kZ+WzVwhj77zkHKbX-PmgfPvCO9qcX07vyC-BG+nDhWxA@mail.gmail.com","threadId":"42214","inReplyTo":"20160508215401.GA59077@book.hvoigt.net","subject":"Re: Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-05-09T17:32:50Z","receivedAt":"2016-05-09T17:32:50Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> Here is what I imagine\n>> When B mirrors from A, B sets up this special ref for its repository,\n>> e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.\n>> (e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,\n>> which has a worktree which contains a .gitmodules files which\n>> sets up\n>>   \"submodule.baz.url = http://B/baz\"\n>>   \"submodule.relativeBase = http://A\"\n>>\n>> That way anyone cloning from B would get\n>> the superproject and the submodule baz from B while the\n>> rest of the submodules are found at A.\n>\n> This sounds sensible. But my imagination of a conflict was in a\n> different way. E.g. project A has a submodule B. And now A has a remote\n> 1 where you publish and maybe another remote 2 where someone else (a\n> colleague?) publishes. Which configuration do you use? Here the two\n> remotes are independent instead of subsequent forks. In this case my\n> solution would be to use the configuration branch from 1 for B when\n> interacting with 1. I do not have completely checked whether we always\n> have a remote at hand for such a resolution.\n\nI think it is the responsibility of the pusher to make sure the\nconfiguration is sane.\nSo if I were to push to remote 2 and you push to remote 1, we'd both configure\nthe special branch of our superprojects for these remotes for that submodule.\n\nIf the superproject has relative urls for the submodule, all we had to do was\nunset (or overwrite) the submodule.baseConfig.\n\n>\n>> When C mirrors from A, they add another branch  refs/meta/submodule-C,\n>> which can either be a fork of refs/meta/submodule-B with some changes on\n>> top of it or it can add a reference to refs/meta/submodule-B, i.e. the\n>> configuration\n>> would be:\n>>\n>>   \"submodule.baseConfig = refs/meta/submodule-B\"\n>>   \"submodule.foo.url = ssh://C/foo\"\n>>\n>> and SUBMODULE_CONFIG would point at refs/meta/submodule-C.\n>>\n>> When cloning from C, the user would get\n>>\n>>  * the superproject from C\n>>  * submodule foo from C\n>>  * submodule baz from B\n>>  * all other submodules from A\n>>\n>> By the inheriting property of the branch of B there are no conflicting values.\n>> C could just overwrite submodule.baseConfig for example.\n>\n> So that means in the default case we create a chain of all previous\n> forks embedded in repository database.\n\nNot necessarily. I was just pointing out that this was possible. The\nintermediate\nparty could decide that their upstream is too unreliable and not point\nto their upstream.\n\nThis would incur the cost of having to clone all submodules and\noverwriting the absolute\nurls. For the relative URLs this would just work as of now.\n\nAll I wanted with that example is to offer the flexibility to not have\nto clone all the\nsubmodule, but I can fork a mega-project with 100s of submodules and maybe\njust fiddle with one of them and then publish that.\n\n> I am not saying that this is\n> necessarily a bad thing but I feel that it is a new property which we\n> should think about. It helps because users will get updated values from\n> sources that are in the chain. On the other hand it adds a lot of\n> dependencies which are point of failures in case a remote disappears. I\n> am undecided on this. I would prefer if we could let people play with it\n> a little (maybe on pu?) and then decide if there are practical pitfalls\n> with this.\n>\n>> > My suggested scheme above does not solve the currently quite typical use\n>> > case where you might 'git fetch' without submodules first and then do\n>> > the submodule fetches during a 'git submodule update'. On the other hand\n>> > in the 'ideal future world' where submodules behave like \"normal files\" the\n>> > fetch will be done during the superproject fetch so in that case we\n>> > could solve such conflicts.\n>> >\n>> > The main thing which we could keep in mind is that we only allow certain\n>> > values in such special refs. E.g. only the ones needed to support the\n>> > fork workflow. BTW, do we actually need to change other values than the\n>> > URL? Addtionally we ignore other values that are more related to the\n>> > overall project structure. E.g. like submodule.<name>.ignore.\n>>\n>> Maybe we want to have a dedicated protocol field, eventually.\n>> A,B,C may have different standards on what they use by default.\n>> e.g. Use ssh at kernel.org, but http in a corporate mirror, because http is\n>> the only protocol not blocked by firewall. So I could imagine that a\n>> complete mirror of submodules with relative URLs wants to only replace\n>> ssh by http.\n>\n> By this you mean 'submodule.relativeBase' that you introduced above\n> right? Or something similar. These values I would still consider them\n> URL'ish. But my question was more geared towards this direction: Are\n> there other values than the ones used to assemble the URL that make\n> sense to share?\n>\n> E.g.: Someone might want to fork a repository and might want to change\n> the default set of submodules that are populated with 'git submodule\n> update --init'. Is this something we should allow via these special refs\n> or is this actually changing the project structure and should also be\n> reflected in project history? IMO the latter is the case.\n\nThat sounds reasonable.\n\n>\n> Only things like the technical organisation (like the place where a\n> repository can be found) justify to be outside of the repository IMO.\n>\n> A repository without submodules does have one collection of remote\n> repository urls. To me adding proper fork support seems be the switch\n> from one collection for one repository to many collections for many\n> repositories. Since this one collection is already outside of the\n> superproject it makes sense to do the same for the submodules. So my\n> question reformulated could be: Are there more values we currently keep\n> inside the repository for submodules that actually belong outside? A\n> good indication could be that they are already outside in the\n> superproject.\n>\n> I did not find any flaw in these statements yet, but maybe I am\n> oversimplifying?\n\nThey sound right to me.\n\n>\n> Cheers Heiko\n\nThanks for the discussion :)\nStefan\n"},{"id":"286273","messageId":"20160511155047.GA3321@sandbox","threadId":"42214","inReplyTo":"xmqqlh3jxkyn.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-05-11T15:50:48Z","receivedAt":"2016-05-11T15:50:48Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Mon, May 09, 2016 at 09:19:44AM -0700, Junio C Hamano wrote:\n> Heiko Voigt <hvoigt@hvoigt.net> writes:\n> \n> >> > - When upstream adds a new submodule, I have to do the same manual\n> >> >   work to change the options for that new submodule.\n> >> \n> >> Because a new module is not automatically \"init\"ed by default?\n> >> \n> >> Isn't \"config only\" vs \"config with gitmodules fallback\" orthogonal\n> >> to that issue?\n> >\n> > What do you mean with \"orthogonal to that issue\"? AFAICS a gitmodule\n> > fallback does not have that issue.\n> >\n> > Actually I would see it more like:\n> > .gitmodule is the default and .git/config a possibility to override.\n> \n> The way I read Jonathan's \"I have to do the same manual...\" above is:\n> \n>   Back when I cloned, the upstream had one submodule A.  I didn't like\n>   some aspect of the configuration for that submodule so I did a\n>   customization in [submodule \"A\"] section of .git/config for it.\n> \n>   Now the upstream added another submodule B.  I want a tweak similar\n>   to what I did to A applied to this one, but that would mean I need\n>   to edit the entry in .git/config copied by \"init\" from .gitmodules.\n> \n> I do not see how difference between \".git/config is the only source\n> of truth\" or \".git/config overrides what is in .gitmodules\" would\n> matter to the above scenario.\n\nI see with that explanation your comment makes sense to me. So what we\nare here talking about is the wish to configure some general user set\nsettings that are applied to a group of/all submodules.\n\nThinking about it: Maybe sticking configurations to the submodule\ngroups, which Stefan Beller introduced in a different topic, could be a\ndirection we can go for such needs.\n\nCheers Heiko\n"},{"id":"286274","messageId":"20160511165400.GB3321@sandbox","threadId":"42214","inReplyTo":"CAGZ79kZ+WzVwhj77zkHKbX-PmgfPvCO9qcX07vyC-BG+nDhWxA@mail.gmail.com","subject":"Re: Re: [RFD/PATCH] submodule doc: describe where we can configure them","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-05-11T16:54:00Z","receivedAt":"2016-05-11T16:54:00Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Mon, May 09, 2016 at 10:32:50AM -0700, Stefan Beller wrote:\n> >> Here is what I imagine\n> >> When B mirrors from A, B sets up this special ref for its repository,\n> >> e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.\n> >> (e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,\n> >> which has a worktree which contains a .gitmodules files which\n> >> sets up\n> >>   \"submodule.baz.url = http://B/baz\"\n> >>   \"submodule.relativeBase = http://A\"\n> >>\n> >> That way anyone cloning from B would get\n> >> the superproject and the submodule baz from B while the\n> >> rest of the submodules are found at A.\n> >\n> > This sounds sensible. But my imagination of a conflict was in a\n> > different way. E.g. project A has a submodule B. And now A has a remote\n> > 1 where you publish and maybe another remote 2 where someone else (a\n> > colleague?) publishes. Which configuration do you use? Here the two\n> > remotes are independent instead of subsequent forks. In this case my\n> > solution would be to use the configuration branch from 1 for B when\n> > interacting with 1. I do not have completely checked whether we always\n> > have a remote at hand for such a resolution.\n> \n> I think it is the responsibility of the pusher to make sure the\n> configuration is sane.\n> So if I were to push to remote 2 and you push to remote 1, we'd both configure\n> the special branch of our superprojects for these remotes for that submodule.\n> \n> If the superproject has relative urls for the submodule, all we had to do was\n> unset (or overwrite) the submodule.baseConfig.\n\nWhat if (because we work together) you and me have both remotes in our\nlocal repository. We only push to our private remotes but fetch from\nboth. Since we work together we also forked the same submodule B and\nhave different URL configurations for it. I push to B1 and you to B2.\nNow we both have two special branches (one from B1 and one from B2) in\nour local repositories, since on either of our private remotes there is\none special branch.\n\nWhich values are valid now? I see you are advocating for a symbolic ref\nSUBMODULE_CONFIG that points to a single special branch in charge, but\nmaybe we can avoid that. In this case there actually is no real\nconflict, since we can just add both remotes B1, B2 to the submodule B.\nWhich one is used is a choice of the user during push.\n\nFor submodule.relativeBase we could try a similar solution and just add\nall remotes that can be constructed with the different configurations.\nProbably under the same name as in the superproject.\n\nSo if we limit ourselves to only allow URL'ish (actually remote'ish is\nprobably a better term) we can actually avoid conflict resolution and\njust add/use them all. If we limit ourselves to the fork use case and my\nhypothesis that we only need to allow remote'ish values in these special\nbranches for it is true, we can actually keep it quite simple and have\nno conflict resolution at all I think (and realize now).\n\nWhat do you think?\n\n> >> When C mirrors from A, they add another branch  refs/meta/submodule-C,\n> >> which can either be a fork of refs/meta/submodule-B with some changes on\n> >> top of it or it can add a reference to refs/meta/submodule-B, i.e. the\n> >> configuration\n> >> would be:\n> >>\n> >>   \"submodule.baseConfig = refs/meta/submodule-B\"\n> >>   \"submodule.foo.url = ssh://C/foo\"\n> >>\n> >> and SUBMODULE_CONFIG would point at refs/meta/submodule-C.\n> >>\n> >> When cloning from C, the user would get\n> >>\n> >>  * the superproject from C\n> >>  * submodule foo from C\n> >>  * submodule baz from B\n> >>  * all other submodules from A\n> >>\n> >> By the inheriting property of the branch of B there are no conflicting values.\n> >> C could just overwrite submodule.baseConfig for example.\n> >\n> > So that means in the default case we create a chain of all previous\n> > forks embedded in repository database.\n> \n> Not necessarily. I was just pointing out that this was possible. The\n> intermediate\n> party could decide that their upstream is too unreliable and not point\n> to their upstream.\n> \n> This would incur the cost of having to clone all submodules and\n> overwriting the absolute\n> urls. For the relative URLs this would just work as of now.\n> \n> All I wanted with that example is to offer the flexibility to not have\n> to clone all the\n> submodule, but I can fork a mega-project with 100s of submodules and maybe\n> just fiddle with one of them and then publish that.\n\nDo you mean 'not having to fork all the submodules' here? Since 'without\ncloning' is already possible, no?\n\nI am assuming you meant fork. So submodule.relativeBase is meant to\nsolve that right? You set it and all relative submodule URLs that are\nnot configured otherwise relate to it.\n\nMy point was about the chaining with submodule.baseConfig. That is not\nnecessary to support partial forks of just a few submodules.\n\nActually while thinking about submodule.relativeBase now, I found it\nmight be nice to extend it a little. Imagine someone wants fork a set of\nsubmodules and specify a relativeBase for them and then someone else\nforking again wants to do that with another set of submodules. I imagine\nsubsequents forks are quite usual in git (like in the kernels workflow).\nMaybe we can extend this scheme a little bit and allow to set\nsubmodule.relativeBase for groups of submodules somehow?\n\n> > I am not saying that this is\n> > necessarily a bad thing but I feel that it is a new property which we\n> > should think about. It helps because users will get updated values from\n> > sources that are in the chain. On the other hand it adds a lot of\n> > dependencies which are point of failures in case a remote disappears. I\n> > am undecided on this. I would prefer if we could let people play with it\n> > a little (maybe on pu?) and then decide if there are practical pitfalls\n> > with this.\n> >\n[...]\n\n> >\n> > Only things like the technical organisation (like the place where a\n> > repository can be found) justify to be outside of the repository IMO.\n> >\n> > A repository without submodules does have one collection of remote\n> > repository urls. To me adding proper fork support seems be the switch\n> > from one collection for one repository to many collections for many\n> > repositories. Since this one collection is already outside of the\n> > superproject it makes sense to do the same for the submodules. So my\n> > question reformulated could be: Are there more values we currently keep\n> > inside the repository for submodules that actually belong outside? A\n> > good indication could be that they are already outside in the\n> > superproject.\n> >\n> > I did not find any flaw in these statements yet, but maybe I am\n> > oversimplifying?\n> \n> They sound right to me.\n\nGreat. Then my simplification suggestion above should work as well.\n\n> Thanks for the discussion :)\nThank you for caring about this topic! I think this is quite some\nimportant work to get submodule forking almost as simple as forking\ntheir superprojects. I am happy to continue this discussion and\nbounce ideas off of each other :)\n\nCheers Heiko\n"}]}