{"thread":{"id":"41348","subject":"[RFC] On the --depth argument when fetching with submodules","startedAt":"2016-02-05T22:48:43Z","lastAt":"2016-02-08T20:38:20Z","messageCount":7,"participants":["Stefan Beller","Junio C Hamano","Fredrik Gustafsson","Lars Schneider","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"277614","messageId":"CAGZ79kbt2-Vm94eTQY0PmJrNwqyTa36FJy5Q+2YBsxu6uYdTmQ@mail.gmail.com","threadId":"41348","inReplyTo":null,"subject":"[RFC] On the --depth argument when fetching with submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-05T22:48:43Z","receivedAt":"2016-02-05T22:48:43Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Currently when cloning a project, including submodules, the --depth argument\nis passed on recursively, i.e. when cloning with \"--depth 2\", both the\nsuperproject as well as the submodule will have a depth of 2.  It is not\ngarantueed that the commits as specified by the superproject are included\nin these 2 commits of the submodule.\n\nIllustration:\n(superproject with depth 2, so A would have more parents, not shown)\n\nsuperproject/master: A <- B\n                    /      \\\nsubmodule/master:  C <- D <- E <- F <- G\n\n(Current behavior is to fetch G and F)\n\nThe submodule is referenced at C and E from the two superproject commits.\nSo it is a reasonable expectation to have C and E included when fetching\nthe superproject with depth of 2.\n\nSo to fetch the correct submodule commits, we need to\n* traverse the superproject and list all submodule commits.\n* fetch these submodule commits (C and E) by sha1\n* in case of shallow clones, I'd propose to have the submodules be shallowed\n  with depth 1, i.e. to fetch C and E and no parents thereof\n* we need to think of a way to preserve the commits in the submodule for\n  later use (i.e. gc must not delete those commits)\n\nFor the later I propose to use a ref in the submodule (refs/meta/superproject ?)\nwhich will be an evil merge of all relevant commits to be preserved for the\nsuperproject. It is an evil merge commit as we do not need to store any worktree\nat all, but we only care about the parents relationship preventing the gc to\ncollect data required from the superproject. The parents should contain all\nbranches or in case of disjoint shallow histories (like in the example above)\nall the relevant commits.\n\nUsing these ideas we can go a step further in submodules and not need branches,\nbut only detached heads, which are directed from the superproject.\n\nAs for the implementation of these ideas, whenever you commit in the\nsuperproject\nwith modification to a submodule, the superproject would need to modify the\nrefs/meta/superproject submodule branch to make sure the submodule is\nsafe against\ngarbage collection.\n\nThanks,\nStefan\n"},{"id":"277619","messageId":"xmqqoabubt5e.fsf@gitster.mtv.corp.google.com","threadId":"41348","inReplyTo":"CAGZ79kbt2-Vm94eTQY0PmJrNwqyTa36FJy5Q+2YBsxu6uYdTmQ@mail.gmail.com","subject":"Re: [RFC] On the --depth argument when fetching with submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-06T00:05:01Z","receivedAt":"2016-02-06T00:05:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> Currently when cloning a project, including submodules, the --depth argument\n> is passed on recursively, i.e. when cloning with \"--depth 2\", both the\n> superproject as well as the submodule will have a depth of 2.  It is not\n> garantueed that the commits as specified by the superproject are included\n> in these 2 commits of the submodule.\n>\n> Illustration:\n> (superproject with depth 2, so A would have more parents, not shown)\n>\n> superproject/master: A <- B\n>                     /      \\\n> submodule/master:  C <- D <- E <- F <- G\n>\n> (Current behavior is to fetch G and F)\n\nI think the issue is deeper than merely \"--depth 2\", and you would\nbe better off stepping back and think about various use cases to\nmake sure that we know what kind of behaviour we want to support\nbefore delving into one particular corner case.  We currently pass\nthe depth recursively, and I do not think it makes much sense, but I\nview it as a secondary question \"among the behaviours we want to\nsupport, which one should be the default?\"  It may turn out that not\npassing it recursively at all, or even passing a different depth, is\na better default, but we wouldn't know until we know what are the\ndesirable behaviour in various workflows.\n\nIf you are actively working on the superproject plus some submodules\nbut you are merely using the submodule you depicted above, not\nworking on changing it, even when you want the full history of the\nsuperproject (i.e. no \"--depth 2\"), you may not want history of the\nsubmodule.  Even though we have a way to say \"I am not interested in\nthis submodule AT ALL\" by not doing \"submodule init\", not having\nanything at all at the path submodule/ may not allow you to build\nthe whole thing, and we currently lack a way to express \"I am not\ninterested in the history of this thing, but I need at least the\ntree that matches the commit referred to by the superproject\".\n\nIf you are working on a single submodule, trying to fix a bug in the\ncontext of the whole project, you might want to have a single-depth\nclone of the superproject and all other submodules, plus the whole\nhistory of the single submodule.\n\nIn either of these examples, the top-level \"--depth\" does not have\nmuch to do with what depth the user wants to use when cloning or\nfetching the submodule repositories.\n\nI have a feeling (but I would not be surprised if somebody who uses\nsubmodules heavily has a counter-example from real life) that\nregardless of \"--depth\" or full clone, fetching the tip of matching\nbranch is not a good default behaviour.  In your picture, even when\ndepth is not given at all, there isn't much point fetching F or G.\n\n> So to fetch the correct submodule commits, we need to\n> * traverse the superproject and list all submodule commits.\n> * fetch these submodule commits (C and E) by sha1\n\nI do not think requiring that C to be fetched when the superproject\nis cloned with --depth=2 (hence A and B are present in the result)\nis a good definition of \"correct submodule commits\".  The initial\nclone could be \"superproject follows --depth, all submodules are\ncloned with --depth=1 at the commits referenced by the superproject\ntree\"--by that definition, you need E but you do not want C.\n\nAs a specification of the behaviour, the above two might work, but I\ndo not think that should be the implementation.  In other words,\n\"The implementation should behave as if it did the above two\" is OK,\nand it is also OK to qualify with further conditions to help the\nimplementation.  For example, the current structure assumes that E\nand C are reachable from \"some\" ref in submodule, so that at least a\nwhole clone of the submodule would give them to you--otherwise you\nwould not be able to even build the superproject at A or B.  Perhaps\nit is OK to further require that, when you are working in a single\nbranch mode and working on 'master', you are required to have\ncommits C and E reachable on the 'master' branch in the submodule,\nand that may lets you limit the need for such scanning of the\nhistory?\n"},{"id":"277648","messageId":"20160206074138.GA30443@paksenarrion.iveqy.com","threadId":"41348","inReplyTo":"xmqqoabubt5e.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] On the --depth argument when fetching with submodules","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2016-02-06T07:41:38Z","receivedAt":"2016-02-06T07:41:38Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Fri, Feb 05, 2016 at 04:05:01PM -0800, Junio C Hamano wrote:\n> Stefan Beller <sbeller@google.com> writes:\n> \n> > Currently when cloning a project, including submodules, the --depth argument\n> > is passed on recursively, i.e. when cloning with \"--depth 2\", both the\n> > superproject as well as the submodule will have a depth of 2.  It is not\n> > garantueed that the commits as specified by the superproject are included\n> > in these 2 commits of the submodule.\n> >\n> > Illustration:\n> > (superproject with depth 2, so A would have more parents, not shown)\n> >\n> > superproject/master: A <- B\n> >                     /      \\\n> > submodule/master:  C <- D <- E <- F <- G\n> >\n> > (Current behavior is to fetch G and F)\n> \n\nThe --depth option to git submodule is something I've wondered for some\ntime if it was correct to implement it or not. It's clearly not a\ncomplete feature yet and it has some weaknesses, the most obvious one\nstated above.\n\nThe reason for implementing --depth in submodules was for people having\nhuge (or many) submodules, that they aren't interested in developing in,\nbut need to download to  build their project. So they want to save time\nand bandwidth.\n\nMy first thought was to implement depth from the sha1 the superproject\nrefered sha1. However, when implementing this, git didn't support\nfetching a sha1 from a remote repository for security reasons (you\nshould never be allowed to fetch a commit that is not reachable from a\nbranch or tag).\n\nWaiting for this to be supported (an (expensive) check could be done if\nthe sha1 asked for exists in any branch or tag), the --depth was added\nand it's a guessing game on how deep you should fetch.\n-- \nFredrik Gustafsson\n\nphone: +46 733-608274\ne-mail: iveqy@iveqy.com\nwebsite: http://www.iveqy.com\n"},{"id":"277691","messageId":"FA2DA97F-D944-4784-8297-E2885F197AC0@gmail.com","threadId":"41348","inReplyTo":"xmqqoabubt5e.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC] On the --depth argument when fetching with submodules","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-02-07T13:32:26Z","receivedAt":"2016-02-07T13:32:26Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 06 Feb 2016, at 01:05, Junio C Hamano <gitster@pobox.com> wrote:\n\n> Stefan Beller <sbeller@google.com> writes:\n> \n>> Currently when cloning a project, including submodules, the --depth argument\n>> is passed on recursively, i.e. when cloning with \"--depth 2\", both the\n>> superproject as well as the submodule will have a depth of 2.  It is not\n>> garantueed that the commits as specified by the superproject are included\n>> in these 2 commits of the submodule.\n>> \n>> Illustration:\n>> (superproject with depth 2, so A would have more parents, not shown)\n>> \n>> superproject/master: A <- B\n>>                    /      \\\n>> submodule/master:  C <- D <- E <- F <- G\n>> \n>> (Current behavior is to fetch G and F)\n> \n> I think the issue is deeper than merely \"--depth 2\", and you would\n> be better off stepping back and think about various use cases to\n> make sure that we know what kind of behaviour we want to support\n> before delving into one particular corner case.  We currently pass\n> the depth recursively, and I do not think it makes much sense, but I\n> view it as a secondary question \"among the behaviours we want to\n> support, which one should be the default?\"  It may turn out that not\n> passing it recursively at all, or even passing a different depth, is\n> a better default, but we wouldn't know until we know what are the\n> desirable behaviour in various workflows.\n> \n> If you are actively working on the superproject plus some submodules\n> but you are merely using the submodule you depicted above, not\n> working on changing it, even when you want the full history of the\n> superproject (i.e. no \"--depth 2\"), you may not want history of the\n> submodule.  Even though we have a way to say \"I am not interested in\n> this submodule AT ALL\" by not doing \"submodule init\", not having\n> anything at all at the path submodule/ may not allow you to build\n> the whole thing, and we currently lack a way to express \"I am not\n> interested in the history of this thing, but I need at least the\n> tree that matches the commit referred to by the superproject\".\n> \n> If you are working on a single submodule, trying to fix a bug in the\n> context of the whole project, you might want to have a single-depth\n> clone of the superproject and all other submodules, plus the whole\n> history of the single submodule.\n> \n> In either of these examples, the top-level \"--depth\" does not have\n> much to do with what depth the user wants to use when cloning or\n> fetching the submodule repositories.\n> \n> I have a feeling (but I would not be surprised if somebody who uses\n> submodules heavily has a counter-example from real life) that\n> regardless of \"--depth\" or full clone, fetching the tip of matching\n> branch is not a good default behaviour.  In your picture, even when\n> depth is not given at all, there isn't much point fetching F or G.\n\nI really wonder in what cases people use the \"--depth\" option, too. \nFor instance I have never used it in either one of the two cases you\ndescribed above. I don't worry about a long running \"clone\" as it \nusually is a one-time operation.\n\nHowever, in case of a continuous integration system that starts with\na clean state in the beginning of every run (e.g. Travis CI) a\n\"clone\" operation is no one-time operation anymore. In this case the\n\"--depth 1\" option makes very much sense to me. This was the situation\nwhere I realized the problem that Stefan wants to tackle here and I\ntried to make it tangible with a test case [1]. \n\nOn top of that I think Git's error message is really confusing if\nyou clone a repo with \"--depth\" that has submodules and Git is not\nfetching the necessary submodule commits:\n\nUnable to checkout '$SHA' in submodule path 'path/to/submodule'\n\nI tried to tackle that with [2] which would detect this case and\nprint the following error instead (slightly changed from the patch):\n\nUnable to checkout '$SHA' in submodule path '/path/to/commit'.\nTry to remove the '--depth' argument on clone!\n\n[1] https://www.mail-archive.com/git%40vger.kernel.org/msg82614.html\n[2] https://www.mail-archive.com/git%40vger.kernel.org/msg82613.html\n\n\n> \n>> So to fetch the correct submodule commits, we need to\n>> * traverse the superproject and list all submodule commits.\n>> * fetch these submodule commits (C and E) by sha1\n> \n> I do not think requiring that C to be fetched when the superproject\n> is cloned with --depth=2 (hence A and B are present in the result)\n> is a good definition of \"correct submodule commits\".  The initial\n> clone could be \"superproject follows --depth, all submodules are\n> cloned with --depth=1 at the commits referenced by the superproject\n> tree\"--by that definition, you need E but you do not want C.\n> \n> As a specification of the behaviour, the above two might work, but I\n> do not think that should be the implementation.  In other words,\n> \"The implementation should behave as if it did the above two\" is OK,\n> and it is also OK to qualify with further conditions to help the\n> implementation.  For example, the current structure assumes that E\n> and C are reachable from \"some\" ref in submodule, so that at least a\n> whole clone of the submodule would give them to you--otherwise you\n> would not be able to even build the superproject at A or B.  Perhaps\n> it is OK to further require that, when you are working in a single\n> branch mode and working on 'master', you are required to have\n> commits C and E reachable on the 'master' branch in the submodule,\n> and that may lets you limit the need for such scanning of the\n> history?\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"277751","messageId":"CAGZ79kYT8EWv6T=3bW_fH+_Q8p74p=JyzbErB+TOPO2VtVQHgQ@mail.gmail.com","threadId":"41348","inReplyTo":"FA2DA97F-D944-4784-8297-E2885F197AC0@gmail.com","subject":"Re: [RFC] On the --depth argument when fetching with submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-08T18:27:41Z","receivedAt":"2016-02-08T18:27:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Feb 7, 2016 at 5:32 AM, Lars Schneider <larsxschneider@gmail.com> wrote:\n>\n> On 06 Feb 2016, at 01:05, Junio C Hamano <gitster@pobox.com> wrote:\n>\n>> Stefan Beller <sbeller@google.com> writes:\n>>\n>>> Currently when cloning a project, including submodules, the --depth argument\n>>> is passed on recursively, i.e. when cloning with \"--depth 2\", both the\n>>> superproject as well as the submodule will have a depth of 2.  It is not\n>>> garantueed that the commits as specified by the superproject are included\n>>> in these 2 commits of the submodule.\n>>>\n>>> Illustration:\n>>> (superproject with depth 2, so A would have more parents, not shown)\n>>>\n>>> superproject/master: A <- B\n>>>                    /      \\\n>>> submodule/master:  C <- D <- E <- F <- G\n>>>\n>>> (Current behavior is to fetch G and F)\n>>\n>> I think the issue is deeper than merely \"--depth 2\", and you would\n>> be better off stepping back and think about various use cases to\n>> make sure that we know what kind of behaviour we want to support\n>> before delving into one particular corner case.  We currently pass\n>> the depth recursively, and I do not think it makes much sense, but I\n>> view it as a secondary question \"among the behaviours we want to\n>> support, which one should be the default?\"  It may turn out that not\n>> passing it recursively at all, or even passing a different depth, is\n>> a better default, but we wouldn't know until we know what are the\n>> desirable behaviour in various workflows.\n>>\n>> If you are actively working on the superproject plus some submodules\n>> but you are merely using the submodule you depicted above, not\n>> working on changing it, even when you want the full history of the\n>> superproject (i.e. no \"--depth 2\"), you may not want history of the\n>> submodule.  Even though we have a way to say \"I am not interested in\n>> this submodule AT ALL\" by not doing \"submodule init\", not having\n>> anything at all at the path submodule/ may not allow you to build\n>> the whole thing, and we currently lack a way to express \"I am not\n>> interested in the history of this thing, but I need at least the\n>> tree that matches the commit referred to by the superproject\".\n>>\n>> If you are working on a single submodule, trying to fix a bug in the\n>> context of the whole project, you might want to have a single-depth\n>> clone of the superproject and all other submodules, plus the whole\n>> history of the single submodule.\n>>\n>> In either of these examples, the top-level \"--depth\" does not have\n>> much to do with what depth the user wants to use when cloning or\n>> fetching the submodule repositories.\n>>\n>> I have a feeling (but I would not be surprised if somebody who uses\n>> submodules heavily has a counter-example from real life) that\n>> regardless of \"--depth\" or full clone, fetching the tip of matching\n>> branch is not a good default behaviour.  In your picture, even when\n>> depth is not given at all, there isn't much point fetching F or G.\n>\n> I really wonder in what cases people use the \"--depth\" option, too.\n> For instance I have never used it in either one of the two cases you\n> described above. I don't worry about a long running \"clone\" as it\n> usually is a one-time operation.\n\nI think there are 3 use cases.\n\n1) You work on the superproject and don't care about the submodules.\nIn this case you want the superproject non-shallow and the submodules\nmay be just fine with depth 1. (Think of libraries pulled in via Git instead\nof via the build system)\n\n2) The superproject is a collection of submodules, i.e. not much content\nin the superproject except for the submodules. You want to work\nin the submodules, i.e. you want the suberproject shallow, and all\nsubmodules deep.\n\n3) same as 2, but you're interested in only one (or a few) submodules,\nwhich means you want superproject and most of the other submodules\nshallow, but one submodule needs to be deep.\n\nSo covering 1 and 2 is easy, 3 is complicated.\nFor 1) we can make it so, that the depth argument is not passed on,\nbut only covers the referenced submodule commits, and then we\nintroduce another switch \"--submodule-depth\" to cover 2).\n\nFor 3 we don't know which submodules the user is interested in,\nso the user needs to unshallow the interesting submodules themselves\nafter doing a \"--depth 1 --submodule--depth 1\" clone. \"--depth 1\" sort of\nimplies \"--submodule--depth 1\", though.\n\n>\n> However, in case of a continuous integration system that starts with\n> a clean state in the beginning of every run (e.g. Travis CI) a\n> \"clone\" operation is no one-time operation anymore. In this case the\n> \"--depth 1\" option makes very much sense to me. This was the situation\n> where I realized the problem that Stefan wants to tackle here and I\n> tried to make it tangible with a test case [1].\n\nThanks for the test! The problem to make it work is in making it working\nin a backwards compatible way. Instead of the branch, you can just pass\na sha1 to git fetch and it sometimes works (if the server permits fetching\narbitrary or hidden sha1s. Though Jonathan noted this check may be in\nthe client only and the server trusts the client on not wanting arbitrary\nsha1s?)\n\nSo for fetching I think we need to have a \"--try-to-get-commit <sha1>\"\nargument for fetch, which depending on the server capabilities and\nthe history obtained otherwise may try again to fetch the exact sha1.\n\n\n>\n> On top of that I think Git's error message is really confusing if\n> you clone a repo with \"--depth\" that has submodules and Git is not\n> fetching the necessary submodule commits:\n>\n> Unable to checkout '$SHA' in submodule path 'path/to/submodule'\n>\n> I tried to tackle that with [2] which would detect this case and\n> print the following error instead (slightly changed from the patch):\n>\n> Unable to checkout '$SHA' in submodule path '/path/to/commit'.\n> Try to remove the '--depth' argument on clone!\n>\n> [1] https://www.mail-archive.com/git%40vger.kernel.org/msg82614.html\n> [2] https://www.mail-archive.com/git%40vger.kernel.org/msg82613.html\n>\n>\n>>\n>>> So to fetch the correct submodule commits, we need to\n>>> * traverse the superproject and list all submodule commits.\n>>> * fetch these submodule commits (C and E) by sha1\n>>\n>> I do not think requiring that C to be fetched when the superproject\n>> is cloned with --depth=2 (hence A and B are present in the result)\n>> is a good definition of \"correct submodule commits\".  The initial\n>> clone could be \"superproject follows --depth, all submodules are\n>> cloned with --depth=1 at the commits referenced by the superproject\n>> tree\"--by that definition, you need E but you do not want C.\n>>\n>> As a specification of the behaviour, the above two might work, but I\n>> do not think that should be the implementation.  In other words,\n>> \"The implementation should behave as if it did the above two\" is OK,\n>> and it is also OK to qualify with further conditions to help the\n>> implementation.  For example, the current structure assumes that E\n>> and C are reachable from \"some\" ref in submodule, so that at least a\n>> whole clone of the submodule would give them to you--otherwise you\n>> would not be able to even build the superproject at A or B.  Perhaps\n>> it is OK to further require that, when you are working in a single\n>> branch mode and working on 'master', you are required to have\n>> commits C and E reachable on the 'master' branch in the submodule,\n>> and that may lets you limit the need for such scanning of the\n>> history?\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"277763","messageId":"xmqq8u2v7y7x.fsf@gitster.mtv.corp.google.com","threadId":"41348","inReplyTo":"CAGZ79kYT8EWv6T=3bW_fH+_Q8p74p=JyzbErB+TOPO2VtVQHgQ@mail.gmail.com","subject":"Re: [RFC] On the --depth argument when fetching with submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-08T20:18:10Z","receivedAt":"2016-02-08T20:18:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> ... Instead of the branch, you can just pass\n> a sha1 to git fetch and it sometimes works (if the server permits fetching\n> arbitrary or hidden sha1s. Though Jonathan noted this check may be in\n> the client only and the server trusts the client on not wanting arbitrary\n> sha1s?)\n\nallowtipsha1inwant and allowreachablesha1inwant are the server side\ncontrols that are by default off.\n\n> So for fetching I think we need to have a \"--try-to-get-commit <sha1>\"\n> argument for fetch, which depending on the server capabilities and\n> the history obtained otherwise may try again to fetch the exact sha1.\n\nHmph, \"try\"?  t5516 seems to show how to do this.\n\nI however have to wonder how we disambiguate a 40-hex that is an\nobject name and a branch whose name happens to be 40-hex.\n"},{"id":"277765","messageId":"20160208203820.GA22317@google.com","threadId":"41348","inReplyTo":"CAGZ79kYT8EWv6T=3bW_fH+_Q8p74p=JyzbErB+TOPO2VtVQHgQ@mail.gmail.com","subject":"Re: [RFC] On the --depth argument when fetching with submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-02-08T20:38:20Z","receivedAt":"2016-02-08T20:38:20Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nStefan Beller wrote:\n\n> 1) You work on the superproject and don't care about the submodules.\n> In this case you want the superproject non-shallow and the submodules\n> may be just fine with depth 1. (Think of libraries pulled in via Git instead\n> of via the build system)\n>\n> 2) The superproject is a collection of submodules, i.e. not much content\n> in the superproject except for the submodules. You want to work\n> in the submodules, i.e. you want the suberproject shallow, and all\n> submodules deep.\n>\n> 3) same as 2, but you're interested in only one (or a few) submodules,\n> which means you want superproject and most of the other submodules\n> shallow, but one submodule needs to be deep.\n>\n> So covering 1 and 2 is easy, 3 is complicated.\n> For 1) we can make it so, that the depth argument is not passed on,\n> but only covers the referenced submodule commits, and then we\n> introduce another switch \"--submodule-depth\" to cover 2).\n>\n> For 3 we don't know which submodules the user is interested in,\n> so the user needs to unshallow the interesting submodules themselves\n> after doing a \"--depth 1 --submodule--depth 1\" clone. \"--depth 1\" sort of\n> implies \"--submodule--depth 1\", though.\n\nWhat does a --submodule-depth other than 1 mean?  For example, if I pass 2\nthere, does that mean I want depth 2 in all sub-sub-submodules, and\nsub-sub-sub-submodules, etc?  This has similar problems to the existing\n\"--depth=2 --recurse-submodules\" options.\n\nIf I understood the use cases you are describing correctly, you'd want two\nrelevant options when cloning:\n\n --depth: how deep of a history to fetch in the superproject\n --shallow-submodules: whether the history in submodules is interesting\n\nIf the history in a submodule is uninteresting and the submodule\nsupports fetch-by-sha1, then a fetch-by-sha1 with depth 1 for any\ncommits needed by the superproject would get all objects needed.\n\n--depth could imply --shallow-submodules in the same way as it implies\n--single-branch because when people pass --depth they are usually\ntrying to avoid transferring too much data (for disk space or clone\ntime reasons).  Then someone wanting non-shallow submodules can\nexplicitly pass --no-shallow-submodules with --depth.\n\n[...]\n> So for fetching I think we need to have a \"--try-to-get-commit <sha1>\"\n> argument for fetch, which depending on the server capabilities and\n> the history obtained otherwise may try again to fetch the exact sha1.\n\nThe case of servers without fetch-by-sha1 is a hard one.\n\n 1. One approach would be to use the ls-remote response to see if\n    there is a ref pointing to the particular SHA-1 needed.  If there\n    is one, we're all set.\n\n 2. If there is no ref pointing to the desired SHA-1, fetching HEAD\n    (or the branch named in the submodule.<name>.branch configuration)\n    with full history is possible to work as a fallback.\n\n 3. If the history of HEAD also doesn't contain the desired SHA-1,\n    fetching the refs named in the default refspec (refs/heads/*,\n    refs/tags/*) might work.\n\n 4. If even then the desired SHA-1 is unreachable, then there's\n    nothing left to try --- it's time to error out and have the user\n    contact their server admin (as they'd already be doing today\n    without --depth).\n\nPerhaps some people would want to error out earlier.  Some\nconfiguration or a --strict-shallow-submodules option could be useful\nto error out if step (1) fails.  That would allow the user to be sure\nthat the submodules are actually fetched without history.\n\nThe missing functionality from fetch/ls-remote/etc is a way to read\nthe server capabilities that were sent with the ref listing to tell\nwhether the server supports fetch-by-sha1.\n\n> On Sun, Feb 7, 2016 at 5:32 AM, Lars Schneider <larsxschneider@gmail.com> wrote:\n\n>> On top of that I think Git's error message is really confusing if\n>> you clone a repo with \"--depth\" that has submodules and Git is not\n>> fetching the necessary submodule commits:\n>>\n>> Unable to checkout '$SHA' in submodule path 'path/to/submodule'\n\nYep.  Thanks for working on it.\n\nThis confusing error can appear without --depth, too (if the commit\nnamed in the submodule is not an ancestor of anything in refs/heads/*\nor refs/tags/*).  For example, this happens in gerrit when people try\nto check out a change under review in a parent project that imports a\nchange in a subproject that has not been accepted yet (it is reachable\nfrom refs/changes/* but not refs/heads/*).\n\nThanks,\nJonathan\n"}]}