{"thread":{"id":"41674","subject":"[RFC/PATCH] clone: add `--shallow-submodules` flag","startedAt":"2016-03-11T23:41:23Z","lastAt":"2016-03-14T18:37:46Z","messageCount":6,"participants":["Stefan Beller","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"280651","messageId":"1457739683-1972-1-git-send-email-sbeller@google.com","threadId":"41674","inReplyTo":null,"subject":"[RFC/PATCH] clone: add `--shallow-submodules` flag","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-03-11T23:41:23Z","receivedAt":"2016-03-11T23:41:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"When creating a shallow clone of a repository with submodules, the depth\nargument does not influence the submodules, i.e. the submodules are done\nas non-shallow clones. It is unclear what the best default is for the\ndepth of submodules of a shallow clone, so we need to have the possibility\nto do all kinds of combinations:\n\n* shallow super project with shallow submodules\n e.g. build bots starting always from scratch. They want to transmit\n the least amount of network data as well as using the least amount\n of space on their hard drive.\n* shallow super project with unshallow submodules\n e.g. The superproject is just there to track a collection of repositories\n and it is not important to have the relationship between the repositories\n intact. However the history of the individual submodules matter.\n* unshallow super project with shallow submodules\n e.g. The superproject is the actual project and the submodule is a\n library which is rarely touched.\n\nThe new switch to select submodules to be shallow or unshallow supports\nall of these three cases.\n\nIt is easy to transition from the first to the second case by just\nunshallowing the submodules (`git submodule foreach git fetch\n--unshallow`), but it is not possible to transition from the second to the\nfirst case (as we wouldd have already transmitted the non shallow over\nthe network). That is why we want to make the first case the default in\ncase of a shallow super project. This leads to the inconvenience in the\nsecond case with the shallow super project and unshallow submodules,\nas you need to pass `--no-shallow-submodules`.\n\nSigned-off-by: Stefan Beller \n---\nA few notes:\n * This applies on top of sb/submodule-parallel-update\n * I am aware of the current release cycle, and I ought to not add shiny\n new features. But scanning the list revealed no bugs I could jump at to fix.\n * Lars made some unit tests for a very similar case a few weeks ago, but they\n were not applied as there was no intention to fix it. So I am hoping we can\n reuse some of these tests for this patch.\n * Currently I have the opinion that thinking about (un)shallow projects should\n be rather binary. Either you want the full history or you want as least as\n possible. (Who wants to have a --depth 42? Some people have argued that you\n can use the depth to avoid large binaries which were part of the history in\n the past. But I'd counter that argument by pointing out the --depth argument\n is a workaround for such a use case. What you really want in that situation\n is a setting to clone history since a certain point in time/DAG, or a setting\n to clone up to a certain depth such that the filesize of the packed objects\n is smaller than some threshold.)\n \n If we were to allow specifying the depth for submodules, we'd need to discuss\n how to specify them for individual submodules i.e. clone submodule A with\n depth 4 and submodule B with depth 10. but that problem is solved easier\n by first doing a shallow clone with depth 1 of all submodules and then deepen\n them individually.\n \n So binary shallowness (depth = 1 or infinity) it is.\n\nThanks,\nStefan\n\n Documentation/git-clone.txt | 13 ++++++++++---\n builtin/clone.c | 6 ++++++\n 2 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 6db7b6d..20a4577 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,8 +14,8 @@ SYNOPSIS\n \t [-o ] [-b ] [-u ] [--reference ]\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch]\n-\t [--recursive | --recurse-submodules] [--jobs ] [--] \n-\t []\n+\t [--recursive | --recurse-submodules] [--[no-]shallow-submodules]\n+\t [--jobs ] [--] []\n \n DESCRIPTION\n -----------\n@@ -190,7 +190,11 @@ objects from the source repository into a pack in the cloned repository.\n \n --depth ::\n \tCreate a 'shallow' clone with a history truncated to the\n-\tspecified number of revisions.\n+\tspecified number of revisions. Implies `--single-branch` unless\n+\t`--no-single-branch` is given to fetch the histories near the\n+\ttips of all branches. This implies `--shallow-submodules`. If\n+\tyou want to have a shallow clone, but full submodules, also pass\n+\t`--no-shallow-submodules`.\n \n --[no-]single-branch::\n \tClone only the history leading to the tip of a single branch,\n@@ -214,6 +218,9 @@ objects from the source repository into a pack in the cloned repository.\n \trepository does not have a worktree/checkout (i.e. if any of\n \t`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)\n \n+--shallow-submodules::\n+\tAll submodules which are cloned, will be shallow.\n+\n --separate-git-dir=::\n \tInstead of placing the cloned repository where it is supposed\n \tto be, place the cloned repository at the specified directory,\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex b004fb4..cfa01fe 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -40,6 +40,7 @@ static const char * const builtin_clone_usage[] = {\n \n static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n+static int option_shallow_submodules = -1;\n static char *option_template, *option_depth;\n static char *option_origin = NULL;\n static char *option_branch = NULL;\n@@ -91,6 +92,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"create a shallow clone of that depth\")),\n \tOPT_BOOL(0, \"single-branch\", &option_single_branch,\n \t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\tOPT_BOOL(0, \"shallow-submodules\", &option_shallow_submodules,\n+\t\t N_(\"any cloned submodules will be shallow\")),\n \tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n@@ -727,6 +730,9 @@ static int checkout(void)\n \t\tstruct argv_array args = ARGV_ARRAY_INIT;\n \t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n \n+\t\tif (option_shallow_submodules && option_depth)\n+\t\t\targv_array_pushf(&args, \"--depth=1\");\n+\n \t\tif (max_jobs != -1)\n \t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n \n-- \n2.7.0.rc0.42.g42a5408.dirty\n"},{"id":"280652","messageId":"xmqqa8m4a5nx.fsf@gitster.mtv.corp.google.com","threadId":"41674","inReplyTo":"1457739683-1972-1-git-send-email-sbeller@google.com","subject":"Re: [RFC/PATCH] clone: add `--shallow-submodules` flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-12T00:41:54Z","receivedAt":"2016-03-12T00:41:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> When creating a shallow clone of a repository with submodules, the depth\n> argument does not influence the submodules, i.e. the submodules are done\n> as non-shallow clones. It is unclear what the best default is for the\n> depth of submodules of a shallow clone, so we need to have the possibility\n> to do all kinds of combinations:\n>\n> * shallow super project with shallow submodules\n> e.g. build bots starting always from scratch. They want to transmit\n> the least amount of network data as well as using the least amount\n> of space on their hard drive.\n> * shallow super project with unshallow submodules\n> e.g. The superproject is just there to track a collection of repositories\n> and it is not important to have the relationship between the repositories\n> intact. However the history of the individual submodules matter.\n> * unshallow super project with shallow submodules\n> e.g. The superproject is the actual project and the submodule is a\n> library which is rarely touched.\n>\n> The new switch to select submodules to be shallow or unshallow supports\n> all of these three cases.\n\nI think something like this is necessary to prime the well, but the\nmore important (and intereseting) bit is how this shallowness is\ngoing to be maintained and carried forward across the future updates\nto the top-level supermodule. A submodule that was cloned at depth=1\ninitially along with its supermodule when the latter was initially\ncloned does not have to be indefinitely kept at depth=1, and there\nwould be a lot of creative ways to make it useful, but the creative\nand useful logic would need a piece of information to tell the\nfuture \"submodule update\" why the submodule repository is shallow to\ntake into account, I would imagine.\n\nIt is somewhat curious that there is no hint left in the submodule\nrepositories (e.g. their configfile) that they are originally\ncreated with an explicit user request \"I said that I want these\nsubmodules to be cloned with depth=1\", from that point of view.\n"},{"id":"280653","messageId":"CAGZ79kYc1WR9CTBi1F3KpLaYRUSxNPQkQ2jk+P09pshR3jG8aw@mail.gmail.com","threadId":"41674","inReplyTo":"xmqqa8m4a5nx.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC/PATCH] clone: add `--shallow-submodules` flag","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-03-12T00:56:33Z","receivedAt":"2016-03-12T00:56:33Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Mar 11, 2016 at 4:41 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> When creating a shallow clone of a repository with submodules, the depth\n>> argument does not influence the submodules, i.e. the submodules are done\n>> as non-shallow clones. It is unclear what the best default is for the\n>> depth of submodules of a shallow clone, so we need to have the possibility\n>> to do all kinds of combinations:\n>>\n>> * shallow super project with shallow submodules\n>> e.g. build bots starting always from scratch. They want to transmit\n>> the least amount of network data as well as using the least amount\n>> of space on their hard drive.\n>> * shallow super project with unshallow submodules\n>> e.g. The superproject is just there to track a collection of repositories\n>> and it is not important to have the relationship between the repositories\n>> intact. However the history of the individual submodules matter.\n>> * unshallow super project with shallow submodules\n>> e.g. The superproject is the actual project and the submodule is a\n>> library which is rarely touched.\n>>\n>> The new switch to select submodules to be shallow or unshallow supports\n>> all of these three cases.\n>\n> I think something like this is necessary to prime the well, but the\n> more important (and intereseting) bit is how this shallowness is\n> going to be maintained and carried forward across the future updates\n> to the top-level supermodule. A submodule that was cloned at depth=1\n> initially along with its supermodule when the latter was initially\n> cloned does not have to be indefinitely kept at depth=1, and there\n> would be a lot of creative ways to make it useful, but the creative\n> and useful logic would need a piece of information to tell the\n> future \"submodule update\" why the submodule repository is shallow to\n> take into account, I would imagine.\n>\n> It is somewhat curious that there is no hint left in the submodule\n> repositories (e.g. their configfile) that they are originally\n> created with an explicit user request \"I said that I want these\n> submodules to be cloned with depth=1\", from that point of view.\n\nWhy is it interesting for submodules but not for standard repositories?\nIf I clone a repository without submodules, it is also not recorded\nthat I cloned with an explicit depth=1. If you fetch, you may end up with\na deeper history as git fetch doesn't do a \"reshallow\" to the configured\ndepth.\n\nAs the depth can easily change I view depth as a measure which is\nonly valid at moment in time, after the operation succeeded we rather\nwant to talk about the cut off points which were introduced by the\nshallow operation? And these are kept as is by default which is sane.\n"},{"id":"280681","messageId":"xmqq60wra40h.fsf@gitster.mtv.corp.google.com","threadId":"41674","inReplyTo":"CAGZ79kYc1WR9CTBi1F3KpLaYRUSxNPQkQ2jk+P09pshR3jG8aw@mail.gmail.com","subject":"Re: [RFC/PATCH] clone: add `--shallow-submodules` flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-12T19:29:50Z","receivedAt":"2016-03-12T19:29:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> Why is it interesting for submodules but not for standard repositories?\n>\n> If I clone a repository without submodules, it is also not recorded\n> that I cloned with an explicit depth=1. If you fetch, you may end up with\n> a deeper history as git fetch doesn't do a \"reshallow\" to the configured\n> depth.\n\nVery simple.\n\nIf you do not have submodule, you would always interact with the\nother side directly with \"git fetch\" or \"git pull\" and have total\ncontrol over when you choose to pass or not to pass extra options to\nchoose to 1. incrementally extend, 2. deepen, or 3. unshallow. The\nuser will always explicitly tell you, and knowing how you got there\nwould not help you, as there is no need to guess for you.\n\nThe user can do the same explicit \"cd dir && git fetch\" update in\neach submodule directory and give appropriate options to choose\namong the three, but I have an impression that your recent work is\ngoing in the direction of making commands that are run in the\nsuperproject recurse into submodules that automatically fetches and\nupdates the history down there, discouraging users from working on\nindividual submodules. You lose the flexibility to explicitly\nchoose among the three for individual submodules, and you may want\nto have some smart in your \"run from the superproject and recurse\"\ntools.\n\nA submodule that was initially cloned with depth=1, perhaps because\nthe user didn't know if the module was interesting to her in the\ncontext of working on the superproject before she had her clone of\nthe superproject hence she only wanted to see what's there, and a\nsubmodule that was not even fetched initially when the superproject\nwas cloned and later was \"submodule init\"ed and fetched with\ndepth=1, would have the same shallow boundary, but the intent of the\nuser would clearly be different in the larger picture. I imagined\nthat your \"run in top-level and recurse to fetch in submodules\"\ntools would benefit if it has more information to intuit what the\nend user meant.\n"},{"id":"280725","messageId":"CAGZ79kb731RzLco_thuucTokLCe4xNMbEod9C=gWG+P+ruih7g@mail.gmail.com","threadId":"41674","inReplyTo":"xmqq60wra40h.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC/PATCH] clone: add `--shallow-submodules` flag","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-03-14T18:17:22Z","receivedAt":"2016-03-14T18:17:22Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sat, Mar 12, 2016 at 11:29 AM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> Why is it interesting for submodules but not for standard repositories?\n>>\n>> If I clone a repository without submodules, it is also not recorded\n>> that I cloned with an explicit depth=1. If you fetch, you may end up with\n>> a deeper history as git fetch doesn't do a \"reshallow\" to the configured\n>> depth.\n>\n> Very simple.\n>\n> If you do not have submodule, you would always interact with the\n> other side directly with \"git fetch\" or \"git pull\" and have total\n> control over when you choose to pass or not to pass extra options to\n> choose to 1. incrementally extend, 2. deepen, or 3. unshallow. The\n> user will always explicitly tell you, and knowing how you got there\n> would not help you, as there is no need to guess for you.\n\nBut the 1. being the default when no options are given, would fit into\nthe story of submodule treatment here, too?\n\nSay you have run\n\n $ git clone --recurse-submodules --shallow-submodules --depth 42\n\nand later\n\n $ git fetch --recurse-submodules\n\nyou'd fetch as usual without moving the anchor point. You could have\noptions like\n\n $ git fetch --recurse-submodules[=label/pattern]\n--unshallow-submodules[=label/pattern]\n\nSo first a switch for submodule behavior during fetch and that switch could\nbe narrowed down to specific submodules only.\n\n\n>\n> The user can do the same explicit \"cd dir && git fetch\" update in\n> each submodule directory and give appropriate options to choose\n> among the three, but I have an impression that your recent work is\n> going in the direction of making commands that are run in the\n> superproject recurse into submodules that automatically fetches and\n> updates the history down there, discouraging users from working on\n> individual submodules.\n\nGlad you see the high level direction where the submodules are heading,\nI was just fixing the most obvious problems (as indicated by their existence\nin the \"repo\" tool).\n\nFor the workflow I would think you'd only operate in the supermodule for\nsynchronizing, e.g.\n\n $ (cd super_project && git pull --recurse-submodules)\n $ (cd super_project/submoduleA && $EDITOR && git add <...> && git\ncommit <...>)\n $ (cd super_project/submoduleB && $EDITOR && git add <...> && git\ncommit <...>)\n # now it becomes a Gerrit specific workflow; no need to commit\nsubmodule changes\n # to the super project, (but it may not do harm?)\n $ (cd super_project && git push --recurse-submodules\n--submodules-only origin HEAD:refs/for/master)\n\nThe last command pushes submodules only as Gerrit will perform the\nsuperproject update\non your behalf. This is needed as submodules are treated as binary\nfiles, i.e. merging\ndiverged submodules is hard. So to avoid diverging submodules, Gerrit\ncan do that for you.\n\nAnother workflow could be to improve the merge algorithm for\nsubmodules, such that\nyou can specify how they should be integrated (merged when no conflict\nin the submodules\noccurs; or rebase the commits in the submodules, altering the commit\nin the superproject.)\n\n> You lose the flexibility to explicitly\n> choose among the three for individual submodules, and you may want\n> to have some smart in your \"run from the superproject and recurse\"\n> tools.\n\nBut that smart can also come from commandline options (or configuration in\nthe super project). So you would suggest to configure the superproject\nto a certain behavior when you clone with submodules in a certain way?\n\n>\n> A submodule that was initially cloned with depth=1, perhaps because\n> the user didn't know if the module was interesting to her in the\n> context of working on the superproject before she had her clone of\n> the superproject hence she only wanted to see what's there, and a\n> submodule that was not even fetched initially when the superproject\n> was cloned and later was \"submodule init\"ed and fetched with\n> depth=1, would have the same shallow boundary, but the intent of the\n> user would clearly be different in the larger picture. I imagined\n> that your \"run in top-level and recurse to fetch in submodules\"\n> tools would benefit if it has more information to intuit what the\n> end user meant.\n\nSo the first one should be shallowed after fetch, but the second would\nfetch or even deepen automatically?\n\nI am not sure if we add too much \"intuitive magic\" here as it would be\nhard to explain why that happens?\n"},{"id":"280728","messageId":"xmqqh9g8aosl.fsf@gitster.mtv.corp.google.com","threadId":"41674","inReplyTo":"CAGZ79kb731RzLco_thuucTokLCe4xNMbEod9C=gWG+P+ruih7g@mail.gmail.com","subject":"Re: [RFC/PATCH] clone: add `--shallow-submodules` flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-14T18:37:46Z","receivedAt":"2016-03-14T18:37:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> you'd fetch as usual without moving the anchor point. You could have\n> options like\n>\n> $ git fetch --recurse-submodules[=label/pattern]\n> --unshallow-submodules[=label/pattern]\n> ...\n> So the first one should be shallowed after fetch, but the second would\n> fetch or even deepen automatically?\n>\n> I am not sure if we add too much \"intuitive magic\" here as it would be\n> hard to explain why that happens?\n\nAll of these are things that need to be thought about when you are\nmaking more things recurse into submodules, as you would want to\nmake sure people can do different things to each of the submodule\nthey have.\n\nAnd if my comment to an RFC/PATCH made you think about them, it\nserved its purpose. I didn't mean to say \"You must implement the\nsmart from the beginning\"--I just meant to say that users will\nexpect more from the recursive behaviour and you must be prepared\nfor it.\n"}]}