{"thread":{"id":"40771","subject":"[RFC] Clone repositories recursive with depth 1","startedAt":"2015-11-11T14:09:18Z","lastAt":"2015-11-14T16:25:15Z","messageCount":6,"participants":["Lars Schneider","Stefan Beller","Fredrik Gustafsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"273190","messageId":"DE5B8B46-B185-4258-A1C8-07E46072CD5D@gmail.com","threadId":"40771","inReplyTo":null,"subject":"[RFC] Clone repositories recursive with depth 1","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-11-11T14:09:18Z","receivedAt":"2015-11-11T14:09:18Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Hi,\n\nI have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:\n\ngit clone --recursive --depth 1 --single-branch \n\nApparently this does not clone the submodules with \"--depth 1\" (using Git 2.4.9). As a workaround I tried:\n\ngit clone --depth 1 --single-branch \ncd \ngit submodule update --init --recursive --depth 1\n\nHowever, this does not work either as I get:\nfatal: reference is not a tree: \nUnable to checkout in submodule path \n\nHow would you clone the repo? Is the behavior above expected? If not, should the \"--depth 1\" flag be applied recursively to all submodules on a clone --recursive? Has a patch implementing this a chance to get in?\n\nThanks,\nLars"},{"id":"273205","messageId":"CAGZ79kbfFhCvQW=_7i4KxjWeh7uYRTTNvLzQUq+CJ641g3=UDg@mail.gmail.com","threadId":"40771","inReplyTo":"DE5B8B46-B185-4258-A1C8-07E46072CD5D@gmail.com","subject":"Re: [RFC] Clone repositories recursive with depth 1","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-11T19:19:15Z","receivedAt":"2015-11-11T19:19:15Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider\n wrote:\n> Hi,\n>\n> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:\n>\n> git clone --recursive --depth 1 --single-branch \n\nThat *should* work, actually.\nHowever looking at the code it does not.\n\nciting from builtin/clone.c:\n\n static struct option builtin_clone_options[] = {\n ...\n OPT_BOOL(0, \"recursive\", &option_recursive,\n N_(\"initialize submodules in the clone\")),\n OPT_BOOL(0, \"recurse-submodules\", &option_recursive,\n N_(\"initialize submodules in the clone\")),\n ...\n };\n ...\n static const char *argv_submodule[] = {\n \"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n };\n\n if (!err && option_recursive)\n err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n\nSo the --depth argument is not passed on, although \"git submodule update\"\ndefinitely supports --depth.\n\nIn an upcoming series (next version of origin/sb/submodule-parallel-update),\nthis will slightly change, such it will be even easier to add the\ndepth argument in\nthere as we construct the argument list in code instead of hard coding\nargv_submodule.\n\nThis may require some discussion whether you expect --depth to be recursed.\n(What if you only want a top level shallow thing?, What if you want to have only\nsubmodules shallow? What is the user expectation here?)\n\n>\n> Apparently this does not clone the submodules with \"--depth 1\" (using Git 2.4.9). As a workaround I tried:\n>\n> git clone --depth 1 --single-branch \n> cd \n> git submodule update --init --recursive --depth 1\n>\n> However, this does not work either as I get:\n> fatal: reference is not a tree: \n> Unable to checkout in submodule path \n\nThat seems like another bug to me.\n\nI just tried to clone a project and populate with submodules later and\nit works as expected without these error messages.\n(I am running some kind of xxx.dirty development version, most likely\norigin/sb/submodule-parallel-update,\nI'll check some other versions, too)\n"},{"id":"273221","messageId":"CAGZ79kazAWj+D85EXUyP9iHgcb=7QpymyaaKEqQRu9a+UO9msw@mail.gmail.com","threadId":"40771","inReplyTo":"CAGZ79kbfFhCvQW=_7i4KxjWeh7uYRTTNvLzQUq+CJ641g3=UDg@mail.gmail.com","subject":"Re: [RFC] Clone repositories recursive with depth 1","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-11T20:09:41Z","receivedAt":"2015-11-11T20:09:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 11, 2015 at 11:19 AM, Stefan Beller wrote:\n> On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider\n> wrote:\n>> Hi,\n>>\n>> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:\n>>\n>> git clone --recursive --depth 1 --single-branch \n>\n> That *should* work, actually.\n> However looking at the code it does not.\n>\n> citing from builtin/clone.c:\n>\n> static struct option builtin_clone_options[] = {\n> ...\n> OPT_BOOL(0, \"recursive\", &option_recursive,\n> N_(\"initialize submodules in the clone\")),\n> OPT_BOOL(0, \"recurse-submodules\", &option_recursive,\n> N_(\"initialize submodules in the clone\")),\n> ...\n> };\n> ...\n> static const char *argv_submodule[] = {\n> \"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n> };\n>\n> if (!err && option_recursive)\n> err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n>\n> So the --depth argument is not passed on, although \"git submodule update\"\n> definitely supports --depth.\n>\n> In an upcoming series (next version of origin/sb/submodule-parallel-update),\n> this will slightly change, such it will be even easier to add the\n> depth argument in\n> there as we construct the argument list in code instead of hard coding\n> argv_submodule.\n>\n> This may require some discussion whether you expect --depth to be recursed.\n> (What if you only want a top level shallow thing?, What if you want to have only\n> submodules shallow? What is the user expectation here?)\n>\n>>\n>> Apparently this does not clone the submodules with \"--depth 1\" (using Git 2.4.9). As a workaround I tried:\n>>\n>> git clone --depth 1 --single-branch \n>> cd \n>> git submodule update --init --recursive --depth 1\n>>\n\nThe workaround works with the origin/master version for me.\n\nNotice the other email thread, which suggests to include --remote into the\ncall to git submodule update depending on a branch config option being\npresent in the .gitmodules file.\n"},{"id":"273242","messageId":"19CE7136-E1FD-495A-9BAC-636164757E04@gmail.com","threadId":"40771","inReplyTo":"CAGZ79kazAWj+D85EXUyP9iHgcb=7QpymyaaKEqQRu9a+UO9msw@mail.gmail.com","subject":"Re: [RFC] Clone repositories recursive with depth 1","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-11-12T09:39:45Z","receivedAt":"2015-11-12T09:39:45Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 11 Nov 2015, at 21:09, Stefan Beller wrote:\n\n> On Wed, Nov 11, 2015 at 11:19 AM, Stefan Beller wrote:\n>> On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider\n>> wrote:\n>>> Hi,\n>>> \n>>> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:\n>>> \n>>> git clone --recursive --depth 1 --single-branch \n>> \n>> That *should* work, actually.\n>> However looking at the code it does not.\n>> \n>> citing from builtin/clone.c:\n>> \n>> static struct option builtin_clone_options[] = {\n>> ...\n>> OPT_BOOL(0, \"recursive\", &option_recursive,\n>> N_(\"initialize submodules in the clone\")),\n>> OPT_BOOL(0, \"recurse-submodules\", &option_recursive,\n>> N_(\"initialize submodules in the clone\")),\n>> ...\n>> };\n>> ...\n>> static const char *argv_submodule[] = {\n>> \"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n>> };\n>> \n>> if (!err && option_recursive)\n>> err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n>> \n>> So the --depth argument is not passed on, although \"git submodule update\"\n>> definitely supports --depth.\n>> \n>> In an upcoming series (next version of origin/sb/submodule-parallel-update),\n>> this will slightly change, such it will be even easier to add the\n>> depth argument in\n>> there as we construct the argument list in code instead of hard coding\n>> argv_submodule.\n>> \n>> This may require some discussion whether you expect --depth to be recursed.\n>> (What if you only want a top level shallow thing?, What if you want to have only\n>> submodules shallow? What is the user expectation here?)\n>> \n>>> \n>>> Apparently this does not clone the submodules with \"--depth 1\" (using Git 2.4.9). As a workaround I tried:\n>>> \n>>> git clone --depth 1 --single-branch \n>>> cd \n>>> git submodule update --init --recursive --depth 1\n>>> \n> \n> The workaround works with the origin/master version for me.\n> \n> Notice the other email thread, which suggests to include --remote into the\n> call to git submodule update depending on a branch config option being\n> present in the .gitmodules file.\n\nCan you check \"[PATCH v2] add test to demonstrate that shallow recursive clones fail\"? This demonstrates the failure that I see. I also tried the \"--remote\" flag but this does not work either (see test case).\nCan you confirm this behavior?\n\nCheers,\nLars"},{"id":"273249","messageId":"CAGZ79kYNCWXVLWAzTCoFfuHGRkJzzUsojGzK9-f=543FVfVMAg@mail.gmail.com","threadId":"40771","inReplyTo":"19CE7136-E1FD-495A-9BAC-636164757E04@gmail.com","subject":"Re: [RFC] Clone repositories recursive with depth 1","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-12T23:47:44Z","receivedAt":"2015-11-12T23:47:44Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"+cc Stanislav, who came up with the other thread for passing --remote\nto git submodule\n\nOn Thu, Nov 12, 2015 at 1:39 AM, Lars Schneider\n wrote:\n>> Notice the other email thread, which suggests to include --remote into the\n>> call to git submodule update depending on a branch config option being\n>> present in the .gitmodules file.\n>\n> Can you check \"[PATCH v2] add test to demonstrate that shallow recursive clones fail\"? This demonstrates the failure that I see. I also tried the \"--remote\" flag but this does not work either (see test case).\n> Can you confirm this behavior?\n>\n> Cheers,\n> Lars\n\nI can confirm it breaks as expected here.\n\nI may have confused you here by pointing to the --remote option.\n\n(git clone is a bit stupid when it comes to submodule handling.)\nAll it does currently is this:\n\n if --recurseSubmodules option or --recursive option is given:\n run: \"git submodule update --init --recursive\"\n\nNo attention is paid to any other option such as --depth.\nThat's all I wanted to point out there.\n\nIdeally we want to add:\n\n If there is a branch configured in the .gitmodules file,\n we would want to add the --remote command\n\n if we have given other options such as --depth or --reference\n we want to pass that along to the called submodule helper.\n\nSo I was looking at the internal code structure and think one of the next\nseries I am going to send will touch the code such that we can incorporate\nthe conditions as outlined above easier, because it is not hardcoded into an\narray [\"git\", \"submodule\", \"update\" \"--init\", \"--recursive\"], as I\nwant to add yet\nanother dynamic option to the submodule helper invocation. (I want to add\n--jobs there)\n\nCheers,\nStefan\n"},{"id":"273334","messageId":"20151114162515.GJ1678@paksenarrion.iveqy.com","threadId":"40771","inReplyTo":"DE5B8B46-B185-4258-A1C8-07E46072CD5D@gmail.com","subject":"Re: [RFC] Clone repositories recursive with depth 1","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2015-11-14T16:25:15Z","receivedAt":"2015-11-14T16:25:15Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Wed, Nov 11, 2015 at 03:09:18PM +0100, Lars Schneider wrote:\n> Apparently this does not clone the submodules with \"--depth 1\" (using Git 2.4.9). As a workaround I tried:\n> \n> git clone --depth 1 --single-branch \n> cd \n> git submodule update --init --recursive --depth 1\n> \n> However, this does not work either as I get:\n> fatal: reference is not a tree: \n> Unable to checkout in submodule path \n\nThis looks like a familiar bug to me. I'm not sure if it's a bug or a\nknown behaviour. When the depth argument was introduced to submodules a\nyear (or two) ago there was a know bug. I not sure if it's fixed or\nnot.\n\nThe problem is that git is/was unable to fetch a sha1 but only a branch\nor a tag. So fetching a submodule will fetch the HEAD of the submodule\nwith the requested depth. Then git will try to checkout a sha1 of that\nsubmodule, which may or maynot exists.\n\nSay that you fetch master of a submodule of depth 1 and do a checkout of\na commit that exists. When someonen else has pushed to that submodule,\nthe commit will not longer be reachable from depth 1 and if someone else\ntries to clone with depth 1 it will fail with the same error message as\nyou got.\n\nThe solution to this is to allow git to fetch the sha1 the superproject\npoints to direct when fetching the submodule.\n\n-- \nFredrik Gustafsson\n\nphone: +46 733-608274\ne-mail: iveqy@iveqy.com\nwebsite: http://www.iveqy.com\n"}]}