{"thread":{"id":"45593","subject":"Bug? git submodule update --reference doesn't use the referenced repository","startedAt":"2017-04-03T03:13:16Z","lastAt":"2017-04-04T02:15:41Z","messageCount":3,"participants":["Maxime Viargues","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"316079","messageId":"35343b75-0aa7-3477-888b-3af5024ae7dd@serato.com","threadId":"45593","inReplyTo":null,"subject":"Bug? git submodule update --reference doesn't use the referenced repository","fromName":"Maxime Viargues","fromEmail":"maxime.viargues@serato.com","sentAt":"2017-04-03T03:13:07Z","receivedAt":"2017-04-03T03:13:16Z","isPatch":false,"sender":{"key":"maxime.viargues@serato.com","avatar":null},"body":"Hi there,\n\nI have been trying to use the --reference option to clone a big \nrepository using a local copy, but I can't manage to make it work using \nsub-module update. I believe this is a bug, unless I missed something.\nI am on Windows, Git 2.12.0\n\nSo the problem is as follow:\n- I have got a repository with multiple sub-modules, say\n     main\n         lib1\n             sub-module1.git\n         lib2\n             sub-module2.git\n- The original repositories are in GitHub, which makes it slow\n- I have done a normal git clone of the entire repository (not bare) and \nput it on a file server, say \\\\fileserver\\ref_repo\\\n(Note that the problem also happens with local copy)\n\nSo if I do a clone to get the repo and all the submodules with...\ngit clone --reference-if-able \\\\fileserver\\ref-repo --recursive \ngit@github.com:company/main\n...then it all works, all the sub-modules get cloned and the it's fast.\n\nNow in my case I am working with Jenkins jobs and I need to first do a \nclone, and then get the sub-modules, but if I do...\ngit clone --reference-if-able \\\\fileserver\\ref-repo \ngit@github.com:company/main (so non-recursive)\ncd main\ngit submodule update --init --reference \\\\fileserver\\ref-repo\n... then this takes ages, as it would normally do without the use of \n--reference. I suspect it's not actually using it.\nThe git clone documentation mentions that the reference is then passed \nto the sub-module clone commands, so I would expect \"git clone \n--recursive\" to work the same as \"git submodule update\", as far as \n--reference is concerned.\n\nI noticed for a single module, doing a...\ngit submodule update --init --reference \n\\\\fileserver\\ref-repo\\lib1\\sub-module1 -- lib1/sub-module1\n...i.e. adding the sub-module path to the reference path, works. Which \nkind of make sense but then how do you do to apply it to all the \nsub-modules? (without writing a script to do that)\n\nIf someone can confirm the problem or explain me what I am dong wrong \nthat would be great.\n\nMaxime\n"},{"id":"316096","messageId":"CAGZ79kYKfKcVcV9ddRHNc-aQ7RpN4ToDs9i=voqXcKTm79j3KA@mail.gmail.com","threadId":"45593","inReplyTo":"35343b75-0aa7-3477-888b-3af5024ae7dd@serato.com","subject":"Re: Bug? git submodule update --reference doesn't use the referenced repository","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-04-03T16:32:50Z","receivedAt":"2017-04-03T16:32:56Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Apr 2, 2017 at 8:13 PM, Maxime Viargues\n<maxime.viargues@serato.com> wrote:\n> Hi there,\n>\n> I have been trying to use the --reference option to clone a big repository\n> using a local copy, but I can't manage to make it work using sub-module\n> update. I believe this is a bug, unless I missed something.\n> I am on Windows, Git 2.12.0\n\nwhich is new enough, that the new --reference code is in. :)\n\n>\n> So the problem is as follow:\n> - I have got a repository with multiple sub-modules, say\n>     main\n>         lib1\n>             sub-module1.git\n>         lib2\n>             sub-module2.git\n> - The original repositories are in GitHub, which makes it slow\n> - I have done a normal git clone of the entire repository (not bare) and put\n> it on a file server, say \\\\fileserver\\ref_repo\\\n> (Note that the problem also happens with local copy)\n>\n> So if I do a clone to get the repo and all the submodules with...\n> git clone --reference-if-able \\\\fileserver\\ref-repo --recursive\n> git@github.com:company/main\n> ...then it all works, all the sub-modules get cloned and the it's fast.\n\ngreat. :)\n\n> Now in my case I am working with Jenkins jobs and I need to first do a\n> clone, and then get the sub-modules, but if I do...\n> git clone --reference-if-able \\\\fileserver\\ref-repo\n> git@github.com:company/main (so non-recursive)\n> cd main\n> git submodule update --init --reference \\\\fileserver\\ref-repo\n> ... then this takes ages, as it would normally do without the use of\n> --reference. I suspect it's not actually using it.\n\nSo to confirm your suspicion, can you run\n\n  GIT_TRACE=1 git clone ...\n  cd main && GIT_TRACE=1 git submodule update ...\n\nto see which child processes are spawned to deal with the submodules?\nAlso to confirm, it is the \"submodule update\" that is taking so long for you?\n\n> The git clone documentation mentions that the reference is then passed to\n> the sub-module clone commands, so I would expect \"git clone --recursive\" to\n> work the same as \"git submodule update\", as far as --reference is concerned.\n\nOh, there we have an opportunity to improve the man page (or the code).\n\n    git clone --reference --recursive ...\n\nwill set the config variables\n\n    git config submodule.alternateLocation superproject\n    git config submodule.alternateErrorStrategy die (or \"info\" for\n--reference-if-able)\n\nand the clone for the submodules (that are an independent process, just\nrun after the clone of the superproject is done) will pickup these\nconfig variables\nand act accordingly.\n\nIf you only run\n\n    git clone --reference ...\n\nthen these variables are not set. Probably they should be set such\nthat the later\ninvocation of \"git submodule update --int\" will behave the same as the git-clone\nof the superproject did.\n\nSo as a workaround for you to get up to speed again, you can just set\nthese config\nvariables yourself before running the \"submodule update --init\" and it\nshould work.\n\n>\n> I noticed for a single module, doing a...\n> git submodule update --init --reference\n> \\\\fileserver\\ref-repo\\lib1\\sub-module1 -- lib1/sub-module1\n> ...i.e. adding the sub-module path to the reference path, works. Which kind\n> of make sense but then how do you do to apply it to all the sub-modules?\n> (without writing a script to do that)\n\nI think that functionality is broken as it takes the same reference\nfor all submodules,\nsuch that you need to go through the submodules one by one and give the\nsubmodule specific reference location.\n\n>\n> If someone can confirm the problem or explain me what I am dong wrong that\n> would be great.\n>\n> Maxime\n\nStefan\n"},{"id":"316126","messageId":"f7c100e2-da76-57e3-1fbc-f71fd57f26eb@serato.com","threadId":"45593","inReplyTo":"CAGZ79kYKfKcVcV9ddRHNc-aQ7RpN4ToDs9i=voqXcKTm79j3KA@mail.gmail.com","subject":"Re: Bug? git submodule update --reference doesn't use the referenced repository","fromName":"Maxime Viargues","fromEmail":"maxime.viargues@serato.com","sentAt":"2017-04-04T02:15:30Z","receivedAt":"2017-04-04T02:15:41Z","isPatch":false,"sender":{"key":"maxime.viargues@serato.com","avatar":null},"body":"On 04-Apr-17 4:32 AM, Stefan Beller wrote:\n> On Sun, Apr 2, 2017 at 8:13 PM, Maxime Viargues\n> <maxime.viargues@serato.com> wrote:\n>> Hi there,\n>>\n>> I have been trying to use the --reference option to clone a big repository\n>> using a local copy, but I can't manage to make it work using sub-module\n>> update. I believe this is a bug, unless I missed something.\n>> I am on Windows, Git 2.12.0\n> which is new enough, that the new --reference code is in. :)\n>\n>> So the problem is as follow:\n>> - I have got a repository with multiple sub-modules, say\n>>      main\n>>          lib1\n>>              sub-module1.git\n>>          lib2\n>>              sub-module2.git\n>> - The original repositories are in GitHub, which makes it slow\n>> - I have done a normal git clone of the entire repository (not bare) and put\n>> it on a file server, say \\\\fileserver\\ref_repo\\\n>> (Note that the problem also happens with local copy)\n>>\n>> So if I do a clone to get the repo and all the submodules with...\n>> git clone --reference-if-able \\\\fileserver\\ref-repo --recursive\n>> git@github.com:company/main\n>> ...then it all works, all the sub-modules get cloned and the it's fast.\n> great. :)\n>\n>> Now in my case I am working with Jenkins jobs and I need to first do a\n>> clone, and then get the sub-modules, but if I do...\n>> git clone --reference-if-able \\\\fileserver\\ref-repo\n>> git@github.com:company/main (so non-recursive)\n>> cd main\n>> git submodule update --init --reference \\\\fileserver\\ref-repo\n>> ... then this takes ages, as it would normally do without the use of\n>> --reference. I suspect it's not actually using it.\n> So to confirm your suspicion, can you run\n>\n>    GIT_TRACE=1 git clone ...\n>    cd main && GIT_TRACE=1 git submodule update ...\n>\n> to see which child processes are spawned to deal with the submodules?\n> Also to confirm, it is the \"submodule update\" that is taking so long for you?\nYes I confirm it's the \"submodule update\" which is taking a long time. \nThe clone with the reference is definitely working.\n\nRunning git submodule update with  \"GIT_TRACE=1\", here is a snippet of \nwhat I get:\n\n10:14:44.924684 git.c:596               trace: exec: 'git-submodule' \n'update' '--init' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:14:44.925684 run-command.c:369       trace: run_command: \n'git-submodule' 'update' '--init' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:14:45.146488 git.c:596               trace: exec: \n'git-sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'\n10:14:45.146488 run-command.c:369       trace: run_command: \n'git-sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'\n10:14:45.231548 git.c:596               trace: exec: \n'git-sh-i18n--envsubst' 'usage: $dashless $USAGE'\n10:14:45.231548 run-command.c:369       trace: run_command: \n'git-sh-i18n--envsubst' 'usage: $dashless $USAGE'\n10:14:45.357059 git.c:371               trace: built-in: git 'rev-parse' \n'--git-dir'\n10:14:45.427806 git.c:371               trace: built-in: git 'rev-parse' \n'--git-path' 'objects'\n10:14:45.487348 git.c:371               trace: built-in: git 'rev-parse' \n'-q' '--git-dir'\n10:14:45.593794 git.c:371               trace: built-in: git 'rev-parse' \n'--show-prefix'\n10:14:45.643162 git.c:371               trace: built-in: git 'rev-parse' \n'--show-toplevel'\n10:14:45.700201 git.c:371               trace: built-in: git \n'submodule--helper' 'init'\n10:14:45.986024 git.c:371               trace: built-in: git \n'submodule--helper' 'update-clone' \n'--reference=\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:14:45.988024 run-command.c:1155      run_processes_parallel: \npreparing to run up to 1 tasks\n10:14:45.988024 run-command.c:369       trace: run_command: \n'submodule--helper' 'clone' '--path' 'lib1/lib1_source' '--name' \n'lib1/lib1_source' '--url' 'git@github.com:company/main-repo-lib1.git' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:15:06.204872 run-command.c:369       trace: run_command: \n'submodule--helper' 'clone' '--path' 'lib2/lib2_source' '--name' \n'lib2/lib2_source' '--url' 'git@github.com:company/main-repo-lib2.git' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:14:46.025555 git.c:371               trace: built-in: git \n'submodule--helper' 'clone' '--path' 'lib1/lib1_source' '--name' \n'lib1/lib1_source' '--url' 'git@github.com:company/main-repo-lib1.git' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:14:46.027555 run-command.c:369       trace: run_command: 'clone' \n'--no-checkout' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo' '--separate-git-dir' \n'D:/tmp2/git_clone_tests/main-repo/.git/modules/lib1/lib1_source' \n'git@github.com:company/main-repo-lib1.git' \n'D:/tmp2/git_clone_tests/main-repo/lib1/lib1_source'\n10:14:46.061305 git.c:371               trace: built-in: git 'clone' \n'--no-checkout' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo' '--separate-git-dir' \n'D:/tmp2/git_clone_tests/main-repo/.git/modules/lib1/lib1_source' \n'git@github.com:company/main-repo-lib1.git' \n'D:/tmp2/git_clone_tests/main-repo/lib1/lib1_source'\n10:14:46.115339 run-command.c:369       trace: run_command: 'ssh' \n'git@github.com' 'git-upload-pack '\\''company/main-repo-lib1.git'\\'''\nCloning into 'D:/tmp2/git_clone_tests/main-repo/lib1/lib1_source'...\n10:14:48.962590 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''//fileserver/Builds/reference_repos/main-repo/.git'\\'''\n10:14:49.103908 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''D:/GitHub/main-repo/.git'\\'''\n10:14:49.184477 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''//fileserver/Builds/reference_repos/main-repo/.git'\\'''\n10:14:49.322365 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''D:/GitHub/main-repo/.git'\\'''\n10:14:52.281044 run-command.c:369       trace: run_command: 'index-pack' \n'--stdin' '--fix-thin' '--keep=fetch-pack 5764 on WIN-1198' \n'--check-self-contained-and-connected'\n10:14:52.315569 git.c:371               trace: built-in: git \n'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 5764 on WIN-1198' \n'--check-self-contained-and-connected'\n10:15:06.119340 run-command.c:369       trace: run_command: 'rev-list' \n'--objects' '--stdin' '--not' '--all' '--quiet'\n10:15:06.170876 git.c:371               trace: built-in: git 'rev-list' \n'--objects' '--stdin' '--not' '--all' '--quiet'\n10:15:13.072336 run-command.c:369       trace: run_command: \n'submodule--helper' 'clone' '--path' 'lib3/lib3_source' '--name' \n'lib3/lib3_source' '--url' 'git@github.com:company/main-repo-lib3' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:15:06.238893 git.c:371               trace: built-in: git \n'submodule--helper' 'clone' '--path' 'lib2/lib2_source' '--name' \n'lib2/lib2_source' '--url' 'git@github.com:company/main-repo-lib2.git' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:15:06.239894 run-command.c:369       trace: run_command: 'clone' \n'--no-checkout' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo' '--separate-git-dir' \n'D:/tmp2/git_clone_tests/main-repo/.git/modules/lib2/lib2_source' \n'git@github.com:company/main-repo-lib2.git' \n'D:/tmp2/git_clone_tests/main-repo/lib2/lib2_source'\n10:15:06.273418 git.c:371               trace: built-in: git 'clone' \n'--no-checkout' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo' '--separate-git-dir' \n'D:/tmp2/git_clone_tests/main-repo/.git/modules/lib2/lib2_source' \n'git@github.com:company/main-repo-lib2.git' \n'D:/tmp2/git_clone_tests/main-repo/lib2/lib2_source'\n10:15:06.309945 run-command.c:369       trace: run_command: 'ssh' \n'git@github.com' 'git-upload-pack '\\''company/main-repo-lib2.git'\\'''\nCloning into 'D:/tmp2/git_clone_tests/main-repo/lib2/lib2_source'...\n10:15:08.210491 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''//fileserver/Builds/reference_repos/main-repo/.git'\\'''\n10:15:08.370561 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''D:/GitHub/main-repo/.git'\\'''\n10:15:08.451234 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''//fileserver/Builds/reference_repos/main-repo/.git'\\'''\n10:15:08.589129 run-command.c:369       trace: run_command: \n'git-upload-pack '\\''D:/GitHub/main-repo/.git'\\'''\n10:15:11.533328 run-command.c:369       trace: run_command: 'index-pack' \n'--stdin' '--fix-thin' '--keep=fetch-pack 9308 on WIN-1198' \n'--check-self-contained-and-connected'\n10:15:11.575862 git.c:371               trace: built-in: git \n'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 9308 on WIN-1198' \n'--check-self-contained-and-connected'\n10:15:12.986776 run-command.c:369       trace: run_command: 'rev-list' \n'--objects' '--stdin' '--not' '--all' '--quiet'\n10:15:13.039314 git.c:371               trace: built-in: git 'rev-list' \n'--objects' '--stdin' '--not' '--all' '--quiet'\n10:15:49.633796 run-command.c:369       trace: run_command: \n'submodule--helper' 'clone' '--path' 'lib4/lib4_source' '--name' \n'lib4/lib4_source' '--url' 'git@github.com:company/main-repo-lib4.git' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:15:13.106441 git.c:371               trace: built-in: git \n'submodule--helper' 'clone' '--path' 'lib3/lib3_source' '--name' \n'lib3/lib3_source' '--url' 'git@github.com:company/main-repo-lib3' \n'--reference' '\\\\fileserver\\Builds\\reference_repos\\main-repo'\n10:15:13.107441 run-command.c:369       trace: run_command: 'clone' \n'--no-checkout' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo' '--separate-git-dir' \n'D:/tmp2/git_clone_tests/main-repo/.git/modules/lib3/lib3_source' \n'git@github.com:company/main-repo-lib3' \n'D:/tmp2/git_clone_tests/main-repo/lib3/lib3_source'\n10:15:13.141464 git.c:371               trace: built-in: git 'clone' \n'--no-checkout' '--reference' \n'\\\\fileserver\\Builds\\reference_repos\\main-repo' '--separate-git-dir' \n'D:/tmp2/git_clone_tests/main-repo/.git/modules/lib3/lib3_source' \n'git@github.com:company/main-repo-lib3' \n'D:/tmp2/git_clone_tests/main-repo/lib3/lib3_source'\n10:15:13.174486 run-command.c:369       trace: run_command: 'ssh' \n'git@github.com' 'git-upload-pack '\\''company/main-repo-lib3'\\'''\n...\n>\n>> The git clone documentation mentions that the reference is then passed to\n>> the sub-module clone commands, so I would expect \"git clone --recursive\" to\n>> work the same as \"git submodule update\", as far as --reference is concerned.\n> Oh, there we have an opportunity to improve the man page (or the code).\n>\n>      git clone --reference --recursive ...\n>\n> will set the config variables\n>\n>      git config submodule.alternateLocation superproject\n>      git config submodule.alternateErrorStrategy die (or \"info\" for\n> --reference-if-able)\n>\n> and the clone for the submodules (that are an independent process, just\n> run after the clone of the superproject is done) will pickup these\n> config variables\n> and act accordingly.\n>\n> If you only run\n>\n>      git clone --reference ...\n>\n> then these variables are not set. Probably they should be set such\n> that the later\n> invocation of \"git submodule update --int\" will behave the same as the git-clone\n> of the superproject did.\n>\n> So as a workaround for you to get up to speed again, you can just set\n> these config\n> variables yourself before running the \"submodule update --init\" and it\n> should work.\nOk I'll try that.\n>> I noticed for a single module, doing a...\n>> git submodule update --init --reference\n>> \\\\fileserver\\ref-repo\\lib1\\sub-module1 -- lib1/sub-module1\n>> ...i.e. adding the sub-module path to the reference path, works. Which kind\n>> of make sense but then how do you do to apply it to all the sub-modules?\n>> (without writing a script to do that)\n> I think that functionality is broken as it takes the same reference\n> for all submodules,\n> such that you need to go through the submodules one by one and give the\n> submodule specific reference location.\nI actually made a script to run it on each submodule, which works but is \nstill quite slow as it cannot be parallelized (git doesn't like multiple \nsubmodule updates running concurrently).\n>\n>> If someone can confirm the problem or explain me what I am dong wrong that\n>> would be great.\n>>\n>> Maxime\n> Stefan\nThanks for you quick answer\n\nMaxime\n"}]}