{"thread":{"id":"30294","subject":"git clone submodules recursive and reference","startedAt":"2012-04-20T15:12:29Z","lastAt":"2012-06-29T20:19:35Z","messageCount":7,"participants":["Samuel Maftoul","Jens Lehmann","Phil Hord"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"189761","messageId":"CAFj+z055ZZc-Am+pEMgjRvHgoxKSRBjnHhABCseC45+8YswKWg@mail.gmail.com","threadId":"30294","inReplyTo":null,"subject":"git clone submodules recursive and reference","fromName":"Samuel Maftoul","fromEmail":"samuel.maftoul@gmail.com","sentAt":"2012-04-20T15:12:29Z","receivedAt":"2012-04-20T15:12:29Z","isPatch":false,"sender":{"key":"samuel.maftoul@gmail.com","avatar":"https://avatars.githubusercontent.com/u/239203?v=4"},"body":"Hello,\n\nI'm using git clone --reference, it works like a charm !\n\nNow, I have submodules, so I call git clone with both --recursive and\n--reference, works only for the repo itself, submodules are being\ncloned without the \"--reference\" option.\n\nWith GIT_TRACE=1 I can see this for the initial repo:\n\ntrace: built-in: git 'clone' '--recursive' '--reference' [...]\n\nAnd this for the submodules:\n\ntrace: built-in: git 'clone' '-n' [...]\n\nfor every submodules.\n\nIs this an intended behavior ?\n\nHow can I force the clones for submodules to be executed with the\n--reference option ?\n\nI tried to wrap /usr/lib/git-core/git-clone, but GIT_TRACE states it,\nit's a builtin and indeed my wrapped version of git clone is not\nexecuted !\n\nThanks\n\n--\n\nSamuel MAFTOUL\n"},{"id":"189769","messageId":"4F91B22A.9000507@web.de","threadId":"30294","inReplyTo":"CAFj+z055ZZc-Am+pEMgjRvHgoxKSRBjnHhABCseC45+8YswKWg@mail.gmail.com","subject":"Re: git clone submodules recursive and reference","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-04-20T18:59:54Z","receivedAt":"2012-04-20T18:59:54Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 20.04.2012 17:12, schrieb Samuel Maftoul:\n> Hello,\n> \n> I'm using git clone --reference, it works like a charm !\n> \n> Now, I have submodules, so I call git clone with both --recursive and\n> --reference, works only for the repo itself, submodules are being\n> cloned without the \"--reference\" option.\n> \n> With GIT_TRACE=1 I can see this for the initial repo:\n> \n> trace: built-in: git 'clone' '--recursive' '--reference' [...]\n> \n> And this for the submodules:\n> \n> trace: built-in: git 'clone' '-n' [...]\n> \n> for every submodules.\n> \n> Is this an intended behavior ?\n\nHmm, to me it looks like passing the --reference option to the clone\nrun in the submodules doesn't make much sense, as that would make\nall submodules and the superproject use the same alternates. And as\nfar as I know sharing objects between different repositories is not\nsupported.\n\n> How can I force the clones for submodules to be executed with the\n> --reference option ?\n\nYou'd have to use \"git clone\" without the --recursive option and\nthen do a \"git submodule update --init --reference ...\".\n"},{"id":"189770","messageId":"CAFj+z04A5v7Cz=Wbqn_TBJQG88rPSfrs4T1=22x1N+v77ZXgYA@mail.gmail.com","threadId":"30294","inReplyTo":"4F91B22A.9000507@web.de","subject":"Re: git clone submodules recursive and reference","fromName":"Samuel Maftoul","fromEmail":"samuel.maftoul@gmail.com","sentAt":"2012-04-20T19:26:22Z","receivedAt":"2012-04-20T19:26:22Z","isPatch":false,"sender":{"key":"samuel.maftoul@gmail.com","avatar":"https://avatars.githubusercontent.com/u/239203?v=4"},"body":"> Hmm, to me it looks like passing the --reference option to the clone\n> run in the submodules doesn't make much sense, as that would make\n> all submodules and the superproject use the same alternates. And as\n> far as I know sharing objects between different repositories is not\n> supported.\n\nI'm sharing objects between repositories by creating a bare\nrepository, adding the remotes for the repositories and fetch them in\nthis bare repo.\nSo for me, it makes sense to pass the \"--reference\" to the submodules\nclone, if submodules remotes are added to this reference bare repo and\nobjects are already fetched (and I'm in this case, as I use a lot of\ndifferent projects that shares the same set of submodules).\n\n>\n>> How can I force the clones for submodules to be executed with the\n>> --reference option ?\n>\n> You'd have to use \"git clone\" without the --recursive option and\n> then do a \"git submodule update --init --reference ...\".\n\nYes, this should make it, but I would have been more happy with a\nsingle command !\n\nThanks Jens !\n"},{"id":"189806","messageId":"4F92BA0C.4030009@web.de","threadId":"30294","inReplyTo":"CAFj+z04A5v7Cz=Wbqn_TBJQG88rPSfrs4T1=22x1N+v77ZXgYA@mail.gmail.com","subject":"Re: git clone submodules recursive and reference","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-04-21T13:45:48Z","receivedAt":"2012-04-21T13:45:48Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 20.04.2012 21:26, schrieb Samuel Maftoul:\n>> Hmm, to me it looks like passing the --reference option to the clone\n>> run in the submodules doesn't make much sense, as that would make\n>> all submodules and the superproject use the same alternates. And as\n>> far as I know sharing objects between different repositories is not\n>> supported.\n\nI take that back, I was thinking about the idea to store the objects\nof all submodules in the superproject's object store and then access\nthem via alternates which was discussed some time ago. That won't\nwork out of the box because the submodule commits would be dangling\nin the superprojects repo.\n\n> I'm sharing objects between repositories by creating a bare\n> repository, adding the remotes for the repositories and fetch them in\n> this bare repo.\n\nThis sounds like a cool way to reduce the disk footprint of the\nrepos on our Jenkins server.\n\n> So for me, it makes sense to pass the \"--reference\" to the submodules\n> clone, if submodules remotes are added to this reference bare repo and\n> objects are already fetched (and I'm in this case, as I use a lot of\n> different projects that shares the same set of submodules).\n\nHow do you fetch then, do you fetch into the referenced repo first\nand then do a fetch in the clones afterwards to just update the refs\nthere? Or is the bare repo just a starting point for the initial\nclone?\n\n>>> How can I force the clones for submodules to be executed with the\n>>> --reference option ?\n>>\n>> You'd have to use \"git clone\" without the --recursive option and\n>> then do a \"git submodule update --init --reference ...\".\n> \n> Yes, this should make it, but I would have been more happy with a\n> single command !\n\nHmm, me thinks we'd have to add a new option for that, and I'm not\nsure it is worth it.\n"},{"id":"189849","messageId":"CAFj+z05G_LLLc=OqZiqKCJPpTZ21Y4W6HTJ6ZitraVZXEQ50-A@mail.gmail.com","threadId":"30294","inReplyTo":"4F92BA0C.4030009@web.de","subject":"Re: git clone submodules recursive and reference","fromName":"Samuel Maftoul","fromEmail":"samuel.maftoul@gmail.com","sentAt":"2012-04-23T08:06:41Z","receivedAt":"2012-04-23T08:06:41Z","isPatch":false,"sender":{"key":"samuel.maftoul@gmail.com","avatar":"https://avatars.githubusercontent.com/u/239203?v=4"},"body":">> I'm sharing objects between repositories by creating a bare\n>> repository, adding the remotes for the repositories and fetch them in\n>> this bare repo.\n>\n> This sounds like a cool way to reduce the disk footprint of the\n> repos on our Jenkins server.\n\nI'm not using --reference for reducing disk footprint, but rather for\ncaching git repos and reducing the impact of slow networks !\nWhy would it reduce the disk footprint ?\n\n>\n>> So for me, it makes sense to pass the \"--reference\" to the submodules\n>> clone, if submodules remotes are added to this reference bare repo and\n>> objects are already fetched (and I'm in this case, as I use a lot of\n>> different projects that shares the same set of submodules).\n>\n> How do you fetch then, do you fetch into the referenced repo first\n> and then do a fetch in the clones afterwards to just update the refs\n> there? Or is the bare repo just a starting point for the initial\n> clone?\n\nYou need to fetch first in the bare repo, than in your clones. When\nyou use --reference, the reference leaves untouched, it's your job to\nupdate the reference (would be nice to have options that allows to\nupdate the reference at the same time that the clone updates, so no\nneed to connect twice to the remote repository).\n\n> Hmm, me thinks we'd have to add a new option for that, and I'm not\n> sure it is worth it.\n\nMaybe it's not worth ...\n"},{"id":"189932","messageId":"4F95C786.40901@web.de","threadId":"30294","inReplyTo":"CAFj+z05G_LLLc=OqZiqKCJPpTZ21Y4W6HTJ6ZitraVZXEQ50-A@mail.gmail.com","subject":"Re: git clone submodules recursive and reference","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-04-23T21:20:06Z","receivedAt":"2012-04-23T21:20:06Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 23.04.2012 10:06, schrieb Samuel Maftoul:\n>>> I'm sharing objects between repositories by creating a bare\n>>> repository, adding the remotes for the repositories and fetch them in\n>>> this bare repo.\n>>\n>> This sounds like a cool way to reduce the disk footprint of the\n>> repos on our Jenkins server.\n> \n> I'm not using --reference for reducing disk footprint, but rather for\n> caching git repos and reducing the impact of slow networks !\n> Why would it reduce the disk footprint ?\n\nBecause the object store of the referenced repo is reused, the cloned\n.git directory takes up less space (I think that is why the man page\ntalks about \"reducing network and local storage costs\").\n\n>>> So for me, it makes sense to pass the \"--reference\" to the submodules\n>>> clone, if submodules remotes are added to this reference bare repo and\n>>> objects are already fetched (and I'm in this case, as I use a lot of\n>>> different projects that shares the same set of submodules).\n>>\n>> How do you fetch then, do you fetch into the referenced repo first\n>> and then do a fetch in the clones afterwards to just update the refs\n>> there? Or is the bare repo just a starting point for the initial\n>> clone?\n> \n> You need to fetch first in the bare repo, than in your clones. When\n> you use --reference, the reference leaves untouched, it's your job to\n> update the reference (would be nice to have options that allows to\n> update the reference at the same time that the clone updates, so no\n> need to connect twice to the remote repository).\n\nI think you could also just fetch in the cloned repos, but then you'll\nhave to download the objects over the network for each clone and also\nwon't share the new objects. So I think your approach makes lots of\nsense, now I'll just have to tune our Jenkins scripts a bit. ;-)\n"},{"id":"194447","messageId":"CABURp0rhsq8at8=8yDgDr4nt1Y4_t=yR=9dzKFpK59ZE6A5mnQ@mail.gmail.com","threadId":"30294","inReplyTo":"4F91B22A.9000507@web.de","subject":"Re: git clone submodules recursive and reference","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2012-06-29T20:19:35Z","receivedAt":"2012-06-29T20:19:35Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Fri, Apr 20, 2012 at 2:59 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n> Am 20.04.2012 17:12, schrieb Samuel Maftoul:\n>> Hello,\n>>\n>> I'm using git clone --reference, it works like a charm !\n>>\n>> Now, I have submodules, so I call git clone with both --recursive and\n>> --reference, works only for the repo itself, submodules are being\n>> cloned without the \"--reference\" option.\n>>\n>> With GIT_TRACE=1 I can see this for the initial repo:\n>>\n>> trace: built-in: git 'clone' '--recursive' '--reference' [...]\n>>\n>> And this for the submodules:\n>>\n>> trace: built-in: git 'clone' '-n' [...]\n>>\n>> for every submodules.\n>>\n>> Is this an intended behavior ?\n>\n> Hmm, to me it looks like passing the --reference option to the clone\n> run in the submodules doesn't make much sense, as that would make\n> all submodules and the superproject use the same alternates. And as\n> far as I know sharing objects between different repositories is not\n> supported.\n\nSuppose we had an option for submodules which would use the _relative_\nsubmodule URL to augment the --reference path.  Would that be tenable,\nor do we need some extra option?\n\nOn my Jenkins server, I have a local mirror of my repos on my Gerrit\nserver.  I intentionally set the mirror paths up to match the layout\nof the Gerrit forest of repositories.  Thus, my relative URLs work for\nlocal clones from this mirror as well.\n\nCurrently I am doing something like this:\n\nmyrepo=/tmp/test-mirror1\nmirror=/var/lib/jenkins/mirror/superproject.git\nremote=gerrit:superproject.git\n\n#-- Clone fresh from local mirrors\ngit clone --recursive ${mirror} ${myrepo}\ncd ${myrepo}\n\n#-- Switch to the remote server URL\ngit config remote.origin.url ${remote}\ngit submodule sync\n\n#-- Checkout remote updates\ngit pull --ff-only --recurse-submodules origin\ngit submodule update\n\n\nIn my tests, this is about twice as fast[*1*] doing it the normal way:\n\ngit clone --recursive ${remote} ${myrepo}\n\n\nBut I would like to just do it like this:\n\ngit clone --reference=${mirror} --recursive ${remote}  ${myrepo}\n\nIt would be silly for all the submodules to use that reference as-is,\nexcept in the weird case where you've pull the remote objects from\nseveral repositories into one bare container.  I would argue we don't\nneed to support the weird case at all, or only with some --weirdo\nswitch added on.  Except there is already precedent for doing this\n\"the weird way\":\n\ngit submodule update --recursive --reference=/var/mirror/foo\n\nThere's nothing there that stops the the absolute reference being used\nfor each submodule, and that's glaringly discongruent to my proposed\naddition to git-clone.  So I can think of two ways to move forward\nwith the relative-reference idea.\n\n1. Use another switch to turn this behavior on\n    git clone --reference=${mirror} \\\n              --submodule-relative-references \\\n              --recursive ${remote} ${myrepo}\n\n2. Use a different switch name for 'reference':\n    git clone --reference-forest=${mirror} \\\n              --recursive ${remote} ${myrepo}\n\nDoes someone have an opinion to guide me?\n\n>> How can I force the clones for submodules to be executed with the\n>> --reference option ?\n>\n> You'd have to use \"git clone\" without the --recursive option and\n> then do a \"git submodule update --init --reference ...\".\n\nOr the way I did it, but my method has its flaws.\n\nPhil\n\n[*1*] While it is twice as fast, it is subject to some fatal errors if\nmy mirrored repos are out of sync. That is, if a mirrored submodule\ndoes not include the commit which is gitlinked from the\nsuper-project's HEAD, then the original recursive clone will fail;\nthen I lose my speedup and possibly error out as well.  --references\ndoes not suffer this same malady.\n"}]}