{"thread":{"id":"38188","subject":"Saving space/network on common repos","startedAt":"2014-12-17T06:58:05Z","lastAt":"2014-12-23T05:36:53Z","messageCount":9,"participants":["Craig Silverstein","Stefan Beller","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"253757","messageId":"CAGXKyzEYhR69w1=4q-xtBagVBwOPqNA9C=AD0bAorB+5eRtVRg@mail.gmail.com","threadId":"38188","inReplyTo":null,"subject":"Saving space/network on common repos","fromName":"Craig Silverstein","fromEmail":"csilvers@khanacademy.org","sentAt":"2014-12-17T06:58:05Z","receivedAt":"2014-12-17T06:58:05Z","isPatch":false,"sender":{"key":"csilvers@khanacademy.org","avatar":null},"body":"At Khan Academy, we are running a Jenkins installation as our build\nserver. By design, our Jenkins machine has several different\ndirectories that each hold a copy of the same git repository. (For\ninstance, Jenkins may be running tests on our repo at several\ndifferent commits at the same time.) When Jenkins decides to run a\ntest -- I'm simplifying a bit -- it will pick one of the copies of the\nrepo, do a 'git fetch origin && git checkout ' and the\nrun the tests.\n\nOur repo has a lot of churn and some big files, and this git fetch can\ntake a long time. I'd like to reduce both the time to fetch and the\ndisk space used by sharing objects between the repo copies.\n\nMy research has turned up three techniques that try to address this use case:\n* git clone --reference\n* git clone --shared\n* git clone , which creates hard links\n\nI can probably use any of these approaches, but git clone --reference\nwould be the easiest to set up. I would do so by creating a 'cache'\nrepo that is just created to serve as a reference and not used in any\nother way, so I wouldn't have to worry about the dangers with pruning,\naccidentally deleting the repo, etc.\n\nMy big concern is that all these methods seem to just affect clone. So:\n\nQuestion 1) If I do 'git clone --reference, will the reference repo be\nused for subsequent fetches as well? What about 'git clone --shared'?\n\nQuestion 2) If I git clone a local repo, will subsequent fetches also\ncreate hard links?\n\nQuestion 3) If the answer to any of the above is yes, how does this\nwork with packing? Say I pack the reference repo (being careful not\nto prune anything). Will subsequent fetches still be able to get the\nobjects they need from the reference repo?\n\nAn added complication is submodules. We have a submodule that is as\nbig and slow to fetch as our main repository.\n\nQuestion 4) Is there a practical way to set up submodules so they can\nuse the same object-sharing framework that the main repo does?\n\nI'm not keen on rewriting .gitmodules in each of my repos, so probably\nsomething that uses info/alternates is the most workable. I have a\nscheme for setting that up that maybe will work, but it's a moot point\nif info/alternates doesn't work for fetching.\n\nI'm wondering if the best approach for us might be to use\nGIT_OBJECT_DIRECTORY: set GIT_OBJECT_DIRECTORY to the shared cached\ndirectory for each of our repos, so they all fetch to the same place.\n\nQuestion 5) I haven't seen this mentioned anywhere else, so I'm\nguessing it won't work. Am I missing a big problem?\n\nQuestion 6) Will git be sad if two different repos that share an\nobject directory, both do 'git fetch' at the same time? I could maybe\nprotect fetches with an flock, but jenkins can do git operations\nbehind my back so it would be easier if I didn't have to worry about\nlocking.\n\nQuestion 7) Is GIT_OBJECT_DIRECTORY supposed to work with subrepos?\nIn my experimentation, it looks like it doesn't: when I run\n'GIT_OBJECT_DIRECTORY=../obj git submodule update --init' it still\nputs the objects in .git/modules//objects/. Is this a bug?\n Is there any way to work around it?\n\nAny suggestions would be appreciated! It feels to me like this is\nsomething that git should support pretty easily given its\narchitecture, but I just don't see a way to do it.\n\nThanks,\ncraig\n"},{"id":"253780","messageId":"CAGZ79kbupVyX=SUSbuAbkX5nA2NJC1R5zCzoVz4-Rdkcc+Xt1Q@mail.gmail.com","threadId":"38188","inReplyTo":"CAGXKyzEYhR69w1=4q-xtBagVBwOPqNA9C=AD0bAorB+5eRtVRg@mail.gmail.com","subject":"Re: Saving space/network on common repos","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2014-12-17T22:01:55Z","receivedAt":"2014-12-17T22:01:55Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"I am not sure if there was any improvement since then, but Junio\nwrote about alternates 2 years ago\nhttp://git-blame.blogspot.com/2012/08/bringing-bit-more-sanity-to-alternates.html\n"},{"id":"253782","messageId":"20141217223215.GO29365@google.com","threadId":"38188","inReplyTo":"CAGXKyzEYhR69w1=4q-xtBagVBwOPqNA9C=AD0bAorB+5eRtVRg@mail.gmail.com","subject":"Re: Saving space/network on common repos","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-12-17T22:32:15Z","receivedAt":"2014-12-17T22:32:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(+cc: Duy who wrote the recent 'checkout --to' patch series)\nHi Craig,\n\nCraig Silverstein wrote:\n\n> By design, our Jenkins machine has several different\n> directories that each hold a copy of the same git repository. (For\n> instance, Jenkins may be running tests on our repo at several\n> different commits at the same time.) When Jenkins decides to run a\n> test -- I'm simplifying a bit -- it will pick one of the copies of the\n> repo, do a 'git fetch origin && git checkout ' and the\n> run the tests.\n\nYou might find 'git new-workdir' from contrib/workdir to be helpful.\nIt lets you attach multiple working copies to a single set of objects\nand refs.\n\nThere's a patch series to move that functionality into core git\nthrough an option \"git checkout --to=\" that creates a\nnew workdir for an existing repository that is currently in the\npu (\"proposed updates\") branch.\n\n[...]\n> An added complication is submodules. We have a submodule that is as\n> big and slow to fetch as our main repository.\n>\n> Question 4) Is there a practical way to set up submodules so they can\n> use the same object-sharing framework that the main repo does?\n\nIt's possible to do, but we haven't written a nice UI for it yet.\n(In other words, you can do this by cloning with --no-recurse-submodules\nand manually creating the submodule workdir in the appropriate place.\nLater calls to \"git submodule update\" will do the right thing.)\n\nThanks for a useful example,\nJonathan\n"},{"id":"253789","messageId":"CAGXKyzEqTik3p=A8NZJ6kUscFjw_Dh1mBPT-ciwq9L8kNKDDig@mail.gmail.com","threadId":"38188","inReplyTo":"20141217223215.GO29365@google.com","subject":"Re: Saving space/network on common repos","fromName":"Craig Silverstein","fromEmail":"csilvers@khanacademy.org","sentAt":"2014-12-17T23:57:32Z","receivedAt":"2014-12-17T23:57:32Z","isPatch":false,"sender":{"key":"csilvers@khanacademy.org","avatar":null},"body":"On Wed, Dec 17, 2014 at 2:32 PM, Jonathan Nieder wrote:\n> You might find 'git new-workdir' from contrib/workdir to be helpful.\n> It lets you attach multiple working copies to a single set of objects\n> and refs.\n\nThanks! That does indeed sound promising -- like a more principled\nversion of my GIT_OBJECT_DIRECTORY suggestion.\n\n>> Question 4) Is there a practical way to set up submodules so they can\n>> use the same object-sharing framework that the main repo does?\n>\n> It's possible to do, but we haven't written a nice UI for it yet.\n> (In other words, you can do this by cloning with --no-recurse-submodules\n> and manually creating the submodule workdir in the appropriate place.\n\nHmm, let me see if I understand you right -- you're suggesting that\nwhen cloning my reference repo, I do\n git clone --no-recurse-submodules \n for (path, url) in `parse-.gitmodules`: git clone url path\n# this is psuedocode, obviously :-)\n\nand then when I want to create a new workdir, I do something like:\n cd reference_repo\n git new-workdir /var/workspace1\n for (path, url) in `parse-.gitmodules`: cd path && git new-workdir\n/var/workspace1/path\n\n? Basically, I'm going back to the old git way of having each\nsubmodule have its own .git directory, rather than having it have a\n.git file with a 'gitdir' entry. Am I understanding this right?\n\nAlso, it seems to me there's the possibility, with git-newdir, that if\nseveral of the workspaces try to fetch at the same time they could\nstep on each others' toes. Is that a problem? I know there's a push\nlock but I don't believe there's a fetch lock, and I could imagine git\ngetting unhappy if two fetches happened in the same repo at the same\ntime.\n\ncraig\n"},{"id":"253791","messageId":"20141218000723.GP29365@google.com","threadId":"38188","inReplyTo":"CAGXKyzEqTik3p=A8NZJ6kUscFjw_Dh1mBPT-ciwq9L8kNKDDig@mail.gmail.com","subject":"Re: Saving space/network on common repos","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-12-18T00:07:23Z","receivedAt":"2014-12-18T00:07:23Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Craig Silverstein wrote:\n> On Wed, Dec 17, 2014 at 2:32 PM, Jonathan Nieder wrote:\n>> Craig Silverstein wrote:\n\n>>> Question 4) Is there a practical way to set up submodules so they can\n>>> use the same object-sharing framework that the main repo does?\n>>\n>> It's possible to do, but we haven't written a nice UI for it yet.\n>> (In other words, you can do this by cloning with --no-recurse-submodules\n>> and manually creating the submodule workdir in the appropriate place.\n>\n> Hmm, let me see if I understand you right -- you're suggesting that\n> when cloning my reference repo, I do\n> git clone --no-recurse-submodules \n> for (path, url) in `parse-.gitmodules`: git clone url path\n> # this is psuedocode, obviously :-)\n>\n> and then when I want to create a new workdir, I do something like:\n> cd reference_repo\n> git new-workdir /var/workspace1\n> for (path, url) in `parse-.gitmodules`: cd path && git new-workdir /var/workspace1/path\n>\n> ? Basically, I'm going back to the old git way of having each\n> submodule have its own .git directory, rather than having it have a\n> .git file with a 'gitdir' entry. Am I understanding this right?\n\nBasically. The initial clone can still use --recurse-submodules.\nWhen you create a new workdir you'd create new workdirs for the\nsubmodules by hand.\n\nA 'git submodule foreach' command in the initial repo can take\ncare of the `parse-.gitmodules` part.\n\n[...]\n> Also, it seems to me there's the possibility, with git-newdir, that if\n> several of the workspaces try to fetch at the same time they could\n> step on each others' toes. Is that a problem? I know there's a push\n> lock but I don't believe there's a fetch lock, and I could imagine git\n> getting unhappy if two fetches happened in the same repo at the same\n> time.\n\nThat's a good question. If concurrent fetches cause trouble then I'd\nconsider it a bug (it's not too different from multiple concurrent\npushes to the same repository, which is a very common thing to do),\nbut I haven't looked carefully into whether such bugs exist.\n\nHopefully someone else can chime in.\n\nThanks,\nJonathan\n"},{"id":"253980","messageId":"CAGXKyzGnnt6bBEPoy4nvNn=zX0hdRHpgvsAe263apaDsVxv0dA@mail.gmail.com","threadId":"38188","inReplyTo":"20141218000723.GP29365@google.com","subject":"Re: Saving space/network on common repos","fromName":"Craig Silverstein","fromEmail":"csilvers@khanacademy.org","sentAt":"2014-12-23T01:00:29Z","receivedAt":"2014-12-23T01:00:29Z","isPatch":false,"sender":{"key":"csilvers@khanacademy.org","avatar":null},"body":"btw, just FYI, the scheme you lay out here doesn't actually work\nas-is. The problem is the config file, which has an entry like:\n worktree = ../../../mysubmodule\nThis depends on the config file living in\n./git/modules/mysubmodule/config. But the proposed scheme moves the\nconfig file to mysubmodule/.git/config, and the relative path is\nbroken.\n\nI'm not sure what the best solution is; the cleanest one requires a\npretty substantial rewrite of git-new-workdir (not that it's such a\ngiant piece of code), and separating out new_workdir from new_gitdir.\nThe smallest one involves having some way to suppress the final 'git\ncheckout -f' (which is the only thing in this script that needs the\nworktree entry to resolve somewhere) to allow for post-script cleanup.\n\ncraig\n\nOn Wed, Dec 17, 2014 at 4:07 PM, Jonathan Nieder wrote:\n> Craig Silverstein wrote:\n>> On Wed, Dec 17, 2014 at 2:32 PM, Jonathan Nieder wrote:\n>>> Craig Silverstein wrote:\n>\n>>>> Question 4) Is there a practical way to set up submodules so they can\n>>>> use the same object-sharing framework that the main repo does?\n>>>\n>>> It's possible to do, but we haven't written a nice UI for it yet.\n>>> (In other words, you can do this by cloning with --no-recurse-submodules\n>>> and manually creating the submodule workdir in the appropriate place.\n>>\n>> Hmm, let me see if I understand you right -- you're suggesting that\n>> when cloning my reference repo, I do\n>> git clone --no-recurse-submodules \n>> for (path, url) in `parse-.gitmodules`: git clone url path\n>> # this is psuedocode, obviously :-)\n>>\n>> and then when I want to create a new workdir, I do something like:\n>> cd reference_repo\n>> git new-workdir /var/workspace1\n>> for (path, url) in `parse-.gitmodules`: cd path && git new-workdir /var/workspace1/path\n>>\n>> ? Basically, I'm going back to the old git way of having each\n>> submodule have its own .git directory, rather than having it have a\n>> .git file with a 'gitdir' entry. Am I understanding this right?\n>\n> Basically. The initial clone can still use --recurse-submodules.\n> When you create a new workdir you'd create new workdirs for the\n> submodules by hand.\n>\n> A 'git submodule foreach' command in the initial repo can take\n> care of the `parse-.gitmodules` part.\n>\n> [...]\n>> Also, it seems to me there's the possibility, with git-newdir, that if\n>> several of the workspaces try to fetch at the same time they could\n>> step on each others' toes. Is that a problem? I know there's a push\n>> lock but I don't believe there's a fetch lock, and I could imagine git\n>> getting unhappy if two fetches happened in the same repo at the same\n>> time.\n>\n> That's a good question. If concurrent fetches cause trouble then I'd\n> consider it a bug (it's not too different from multiple concurrent\n> pushes to the same repository, which is a very common thing to do),\n> but I haven't looked carefully into whether such bugs exist.\n>\n> Hopefully someone else can chime in.\n>\n> Thanks,\n> Jonathan\n"},{"id":"253982","messageId":"20141223013347.GV29365@google.com","threadId":"38188","inReplyTo":"CAGXKyzGnnt6bBEPoy4nvNn=zX0hdRHpgvsAe263apaDsVxv0dA@mail.gmail.com","subject":"Re: Saving space/network on common repos","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-12-23T01:33:47Z","receivedAt":"2014-12-23T01:33:47Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Craig Silverstein wrote:\n\n> btw, just FYI, the scheme you lay out here doesn't actually work\n> as-is. The problem is the config file, which has an entry like:\n> worktree = ../../../mysubmodule\n> This depends on the config file living in\n> ./git/modules/mysubmodule/config. But the proposed scheme moves the\n> config file to mysubmodule/.git/config, and the relative path is\n> broken.\n\n*puzzled* Can you give a transcript illustrating this happening?\n\nSubmodules with .git directory within their worktree or under\n.git/modules/ are both supposed to work. And in either case, having\nrefs/ and objects/ as symlinks should work fine. When git new-workdir\ncreates a new workdir, it has its own new and separate config file, so\nI don't think that is the source of trouble.\n\nThanks,\nJonathan\n"},{"id":"253987","messageId":"20141223031233.GX29365@google.com","threadId":"38188","inReplyTo":"CAGXKyzGnnt6bBEPoy4nvNn=zX0hdRHpgvsAe263apaDsVxv0dA@mail.gmail.com","subject":"Re: Saving space/network on common repos","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-12-23T03:12:33Z","receivedAt":"2014-12-23T03:12:33Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Craig Silverstein wrote:\n\n> btw, just FYI, the scheme you lay out here doesn't actually work\n> as-is. The problem is the config file, which has an entry like:\n> worktree = ../../../mysubmodule\n> This depends on the config file living in\n> ./git/modules/mysubmodule/config. But the proposed scheme moves the\n> config file to mysubmodule/.git/config, and the relative path is\n> broken.\n\nAs was pointed out to me privately, the behavior is exactly as you\ndescribed and I had confused myself by looking at directory that\nwasn't even made with git-new-workdir. Sorry for the nonsense.\n\nWorkdirs share a single config file because information associated to\nbranches set by \"git branch --set-upstream-to\", \"git branch\n--edit-description\", \"git remote\", and so on are stored in the config\nfile.\n\nThe 'git checkout --to' series in \"pu\" avoids this problem by ignoring\ncore.bare and core.worktree in worktrees created with 'git checkout --to'.\nTo try it:\n\n\tgit clone https://kernel.googlesource.com/pub/scm/git/git\n\tcd git\n\tgit merge 'origin/pu^{/nd/multiple-work-trees}^2'\n\tmake\n\tPATH=$(pwd)/bin-wrappers:$PATH\n\n\tgit checkout --to=../experiment next\n\nThis seems like good motivation to try to get that series in good\nshape and release it soon.\n\nThanks again,\nJonathan\n"},{"id":"253989","messageId":"CAGXKyzFXVODvbqzO3CAQRr2S4dDbe95TGc=8iYgcBkKpAPiXSg@mail.gmail.com","threadId":"38188","inReplyTo":"20141223031233.GX29365@google.com","subject":"Re: Saving space/network on common repos","fromName":"Craig Silverstein","fromEmail":"csilvers@khanacademy.org","sentAt":"2014-12-23T05:36:53Z","receivedAt":"2014-12-23T05:36:53Z","isPatch":false,"sender":{"key":"csilvers@khanacademy.org","avatar":null},"body":"} This seems like good motivation to try to get that series in good\nshape and release it soon.\n\nI was going to spend some time tomorrow (if I can find any :-) )\ntrying to fix up the contrib script to work with submodules, or at\nleast the kind that we use. Is that something that's worth the time\nto do, or would we be better off just waiting for the work-tree stuff\nto get released? If I do end up doing it, would you be interested in\na pull request (or however patches are submitted in the git world)?\n\ncraig\n\nOn Mon, Dec 22, 2014 at 7:12 PM, Jonathan Nieder wrote:\n> Craig Silverstein wrote:\n>\n>> btw, just FYI, the scheme you lay out here doesn't actually work\n>> as-is. The problem is the config file, which has an entry like:\n>> worktree = ../../../mysubmodule\n>> This depends on the config file living in\n>> ./git/modules/mysubmodule/config. But the proposed scheme moves the\n>> config file to mysubmodule/.git/config, and the relative path is\n>> broken.\n>\n> As was pointed out to me privately, the behavior is exactly as you\n> described and I had confused myself by looking at directory that\n> wasn't even made with git-new-workdir. Sorry for the nonsense.\n>\n> Workdirs share a single config file because information associated to\n> branches set by \"git branch --set-upstream-to\", \"git branch\n> --edit-description\", \"git remote\", and so on are stored in the config\n> file.\n>\n> The 'git checkout --to' series in \"pu\" avoids this problem by ignoring\n> core.bare and core.worktree in worktrees created with 'git checkout --to'.\n> To try it:\n>\n> git clone https://kernel.googlesource.com/pub/scm/git/git\n> cd git\n> git merge 'origin/pu^{/nd/multiple-work-trees}^2'\n> make\n> PATH=$(pwd)/bin-wrappers:$PATH\n>\n> git checkout --to=../experiment next\n>\n> This seems like good motivation to try to get that series in good\n> shape and release it soon.\n>\n> Thanks again,\n> Jonathan\n"}]}