{"thread":{"id":"22518","subject":"Dealing with many many git repos in a /home directory","startedAt":"2010-02-04T08:29:46Z","lastAt":"2010-02-04T17:35:11Z","messageCount":6,"participants":["demerphq","Alex Riesen","Martin Langhoff","Sergio","Andreas Schwab","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"133593","messageId":"9b18b3111002040029x1c7de0afw4a5ef883588f7a18@mail.gmail.com","threadId":"22518","inReplyTo":null,"subject":"Dealing with many many git repos in a /home directory","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2010-02-04T08:29:46Z","receivedAt":"2010-02-04T08:29:46Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"At $work we have a host where we have about 50-100 users each with\ntheir own private copies of the same repos. These are cloned froma\nremote via git/ssh and are not thus automatically hardlinking their\nobject stores.\n\nThis is starting to take a lot of space.\n\nI was thinking it should be possible to hardlink all of the objects in\nthe different repos to a canonical single copy.\n\nWould i be correct in thinking that if i have to repos with an\nequivalent  .git/objects/../..... file in them that the files are\nnecessarily identical and one can be replaced by a hardlink to the\nother?\n\nIf this is correct then is there some tool known to the list that\nalready does this?  I whipped this together:\n\nfind /home -regex .\\*/.git/objects/.\\* | perl -lne'if\n(m!(\\.git/objects/../.+)!) { if (my $t= $seen{$1}) { link $t,$_ } else\n{ $seen{$1}=$_ } }'\n\nBut a proper script with a sign off of some git dev would make me feel\na lot safer :-)\n\ncheers,\nYves\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"133597","messageId":"81b0412b1002040157r7af2a587y9cfc7189fc8ad632@mail.gmail.com","threadId":"22518","inReplyTo":"9b18b3111002040029x1c7de0afw4a5ef883588f7a18@mail.gmail.com","subject":"Re: Dealing with many many git repos in a /home directory","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2010-02-04T09:57:55Z","receivedAt":"2010-02-04T09:57:55Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On Thu, Feb 4, 2010 at 09:29, demerphq <demerphq@gmail.com> wrote:\n> Would i be correct in thinking that if i have to repos with an\n> equivalent  .git/objects/../..... file in them that the files are\n> necessarily identical and one can be replaced by a hardlink to the\n> other?\n\nYes, but you probably wont save as much as you'd like: think about the users\nwho *do* repack their repositories. The .pack files will be all different.\n"},{"id":"133601","messageId":"46a038f91002040700x623a686ek4bc35d20f5d9cb86@mail.gmail.com","threadId":"22518","inReplyTo":"9b18b3111002040029x1c7de0afw4a5ef883588f7a18@mail.gmail.com","subject":"Re: Dealing with many many git repos in a /home directory","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2010-02-04T15:00:38Z","receivedAt":"2010-02-04T15:00:38Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Thu, Feb 4, 2010 at 3:29 AM, demerphq <demerphq@gmail.com> wrote:\n> This is starting to take a lot of space.\n\nWhat I used to do was to\n\n - have a \"canonical\" local bare repo for each major project, fetching\nand repacking nightly\n\n - a script that \"injects\" an \"alternates\" entry to matching user\nrepos -- logic to look at a repo and decide which alternate to hook it\nto is left to the reader.\n\n - optional: automating repacks on users repos\n\nAs users repack, their \"local\" packs will only have the objects that\nare not shared with the canonical repos. With Moodle repos, this was a\n200MB savings per repo.\n\nAnd the kernel keeps one set of packfiles in buffers, so everyone gets\nmuch faster gitk / gitlog / blame...\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"133602","messageId":"loom.20100204T160327-458@post.gmane.org","threadId":"22518","inReplyTo":"81b0412b1002040157r7af2a587y9cfc7189fc8ad632@mail.gmail.com","subject":"Re: Dealing with many many git repos in a /home directory","fromName":"Sergio","fromEmail":"sergio.callegari@gmail.com","sentAt":"2010-02-04T15:20:18Z","receivedAt":"2010-02-04T15:20:18Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Alex Riesen <raa.lkml <at> gmail.com> writes:\n\n> \n> On Thu, Feb 4, 2010 at 09:29, demerphq <demerphq <at> gmail.com> wrote:\n> > Would i be correct in thinking that if i have to repos with an\n> > equivalent  .git/objects/../..... file in them that the files are\n> > necessarily identical and one can be replaced by a hardlink to the\n> > other?\n> \n> Yes, but you probably wont save as much as you'd like: think about the\n> users\n> who *do* repack their repositories. The .pack files will be all\n> different.\n> \n\n\nMaybe you can:\n\nfor each repo\n  clone it to some place\n  pack it with gc --aggressive\n  take the resulting pack and move it (and the associated index) somewhere\n  make in the same place a file with the same hash as the pack and extension\n    keep and possibly, inside, some note about its content (e.q. what repo\n    was cloned and at what state/time it was so frozen).\n  ask the users to go in the .git/objects/packs dir of their private copy\n    of the corresponding repo and hardlink there the .pack, .idx, .keep\n    file that you have prepared\n  ask the users to invoke git gc\n\nBefore actually doing that on something important, maybe wait have the\nconfirmation from some developer that there is not something flawed in the\napproach.\n\nPersonally, I tend to use keep files a lot because I need to keep two\nmachines synchronized using \"unison\". Without keep files, large packs are\nchanged at every gc and the synchronization takes ages. By \"freezing\" a\nstable subset of my objects I maintain the changing packs much smaller and\nreduce the amount of data that needs to be carried over by unison to keep\nthe two machines in sync.\n"},{"id":"133603","messageId":"m24olxuicm.fsf@igel.home","threadId":"22518","inReplyTo":"9b18b3111002040029x1c7de0afw4a5ef883588f7a18@mail.gmail.com","subject":"Re: Dealing with many many git repos in a /home directory","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2010-02-04T15:32:41Z","receivedAt":"2010-02-04T15:32:41Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"demerphq <demerphq@gmail.com> writes:\n\n> At $work we have a host where we have about 50-100 users each with\n> their own private copies of the same repos. These are cloned froma\n> remote via git/ssh and are not thus automatically hardlinking their\n> object stores.\n>\n> This is starting to take a lot of space.\n\nCreate local mirrors of the remote repos (and update them regularily)\nand ask the users to borrow from them.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"133623","messageId":"alpine.LFD.2.00.1002041207330.1681@xanadu.home","threadId":"22518","inReplyTo":"9b18b3111002040029x1c7de0afw4a5ef883588f7a18@mail.gmail.com","subject":"Re: Dealing with many many git repos in a /home directory","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-04T17:35:11Z","receivedAt":"2010-02-04T17:35:11Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 4 Feb 2010, demerphq wrote:\n\n> At $work we have a host where we have about 50-100 users each with\n> their own private copies of the same repos. These are cloned froma\n> remote via git/ssh and are not thus automatically hardlinking their\n> object stores.\n> \n> This is starting to take a lot of space.\n\nYou should keep a pristine copy of that common repository on that host \nand make it readable to everyone, and then ask your users to use the \n--reference argument with 'git clone' to borrow as much as possible from \nthat common repository.\n\nFor those who already cloned the repository in full i.e. without the \n--reference switch, then it is possible to fix the situation simply by \nadding the full path to the common repository's .git/objects directory \nin their own .git/objects/info/alternates (create it if it doesn't \nexist) and then run 'git gc'.  That's what the --reference argument to \nthe clone command does: setting up that .git/objects/info/alternates \nfile.\n\n> I was thinking it should be possible to hardlink all of the objects in\n> the different repos to a canonical single copy.\n> \n> Would i be correct in thinking that if i have to repos with an\n> equivalent  .git/objects/../..... file in them that the files are\n> necessarily identical and one can be replaced by a hardlink to the\n> other?\n\nYes, you could do that.  However you'll save very little by doing that \nas the bulk of a repository content is normally stored into pack files, \nand those may differ from one repository to another depending on what \nexactly the pack contains.  The alternates mechanism is more powerful as \nit lets Git fetch objects from the canonical repository packed or not, \nand more importantly it avoids creating local copy of new objects if \nthey already exists in that canonical copy meaning that you don't have \nto constantly search in every user's repository for potential new \nobjects to hardlink.\n\n> If this is correct then is there some tool known to the list that\n> already does this?  I whipped this together:\n\nThe \"tool\" exists in Git already and is what I describe above.  The \nactual tool you might need is probably a script to populate that \n.git/objects/info/alternates file in all your users' repositoryes and \nmaybe run ,git gc' on their behalf.\n\n\nNicolas\n"}]}