{"thread":{"id":"29182","subject":"Any tips for improving the performance of cloning large repositories?","startedAt":"2011-12-16T13:02:12Z","lastAt":"2011-12-16T18:37:14Z","messageCount":7,"participants":["Alex Bennee","Hallvard Breien Furuseth","Seth Robertson","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"181293","messageId":"CAJ-05NPP7aCcr_SYxLYk8U1entDMv0aF2Me3cTGmOLjYqFKUOA@mail.gmail.com","threadId":"29182","inReplyTo":null,"subject":"Any tips for improving the performance of cloning large repositories?","fromName":"Alex Bennee","fromEmail":"kernel-hacker@bennee.com","sentAt":"2011-12-16T13:02:12Z","receivedAt":"2011-12-16T13:02:12Z","isPatch":false,"sender":{"key":"kernel-hacker@bennee.com","avatar":null},"body":"Hi,\n\nWe've migrated our old CVS repository into GIT without too many\nissues. However now we are rolling out the usage of the new repository\nwe are hitting some performance bottlenecks, especially on the initial\nclone (something our buildbot instance does a lot).\n\nOur repo is large, my .git is around 2.5G although the central repo\nhas a 1.7Gb single pack file. However some machines handle the cloning\nbetter than others. For one thing the clone process seems to involve\nthe receiving side needing a large glob of memory which causes\nproblems when there is memory pressure.\n\nI've tried tweaking the pack size from unlimited to 256m but this\nseems to have increased the clone time as the receiving end attempts\nto re-pack everything back into an uber-pack.\n\nAnother thing that I've noticed is very high systime on the receiving\nmachines as ethernet and disk I/O is heavily hit.\n\nSo what I'm looking for are some tips on how I can tweak\nconfigurations to make the clone process a little less I/O and memory\nheavy. Any suggestions?\n\nOne thing I did try was a rsync'ed local repo in /var/cache/repos\nwhich the clone command used for reference with something like:\n\ngit clone --local --reference /var/cache/repo.git git://repo/repo.git\n\nBut that didn't help as it seems to copy the whole thing anyway.\n\n-- \nAlex, homepage: http://www.bennee.com/~alex/\nhttp://www.half-llama.co.uk\n"},{"id":"181294","messageId":"hbf.20111216yufz@bombur.uio.no","threadId":"29182","inReplyTo":"CAJ-05NPP7aCcr_SYxLYk8U1entDMv0aF2Me3cTGmOLjYqFKUOA@mail.gmail.com","subject":"Re: Any tips for improving the performance of cloning large repositories?","fromName":"Hallvard Breien Furuseth","fromEmail":"h.b.furuseth@usit.uio.no","sentAt":"2011-12-16T13:11:11Z","receivedAt":"2011-12-16T13:11:11Z","isPatch":false,"sender":{"key":"h.b.furuseth@usit.uio.no","avatar":null},"body":"Alex Bennee writes:\n> We've migrated our old CVS repository into GIT without too many\n> issues. However now we are rolling out the usage of the new repository\n> we are hitting some performance bottlenecks, especially on the initial\n> clone (something our buildbot instance does a lot).\n\nDo you often need to clone from a remote? Instead of cloning from a\nlocal (git clone --mirror) which gets auto-updated from the remote.\n\nCould the buildbot make do with a shallow repo, clone --depth ?\n\n-- \nHallvard\n"},{"id":"181295","messageId":"hbf.20111216zcin@bombur.uio.no","threadId":"29182","inReplyTo":"hbf.20111216yufz@bombur.uio.no","subject":"Re: Any tips for improving the performance of cloning large repositories?","fromName":"Hallvard Breien Furuseth","fromEmail":"h.b.furuseth@usit.uio.no","sentAt":"2011-12-16T13:39:05Z","receivedAt":"2011-12-16T13:39:05Z","isPatch":false,"sender":{"key":"h.b.furuseth@usit.uio.no","avatar":null},"body":"I wrote:\n> Do you often need to clone from a remote? Instead of cloning from a\n> local (git clone --mirror) which gets auto-updated from the remote.\n\nEr, obviously not, since you tried that with rsync. Create the mirror\nwith 'git clone --mirror', then update it with 'git fetch' rather than\nrsync.\n\n-- \nHallvard\n"},{"id":"181297","messageId":"201112161414.pBGEExLJ006769@no.baka.org","threadId":"29182","inReplyTo":"hbf.20111216zcin@bombur.uio.no","subject":"Re: Any tips for improving the performance of cloning large repositories?","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2011-12-16T14:14:59Z","receivedAt":"2011-12-16T14:14:59Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message , Hallvard Breien Furuseth writes:\n\n I wrote:\n > Do you often need to clone from a remote? Instead of cloning from a\n > local (git clone --mirror) which gets auto-updated from the remote.\n\n Er, obviously not, since you tried that with rsync. Create the mirror\n with 'git clone --mirror', then update it with 'git fetch' rather than\n rsync.\n\nIf you really need to perform a full clone from the buildbot with or\nwithout a different working directory (for instance if you have\nbuildbots/checkout users running in parallel where multiple users need\na consistent HEAD for multiple sequential operations) then instead\nconsider cloning with --reference or --shared. There are severe\nrestrictions on what you should do with aggressive sharing (man\ngit-clone), but if all you are doing is normal checkouts, tags,\ncommits, etc, then it would be just fine. Of course remember to add a\nremote for the real upstream if you are planning on pushing\nchanges/tags back.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"181299","messageId":"CAJ-05NPbRmyx=a+U7BK4rNShBgaXj+g-Bwc1aBDDb3N0VPBW=A@mail.gmail.com","threadId":"29182","inReplyTo":"201112161414.pBGEExLJ006769@no.baka.org","subject":"Re: Any tips for improving the performance of cloning large repositories?","fromName":"Alex Bennee","fromEmail":"kernel-hacker@bennee.com","sentAt":"2011-12-16T15:28:07Z","receivedAt":"2011-12-16T15:28:07Z","isPatch":false,"sender":{"key":"kernel-hacker@bennee.com","avatar":null},"body":"On 16 December 2011 14:14, Seth Robertson wrote:\n>\n> In message , Hallvard Breien Furuseth writes:\n>\n>    I wrote:\n>    > Do you often need to clone from a remote?  Instead of cloning from a\n>    > local (git clone --mirror) which gets auto-updated from the remote.\n>\n>    Er, obviously not, since you tried that with rsync.  Create the mirror\n>    with 'git clone --mirror', then update it with 'git fetch' rather than\n>    rsync.\n>\n> If you really need to perform a full clone from the buildbot with or\n> without a different working directory (for instance if you have\n> buildbots/checkout users running in parallel where multiple users need\n> a consistent HEAD for multiple sequential operations) then instead\n> consider cloning with --reference or --shared.\n\nWell that's counter intuitive....\n\n - reverting the original repo to one big pack speeds up the clone\n - adding a --local --reference mirror slows it down\n\nTimings:\n\n14:41 ajb@vsbldhost/i686 [ajb] >time git clone git://engbot/repo.git\ntest-clone-bigpack.git\nInitialized empty Git repository in /scratch/ajb/test-clone-bigpack.git/.git/\nremote: Counting objects: 371220, done.\nremote: Compressing objects: 100% (88900/88900), done.\nremote: Total 371220 (delta 274586), reused 371220 (delta 274586)\nReceiving objects: 100% (371220/371220), 1.78 GiB | 20.10 MiB/s, done.\nResolving deltas: 100% (274586/274586), done.\nChecking out files: 100% (42909/42909), done.\n\nreal 8m53.008s\nuser 2m53.151s\nsys 7m16.339s\n\n14:53 ajb@vsbldhost/i686 [ajb] >time git clone --local --reference\n/var/cache/repos/repo.git git://engbot/repo.git te\nst-clone-local.git\nInitialized empty Git repository in /scratch/ajb/test-clone-local.git/.git/\nChecking out files: 100% (42909/42909), done.\n\nreal 14m6.333s\nuser 1m6.844s\nsys 12m44.676s\n\nTwo things are odd. The first is the clone \"hung\" at around 22%\nchecking out the files for ~ 10 minutes before finishing the remaining\n70% in a few seconds. Secondly is seems in both cases the systime is\nquite high.\n\n-- \nAlex, homepage: http://www.bennee.com/~alex/\nhttp://www.half-llama.co.uk\n"},{"id":"181305","messageId":"7vzkesigw9.fsf@alter.siamese.dyndns.org","threadId":"29182","inReplyTo":"CAJ-05NPbRmyx=a+U7BK4rNShBgaXj+g-Bwc1aBDDb3N0VPBW=A@mail.gmail.com","subject":"Re: Any tips for improving the performance of cloning large repositories?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-16T17:08:06Z","receivedAt":"2011-12-16T17:08:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Alex Bennee writes:\n\n> Well that's counter intuitive....\n>\n> - reverting the original repo to one big pack speeds up the clone\n> - adding a --local --reference mirror slows it down\n\nNeither is. Read what \"--local\" says in the help text of clone. It\ndisables the git aware clever optimization.\n"},{"id":"181316","messageId":"CAJ-05NNcORg2H_j3g-h-eDXuuGit_hNH6PG9tNq30QJwJMAPWg@mail.gmail.com","threadId":"29182","inReplyTo":"7vzkesigw9.fsf@alter.siamese.dyndns.org","subject":"Re: Any tips for improving the performance of cloning large repositories?","fromName":"Alex Bennee","fromEmail":"kernel-hacker@bennee.com","sentAt":"2011-12-16T18:37:14Z","receivedAt":"2011-12-16T18:37:14Z","isPatch":false,"sender":{"key":"kernel-hacker@bennee.com","avatar":null},"body":"On 16 December 2011 17:08, Junio C Hamano wrote:\n> Alex Bennee writes:\n>\n>> Well that's counter intuitive....\n>>\n>>  - reverting the original repo to one big pack speeds up the clone\n>>  - adding a --local --reference mirror slows it down\n>\n> Neither is. Read what \"--local\" says in the help text of clone. It\n> disables the git aware clever optimization.\n\nOK that's not how I read the man page:\n\n --local, -l\n When the repository to clone from is on a local machine,\nthis flag bypasses the normal \"git aware\" transport\n mechanism and clones the repository by making a copy of\nHEAD and everything under objects and refs directories.\n\nSo this says it skips \"git aware\" (whatever that means)\n\n The files under .git/objects/ directory are hardlinked to\nsave space when possible. This is now the default when\n the source repository is specified with /path/to/repo\nsyntax, so it essentially is a no-op option. To force\n copying instead of hardlinking (which may be desirable if\nyou are trying to make a back-up of your repository),\n but still avoid the usual \"git aware\" transport mechanism,\n--no-hardlinks can be used.\n\nAnd this says that objects on the local file-system are hardlinked\n(rather than copied) which I assumed was a optimal approach.\n\n --no-hardlinks\n Optimize the cloning process from a repository on a local\nfilesystem by copying files under .git/objects\n directory.\n\nI'm not sure how this is an optimization? This means more copying\nrather than linking right?\n\n-- \nAlex, homepage: http://www.bennee.com/~alex/\nhttp://www.half-llama.co.uk\n"}]}