Re: Any tips for improving the performance of cloning large repositories?
- From
- Alex Bennee <kernel-hacker@bennee.com>
- Date
- Dec 16, 2011, 15:28 UTC
- Message-ID
- <CAJ-05NPbRmyx=a+U7BK4rNShBgaXj+g-Bwc1aBDDb3N0VPBW=A@mail.gmail.com>
- In-Reply-To
- <201112161414.pBGEExLJ006769@no.baka.org>
On 16 December 2011 14:14, Seth Robertson <in-gitvger@baka.org> wrote:
Show 16 quoted lines
> > In message <hbf.20111216zcin@bombur.uio.no>, Hallvard Breien Furuseth writes: > > I wrote: > > Do you often need to clone from a remote? Instead of cloning from a > > local (git clone --mirror) which gets auto-updated from the remote. > > Er, obviously not, since you tried that with rsync. Create the mirror > with 'git clone --mirror', then update it with 'git fetch' rather than > rsync. > > If you really need to perform a full clone from the buildbot with or > without a different working directory (for instance if you have > buildbots/checkout users running in parallel where multiple users need > a consistent HEAD for multiple sequential operations) then instead > consider cloning with --reference or --shared.
Well that's counter intuitive....
- reverting the original repo to one big pack speeds up the clone - adding a --local --reference mirror slows it down
Timings:
14:41 ajb@vsbldhost/i686 [ajb] >time git clone git://engbot/repo.git test-clone-bigpack.git Initialized empty Git repository in /scratch/ajb/test-clone-bigpack.git/.git/ remote: Counting objects: 371220, done. remote: Compressing objects: 100% (88900/88900), done. remote: Total 371220 (delta 274586), reused 371220 (delta 274586) Receiving objects: 100% (371220/371220), 1.78 GiB | 20.10 MiB/s, done. Resolving deltas: 100% (274586/274586), done. Checking out files: 100% (42909/42909), done.
real 8m53.008s user 2m53.151s sys 7m16.339s
14:53 ajb@vsbldhost/i686 [ajb] >time git clone --local --reference /var/cache/repos/repo.git git://engbot/repo.git te st-clone-local.git Initialized empty Git repository in /scratch/ajb/test-clone-local.git/.git/ Checking out files: 100% (42909/42909), done.
real 14m6.333s user 1m6.844s sys 12m44.676s
Two things are odd. The first is the clone "hung" at around 22% checking out the files for ~ 10 minutes before finishing the remaining 70% in a few seconds. Secondly is seems in both cases the systime is quite high.
-- Alex, homepage: http://www.bennee.com/~alex/ http://www.half-llama.co.uk