Optimizing cloning of a high object count repository

> Hi,
>  there are currently different ideas to move gentoo's cvs repository to an
>  other scm. Current tests showed that svn will not make anything better (it
>  gets in most perfomance and size based benchmarks even worse). Another idea is
>  to move to git. It looks really promising in size based benchmarks but cloning
>  seems nearly impossible. The current test repository is available at
>  git://git.overlays.gentoo.org/exp/gentoo-x86.git and is around 900MB in size
>  and has 4696137 objects. It really takes ages to do the counting of the
>  objects on the server and compressing takes much longer.
>  The size of the linux repository seems to be smaller but in the same range
>  object count and repository size but clones are much much faster. Is there any
>  way to optimize the server operations like counting and compressing of objects
>  to get the same speed as we get from git.kernel.org (which does it in nearly
>  no time and the only limiting factor seems to be my bandwith)?
>  The only other information I have is that Robin H. Johnson made a single
>  ~910MiB pack for the whole repository.

> >  The size of the linux repository seems to be smaller but in the same
> > range object count and repository size but clones are much much faster.
> > Is there any way to optimize the server operations like counting and
> > compressing of objects to get the same speed as we get from
> > git.kernel.org (which does it in nearly no time and the only limiting
> > factor seems to be my bandwith)?
> >  The only other information I have is that Robin H. Johnson made a single
> >  ~910MiB pack for the whole repository.
>
> Make yearly packed repository snapshots and publish them via http.
> People can wget the latest snapshot, then pull updates later.

> Resul Cetin wrote:
> > That would be a workaround but it doesn't explain why git.kernel.org
> > deliveres torvalds repository without any notable counting and
> > compressing time.
>
> If I remember right, git.kernel.org is a quite beefy machine.  But
> then again it has a lot more traffic too.  It might be interesting
> to know what machine you're on, compared to git.kernel.org.

> On Saturday 13 December 2008 16:46:50 you wrote:
> [...]
> > >  The size of the linux repository seems to be smaller but in the same
> > > range object count and repository size but clones are much much faster.
> > > Is there any way to optimize the server operations like counting and
> > > compressing of objects to get the same speed as we get from
> > > git.kernel.org (which does it in nearly no time and the only limiting
> > > factor seems to be my bandwith)?
> > >  The only other information I have is that Robin H. Johnson made a single
> > >  ~910MiB pack for the whole repository.
> >
> > Make yearly packed repository snapshots and publish them via http.
> > People can wget the latest snapshot, then pull updates later.
> That would be a workaround but it doesn't explain why git.kernel.org deliveres 
> torvalds repository without any notable counting and compressing time. Maybe 
> it has something todo with the config I found inside the repository:
> http://git.overlays.gentoo.org/gitroot/exp/gentoo-x86.git/config
> It says that it isnt a bare repository.

> On Sat, 13 Dec 2008, Resul Cetin wrote:
> 
> > On Saturday 13 December 2008 16:46:50 you wrote:
> > [...]
> > > >  The size of the linux repository seems to be smaller but in the same
> > > > range object count and repository size but clones are much much faster.
> > > > Is there any way to optimize the server operations like counting and
> > > > compressing of objects to get the same speed as we get from
> > > > git.kernel.org (which does it in nearly no time and the only limiting
> > > > factor seems to be my bandwith)?
> > > >  The only other information I have is that Robin H. Johnson made a single
> > > >  ~910MiB pack for the whole repository.
> > >
> > > Make yearly packed repository snapshots and publish them via http.
> > > People can wget the latest snapshot, then pull updates later.
> > That would be a workaround but it doesn't explain why git.kernel.org deliveres 
> > torvalds repository without any notable counting and compressing time. Maybe 
> > it has something todo with the config I found inside the repository:
> > http://git.overlays.gentoo.org/gitroot/exp/gentoo-x86.git/config
> > It says that it isnt a bare repository.
> 
> That's not relevant.
> 
> The counting time is a bit unfortunate (although I have plans to speed 
> that up, if only I can find the time).
> 
> You should be able to skip the compression time entirely though, if you 
> do repack the repository first.  And you want it to be as tightly packed 
> as possible for public access.  I'm currently cloning it and the 
> counting phase is not _that_ bad compared to the compression phase.  Try 
> something like 'git repack -a -f -d --window=200' and let it run 
> overnight if necessary.  You need to do this only once, and preferably 
> on a machine with lots of RAM, and preferably on a 64-bit machine.  Once 
> this is done then things should go much more smoothly afterwards.