Re: Performance issue: initial git clone causes massive repack

From: Mark Levedahl <mlevedahl@gmail.com>
Date: Apr 11, 2009, 17:24 UTC
Message-ID: <grqjo1$at2$1@ger.gmane.org>
In-Reply-To: <20090404220743.GA869@curie-int>

Robin H. Johnson wrote:

Show 17 quoted lines

> Hi,
> 
> This is a first in my series of mails over the next few days, on issues
> that we've run into planning a potential migration for Gentoo's
> repository into Git.
> 
> Our full repository conversion is large, even after tuning the
> repacking, the packed repository is between 1.4 and 1.6GiB. As of Feburary
> 4th, 2009, it contained 4886949 objects. It is not suitable for
> splitting into submodules either unfortunately - we have a lot of
> directory moves that would cause submodule bloat.
> 
> During an initial clone, I see that git-upload-pack invokes
> pack-objects, despite the ENTIRE repository already being packed - no
> loose objects whatsoever. git-upload-pack then seems to buffer in
> memory.
>

Have you considered using a bundle as part of the initial clone process? The idea would be to periodically create a bundle

	git bundle create <somename>.bundle [list of refs]

and publish that on your website. A new user would then do

	wget $uri-of-bundle
	git clone <somename>.bundle
	cd $somename
	git remote add origin $origin
	git fetch

and they have the current repo. As the bundle is a file, it can be distributed by torrent or other method. The expense of creating the pack in the bundle is paid exactly once when the bundle is created.

Mark