Re: Figured out how to get Mozilla into git

From: Nicolas Pitre <nico@cam.org>
Date: Jun 9, 2006, 16:11 UTC
Message-ID: <Pine.LNX.4.64.0606091127540.19403@localhost.localdomain>
In-Reply-To: <Pine.LNX.4.64.0606090745390.5498@g5.osdl.org>

On Fri, 9 Jun 2006, Linus Torvalds wrote:

Show 12 quoted lines

> 
> 
> On Fri, 9 Jun 2006, Jakub Narebski wrote:
> > Jon Smirl wrote:
> > 
> > >> git-repack -a -d but it OOMs on my 2GB+2GBswap machine :(
> > > 
> > > We are all having problems getting this to run on 32 bit machines with
> > > the 3-4GB process size limitations.
> > 
> > Is that expected (for 10GB repository if I remember correctly), or is there
> > some way to avoid this OOM?

What was that 10GB related to, exactly? The original CVS repo, or the unpacked GIT repo?

Show 5 quoted lines

> So a single 2GB pack is already very much pushing it. It's really really 
> hard to map in a 2GB file on a 32-bit platform: your VM is usually 
> fragmented enough that it simply isn't practical. In fact, I think the 
> limit for _practical_ usage of single packs is probably somewhere in the 
> half-gig region, unless you just have 64-bit machines.

Sure, but have we already reached that size?

The historic Linux repo currently repacks itself into a ~175MB pack for 63428 commits.

The current Linux repo is ~103MB with a much shorter history (27153 commits).

Given the above we can estimate the size of the kernel repository after x commits as follows:

	slope = (175 - 103) / (63428 - 27153) = approx 2KB per commit

	initial size = 175 - .001985 * 63428 = 49MB

So the initial kernel commit is about 49MB in size which is coherent with the corresponding compressed tarball. Subsequent commits are 2KB in size on average. Given that it will take about 233250 commits before the kernel reaches the half gigabyte pack file, and given the current commit rate (approx 23700 commits per year), that means we still have nearly 9 years to go. And at that point 64-bit machines are likely to be the norm.

So given those numbers I don't think this is really an issue. The Linux kernel is a rather huge and pretty active project to base comparisons against. The Mozilla repository might be difficult to import and repack, but once repacked it should still be pretty usable now even on a 32-bit machine even with a single pack.

Otherwise that should be quite easy to add a batch size argument to git-repack so git-rev-list and git-pack-objects are called multiple times with sequential commit ranges to create a repo with multiple packs.

Nicolas