Re: Figured out how to get Mozilla into git

From: Nicolas Pitre <nico@cam.org>
Date: Jun 9, 2006, 17:38 UTC
Message-ID: <Pine.LNX.4.64.0606091326550.2703@localhost.localdomain>
In-Reply-To: <Pine.LNX.4.64.0606090926550.5498@g5.osdl.org>

On Fri, 9 Jun 2006, Linus Torvalds wrote:

Show 28 quoted lines

> 
> 
> On Fri, 9 Jun 2006, Nicolas Pitre wrote:
> > 
> > > So a single 2GB pack is already very much pushing it. It's really really 
> > > hard to map in a 2GB file on a 32-bit platform: your VM is usually 
> > > fragmented enough that it simply isn't practical. In fact, I think the 
> > > limit for _practical_ usage of single packs is probably somewhere in the 
> > > half-gig region, unless you just have 64-bit machines.
> > 
> > Sure, but have we already reached that size?
> 
> Not for the Linux repos.
> 
> But apparently the mozilla repo ends up being 2GB in git. From Martin:
> 
>   >> oh, I went back to a cvsimport that I started a couple days ago.
>   >> Completed with no problems...
>   >> 
>   >> Last commit:
>   >> commit 5ecb56b9c4566618fad602a8da656477e4c6447a
>   >> Author: wtchang%redhat.com <wtchang%redhat.com>
>   >> Date:   Fri Jun 2 17:20:37 2006 +0000
>   >> 
>   >>    Import NSPR 4.6.2 and NSS 3.11.1
>   >> 
>   >> mozilla.git$ du -sh .git/
>   >> 2.0G    .git/

He also sais:

| git-repack -a -d but it OOMs on my 2GB+2GBswap machine :(

> now that was done with _incremental_ repacking (ie his .git directory
> won't be just one large pack),

So given the nature of packs, incrementally packing an imported repository _might_ cause worse problems since each pack must be self referenced by definition. That means you may end up with multiple revisions of the same file distributed amongst as many packs hence none of those revisions are ever deltified, and to repack that you currently have to mmap all those packs at once.

> but I bet that if you were to clone it
> (without using the "-l" flag or rsync/http), you'd end up with serious
> trouble because of the single-pack limit.

Maybe that single pack would instead be under the 512MB limit? I'd be curious to know.

> So we're starting to see archives where single packs are problematic for
> a 32-bit architecture.

Depending on the operation, the single pack might actually be better, especially for a full clone where everything gets mapped. Multiple packs will always take more space, which is fine if you don't need access to all objects at once since individual packs are small, but the whole of them (when repacking or cloning) isn't.

Nicolas