Re: [RFC] Packing large repositories

From: Nicolas Pitre <nico@cam.org>
Date: Mar 30, 2007, 13:01 UTC
Message-ID: <alpine.LFD.0.83.0703300851270.3041@xanadu.home>
In-Reply-To: <20070330062324.GU13247@spearce.org>

On Fri, 30 Mar 2007, Shawn O. Pearce wrote:

Show 18 quoted lines

> Linus Torvalds <torvalds@linux-foundation.org> wrote:
> > On Wed, 28 Mar 2007, Dana How wrote:
> > > Of course this is unusable, since object_entry's in an .idx
> > > file have only 32 bits in their offset fields.  I conclude that
> > > for such large projects,  git-repack/git-pack-objects would need
> > > new options to control maximum packfile size.
> > 
> > Either that, or update the index file format. I think that your approach 
> > of having a size limiter is actually the *better* one, though. 
> 
> Nico and I were hoping we could push the index file format change back
> until pack v4 was also worthy of merging.  So I had also started work
> on an index-pack splitter:
> 
>   URL:    git://repo.or.cz/git/fastimport.git
>   Branch: sp/splitpack
>  
> Its far from complete.

Well... actually I really think the best solution might simply be a new index format right now. All the preparatory work has been done already anyway.

IMHO that's the solution that would require the least work at this point, with the least possibility of issues/bugs.

Pack v4 could just use an index v3 which would be almost the same as index v2.

Remains only to determine what this new index format should look like. I think that the SHA1 table and the offset table should be separate. For one it will require less mmap space to binary search through the SHA1 table, and it will make things much easier if pack v4 stores the SHA1 table itself.

Nicolas