Re: [RFC] Packing large repositories

From: Shawn O. Pearce <spearce@spearce.org>
Date: Mar 30, 2007, 06:23 UTC
Message-ID: <20070330062324.GU13247@spearce.org>
In-Reply-To: <Pine.LNX.4.64.0703280943450.6730@woody.linux-foundation.org>

Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 8 quoted lines

> On Wed, 28 Mar 2007, Dana How wrote:
> > Of course this is unusable, since object_entry's in an .idx
> > file have only 32 bits in their offset fields.  I conclude that
> > for such large projects,  git-repack/git-pack-objects would need
> > new options to control maximum packfile size.
> 
> Either that, or update the index file format. I think that your approach 
> of having a size limiter is actually the *better* one, though.

Nico and I were hoping we could push the index file format change back until pack v4 was also worthy of merging. So I had also started work on an index-pack splitter:

  URL:    git://repo.or.cz/git/fastimport.git
  Branch: sp/splitpack
 
Its far from complete.

Show 5 quoted lines

> Well, it does affect fetching, in that "git index-pack" obviously would 
> also need to be taught how to split the resulting indexed packs up into 
> multiple smaller ones from one large incoming one. But that shouldn't be 
> fundamentally hard either, apart from the inconvenience of having to 
> rewrite the object count in the pack headers..

We already do this if its a thin-pack that is being made non-thin.
So its annoying, yea, but we already have one toe in that particular
pond...

> To avoid that issue, it may be that it's actually better to split things 
> up at pack-generation time *even* for the case of --stdout

This is actually a pretty good idea.  index-pack knows when the end
of its current packfile is; if there are 12 more bytes available in
the input stream and it looks like another pack header then it can
just restart itself over again with that next pack.  This is actually
a pretty small change in index-pack, certainly a lot less intrusive
than what I was trying to do in my sp/splitpack topic above.

-- 
Shawn.