Re: Mozilla .git tree

From: Linus Torvalds <torvalds@osdl.org>
Date: Sep 2, 2006, 18:56 UTC
Message-ID: <Pine.LNX.4.64.0609021151280.27779@g5.osdl.org>
In-Reply-To: <20060902173922.GA27826@spearce.org>

On Sat, 2 Sep 2006, Shawn Pearce wrote:

Show 13 quoted lines

> Jon Smirl <jonsmirl@gmail.com> wrote:
> > If you're going to redo the pack formats another big win for the
> > Mozilla pack is to convert pack internal sha1 references into file
> > offsets.within the pack. Doing that will take around 30MB off from the
> > Mozilla pack size. sha1's are not compressible so this is a direct
> > savings.
> 
> Right now Junio's working on the index to break the 4 GiB barrier.
> I think Junio and Nico have already agreed to change the base SHA1
> to be an offset instead; though this is an issue for the current
> way the base gets written out behind the delta as you need to know
> exactly how many bytes the delta is going to be so you can correctly
> compute the offset.

I think moving the delta to after the base object is fine, and solves that problem.

However, the "--thin" pack problem is potentially worse, since the object you delta against doesn't even _exist_, and as such you'd then end up having two totally different formats (or perhaps we'd add a new object type for use in thin packs: a "dummy object" that says "my type is <so-and-so> and my SHA1 is <so-and-so>, you'd better have me already".

I'm not actually personally convinced that we need to solve the 4GB pack-file problem right now. Does anybody have that problem yet? The "partial mapping" issue is a much bigger one, I suspect, even if the tree delta stuff seems to have fixed it largely (yes?) for the mozilla fast-import pack-file.

So in many ways, I'd think that the 4GB problem is one where we are perfectly happy to _know_ that a solution exists, rather than have to actually solve it today.

Of course, if we change the pack/index-file format for other reasons, then we should obviously fix the 4G issue at the same time, but were there actually any real other reasons at this point?

			Linus

-- 
VGER BF report: U 0.500001