Re: [PATCH] Support 64-bit indexes for pack files.

From: Nicolas Pitre <nico@cam.org>
Date: Feb 28, 2007, 04:12 UTC
Message-ID: <alpine.LRH.0.82.0702272258080.29426@xanadu.home>
In-Reply-To: <20070228035248.GB5597@spearce.org>

On Tue, 27 Feb 2007, Shawn O. Pearce wrote:

Show 14 quoted lines

> Nicolas Pitre <nico@cam.org> wrote:
> > On Tue, 27 Feb 2007, Geert Bosch wrote:
> > > For smaller packs, the I/O is all going to be buffered anyway,
> > > but if we're going to have >4GB pack files, it adds a lot of real
> > > I/O  and SHA1 computation for no good reason. If we get a rare chance
> > > to have a new pack format, why not fix this wart at the same time?
> > 
> > Fair enough.
> 
> OK, so lets say that if both ends of a network transport support
> pack v4 then we can use pack v4.  If pack v4 omits the count field
> from its header (because its easily derived or obtained from the
> index, and doesn't add any additional data protection over the
> SHA-1s)

I don't agree we should omit it at all.

I agree it might just not be included in the whole pack SHA1. Actually the whole pack header might be ommitted from the pack SHA1. This way, if data is appended at the end like index-pack does in the thin pack case, the header can be fixed up without invalidating the SHA1 that was computed while receiving the pack. Why the whole header? Well who knows if we won't dynamically change the pack version while receiving it as well in the future. In fact, we could just add the header to the SHA1 checksum but after the data payload (for the checksum only not in the actual pack). this way we preserve the same protection/validity strength as we have today.

Show 5 quoted lines

> why not add some machine-readable sideband that can provide
> transfer progress?
> 
> I think we would want four values, number of objects (sent/total)
> and uncompressed bytes (sent/total), to send to the client.

Uncompressed bytes is meaningless when evaluating throughput of compressed data since compression ratio is variable during a single fetch. It doesn't provide more accurate progress than what we have today with number of objects either.

Show 6 quoted lines

> Estimating the total uncompressed bytes is very easy in pack-objects
> before we start sending even the header; actually if we are reusing
> a majority of the objects from an existing packfile we even have
> a good approximation of the compressed size ready.  That would
> give the client a reasonable progress meter; certainly better than
> nothing at all!  ;-)

IMHO it is not worth it. And it won't help git-pack-index if run on a local pack either.

Nicolas