Re: Mozilla .git tree

From: Junio C Hamano <junkio@cox.net>
Date: Aug 30, 2006, 11:42 UTC
Message-ID: <7vzmdmh2lu.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <Pine.LNX.4.64.0608300124550.9796@xanadu.home>

Nicolas Pitre <nico@cam.org> writes:

> And the zlib header contains a CRC which we're about to use for 
> validating the data when doing delta data reuse in order to prevent pack 
> corruption propagation like the one recently posted on the list.

Ah, never thought of using the CRC directly. I was thinking about inflating into void and see if it succeeds, which as you say is perhaps quite expensive.

This brings me back to my pet-peeve, though. I do not think zlib stream seeks back and leaves some clue at the beginning to tell me the deflated length, so it is quite hard to find where each deflated stream ends in a packfile cheaply. Loose objects (with new or legacy style header) are easy (st.st_size is available), but I do not think of a way short of building a reverse index of pack .idx file, which means I am already talking about not so cheap way X-<.

It might be a reason to define a new .idx format. We could lift 32-bit offset limitation while we are at it. Each entry could have 20-byte hash, 64-bit offset into the corresponding .pack, and 32-bit deflated length (heh, why not make it 64-bit while we are at it). Luckily, .idx _is_ a local matter so we can even have a flag day and tell people to run the updated index-pack on existing packfiles to regenerate .idx.

Show 6 quoted lines

> Using an offset instead of a sha1 to reference a delta base object is 
> certainly a good idea though.  But I'd use the same variable encoding as 
> the object size to avoid the 32-bit limit issue.  When generating a thin 
> pack the real sha1 of the delta object could be substituted for the 
> offset quite easily if the base object is not sent a part of the same 
> pack.

That sounds quite a reasonable suggestion. I love this kind of moment when I find us very fortunate to have bright people on the list ;-).