Re: Handling large files with GIT

From: Junio C Hamano <junkio@cox.net>
Date: Feb 8, 2006, 20:11 UTC
Message-ID: <7v4q39pq4t.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <Pine.LNX.4.64.0602080853480.2458@g5.osdl.org>

Linus Torvalds <torvalds@osdl.org> writes:

> Side note: the original explicit git "delta" objects by Nicolas Pitre 
> would have handled this large-file-case much more gracefully.

True.

> The pack-files had absolutely huge advantages, though, so I think we (I) 
> did the right thing there in making the delta code only a very specific 
> special case..

Well the blame for ripping that out falls on me, actually...

> It is possible that we could re-introduce the "explicit delta" object, 
> though (it's not incompatible with also doing pack-files, it's just that 
> pack-files made 99% of all the arguments for an explicit delta go away).

I do not remember we had 'rev-list --objects' support for Nico's explicit delta object chains. If we didn't that would be a new development that needs to be done to resurrect it. I know pack-objects never had support for it so obviously that needs to be added as well. Probably explicit delta objects should always be packed in full without spending cost to find delta candidates.

Personally I feel that post-1.2.0 would be a good time to start looking at enhancing the pack generation chain, rev-list piped to pack-objects. This "large files" use case is helped by less self-contained packs while "shallow clone" use case we discussed earlier is helped by more self-contained packs (we had a discussion long time ago on this and I think we have the code to do so [*1*]).

An addition to pack-objects is needed to make it capable to read a list of objects that we do not want to include in the resulting pack but can be used as base objects for delitified.

BTW, as to the "shallow clone", I changed my mind and am inclined to agree with Johannes that handling cut-offs differently from grafts is easier for dealing with later "give me more history" operation, so I am planning to chuck my jc/clone topic branch that I have included in the proposed updates so far.

[Footnote]

*1* http://article.gmane.org/gmane.comp.version-control.git/5779