Re: Git Large Object Support Proposal

From: david@lang.hm <david@lang.hm>
Date: Mar 19, 2009, 23:42 UTC
Message-ID: <alpine.DEB.1.10.0903191625360.16753@asgard.lang.hm>
In-Reply-To: <7veiwt6t6a.fsf@gitster.siamese.dyndns.org>

On Thu, 19 Mar 2009, Junio C Hamano wrote:

Show 18 quoted lines

> Scott Chacon <schacon@gmail.com> writes:
>
>> But where Git instead stores a stub object and the large binary object
>> is pulled in via a separate mechanism. I was thinking that the client
>> could set a max file size and when a binary object larger than that is
>> staged, Git instead writes a stub blob like:
>>
>> ==
>> blob [size]\0
>> [sha of large blob]
>> ==
>
> An immediate pair of questions are, if you can solve the issue by
> delegating large media to somebody else (i.e. "media server"), and that
> somebody else can solve the issues you are having, (1) what happens if you
> lower that "large" threashold to "0 byte"?  Does that somebody else still
> work fine, and does the git that uses indirection also still work fine?
> If so why are you using git instead of that somebody else altogether?

ideally the difference between useing git with 'large' set to 0 and git with no pack file should be an extra lookup for the indirection.

it may be that some other file manipulation may not be possible for 'large' files, resulting in some reduced functionality.

in any case, the added efficiancy of using pack files (both for local storage and for network transport) will make handling the 'large' files worse than the same size files through git (assuming that they can benifit from delta compression)

> and
> (2) what prevents us from stealing the trick that somebody else uses so
> that git itself can natively handle large blobs without indirection?

the key thing is that large files do not get mmaped or considered for inclusion in pack files (including cloning and pulling pack files)

to make them full first-class citizens you would need to make alternate code paths for everything that currently does mmap, making those paths either process the file a different way. in the long run that may be the best thing to do, but that's a lot of change compared to the proposed change.

> Without thinking the ramifications through myself, this sounds pretty much
> like a band-aid and will nend up hitting the same "blob is larger than we
> can handle" issue when you follow the indirection eventually, but that is
> just my gut feeling.

it depends on what you are doing with that file when you get to it. if you have to mmap it you may run into the same problem. but if the file is a streaming video, you can transport it around (with rsync, http, etc) without a problem, and using the file (playing the video) never keeps much of the file in memory, so it will be very useful on systems that would never have a chance of accessing the entire file through mmap.

David Lang