Re: how to keep git-fetch from running out of memory?

From: Shawn Pearce <spearce@spearce.org>
Date: May 15, 2011, 20:37 UTC
Message-ID: <BANLkTi=+k_qh_RzuL73NPnEEcpA=0B8X_Q@mail.gmail.com>
In-Reply-To: <7vboz3pv7x.fsf@alter.siamese.dyndns.org>

On Sun, May 15, 2011 at 13:25, Junio C Hamano <gitster@pobox.com> wrote:

Show 6 quoted lines

> Shawn Pearce <spearce@spearce.org> writes:
>
>> However, that may still be insufficient. A big object must still be
>> allocated in memory in order to compute its SHA-1.
>
> I wonder if we can stream this?

If its a non-delta, yes. We already do this in JGit for blobs over 50M. (Trees, commits and tags are not streamed because we might need to do fsck on them and the fsck code for these object types requires the entire object in a single buffer.)

Its probably not a huge change to index-pack.c. But I haven't dug around in there in a while. I wish I could say I will look at this on Monday, but I don't have the time.

-- 
Shawn.