Re: [PATCH] fast-import: Stream very large blobs directly to pack

From: Nicolas Pitre <nico@fluxnic.net>
Date: Jan 29, 2010, 02:33 UTC
Message-ID: <alpine.LFD.2.00.1001282125410.1681@xanadu.home>
In-Reply-To: <20100129012350.GD20488@spearce.org>

On Thu, 28 Jan 2010, Shawn O. Pearce wrote:

Show 5 quoted lines

> If a blob is larger than the configured big-file-threshold, instead
> of reading it into a single buffer obtained from malloc, stream it
> onto the end of the current pack file.  Streaming the larger objects
> into the pack avoids the 4+ GiB memory footprint that occurs when
> fast-import is processing 2+ GiB blobs.

Yeah. I've had that item on my todo list for ages now. This big-file-threshold principle has to be applied to 'git add' too so a big blob is stored in pack file form right away, and used to bypass delta searching in 'git pack-objects', used to skip the diff machinery, and so on.

Nicolas