Re: large files and low memory

From: Shawn Pearce <spearce@spearce.org>
Date: Oct 4, 2010, 19:11 UTC
Message-ID: <AANLkTin-mbzt93DWtp71vYBEUcDLHgo=G-6zynT5NC_r@mail.gmail.com>
In-Reply-To: <20101004185854.GA6466@burratino>

On Mon, Oct 4, 2010 at 11:58 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:

Show 13 quoted lines

> Shawn Pearce wrote:
>
>> The mmap() isn't the problem.  Its the allocation of a buffer that is
>> larger than the file in order to hold the result of deflating the file
>> before it gets written to disk.
>
> Wasn't this already fixed, at least in some cases?
>
> commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)
> Author: Nicolas Pitre <nico@fluxnic.net>
> Date:   Sat Feb 20 23:27:31 2010 -0500
>
>    sha1_file: don't malloc the whole compressed result when writing out objects

This change only removes the deflate copy. But due to the SHA-1 consistency issue I alluded to earlier, I think we're still making a full copy of the file in memory before we SHA-1 it or deflate it. So Nico halved the memory usage, but we're still using 1x the size of the file rather than ~2x.

-- 
Shawn.