From: Jeff King <peff@peff.net>
Date: Wed, 18 Dec 2013 22:09:23 GMT
Subject: Re: "git fsck" fails on malloc of 80 G
Message-ID: <20131218220922.GA16347@sigill.intra.peff.net>
In-Reply-To: <201312182108.rBIL8lAo015570@hobgoblin.ariadne.com>

On Wed, Dec 18, 2013 at 04:08:47PM -0500, Dale R. Worley wrote:

> > From: Jeff King <peff@peff.net>
> 
> > One of the problems I ran into recently is that
> > corrupt data can cause it to make a large allocation
> 
> One thing I notice is that in unpack_compressed_entry() in
> sha1_file.c, there is a mallocz of "size" bytes.  It appears that
> "size" is the size of the object that is being unpacked.  If so, this
> code cannot be correct, because it assumes that any file that is
> stored in the repository can be put into a buffer allocated in RAM.

For some definition of correct. Git does load whole-blobs into memory in
several places. Some code paths _can_ stream, but they do not stream
deltas, and the diff engine definitely wants the whole thing in-core.

So you are reading it right. If you want to work on changing it, be my
guest, but it's a non-trivial fix. ;)

-Peff