git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: "git fsck" fails on malloc of 80 G

From
Jeff King <peff@peff.net>
Date
Dec 18, 2013, 22:09 UTC
Message-ID
<20131218220922.GA16347@sigill.intra.peff.net>
In-Reply-To
<201312182108.rBIL8lAo015570@hobgoblin.ariadne.com>
On Wed, Dec 18, 2013 at 04:08:47PM -0500, Dale R. Worley wrote:
Show 10 quoted lines
> > From: Jeff King <peff@peff.net>
> 
> > One of the problems I ran into recently is that
> > corrupt data can cause it to make a large allocation
> 
> One thing I notice is that in unpack_compressed_entry() in
> sha1_file.c, there is a mallocz of "size" bytes.  It appears that
> "size" is the size of the object that is being unpacked.  If so, this
> code cannot be correct, because it assumes that any file that is
> stored in the repository can be put into a buffer allocated in RAM.

For some definition of correct. Git does load whole-blobs into memory in several places. Some code paths _can_ stream, but they do not stream deltas, and the diff engine definitely wants the whole thing in-core.

So you are reading it right. If you want to work on changing it, be my guest, but it's a non-trivial fix. ;)

-Peff
Previous: Dale R. Worley
Message 6 of 6 in “"git fsck" fails on malloc of 80 G”
  1. Dale R. WorleyDec 16, 2013
  2. Jeff KingDec 16, 2013
  3. Dale R. WorleyDec 18, 2013
  4. Jeff KingDec 18, 2013
  5. Dale R. WorleyDec 18, 2013
  6. Jeff KingDec 18, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.