threads / discuss / 27355

how to keep git-fetch from running out of memory?

Subject: how to keep git-fetch from running out of memory?

## tl;dr

4 messages between May 15, 2011 and May 15, 2011.

replies: 3people: 3as markdown or json

Kartik Agaram· May 15, 2011, 05:24 UTC · lore

I have a git repo with some large files that I'm no longer able to update. git fetch keeps running out of memory:

  fatal: Out of memory, malloc failed
  fatal: unpack-objects died with error code 128

Anybody know how to keep it from compressing the refs into packfiles? I've experimented with core.compression, pack.compression, pack.windowMemory, pack.packSizeLimit, all without luck :(

Shawn Pearce· May 15, 2011, 19:13 UTC · re: Kartik Agaram · lore

Re: how to keep git-fetch from running out of memory?

On Sat, May 14, 2011 at 22:24, Kartik Agaram <ak@akkartik.com> wrote:

Show 9 quoted lines

> I have a git repo with some large files that I'm no longer able to
> update. git fetch keeps running out of memory:
>
>  fatal: Out of memory, malloc failed
>  fatal: unpack-objects died with error code 128
>
> Anybody know how to keep it from compressing the refs into packfiles?
> I've experimented with core.compression, pack.compression,
> pack.windowMemory, pack.packSizeLimit, all without luck :(

Instead of playing with these settings, try transfer.unpackLimit 1. It will force the code to use index-pack rather than unpack-objects, which has a different memory profile.

However, that may still be insufficient. A big object must still be allocated in memory in order to compute its SHA-1. If you don't have sufficient memory, you need to increase your ulimits, or reconfigure your system to have more virtual memory available to the process.

-- 
Shawn.

Junio C Hamano· May 15, 2011, 20:25 UTC · re: Shawn Pearce · lore

Re: how to keep git-fetch from running out of memory?

Shawn Pearce <spearce@spearce.org> writes:

> However, that may still be insufficient. A big object must still be
> allocated in memory in order to compute its SHA-1.

I wonder if we can stream this?

Shawn Pearce· May 15, 2011, 20:37 UTC · re: Junio C Hamano · lore

Re: how to keep git-fetch from running out of memory?

On Sun, May 15, 2011 at 13:25, Junio C Hamano <gitster@pobox.com> wrote:

Show 6 quoted lines

> Shawn Pearce <spearce@spearce.org> writes:
>
>> However, that may still be insufficient. A big object must still be
>> allocated in memory in order to compute its SHA-1.
>
> I wonder if we can stream this?

If its a non-delta, yes. We already do this in JGit for blobs over 50M. (Trees, commits and tags are not streamed because we might need to do fsck on them and the fsck code for these object types requires the entire object in a single buffer.)

Its probably not a huge change to index-pack.c. But I haven't dug around in there in a while. I wish I could say I will look at this on Monday, but I don't have the time.

-- 
Shawn.