{"thread":{"id":"27355","subject":"how to keep git-fetch from running out of memory?","startedAt":"2011-05-15T05:24:00Z","lastAt":"2011-05-15T20:37:27Z","messageCount":4,"participants":["Kartik Agaram","Shawn Pearce","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"167881","messageId":"BANLkTinnOTr+PUzBMpx1DLcxs8CVWjy33A@mail.gmail.com","threadId":"27355","inReplyTo":null,"subject":"how to keep git-fetch from running out of memory?","fromName":"Kartik Agaram","fromEmail":"ak@akkartik.com","sentAt":"2011-05-15T05:24:00Z","receivedAt":"2011-05-15T05:24:00Z","isPatch":false,"sender":{"key":"ak@akkartik.com","avatar":"https://gravatar.com/avatar/bfa044e06852c348b2777d9a9a558c179429443f5d7b65d97291ae787b8ccc52?d=mp&s=160"},"body":"I have a git repo with some large files that I'm no longer able to\nupdate. git fetch keeps running out of memory:\n\n  fatal: Out of memory, malloc failed\n  fatal: unpack-objects died with error code 128\n\nAnybody know how to keep it from compressing the refs into packfiles?\nI've experimented with core.compression, pack.compression,\npack.windowMemory, pack.packSizeLimit, all without luck :(\n"},{"id":"167903","messageId":"BANLkTinRkePRMQzQS7k=3bSTxTJRVpG39Q@mail.gmail.com","threadId":"27355","inReplyTo":"BANLkTinnOTr+PUzBMpx1DLcxs8CVWjy33A@mail.gmail.com","subject":"Re: how to keep git-fetch from running out of memory?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-15T19:13:55Z","receivedAt":"2011-05-15T19:13:55Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sat, May 14, 2011 at 22:24, Kartik Agaram <ak@akkartik.com> wrote:\n> I have a git repo with some large files that I'm no longer able to\n> update. git fetch keeps running out of memory:\n>\n>  fatal: Out of memory, malloc failed\n>  fatal: unpack-objects died with error code 128\n>\n> Anybody know how to keep it from compressing the refs into packfiles?\n> I've experimented with core.compression, pack.compression,\n> pack.windowMemory, pack.packSizeLimit, all without luck :(\n\nInstead of playing with these settings, try transfer.unpackLimit 1. It\nwill force the code to use index-pack rather than unpack-objects,\nwhich has a different memory profile.\n\nHowever, that may still be insufficient. A big object must still be\nallocated in memory in order to compute its SHA-1. If you don't have\nsufficient memory, you need to increase your ulimits, or reconfigure\nyour system to have more virtual memory available to the process.\n\n-- \nShawn.\n"},{"id":"167919","messageId":"7vboz3pv7x.fsf@alter.siamese.dyndns.org","threadId":"27355","inReplyTo":"BANLkTinRkePRMQzQS7k=3bSTxTJRVpG39Q@mail.gmail.com","subject":"Re: how to keep git-fetch from running out of memory?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-15T20:25:22Z","receivedAt":"2011-05-15T20:25:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shawn Pearce <spearce@spearce.org> writes:\n\n> However, that may still be insufficient. A big object must still be\n> allocated in memory in order to compute its SHA-1.\n\nI wonder if we can stream this?\n"},{"id":"167922","messageId":"BANLkTi=+k_qh_RzuL73NPnEEcpA=0B8X_Q@mail.gmail.com","threadId":"27355","inReplyTo":"7vboz3pv7x.fsf@alter.siamese.dyndns.org","subject":"Re: how to keep git-fetch from running out of memory?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-15T20:37:27Z","receivedAt":"2011-05-15T20:37:27Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, May 15, 2011 at 13:25, Junio C Hamano <gitster@pobox.com> wrote:\n> Shawn Pearce <spearce@spearce.org> writes:\n>\n>> However, that may still be insufficient. A big object must still be\n>> allocated in memory in order to compute its SHA-1.\n>\n> I wonder if we can stream this?\n\nIf its a non-delta, yes. We already do this in JGit for blobs over\n50M. (Trees, commits and tags are not streamed because we might need\nto do fsck on them and the fsck code for these object types requires\nthe entire object in a single buffer.)\n\nIts probably not a huge change to index-pack.c. But I haven't dug\naround in there in a while. I wish I could say I will look at this on\nMonday, but I don't have the time.\n\n-- \nShawn.\n"}]}