{"thread":{"id":"39487","subject":"git-repack keeps running out of memory","startedAt":"2015-06-01T18:39:04Z","lastAt":"2015-06-02T00:04:22Z","messageCount":5,"participants":["Phil Susi","Junio C Hamano","Jeff King","Phillip Susi"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"262637","messageId":"556CA6C8.6000804@ubuntu.com","threadId":"39487","inReplyTo":null,"subject":"git-repack keeps running out of memory","fromName":"Phil Susi","fromEmail":"psusi@ubuntu.com","sentAt":"2015-06-01T18:39:04Z","receivedAt":"2015-06-01T18:39:04Z","isPatch":false,"sender":{"key":"psusi@ubuntu.com","avatar":null},"body":"I keep having git-repack run out of virtual memory ( 32 bit system ) \nwhen trying to repack my linux kernel repo.  It keeps making it right up \nto 99% then barfing saying mmap failed: Cannot allocate memory.\n\nI thought I could help this by limiting the pack size, and using \n--window-memory to limit the memory usage, but it still happens with \nthis full command line:\n\ngit repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250 \n--window-memory=500m\n\nThe key factor seems to be the --window... with 50 it works fine, but \nwith 100 or more, even with the very low --window-memory limit, it crashes.\n"},{"id":"262638","messageId":"xmqqlhg35ky2.fsf@gitster.dls.corp.google.com","threadId":"39487","inReplyTo":"556CA6C8.6000804@ubuntu.com","subject":"Re: git-repack keeps running out of memory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-06-01T18:43:01Z","receivedAt":"2015-06-01T18:43:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phil Susi <psusi@ubuntu.com> writes:\n\n> I keep having git-repack run out of virtual memory ( 32 bit system )\n> when trying to repack my linux kernel repo.  It keeps making it right\n> up to 99% then barfing saying mmap failed: Cannot allocate memory.\n>\n> I thought I could help this by limiting the pack size, and using\n> --window-memory to limit the memory usage, but it still happens with\n> this full command line:\n>\n> git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250\n> --window-memory=500m\n>\n> The key factor seems to be the --window... with 50 it works fine, but\n> with 100 or more, even with the very low --window-memory limit, it\n> crashes.\n\nUnfortunately, that is totally expected.  Window tells us to keep\nenough information to compare with that many objects in-core.  I do\nnot think max-pack-size would affect much.\n"},{"id":"262642","messageId":"556CAD64.8000208@ubuntu.com","threadId":"39487","inReplyTo":"xmqqlhg35ky2.fsf@gitster.dls.corp.google.com","subject":"Re: git-repack keeps running out of memory","fromName":"Phil Susi","fromEmail":"psusi@ubuntu.com","sentAt":"2015-06-01T19:07:16Z","receivedAt":"2015-06-01T19:07:16Z","isPatch":false,"sender":{"key":"psusi@ubuntu.com","avatar":null},"body":"On 6/1/2015 2:43 PM, Junio C Hamano wrote:\n> Phil Susi <psusi@ubuntu.com> writes:\n>\n>> I keep having git-repack run out of virtual memory ( 32 bit system )\n>> when trying to repack my linux kernel repo.  It keeps making it right\n>> up to 99% then barfing saying mmap failed: Cannot allocate memory.\n>>\n>> I thought I could help this by limiting the pack size, and using\n>> --window-memory to limit the memory usage, but it still happens with\n>> this full command line:\n>>\n>> git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250\n>> --window-memory=500m\n>>\n>> The key factor seems to be the --window... with 50 it works fine, but\n>> with 100 or more, even with the very low --window-memory limit, it\n>> crashes.\n>\n> Unfortunately, that is totally expected.  Window tells us to keep\n> enough information to compare with that many objects in-core.  I do\n> not think max-pack-size would affect much.\n\nIt's more the --window-memory argument that is important here: it is \nsupposed to prevent exactly this problem.  I guess I tried adding the \n--max-pack-size as well on the off chance that it would help.\n"},{"id":"262652","messageId":"20150601205854.GA21144@peff.net","threadId":"39487","inReplyTo":"556CAD64.8000208@ubuntu.com","subject":"Re: git-repack keeps running out of memory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-06-01T20:58:54Z","receivedAt":"2015-06-01T20:58:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 01, 2015 at 03:07:16PM -0400, Phil Susi wrote:\n\n> It's more the --window-memory argument that is important here: it is\n> supposed to prevent exactly this problem.  I guess I tried adding the\n> --max-pack-size as well on the off chance that it would help.\n\nHow many processors do you have? The window-memory is per-thread. Try\nwith --threads=1.\n\n-Peff\n"},{"id":"262681","messageId":"556CF306.6000308@ubuntu.com","threadId":"39487","inReplyTo":"20150601205854.GA21144@peff.net","subject":"Re: git-repack keeps running out of memory","fromName":"Phillip Susi","fromEmail":"psusi@ubuntu.com","sentAt":"2015-06-02T00:04:22Z","receivedAt":"2015-06-02T00:04:22Z","isPatch":false,"sender":{"key":"psusi@ubuntu.com","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA512\n\nOn 06/01/2015 04:58 PM, Jeff King wrote:\n> How many processors do you have? The window-memory is per-thread.\n> Try with --threads=1.\n\nAhh, right... 8... that explains it ;)\n\n\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1\n\niQEcBAEBCgAGBQJVbPMBAAoJENRVrw2cjl5R2rkH/0qa9HerYM7Bw5qfcV3zahFz\n5g/NGoW3M9IutqqLiyDzAQboj62+TzrEuM/fo7BgZrKYBsA7/4SNdJMmabaUb5Kr\nrtW1qfpErIHUQUY0wAxBwXX7gpihB3HTYRsZAX+7Itfae7bw8r3krj/L1Ty2nj3Y\nQLIm9uwijiBNpHoF9Jnl6gbklaVFbBBfdGH2uPLzoso5YdlJ0L6o4VDmeD55njjx\n30cOLDwInQ4zxdGKBH1SaIXGq+2Vi1txLrzYaaEOA+NW9+Rp7J9Anqcv/IH5yyX/\ny9H0yDXXP/3gPwTHvYiRTVWMjAvjxSdc4n4ZgySLBPTruW1utxjPdaTSffm2xsg=\n=wza7\n-----END PGP SIGNATURE-----\n"}]}