threads / discuss / 39487

git-repack keeps running out of memory

Subject: git-repack keeps running out of memory

## tl;dr

5 messages between Jun 1, 2015 and Jun 2, 2015.

replies: 4people: 3as markdown or json

Phil Susi· Jun 1, 2015, 18:39 UTC · lore

I keep having git-repack run out of virtual memory ( 32 bit system ) when trying to repack my linux kernel repo. It keeps making it right up to 99% then barfing saying mmap failed: Cannot allocate memory.

I thought I could help this by limiting the pack size, and using --window-memory to limit the memory usage, but it still happens with this full command line:

git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250 --window-memory=500m

The key factor seems to be the --window... with 50 it works fine, but with 100 or more, even with the very low --window-memory limit, it crashes.

Junio C Hamano· Jun 1, 2015, 18:43 UTC · re: Phil Susi · lore

Re: git-repack keeps running out of memory

Phil Susi <psusi@ubuntu.com> writes:
Show 14 quoted lines
> I keep having git-repack run out of virtual memory ( 32 bit system )
> when trying to repack my linux kernel repo.  It keeps making it right
> up to 99% then barfing saying mmap failed: Cannot allocate memory.
>
> I thought I could help this by limiting the pack size, and using
> --window-memory to limit the memory usage, but it still happens with
> this full command line:
>
> git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250
> --window-memory=500m
>
> The key factor seems to be the --window... with 50 it works fine, but
> with 100 or more, even with the very low --window-memory limit, it
> crashes.

Unfortunately, that is totally expected. Window tells us to keep enough information to compare with that many objects in-core. I do not think max-pack-size would affect much.

Phil Susi· Jun 1, 2015, 19:07 UTC · re: Junio C Hamano · lore

Re: git-repack keeps running out of memory

On 6/1/2015 2:43 PM, Junio C Hamano wrote:
Show 20 quoted lines
> Phil Susi <psusi@ubuntu.com> writes:
>
>> I keep having git-repack run out of virtual memory ( 32 bit system )
>> when trying to repack my linux kernel repo.  It keeps making it right
>> up to 99% then barfing saying mmap failed: Cannot allocate memory.
>>
>> I thought I could help this by limiting the pack size, and using
>> --window-memory to limit the memory usage, but it still happens with
>> this full command line:
>>
>> git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250
>> --window-memory=500m
>>
>> The key factor seems to be the --window... with 50 it works fine, but
>> with 100 or more, even with the very low --window-memory limit, it
>> crashes.
>
> Unfortunately, that is totally expected.  Window tells us to keep
> enough information to compare with that many objects in-core.  I do
> not think max-pack-size would affect much.

It's more the --window-memory argument that is important here: it is supposed to prevent exactly this problem. I guess I tried adding the --max-pack-size as well on the off chance that it would help.

Jeff King· Jun 1, 2015, 20:58 UTC · re: Phil Susi · lore

Re: git-repack keeps running out of memory

On Mon, Jun 01, 2015 at 03:07:16PM -0400, Phil Susi wrote:
> It's more the --window-memory argument that is important here: it is
> supposed to prevent exactly this problem.  I guess I tried adding the
> --max-pack-size as well on the off chance that it would help.

How many processors do you have? The window-memory is per-thread. Try with --threads=1.

-Peff
Phillip Susi· Jun 2, 2015, 00:04 UTC · re: Jeff King · lore

Re: git-repack keeps running out of memory

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
On 06/01/2015 04:58 PM, Jeff King wrote:
> How many processors do you have? The window-memory is per-thread.
> Try with --threads=1.
Ahh, right... 8... that explains it ;)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJVbPMBAAoJENRVrw2cjl5R2rkH/0qa9HerYM7Bw5qfcV3zahFz 5g/NGoW3M9IutqqLiyDzAQboj62+TzrEuM/fo7BgZrKYBsA7/4SNdJMmabaUb5Kr rtW1qfpErIHUQUY0wAxBwXX7gpihB3HTYRsZAX+7Itfae7bw8r3krj/L1Ty2nj3Y QLIm9uwijiBNpHoF9Jnl6gbklaVFbBBfdGH2uPLzoso5YdlJ0L6o4VDmeD55njjx 30cOLDwInQ4zxdGKBH1SaIXGq+2Vi1txLrzYaaEOA+NW9+Rp7J9Anqcv/IH5yyX/ y9H0yDXXP/3gPwTHvYiRTVWMjAvjxSdc4n4ZgySLBPTruW1utxjPdaTSffm2xsg= =wza7 -----END PGP SIGNATURE-----

← back to recent threads