threads / discuss / 28390

git gc exit with out of memory, malloc failed error

Subject: git gc exit with out of memory, malloc failed error

## tl;dr

5 messages between Sep 15, 2011 and Sep 15, 2011.

replies: 4people: 4as markdown or json

Alexander Kostikov· Sep 15, 2011, 01:33 UTC · lore

I'm new to git and I'm getting the following out of memory error on git gc:

$ git gc Counting objects: 80818, done. Delta compression using up to 8 threads. fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes) error: failed to run repack

The only advice I found in the internet suggested to run repack with --window-memory parameter specified. But this call also fails:

$ git repack -adf --window-memory=0 Counting objects: 80818, done. Delta compression using up to 8 threads. warning: suboptimal pack - out of memory fatal: Out of memory, malloc failed (tried to allocate 24356363 bytes)

How do I cleanup my repository?

$ git version git version 1.7.6.msysgit.0

OS: Windows Server 2008 R2 SP1 (x64)
Physical memory: 24 GB
The commands listed were executed under x64 console process.

-- 
Thanks,
Alexander Kostikov

David Michael Barr· Sep 15, 2011, 02:20 UTC · re: Alexander Kostikov · lore

Re: git gc exit with out of memory, malloc failed error

On Thu, Sep 15, 2011 at 11:33 AM, Alexander Kostikov <alex.kostikov@gmail.com> wrote:

Show 34 quoted lines

> I'm new to git and I'm getting the following out of memory error on git gc:
>
> $ git gc
> Counting objects: 80818, done.
> Delta compression using up to 8 threads.
> fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
> error: failed to run repack
>
> The only advice I found in the internet suggested to run repack with
> --window-memory parameter specified. But this call also fails:
>
> $ git repack -adf --window-memory=0
> Counting objects: 80818, done.
> Delta compression using up to 8 threads.
> warning: suboptimal pack - out of memory
> fatal: Out of memory, malloc failed (tried to allocate 24356363 bytes)
>
> How do I cleanup my repository?
>
> $ git version
> git version 1.7.6.msysgit.0
>
> OS: Windows Server 2008 R2 SP1 (x64)
> Physical memory: 24 GB
> The commands listed were executed under x64 console process.
>
> --
> Thanks,
> Alexander Kostikov
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Hi,

My understanding is that msysgit is a 32-bit build. So if your existing pack is ~2GB, repack will fail. Also, I think that setting window-memory to 0 means no limit, which is not what you want. One value I have seen suggested is 256m. In my experience, peak memory consumption of repack is proportional to the value of --window, so you might want to try tweaking that. There are quite a few config parameters that affect the memory consumption of repack.

-- 
David Barr

Alexander Kostikov· Sep 15, 2011, 02:42 UTC · re: David Michael Barr · lore

Re: git gc exit with out of memory, malloc failed error

Thanks! --window-memory=50m worked for my project. 256m was still throwing error. Probably real memory consumption is close to threads number * window memory.

On Wed, Sep 14, 2011 at 7:20 PM, David Michael Barr <davidbarr@google.com> wrote:

Show 53 quoted lines

> On Thu, Sep 15, 2011 at 11:33 AM, Alexander Kostikov
> <alex.kostikov@gmail.com> wrote:
>> I'm new to git and I'm getting the following out of memory error on git gc:
>>
>> $ git gc
>> Counting objects: 80818, done.
>> Delta compression using up to 8 threads.
>> fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
>> error: failed to run repack
>>
>> The only advice I found in the internet suggested to run repack with
>> --window-memory parameter specified. But this call also fails:
>>
>> $ git repack -adf --window-memory=0
>> Counting objects: 80818, done.
>> Delta compression using up to 8 threads.
>> warning: suboptimal pack - out of memory
>> fatal: Out of memory, malloc failed (tried to allocate 24356363 bytes)
>>
>> How do I cleanup my repository?
>>
>> $ git version
>> git version 1.7.6.msysgit.0
>>
>> OS: Windows Server 2008 R2 SP1 (x64)
>> Physical memory: 24 GB
>> The commands listed were executed under x64 console process.
>>
>> --
>> Thanks,
>> Alexander Kostikov
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> Hi,
>
> My understanding is that msysgit is a 32-bit build.
> So if your existing pack is ~2GB, repack will fail.
> Also, I think that setting window-memory to 0
> means no limit, which is not what you want.
> One value I have seen suggested is 256m.
> In my experience, peak memory consumption of
> repack is proportional to the value of --window,
> so you might want to try tweaking that.
> There are quite a few config parameters that
> affect the memory consumption of repack.
>
> --
> David Barr
>

-- Your sincerely, Alexander Kostikov

Brandon Casey· Sep 15, 2011, 02:29 UTC · lore

Re: git gc exit with out of memory, malloc failed error

[resend since gmail's Rich formatting was enabled]

On Wed, Sep 14, 2011 at 8:33 PM, Alexander Kostikov <alex.kostikov@gmail.com> wrote:

Show 8 quoted lines

>
> I'm new to git and I'm getting the following out of memory error on git gc:
>
> $ git gc
> Counting objects: 80818, done.
> Delta compression using up to 8 threads.
> fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
> error: failed to run repack

Try reducing the number of threads that are used.  You must have some
pretty large objects if you have 24GB and ran out of memory.  The
following will configure git to use only one thread.
   git config pack.threads=1
-Brandon

Carlos Martín Nieto· Sep 15, 2011, 09:16 UTC · re: Brandon Casey · lore

Re: git gc exit with out of memory, malloc failed error

On Wed, 2011-09-14 at 21:29 -0500, Brandon Casey wrote:

Show 15 quoted lines

> [resend since gmail's Rich formatting was enabled]
> 
> On Wed, Sep 14, 2011 at 8:33 PM, Alexander Kostikov
> <alex.kostikov@gmail.com> wrote:
> >
> > I'm new to git and I'm getting the following out of memory error on git gc:
> >
> > $ git gc
> > Counting objects: 80818, done.
> > Delta compression using up to 8 threads.
> > fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
> > error: failed to run repack
> 
> Try reducing the number of threads that are used.  You must have some
> pretty large objects if you have 24GB and ran out of memory.  The

IIRC the 32-bit Windows builds limit each user process to 2GB. If msysgit is built in 32-bit mode, it's possible (probable even, in order to remain backwards-compatible) that it's hitting this limit, rather than the physical limit of the machine. If it's using 8 threads, it means that there are 256MB available for each thread.

cmn