RE: [BUG] malloc error when using large file.

From: Emmanuel Puerto <emmanuel.puerto@maxim-ic.com>
Date: Jun 24, 2009, 08:35 UTC
Message-ID: <B03A893ADFE80748942D13AFAF2845190C2D5D3FB0@ITSVLEX06.it.maxim-ic.internal>
In-Reply-To: <alpine.LFD.2.01.0906231412070.3240@localhost.localdomain>

Hi Linus,

Show 26 quoted lines

> > 
> > 
> > > Many operations in git assume that a whole file can be held in memory at
> > > once. Do you actually have 2G available on the machine? Are there any OS
> > > limits that might be a problem?
> > 
> > Thanks for your answer Jeff, 
> > 
> > Yes I have more that 2 Go available when a do it, and after this 
> > allocation error if I try to do a 'git add' with a 9Go file, I did not 
> > have any issue.
>
> Can you add the 2GB files individually, using
>
>	for i in All-f00?.vmdk; do git add $i; done
>
> or similar?
>
> It does sound like an unlucky malloc VM space fragmentation issue, 
> although since the files you add are the same size I'd have not really 
> expected that with any normal malloc library. But with some mixed small 
> and large allocations, I could imagine that the fragmentation happens, and 
> you end up having huge areas that aren't _quite_ useful for the next file 
> due to having some small allocation taking part of it.
>
>		Linus

Sorry, But I have already made this without any success. I really think that the issue is the size of this file. But we never know so I redo it again and here you have the result:

$ free
             total       used       free     shared    buffers     cached
Mem:       3954752     662264    3292488          0      17304     233092
-/+ buffers/cache:     411868    3542884
Swap:      8385920          0    8385920
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ for i in All-f00?.vmdk; do git add $i; done
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ free
             total       used       free     shared    buffers     cached
Mem:       3954752    3928772      25980          0       6052    3510268
-/+ buffers/cache:     412452    3542300
Swap:      8385920         88    8385832
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$

I do nothing else, only a top to know the memory use and I see that just before this 'malloc failed', I have something like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
  4323 epuerto   20   0 2115m 2.0g 2.0g D 20.2 51.7   0:24.64 git

And you can see that it appear some memory allocation is not free after this 'malloc failed' error.

If you have any suggestion, any test (or beta version with more debug message) I can do, you are welcome.

Best regards, Emmanuel.