threads / bug / 19897

[BUG] malloc error when using large file.

Subject: [BUG] malloc error when using large file.

## tl;dr

6 messages between Jun 22, 2009 and Jun 24, 2009.

replies: 5people: 3as markdown or json

Emmanuel Puerto· Jun 22, 2009, 09:54 UTC · lore

Hi All,

I use Git with some Virtual Machine and I have an issue with 2 of them. When I use some VM I have created with a "pre-allocated" disk split into a 2Go file. For example a "ls" give:

-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f001.vmdk -rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f002.vmdk -rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f003.vmdk -rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f004.vmdk -rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f005.vmdk -rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f006.vmdk -rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f007.vmdk -rwx------ 1 epuerto epuerto 1835008 mars 10 21:44 All-f008.vmdk -rwx------ 1 epuerto epuerto 1007 juin 18 09:30 All.vmdk -rw------- 1 epuerto epuerto 548 juin 18 09:40 devel -rw------- 1 epuerto epuerto 1538850816 juin 18 17:05 devel-s001 -rw------- 1 epuerto epuerto 1125384192 juin 18 17:05 devel-s002 -rw------- 1 epuerto epuerto 1384382464 juin 18 17:05 devel-s003 -rw------- 1 epuerto epuerto 1632763904 juin 18 17:05 devel-s004 -rw------- 1 epuerto epuerto 1572864 juin 18 17:05 devel-s005 -rwx------ 1 epuerto epuerto 8684 juin 18 17:07 Ubuntu.nvram -rw------- 1 epuerto epuerto 536870912 juin 18 17:06 Ubuntu.vmem -rwx------ 1 epuerto epuerto 0 mars 10 21:31 Ubuntu.vmsd -rw------- 1 epuerto epuerto 135732052 juin 18 17:07 Ubuntu.vmss -rwx------ 1 epuerto epuerto 2908 juin 18 17:07 Ubuntu.vmx -rwx------ 1 epuerto epuerto 261 mars 10 21:31 Ubuntu.vmxf

So if I just do a "git init" and after do a "git add -A" I have the following error: fatal: Out of memory, malloc failed

no more information is given if I use the verbose option of git (git add -v -A).

In fact the error appear when I try to add file name "all-f00?.vmdk"

It appears that this size of file is not accepted by git. If I do the same with other VM that they have a biggest size or a smallest size, all it is ok

For example: -rw------- 1 epuerto epuerto 8684 avril 6 15:56 CentOs 5.2 Dev.nvram -rw------- 1 epuerto epuerto 8684 juin 18 11:36 CentOs 5.3 Dev.nvram -rw------- 1 epuerto epuerto 536870912 juin 18 11:36 CentOs 5.3 Dev.vmem -rw------- 1 epuerto epuerto 0 avril 3 12:26 CentOs 5.3 Dev.vmsd -rw------- 1 epuerto epuerto 18408288 juin 18 11:36 CentOs 5.3 Dev.vmss -rwxr-xr-x 1 epuerto epuerto 2239 juin 18 11:36 CentOs 5.3 Dev.vmx -rw------- 1 epuerto epuerto 269 avril 6 10:43 CentOs 5.3 Dev.vmxf -rw------- 1 epuerto epuerto 1766981632 juin 18 11:36 sda-s001.vmdk -rw------- 1 epuerto epuerto 2094596096 juin 18 11:36 sda-s002.vmdk -rw------- 1 epuerto epuerto 1808138240 juin 18 11:36 sda-s003.vmdk -rw------- 1 epuerto epuerto 1429667840 juin 18 11:36 sda-s004.vmdk -rw------- 1 epuerto epuerto 65536 juin 18 11:36 sda-s005.vmdk -rw------- 1 epuerto epuerto 907 juin 18 10:57 sda.vmdk

OR

-rwxrwxrwx 1 epuerto epuerto 8684 juin 17 21:08 FreeBSD 64-bit.nvram -rwxrwxrwx 1 epuerto epuerto 1114112 mars 27 2008 FreeBSD 64-bit.vmdk -rwxrwxrwx 1 epuerto epuerto 0 mars 27 2008 FreeBSD 64-bit.vmsd -rwxrwxrwx 1 epuerto epuerto 1652 juin 17 19:48 FreeBSD 64-bit.vmx -rwxrwxrwx 1 epuerto epuerto 269 mars 27 2008 FreeBSD 64-bit.vmxf -rwxrwxrwx 1 epuerto epuerto 9118810112 juin 17 21:08 hdd.vmdk

My primary OS (where I use Git) is $uname -a Linux epuerto-FC10.innova-card.internal 2.6.27.24-170.2.68.fc10.x86_64 #1 SMP Wed May 20 22:47:23 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

And my git version is: $ git --version git version 1.6.0.6

my config is: ThinkPad T400 with a Intel(R) Core(TM)2 Duo CPU T9400 and 4Go

Thanks by advance for your help.

Best regards, Emmanuel.

Before you print Think about ENVIRONMENTAL responsibility and commitment

Emmanuel PUERTO Senior MTS, Field Applications (Microcontroller) Linux Embedded Architect

MAXIM Integrated Products, Inc. ZI Athelia IV - Le Forum, Bat A Quartier Roumagoua 13600 La Ciotat - France Phone : +33 44 298 1338 Fax : +33 44 208 3319 Email : emmanuel.puerto@maxim-ic.com Web : www.maxim-ic.com

Jeff King· Jun 22, 2009, 21:15 UTC · re: Emmanuel Puerto · lore

Re: [BUG] malloc error when using large file.

On Mon, Jun 22, 2009 at 02:54:18AM -0700, Emmanuel Puerto wrote:

Show 7 quoted lines

> I use Git with some Virtual Machine and I have an issue with 2 of them.
> When I use some VM I have created with a "pre-allocated" disk split
> into a 2Go file. For example a "ls" give:
> [...]
> So if I just do a "git init" and after do a "git add -A" I have the
> following error:
> fatal: Out of memory, malloc failed

Many operations in git assume that a whole file can be held in memory at once. Do you actually have 2G available on the machine? Are there any OS limits that might be a problem?

-Peff

Emmanuel Puerto· Jun 23, 2009, 07:05 UTC · re: Jeff King · lore

RE: [BUG] malloc error when using large file.

> Many operations in git assume that a whole file can be held in memory at
> once. Do you actually have 2G available on the machine? Are there any OS
> limits that might be a problem?

Thanks for your answer Jeff,

Yes I have more that 2 Go available when a do it, and after this allocation error if I try to do a 'git add' with a 9Go file, I did not have any issue.

Emmanuel.

Jeff King· Jun 23, 2009, 18:42 UTC · re: Emmanuel Puerto · lore

Re: [BUG] malloc error when using large file.

On Tue, Jun 23, 2009 at 12:05:11AM -0700, Emmanuel Puerto wrote:

> Yes I have more that 2 Go available when a do it, and after this
> allocation error if I try to do a 'git add' with a 9Go file, I did not
> have any issue.

It may be that doing a whole bunch of files in one process is leading to some memory fragmentation that doesn't let malloc get a contiguous 2G slice. I'm not sure what else to suggest trying, except perhaps to search the list archives for past discussions on memory fragmentation.

Maybe somebody else has suggestions.

-Peff

Linus Torvalds· Jun 23, 2009, 21:15 UTC · re: Emmanuel Puerto · lore

RE: [BUG] malloc error when using large file.

On Tue, 23 Jun 2009, Emmanuel Puerto wrote:

Show 11 quoted lines

> 
> 
> > Many operations in git assume that a whole file can be held in memory at
> > once. Do you actually have 2G available on the machine? Are there any OS
> > limits that might be a problem?
> 
> Thanks for your answer Jeff, 
> 
> Yes I have more that 2 Go available when a do it, and after this 
> allocation error if I try to do a 'git add' with a 9Go file, I did not 
> have any issue.

Can you add the 2GB files individually, using

	for i in All-f00?.vmdk; do git add $i; done

or similar?

It does sound like an unlucky malloc VM space fragmentation issue, although since the files you add are the same size I'd have not really expected that with any normal malloc library. But with some mixed small and large allocations, I could imagine that the fragmentation happens, and you end up having huge areas that aren't _quite_ useful for the next file due to having some small allocation taking part of it.

		Linus

Emmanuel Puerto· Jun 24, 2009, 08:35 UTC · re: Linus Torvalds · lore

RE: [BUG] malloc error when using large file.

Hi Linus,

Show 26 quoted lines

> > 
> > 
> > > Many operations in git assume that a whole file can be held in memory at
> > > once. Do you actually have 2G available on the machine? Are there any OS
> > > limits that might be a problem?
> > 
> > Thanks for your answer Jeff, 
> > 
> > Yes I have more that 2 Go available when a do it, and after this 
> > allocation error if I try to do a 'git add' with a 9Go file, I did not 
> > have any issue.
>
> Can you add the 2GB files individually, using
>
>	for i in All-f00?.vmdk; do git add $i; done
>
> or similar?
>
> It does sound like an unlucky malloc VM space fragmentation issue, 
> although since the files you add are the same size I'd have not really 
> expected that with any normal malloc library. But with some mixed small 
> and large allocations, I could imagine that the fragmentation happens, and 
> you end up having huge areas that aren't _quite_ useful for the next file 
> due to having some small allocation taking part of it.
>
>		Linus

Sorry, But I have already made this without any success. I really think that the issue is the size of this file. But we never know so I redo it again and here you have the result:

$ free
             total       used       free     shared    buffers     cached
Mem:       3954752     662264    3292488          0      17304     233092
-/+ buffers/cache:     411868    3542884
Swap:      8385920          0    8385920
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ for i in All-f00?.vmdk; do git add $i; done
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ free
             total       used       free     shared    buffers     cached
Mem:       3954752    3928772      25980          0       6052    3510268
-/+ buffers/cache:     412452    3542300
Swap:      8385920         88    8385832
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$

I do nothing else, only a top to know the memory use and I see that just before this 'malloc failed', I have something like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
  4323 epuerto   20   0 2115m 2.0g 2.0g D 20.2 51.7   0:24.64 git

And you can see that it appear some memory allocation is not free after this 'malloc failed' error.

If you have any suggestion, any test (or beta version with more debug message) I can do, you are welcome.

Best regards, Emmanuel.