{"thread":{"id":"19897","subject":"[BUG] malloc error when using large file.","startedAt":"2009-06-22T09:54:18Z","lastAt":"2009-06-24T08:35:58Z","messageCount":6,"participants":["Emmanuel Puerto","Jeff King","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"116753","messageId":"B03A893ADFE80748942D13AFAF2845190C2D5D388A@ITSVLEX06.it.maxim-ic.internal","threadId":"19897","inReplyTo":null,"subject":"[BUG] malloc error when using large file.","fromName":"Emmanuel Puerto","fromEmail":"emmanuel.puerto@maxim-ic.com","sentAt":"2009-06-22T09:54:18Z","receivedAt":"2009-06-22T09:54:18Z","isPatch":false,"sender":{"key":"emmanuel.puerto@maxim-ic.com","avatar":null},"body":"Hi All,\n\nI use Git with some Virtual Machine and I have an issue with 2 of them.\nWhen I use some VM I have created with a \"pre-allocated\" disk split into a 2Go file. For example a \"ls\" give:\n\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f001.vmdk\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f002.vmdk\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f003.vmdk\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f004.vmdk\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f005.vmdk\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f006.vmdk\n-rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f007.vmdk\n-rwx------ 1 epuerto epuerto    1835008 mars 10 21:44 All-f008.vmdk\n-rwx------ 1 epuerto epuerto       1007 juin 18 09:30 All.vmdk\n-rw------- 1 epuerto epuerto        548 juin 18 09:40 devel\n-rw------- 1 epuerto epuerto 1538850816 juin 18 17:05 devel-s001\n-rw------- 1 epuerto epuerto 1125384192 juin 18 17:05 devel-s002\n-rw------- 1 epuerto epuerto 1384382464 juin 18 17:05 devel-s003\n-rw------- 1 epuerto epuerto 1632763904 juin 18 17:05 devel-s004\n-rw------- 1 epuerto epuerto    1572864 juin 18 17:05 devel-s005\n-rwx------ 1 epuerto epuerto       8684 juin 18 17:07 Ubuntu.nvram\n-rw------- 1 epuerto epuerto  536870912 juin 18 17:06 Ubuntu.vmem\n-rwx------ 1 epuerto epuerto          0 mars 10 21:31 Ubuntu.vmsd\n-rw------- 1 epuerto epuerto  135732052 juin 18 17:07 Ubuntu.vmss\n-rwx------ 1 epuerto epuerto       2908 juin 18 17:07 Ubuntu.vmx\n-rwx------ 1 epuerto epuerto        261 mars 10 21:31 Ubuntu.vmxf\n\nSo if I just do a \"git init\" and after do a \"git add -A\" I have the following error:\nfatal: Out of memory, malloc failed\n\nno more information is given if I use the verbose option of git (git add -v -A).\n\nIn fact the error appear when I try to add file name \"all-f00?.vmdk\"\n\nIt appears that this size of file is not accepted by git.\nIf I do the same with other VM that they have a biggest size or a smallest size, all it is ok\n\nFor example:\n-rw------- 1 epuerto epuerto       8684 avril  6 15:56 CentOs 5.2 Dev.nvram\n-rw------- 1 epuerto epuerto       8684 juin 18 11:36 CentOs 5.3 Dev.nvram\n-rw------- 1 epuerto epuerto  536870912 juin 18 11:36 CentOs 5.3 Dev.vmem\n-rw------- 1 epuerto epuerto          0 avril  3 12:26 CentOs 5.3 Dev.vmsd\n-rw------- 1 epuerto epuerto   18408288 juin 18 11:36 CentOs 5.3 Dev.vmss\n-rwxr-xr-x 1 epuerto epuerto       2239 juin 18 11:36 CentOs 5.3 Dev.vmx\n-rw------- 1 epuerto epuerto        269 avril  6 10:43 CentOs 5.3 Dev.vmxf\n-rw------- 1 epuerto epuerto 1766981632 juin 18 11:36 sda-s001.vmdk\n-rw------- 1 epuerto epuerto 2094596096 juin 18 11:36 sda-s002.vmdk\n-rw------- 1 epuerto epuerto 1808138240 juin 18 11:36 sda-s003.vmdk\n-rw------- 1 epuerto epuerto 1429667840 juin 18 11:36 sda-s004.vmdk\n-rw------- 1 epuerto epuerto      65536 juin 18 11:36 sda-s005.vmdk\n-rw------- 1 epuerto epuerto        907 juin 18 10:57 sda.vmdk\n\nOR\n\n-rwxrwxrwx 1 epuerto epuerto       8684 juin 17 21:08 FreeBSD 64-bit.nvram\n-rwxrwxrwx 1 epuerto epuerto    1114112 mars 27  2008 FreeBSD 64-bit.vmdk\n-rwxrwxrwx 1 epuerto epuerto          0 mars 27  2008 FreeBSD 64-bit.vmsd\n-rwxrwxrwx 1 epuerto epuerto       1652 juin 17 19:48 FreeBSD 64-bit.vmx\n-rwxrwxrwx 1 epuerto epuerto        269 mars 27  2008 FreeBSD 64-bit.vmxf\n-rwxrwxrwx 1 epuerto epuerto 9118810112 juin 17 21:08 hdd.vmdk\n\n\nMy primary OS (where I use Git) is \n$uname -a\nLinux epuerto-FC10.innova-card.internal 2.6.27.24-170.2.68.fc10.x86_64 #1 SMP Wed May 20 22:47:23 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux\n\nAnd my git version is:\n$ git --version\ngit version 1.6.0.6\n\nmy config is:\nThinkPad T400 with a Intel(R) Core(TM)2 Duo CPU T9400 and 4Go\n\nThanks by advance for your help.\n\nBest regards,\nEmmanuel.\n\nBefore you print Think about \nENVIRONMENTAL responsibility and commitment\n\nEmmanuel PUERTO\nSenior MTS, Field Applications (Microcontroller)\nLinux Embedded Architect\n\nMAXIM Integrated Products, Inc.\nZI Athelia IV - Le Forum, Bat A\nQuartier Roumagoua\n13600 La Ciotat - France\nPhone : +33 44 298 1338\nFax   : +33 44 208 3319\nEmail : emmanuel.puerto@maxim-ic.com\nWeb   : www.maxim-ic.com\n"},{"id":"116788","messageId":"20090622211542.GA19364@coredump.intra.peff.net","threadId":"19897","inReplyTo":"B03A893ADFE80748942D13AFAF2845190C2D5D388A@ITSVLEX06.it.maxim-ic.internal","subject":"Re: [BUG] malloc error when using large file.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-06-22T21:15:42Z","receivedAt":"2009-06-22T21:15:42Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 22, 2009 at 02:54:18AM -0700, Emmanuel Puerto wrote:\n\n> I use Git with some Virtual Machine and I have an issue with 2 of them.\n> When I use some VM I have created with a \"pre-allocated\" disk split\n> into a 2Go file. For example a \"ls\" give:\n> [...]\n> So if I just do a \"git init\" and after do a \"git add -A\" I have the\n> following error:\n> fatal: Out of memory, malloc failed\n\nMany operations in git assume that a whole file can be held in memory at\nonce. Do you actually have 2G available on the machine? Are there any OS\nlimits that might be a problem?\n\n-Peff\n"},{"id":"116806","messageId":"B03A893ADFE80748942D13AFAF2845190C2D5D3BDC@ITSVLEX06.it.maxim-ic.internal","threadId":"19897","inReplyTo":"20090622211542.GA19364@coredump.intra.peff.net","subject":"RE: [BUG] malloc error when using large file.","fromName":"Emmanuel Puerto","fromEmail":"emmanuel.puerto@maxim-ic.com","sentAt":"2009-06-23T07:05:11Z","receivedAt":"2009-06-23T07:05:11Z","isPatch":false,"sender":{"key":"emmanuel.puerto@maxim-ic.com","avatar":null},"body":"\n\n> Many operations in git assume that a whole file can be held in memory at\n> once. Do you actually have 2G available on the machine? Are there any OS\n> limits that might be a problem?\n\nThanks for your answer Jeff, \n\nYes I have more that 2 Go available when a do it, and after this allocation error if I try to do a 'git add' with a 9Go file, I did not have any issue.\n\nEmmanuel.\n"},{"id":"116834","messageId":"20090623184208.GA8165@coredump.intra.peff.net","threadId":"19897","inReplyTo":"B03A893ADFE80748942D13AFAF2845190C2D5D3BDC@ITSVLEX06.it.maxim-ic.internal","subject":"Re: [BUG] malloc error when using large file.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-06-23T18:42:08Z","receivedAt":"2009-06-23T18:42:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 23, 2009 at 12:05:11AM -0700, Emmanuel Puerto wrote:\n\n> Yes I have more that 2 Go available when a do it, and after this\n> allocation error if I try to do a 'git add' with a 9Go file, I did not\n> have any issue.\n\nIt may be that doing a whole bunch of files in one process is leading to\nsome memory fragmentation that doesn't let malloc get a contiguous 2G\nslice. I'm not sure what else to suggest trying, except perhaps to\nsearch the list archives for past discussions on memory fragmentation.\n\nMaybe somebody else has suggestions.\n\n-Peff\n"},{"id":"116836","messageId":"alpine.LFD.2.01.0906231412070.3240@localhost.localdomain","threadId":"19897","inReplyTo":"B03A893ADFE80748942D13AFAF2845190C2D5D3BDC@ITSVLEX06.it.maxim-ic.internal","subject":"RE: [BUG] malloc error when using large file.","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-06-23T21:15:21Z","receivedAt":"2009-06-23T21:15:21Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 23 Jun 2009, Emmanuel Puerto wrote:\n> \n> \n> > Many operations in git assume that a whole file can be held in memory at\n> > once. Do you actually have 2G available on the machine? Are there any OS\n> > limits that might be a problem?\n> \n> Thanks for your answer Jeff, \n> \n> Yes I have more that 2 Go available when a do it, and after this \n> allocation error if I try to do a 'git add' with a 9Go file, I did not \n> have any issue.\n\nCan you add the 2GB files individually, using\n\n\tfor i in All-f00?.vmdk; do git add $i; done\n\nor similar?\n\nIt does sound like an unlucky malloc VM space fragmentation issue, \nalthough since the files you add are the same size I'd have not really \nexpected that with any normal malloc library. But with some mixed small \nand large allocations, I could imagine that the fragmentation happens, and \nyou end up having huge areas that aren't _quite_ useful for the next file \ndue to having some small allocation taking part of it.\n\n\t\tLinus\n"},{"id":"116857","messageId":"B03A893ADFE80748942D13AFAF2845190C2D5D3FB0@ITSVLEX06.it.maxim-ic.internal","threadId":"19897","inReplyTo":"alpine.LFD.2.01.0906231412070.3240@localhost.localdomain","subject":"RE: [BUG] malloc error when using large file.","fromName":"Emmanuel Puerto","fromEmail":"emmanuel.puerto@maxim-ic.com","sentAt":"2009-06-24T08:35:58Z","receivedAt":"2009-06-24T08:35:58Z","isPatch":false,"sender":{"key":"emmanuel.puerto@maxim-ic.com","avatar":null},"body":"Hi Linus,\n\n> > \n> > \n> > > Many operations in git assume that a whole file can be held in memory at\n> > > once. Do you actually have 2G available on the machine? Are there any OS\n> > > limits that might be a problem?\n> > \n> > Thanks for your answer Jeff, \n> > \n> > Yes I have more that 2 Go available when a do it, and after this \n> > allocation error if I try to do a 'git add' with a 9Go file, I did not \n> > have any issue.\n>\n> Can you add the 2GB files individually, using\n>\n>\tfor i in All-f00?.vmdk; do git add $i; done\n>\n> or similar?\n>\n> It does sound like an unlucky malloc VM space fragmentation issue, \n> although since the files you add are the same size I'd have not really \n> expected that with any normal malloc library. But with some mixed small \n> and large allocations, I could imagine that the fragmentation happens, and \n> you end up having huge areas that aren't _quite_ useful for the next file \n> due to having some small allocation taking part of it.\n>\n>\t\tLinus\n\nSorry, But I have already made this without any success. I really think that the issue is the size of this file. But we never know so I redo it again and here you have the result:\n\n$ free\n             total       used       free     shared    buffers     cached\nMem:       3954752     662264    3292488          0      17304     233092\n-/+ buffers/cache:     411868    3542884\nSwap:      8385920          0    8385920\n[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ for i in All-f00?.vmdk; do git add $i; done\nfatal: Out of memory, malloc failed\nfatal: Out of memory, malloc failed\nfatal: Out of memory, malloc failed\nfatal: Out of memory, malloc failed\nfatal: Out of memory, malloc failed\nfatal: Out of memory, malloc failed\nfatal: Out of memory, malloc failed\n[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ free\n             total       used       free     shared    buffers     cached\nMem:       3954752    3928772      25980          0       6052    3510268\n-/+ buffers/cache:     412452    3542300\nSwap:      8385920         88    8385832\n[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$\n\nI do nothing else, only a top to know the memory use and I see that just before this 'malloc failed', I have something like this:\n\n  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            \n  4323 epuerto   20   0 2115m 2.0g 2.0g D 20.2 51.7   0:24.64 git                \n\nAnd you can see that it appear some memory allocation is not free after this 'malloc failed' error.\n\nIf you have any suggestion, any test (or beta version with more debug message) I can do, you are welcome.\n\nBest regards,\nEmmanuel.\n"}]}