"malloc failed"

> I've been abusing Git for a purpose it wasn't intended to serve:
> archiving a large number of files with many duplicates and
> near-duplicates.  Every once in a while, when trying to do something
> really big, it tells me "malloc failed" and bails out (I think it's
> during "git add" but because of the way I issued the commands I can't
> tell: it could have been a commit or a gc).  This is on a 64-bit linux
> machine with 8G of ram and plenty of swap space, so I'm surprised.
> 
> Git is doing an amazing job at archiving and compressing all this stuff
> I'm putting in it, but I have to do it a wee bit at a time or it craps
> out.  Bug?

> David Abrahams <dave@boostpro.com> wrote:
>> I've been abusing Git for a purpose it wasn't intended to serve:
>> archiving a large number of files with many duplicates and
>> near-duplicates.  Every once in a while, when trying to do something
>> really big, it tells me "malloc failed" and bails out (I think it's
>> during "git add" but because of the way I issued the commands I can't
>> tell: it could have been a commit or a gc).  This is on a 64-bit linux
>> machine with 8G of ram and plenty of swap space, so I'm surprised.
>> 
>> Git is doing an amazing job at archiving and compressing all this stuff
>> I'm putting in it, but I have to do it a wee bit at a time or it craps
>> out.  Bug?
>
> No, not really.  Above you said you are "abusing git for a purpose
> it wasn't intended to serve"...

> It was mostly designed for source code, where the majority of the
> data stored in it is some form of text file written by a human.
>
> By their very nature these files need to be relatively short (e.g.
> under 1 MB each) as no human can sanely maintain a text file that
> large without breaking it apart into different smaller files (like
> the source code for an operating system kernel).
>
> As a result of this approach, the git code assumes it can malloc()
> at least two blocks large enough for each file: one of the fully
> decompressed content, and another for the fully compressed content.
> Try doing git add on a large file and its very likely malloc
> will fail due to ulimit issues, or you just don't have enough
> memory/address space to go around.

> git gc likewise needs a good chunk of memory, but it shouldn't
> usually report "malloc failed".  Usually in git gc if a malloc fails
> it prints a warning and degrades the quality of its data compression.
> But there are critical bookkeeping data structures where we must be
> able to malloc the memory, and if those fail because we've already
> exhausted the heap early on, then yea, it can fail too.

> Every once in a while, when trying to do something really big, it tells 
> me "malloc failed" and bails out (I think it's during "git add" but 
> because of the way I issued the commands I can't tell: it could have 
> been a commit or a gc).  This is on a 64-bit linux machine with 8G of 
> ram and plenty of swap space, so I'm surprised.

> I've been abusing Git for a purpose it wasn't intended to serve:
> archiving a large number of files with many duplicates and
> near-duplicates.  Every once in a while, when trying to do something
> really big, it tells me "malloc failed" and bails out (I think it's
> during "git add" but because of the way I issued the commands I can't
> tell: it could have been a commit or a gc).  This is on a 64-bit linux
> machine with 8G of ram and plenty of swap space, so I'm surprised.
> 
> Git is doing an amazing job at archiving and compressing all this stuff
> I'm putting in it, but I have to do it a wee bit at a time or it craps
> out.  Bug?

> On Wed, 28 Jan 2009 00:02:25 -0500, Jeff King <peff@peff.net> wrote:
>> On Tue, Jan 27, 2009 at 10:04:42AM -0500, David Abrahams wrote:
>> 
>>> I've been abusing Git for a purpose it wasn't intended to serve:
>>> archiving a large number of files with many duplicates and
>>> near-duplicates.  Every once in a while, when trying to do something
>>> really big, it tells me "malloc failed" and bails out (I think it's
>>> during "git add" but because of the way I issued the commands I can't
>>> tell: it could have been a commit or a gc).  This is on a 64-bit linux
>>> machine with 8G of ram and plenty of swap space, so I'm surprised.
>>> 
>>> Git is doing an amazing job at archiving and compressing all this stuff
>>> I'm putting in it, but I have to do it a wee bit at a time or it craps
>>> out.  Bug?
>> 
>> How big is the repository? How big are the biggest files? I have a
>> 3.5G repo with files ranging from a few bytes to about 180M. I've never
>> run into malloc problems or gone into swap on my measly 1G box.
>> How does your dataset compare?
>
> I'll try to do some research.  Gotta go pick up my boy now...

> Subject: [PATCH] avoid 31-bit truncation in write_loose_object
>
> The size of the content we are adding may be larger than
> 2.1G (i.e., "git add gigantic-file"). Most of the code-path
> to do so uses size_t or unsigned long to record the size,
> but write_loose_object uses a signed int.

> On Thu, Jan 29, 2009 at 12:20:41AM -0500, Jeff King wrote:
>
>> Ok, that _is_ big. ;) I wouldn't be surprised if there is some corner of
>> the code that barfs on a single object that doesn't fit in a signed
>> 32-bit integer; I don't think we have any test coverage for stuff that
>> big.
>
> Sure enough, that is the problem. With the patch below I was able to
> "git add" and commit a 3 gigabyte file of random bytes (so even the
> deflated object was 3G).
>
> I think it might be worth applying as a general cleanup, but I have no
> idea if other parts of the system might barf on such an object.
>
> -- >8 --
> Subject: [PATCH] avoid 31-bit truncation in write_loose_object
>
> The size of the content we are adding may be larger than
> 2.1G (i.e., "git add gigantic-file"). Most of the code-path
> to do so uses size_t or unsigned long to record the size,
> but write_loose_object uses a signed int.
>
> On platforms where "int" is 32-bits (which includes x86_64
> Linux platforms), we end up passing malloc a negative size.

> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  sha1_file.c |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/sha1_file.c b/sha1_file.c
> index 360f7e5..8868b80 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -2340,7 +2340,8 @@ static int create_tmpfile(char *buffer, size_t bufsiz, const
> char *filename)
>  static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
>  			      void *buf, unsigned long len, time_t mtime)
>  {
> -	int fd, size, ret;
> +	int fd, ret;
> +	size_t size;
>  	unsigned char *compressed;
>  	z_stream stream;
>  	char *filename;

> on Thu Jan 29 2009, Jeff King <peff-AT-peff.net> wrote:
> 
>> On Thu, Jan 29, 2009 at 12:20:41AM -0500, Jeff King wrote:
>>
>>> Ok, that _is_ big. ;) I wouldn't be surprised if there is some corner of
>>> the code that barfs on a single object that doesn't fit in a signed
>>> 32-bit integer; I don't think we have any test coverage for stuff that
>>> big.
>> Sure enough, that is the problem. With the patch below I was able to
>> "git add" and commit a 3 gigabyte file of random bytes (so even the
>> deflated object was 3G).
>>
>> I think it might be worth applying as a general cleanup, but I have no
>> idea if other parts of the system might barf on such an object.
>>
>> -- >8 --
>> Subject: [PATCH] avoid 31-bit truncation in write_loose_object
>>
>> The size of the content we are adding may be larger than
>> 2.1G (i.e., "git add gigantic-file"). Most of the code-path
>> to do so uses size_t or unsigned long to record the size,
>> but write_loose_object uses a signed int.
>>
>> On platforms where "int" is 32-bits (which includes x86_64
>> Linux platforms), we end up passing malloc a negative size.
> 
> 
> Good work.  I don't know if this matters to you, but I think on a 32-bit
> platform you'll find that size_t, which is supposed to be able to hold
> the size of the largest representable *memory block*, is only 4 bytes
> large:
> 
>   #include <limits.h>
>   #include <stdio.h>
> 
>   int main()
>   {
>     printf("sizeof(size_t) = %d", sizeof(size_t));
>   }
> 
> Prints "sizeof(size_t) = 4" on my core duo.
> 

> My case: I have a 500 MB repository with a 1GB working tree, with
> binary files ranging from 100KB to 50MB and a few thousand source
> files.
> 
> I have two branches ('master' and 'cmake') and the latter has suffered
> a huge hierarchy reorganization.
> 
> When I merge 'master' in 'cmake', if I use the 'subtree' strategy, it
> works fine. If I use any other strategy, after a couple of minutes I
> receive a "malloc failed" and the tree is all messed up. As I said, on
> Linux it works fine, so maybe it's a Windows-specific problem.