Re: "malloc failed"

From: David Abrahams <dave@boostpro.com>
Date: Jan 29, 2009, 13:10 UTC
Message-ID: <87pri6qmvm.fsf@mcbain.luannocracy.com>
In-Reply-To: <20090129055633.GA32609@coredump.intra.peff.net>

on Thu Jan 29 2009, Jeff King <peff-AT-peff.net> wrote:

Show 24 quoted lines

> On Thu, Jan 29, 2009 at 12:20:41AM -0500, Jeff King wrote:
>
>> Ok, that _is_ big. ;) I wouldn't be surprised if there is some corner of
>> the code that barfs on a single object that doesn't fit in a signed
>> 32-bit integer; I don't think we have any test coverage for stuff that
>> big.
>
> Sure enough, that is the problem. With the patch below I was able to
> "git add" and commit a 3 gigabyte file of random bytes (so even the
> deflated object was 3G).
>
> I think it might be worth applying as a general cleanup, but I have no
> idea if other parts of the system might barf on such an object.
>
> -- >8 --
> Subject: [PATCH] avoid 31-bit truncation in write_loose_object
>
> The size of the content we are adding may be larger than
> 2.1G (i.e., "git add gigantic-file"). Most of the code-path
> to do so uses size_t or unsigned long to record the size,
> but write_loose_object uses a signed int.
>
> On platforms where "int" is 32-bits (which includes x86_64
> Linux platforms), we end up passing malloc a negative size.

Good work. I don't know if this matters to you, but I think on a 32-bit platform you'll find that size_t, which is supposed to be able to hold the size of the largest representable *memory block*, is only 4 bytes large:

  #include <limits.h>
  #include <stdio.h>

  int main()
  {
    printf("sizeof(size_t) = %d", sizeof(size_t));
  }

Prints "sizeof(size_t) = 4" on my core duo.

Show 20 quoted lines

> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  sha1_file.c |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/sha1_file.c b/sha1_file.c
> index 360f7e5..8868b80 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -2340,7 +2340,8 @@ static int create_tmpfile(char *buffer, size_t bufsiz, const
> char *filename)
>  static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
>  			      void *buf, unsigned long len, time_t mtime)
>  {
> -	int fd, size, ret;
> +	int fd, ret;
> +	size_t size;
>  	unsigned char *compressed;
>  	z_stream stream;
>  	char *filename;

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com