Re: git fsck failure on OS X with files >= 4 GiB

From: Filipe Cabecinhas <filcab@gmail.com>
Date: Oct 29, 2015, 06:46 UTC
Message-ID: <CAEDE853qg_J_asKf4SsE7mB8L7E0Tb3ReJnNWDfAfXfPw+tjhQ@mail.gmail.com>
In-Reply-To: <CAG3jReJn2Pz6-bXLw6baOZaE1BHYiC+1-zN0eagigfG3umWpJA@mail.gmail.com>

I did some debugging, and it seems CC_SHA1_Update (used by write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile) takes a uint32_t as a "length" parameter, which explains why it stops working at 4GiB (UINT_MAX+1).

In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:

typedef uint32_t CC_LONG; /* 32 bit unsigned integer */ //... extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)

A possible fix would be to either call SHA1_Update with a maximum of UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X which can handle data longer than UINT_MAX.

I'm not sure what the git maintainers would prefer.

Regards,

  Filipe

On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola <rafael.espindola@gmail.com> wrote:

Show 30 quoted lines

>
> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>
> Create two files with just 0s:
>
> -rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
> -rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib
>
>
> and run
>
> git init
> git add one-less-than-4gib
> git commit -m bar
> git fsck
> git add exactly-4gib
> git commit -m bar
> git fsck
>
> The first fsck will run with no problems, but the second one fails:
>
> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
> is corrupt
>
> Using the very same revision on freebsd doesn't cause any errors.
>
> Cheers,
> Rafael