Re: git fsck failure on OS X with files >= 4 GiB

From

Atousa Duprat <atousa.p@gmail.com>

Date

Oct 29, 2015, 16:02 UTC

Message-ID

<CA+izobtdwszVrYsnKU=_ytLuNbPGyRe_7kXqyrQO7u5Lo+OdPg@mail.gmail.com>

In-Reply-To

<CAEDE853n2HR-SK9_sGn8n3j0xoTb3eQa86UvdW0DBrc+Z1sDrg@mail.gmail.com>

Atousa

--

[PATCH] Limit the size of the data block passed to SHA1_Update()

This avoids issues where OS-specific implementations use
a 32-bit integer to specify block size.  Limit currently
set to 1GiB.
---
 cache.h | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/cache.h b/cache.h
index 79066e5..c305985 100644
--- a/cache.h
+++ b/cache.h
@@ -14,10 +14,28 @@
 #ifndef git_SHA_CTX
 #define git_SHA_CTX SHA_CTX
 #define git_SHA1_Init SHA1_Init
-#define git_SHA1_Update SHA1_Update
 #define git_SHA1_Final SHA1_Final
 #endif

+#define SHA1_MAX_BLOCK_SIZE (1024*1024*1024)
+
+static inline int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
+{
+ size_t nr;
+ size_t total = 0;
+ char *cdata = (char*)data;
+ while(len > 0) {
+ nr = len;
+ if(nr > SHA1_MAX_BLOCK_SIZE)
+ nr = SHA1_MAX_BLOCK_SIZE;
+ SHA1_Update(c, cdata, nr);
+ total += nr;
+ cdata += nr;
+ len -= nr;
+ }
+ return total;
+}
+
 #include <zlib.h>
 typedef struct git_zstream {
  z_stream z;

-- 
2.4.9 (Apple Git-60)


On Thu, Oct 29, 2015 at 8:15 AM, Filipe Cabecinhas <filcab@gmail.com> wrote:
> Defining BLK_SHA1 = YesPlease (when calling make) should just change
> the SHA functions, instead of completely removing OpenSSL or
> CommonCrypto.
>
> Regards,
>   Filipe
>
>
> On Thu, Oct 29, 2015 at 3:46 AM, Rafael Espíndola
> <rafael.espindola@gmail.com> wrote:
>> Awesome, building with
>>
>> NO_OPENSSL = 1
>> NO_GETTEXT = 1
>>
>> produces a working git :-)
>>
>> Cheers,
>> Rafael
>>
>>
>> On 28 October 2015 at 23:37, Filipe Cabecinhas <filcab@gmail.com> wrote:
>>> I did some debugging, and it seems CC_SHA1_Update (used by
>>> write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile)
>>> takes a uint32_t as a "length" parameter, which explains why it stops
>>> working at 4GiB (UINT_MAX+1).
>>>
>>> In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:
>>>
>>> typedef uint32_t CC_LONG;       /* 32 bit unsigned integer */
>>> //...
>>> extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)
>>>
>>> A possible fix would be to either call SHA1_Update with a maximum of
>>> UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X
>>> which can handle data longer than UINT_MAX.
>>>
>>> I'm not sure what the git maintainers would prefer.
>>>
>>> Regards,
>>>
>>>   Filipe
>>>
>>> On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola
>>> <rafael.espindola@gmail.com> wrote:
>>>>
>>>> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
>>>> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>>>>
>>>> Create two files with just 0s:
>>>>
>>>> -rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
>>>> -rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib
>>>>
>>>>
>>>> and run
>>>>
>>>> git init
>>>> git add one-less-than-4gib
>>>> git commit -m bar
>>>> git fsck
>>>> git add exactly-4gib
>>>> git commit -m bar
>>>> git fsck
>>>>
>>>> The first fsck will run with no problems, but the second one fails:
>>>>
>>>> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
>>>> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
>>>> is corrupt
>>>>
>>>> Using the very same revision on freebsd doesn't cause any errors.
>>>>
>>>> Cheers,
>>>> Rafael
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Atousa Pahlevan, PhD
M.Math. University of Waterloo, Canada
Ph.D. Department of Computer Science, University of Victoria, Canada
Voice: 415-341-6206
Email: apahlevan@ieee.org
Website: www.apahlevan.org