Large object issue (Windows)

> Hi all,
> 
> While investigating the last issue I reported (and fixed) I was trying
> to come up with a good test case for repos with large objects. In the
> process I found an issue on Windows with objects at least 4g large:
> 
> git init test
> cd test
> echo "*.exe binary" > .gitattributes
> truncate -s 4g nullbytes.exe
> git stage .
> git commit -m "Test"
> # This will break, complaining that the object is corrupt.
> git fsck --full
> # This will also break, complaining that the object is corrupt.
> #git gc
> 
> I did some investigation and I think that this is a porting issue.
> unpack_object_header_buffer in packfile.c uses an unsigned long for the
> size. On Linux this will be 64 bits (at least on the Linux systems I've
> tried) but on Windows it's 32 bits. The code then decides that the
> object header is bad and bombs. However, if I move the repo to a Linux
> machine it can handle the data just fine. (And ironically git generated
> the object header when storing the object!)
> 
> Is there any reason not to switch the unsigned longs in
> unpack_object_header_buffer (and its callers, wherever that may lead)
> to uint64_t? (Or any potential pitfalls in doing so that I would need
> to look out for?)

> On Mon, Mar 04, 2019 at 07:04:02PM -0500, Patrick Hogg wrote:
>> Hi all,
>>
>> While investigating the last issue I reported (and fixed) I was trying
>> to come up with a good test case for repos with large objects. In the
>> process I found an issue on Windows with objects at least 4g large:
>>
>> git init test
>> cd test
>> echo "*.exe binary" > .gitattributes
>> truncate -s 4g nullbytes.exe
>> git stage .
>> git commit -m "Test"
>> # This will break, complaining that the object is corrupt.
>> git fsck --full
>> # This will also break, complaining that the object is corrupt.
>> #git gc
>>
>> I did some investigation and I think that this is a porting issue.
>> unpack_object_header_buffer in packfile.c uses an unsigned long for the
>> size. On Linux this will be 64 bits (at least on the Linux systems I've
>> tried) but on Windows it's 32 bits. The code then decides that the
>> object header is bad and bombs. However, if I move the repo to a Linux
>> machine it can handle the data just fine. (And ironically git generated
>> the object header when storing the object!)
>>
>> Is there any reason not to switch the unsigned longs in
>> unpack_object_header_buffer (and its callers, wherever that may lead)
>> to uint64_t? (Or any potential pitfalls in doing so that I would need
>> to look out for?)
> It's known that there are several problems with this, affecting various
> parts of the code. Patches to fix this are of course welcome.
>
> I think we've chosen to specify size_t for values which are stored
> entirely in memory, since a buffer can't be larger than this size, and
> off_t for sizes which refer to files or object sizes. The latter will be
> 64-bit on 32-bit systems when compiled with _FILE_OFFSET_BITS set to 64,
> while the former will be 32-bit.