From: Philip Oakley Date: Thu, 07 Mar 2019 17:29:52 GMT Subject: Re: Large object issue (Windows) Message-ID: <44f5c2ae-149f-2b7c-77ac-90476ed7bd9d@iee.org> In-Reply-To: <20190305033526.GH601925@genre.crustytoothpaste.net> On 05/03/2019 03:35, brian m. carlson wrote: > On Mon, Mar 04, 2019 at 07:04:02PM -0500, Patrick Hogg wrote: >> Hi all, >> >> While investigating the last issue I reported (and fixed) I was trying >> to come up with a good test case for repos with large objects. In the >> process I found an issue on Windows with objects at least 4g large: >> >> git init test >> cd test >> echo "*.exe binary" > .gitattributes >> truncate -s 4g nullbytes.exe >> git stage . >> git commit -m "Test" >> # This will break, complaining that the object is corrupt. >> git fsck --full >> # This will also break, complaining that the object is corrupt. >> #git gc >> >> I did some investigation and I think that this is a porting issue. >> unpack_object_header_buffer in packfile.c uses an unsigned long for the >> size. On Linux this will be 64 bits (at least on the Linux systems I've >> tried) but on Windows it's 32 bits. The code then decides that the >> object header is bad and bombs. However, if I move the repo to a Linux >> machine it can handle the data just fine. (And ironically git generated >> the object header when storing the object!) >> >> Is there any reason not to switch the unsigned longs in >> unpack_object_header_buffer (and its callers, wherever that may lead) >> to uint64_t? (Or any potential pitfalls in doing so that I would need >> to look out for?) > It's known that there are several problems with this, affecting various > parts of the code. Patches to fix this are of course welcome. > > I think we've chosen to specify size_t for values which are stored > entirely in memory, since a buffer can't be larger than this size, and > off_t for sizes which refer to files or object sizes. The latter will be > 64-bit on 32-bit systems when compiled with _FILE_OFFSET_BITS set to 64, > while the former will be 32-bit. Hi Patrick, There is also a thread on the Git-for-Windows list at https://github.com/git-for-windows/git/issues/1063 and also here at https://public-inbox.org/git/994568940.109648.1548957557643@ox.hosteurope.de/ Part of the issues is that zlib on windows 'sort of' fails to do >4Gb - see their FAQ32 - in that the length value is only 'long' which is only 32 bit, while in fact the zlib copes fine but returns a length modulo that limit. Trying to get all the places that should be upcast to size_t (ptr) or ptdiff_t rather than coerced down to windows 32bit long is part of the struggle. Philip