git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Large object issue (Windows)

From
brian m. carlson <sandals@crustytoothpaste.net>
Date
Mar 5, 2019, 03:35 UTC
Message-ID
<20190305033526.GH601925@genre.crustytoothpaste.net>
In-Reply-To
<CAFOcBz=+XZtcxPxeE3j6dpHXA79R_BOyDCtNbwSu+nbBpPzGrw@mail.gmail.com>
On Mon, Mar 04, 2019 at 07:04:02PM -0500, Patrick Hogg wrote:
Show 29 quoted lines
> Hi all,
> 
> While investigating the last issue I reported (and fixed) I was trying
> to come up with a good test case for repos with large objects. In the
> process I found an issue on Windows with objects at least 4g large:
> 
> git init test
> cd test
> echo "*.exe binary" > .gitattributes
> truncate -s 4g nullbytes.exe
> git stage .
> git commit -m "Test"
> # This will break, complaining that the object is corrupt.
> git fsck --full
> # This will also break, complaining that the object is corrupt.
> #git gc
> 
> I did some investigation and I think that this is a porting issue.
> unpack_object_header_buffer in packfile.c uses an unsigned long for the
> size. On Linux this will be 64 bits (at least on the Linux systems I've
> tried) but on Windows it's 32 bits. The code then decides that the
> object header is bad and bombs. However, if I move the repo to a Linux
> machine it can handle the data just fine. (And ironically git generated
> the object header when storing the object!)
> 
> Is there any reason not to switch the unsigned longs in
> unpack_object_header_buffer (and its callers, wherever that may lead)
> to uint64_t? (Or any potential pitfalls in doing so that I would need
> to look out for?)

It's known that there are several problems with this, affecting various parts of the code. Patches to fix this are of course welcome.

I think we've chosen to specify size_t for values which are stored entirely in memory, since a buffer can't be larger than this size, and off_t for sizes which refer to files or object sizes. The latter will be 64-bit on 32-bit systems when compiled with _FILE_OFFSET_BITS set to 64, while the former will be 32-bit.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
Previous: Patrick HoggNext: Philip Oakley
Message 2 of 3 in “Large object issue (Windows)”
  1. Patrick HoggMar 5, 2019
  2. brian m. carlsonMar 5, 2019
  3. Philip OakleyMar 7, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.