From: Patrick Steinhardt Date: Wed, 18 Nov 2026 13:33:12 GMT Subject: Re: [PATCH v2 1/4] unpack-objects: use size_t for object sizes Message-ID: In-Reply-To: On Wed, Nov 18, 2026 at 07:56:35AM +0100, Elia Pinto wrote: > Since f2063855fb (index-pack, unpack-objects: use size_t for object > size, 2026-05-08), unpack_one() decodes the size of an object into a > size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed > blob as size_t, 2026-11-06) passes it on as such to stream_blob(), > which handles blobs larger than core.bigFileThreshold. The rest of > builtin/unpack-objects.c still takes the size as an "unsigned long": > > - unpack_non_delta_entry() and get_data(), used for commits, trees, > tags and blobs below core.bigFileThreshold, and for all blobs with > "git unpack-objects -n", which never streams; > > - unpack_delta_entry() and the functions that keep and resolve > deltas; > > - struct obj_buffer, which keeps objects around for --strict. > > On platforms where "long" is 32 bits while size_t is 64, such as 64-bit > Windows, the size of an object of 4GiB or more is therefore truncated: > > - For an object that is not streamed, get_data() allocates a buffer > of the truncated size, inflate runs out of room in it, and > unpack-objects fails with "inflate returned -5". Truncating the > size to 16 bits in get_data() and unpack_non_delta_entry() > reproduces this with a pack holding a 70000-byte blob, both with > and without -n. > > - unpack_delta_entry() reads the base of a delta through > cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even > when the result would be smaller. > > - The result of a delta reaches write_object() as a size_t, but > write_object() takes it as an "unsigned long". A result of 4GiB or > more is silently truncated there, and an object with the wrong > contents, hence the wrong name, is written. > > Use size_t across builtin/unpack-objects.c. The functions that > write_object() hands objects to, odb_write_object() and > parse_object_buffer(), still take an "unsigned long", as does > fsck_object(); they are updated separately. Until then, move the size > cast from unpack_delta_entry() to write_object(), where the size is > passed to them. fsck_object() only sees objects that went through > parse_object_buffer() first, and so does write_cached_object(), which > writes them out later; neither needs a cast of its own. > > As a result, on those platforms: > > - "git unpack-objects -n" handles objects of any size; > > - a delta with a base of 4GiB or more and a smaller result is > resolved; > > - an object of 4GiB or more that is not streamed and has to be > written, including the result of a delta, makes unpack-objects die > with "object too large to read on this platform" instead of failing > with "inflate returned -5" or being written truncated. > > Truncating the size to 16 bits where write_object() passes it on > confirms the last point for a 70000-byte blob: with the cast, > unpack-objects dies with "object too large to read on this platform: > 70000 is cut off to 4464"; without it, it exits with 0 and writes a > truncated blob under another name. > > Triggering this requires processing an object of 4GiB or more on a > platform where "long" is 32 bits while size_t is 64, so no test is > added. Phew, this commit message really is extremely verbose, and I eventually lost steam reading through it halfways. It's good to document what you're doing, but this here might have gone off the rails a bit with that. What's worth documenting is the unexpected parts in your patch, and things that may not be immediately obvious to a reviewer. But you also document what specifically you are doing to what function, and that's quite obvious from the diff itself. So overall I bet that this message could be condensed to half or even a third of its size without losing much information. > diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c > index d2f25bed49..9bd355b2c4 100644 > --- a/builtin/unpack-objects.c > +++ b/builtin/unpack-objects.c > @@ -269,10 +269,11 @@ static void added_object(unsigned nr, enum object_type type, > * to be checked at the end. > */ > static void write_object(unsigned nr, enum object_type type, > - void *buf, unsigned long size) > + void *buf, size_t size) > { > if (!strict) { > - if (odb_write_object(the_repository->objects, buf, size, type, > + if (odb_write_object(the_repository->objects, buf, > + cast_size_t_to_ulong(size), type, > &obj_list[nr].oid) < 0) > die("failed to write object"); > added_object(nr, type, buf, size); These casts here make me a bit unhappy, as you also have a patch series in flight that fix `odb_write_object()`. So you'll have to send another patch series after both have landed to remove those again. Would it make more sense to defer this series here until the other one lands? > @@ -299,7 +301,7 @@ static void write_object(unsigned nr, enum object_type type, > &obj_list[nr].oid); > added_object(nr, type, buf, size); > obj = parse_object_buffer(the_repository, &obj_list[nr].oid, > - type, size, buf, > + type, cast_size_t_to_ulong(size), buf, > &eaten); > if (!obj) > die("invalid %s", type_name(type)); Likewise, even though I don't know whether you also have a patch series in flight fixing this one from the top of my head. Patrick