From: Elia Pinto Date: Wed, 18 Nov 2026 06:56:34 GMT Subject: [PATCH v2 0/4] use size_t when receiving and verifying packs Message-ID: In-Reply-To: This series fixes object sizes that are truncated when receiving and verifying packs. Most of them are kept in an "unsigned long", which is a problem where "long" is 32 bits while size_t is 64, such as on 64-bit Windows; 2/4 is about an "unsigned int", which is 32 bits everywhere. 1/4 unpack-objects fails with "inflate returned -5" on an object of 4GiB or more that is not streamed, dies on a delta base that large, and silently writes a truncated object for a delta result that large. With 1/4, "unpack-objects -n" and large delta bases work, and writing such an object, unless it is streamed, dies with a clear error. 2/4 "index-pack --fix-thin" with a local base of 4GiB or more compresses only part of it and dies later with "serious inflate inconsistency" (all platforms). 3/4 index-pack reports a false "SHA1 COLLISION FOUND" for a large object that already exists locally. 4/4 "git fsck" checks only a prefix of large objects in packs. None of them comes with a test, as each needs an object of 4GiB or more; 1/4 and 2/4 describe how I checked them with the size truncated to 16 bits. Changes since v1: - Rebased onto master at e1ec09b23c (The 22nd batch). v1's 1/4 no longer applied: e07ac16d1c (unpack-objects: pass the size of a streamed blob as size_t, 2026-11-06) by Kristofer Karlsson, now in master, makes the same change to stream_blob(). 1/4 drops that hunk; Kristofer is Cc'd. - The message of 1/4 used stream_blob() as its only example, so it is rewritten around the paths that are still affected: objects that are not streamed, all blobs with "git unpack-objects -n", and deltas. - 1/4 no longer drops the cast_size_t_to_ulong() in unpack_delta_entry(), but moves it to write_object(), where the size is passed to odb_write_object() and parse_object_buffer(). v1 let objects of 4GiB or more reach these functions and be written truncated. - 4/4 says that parse_object_buffer() and fsck_object() still see a truncated size until they take a size_t. - The messages of 1/4, 3/4 and 4/4 now say that the problem is "long" being 32 bits while size_t is 64. The code of 2/4, 3/4 and 4/4 is unchanged. There were no comments on v1. v1 left pack-objects for a later series. That fix has since been merged as ep/pack-objects-streamed-large-blobs-fix (ed0aed1e17); it covers the streamed path and explains why the other truncations in pack-objects cannot be reached. The callees odb_write_object(), parse_object_buffer() and fsck_object() still take an "unsigned long" on master. They are changed by two other series of mine, which are not queued yet: https://lore.kernel.org/git/cover.1793444420.git.gitter.spiros@gmail.com/ https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/ The series is based on master at e1ec09b23c. Elia Pinto (4): unpack-objects: use size_t for object sizes index-pack: use size_t in write_compressed() index-pack: use size_t for object sizes pack-check: use size_t for the size passed to verify callbacks builtin/fsck.c | 2 +- builtin/index-pack.c | 14 +++++++------- builtin/unpack-objects.c | 42 ++++++++++++++++++++-------------------- odb.h | 2 +- pack.h | 2 +- 5 files changed, 31 insertions(+), 31 deletions(-) Range-diff against v1: 1: f259eeb6a6 ! 1: b2c3d349a4 unpack-objects: use size_t for object sizes @@ Metadata ## Commit message ## unpack-objects: use size_t for object sizes - builtin/unpack-objects.c uses "unsigned long" to track object sizes - throughout the file, while sizes read from packs are handled as size_t. - On platforms where "long" is 32 bits, this truncates the size for - objects of 4GiB or more. + Since f2063855fb (index-pack, unpack-objects: use size_t for object + size, 2026-05-08), unpack_one() decodes the size of an object into a + size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed + blob as size_t, 2026-11-06) passes it on as such to stream_blob(), + which handles blobs larger than core.bigFileThreshold. The rest of + builtin/unpack-objects.c still takes the size as an "unsigned long": - When handling large objects on those platforms, unpack-objects fails - rather than corrupting data. For instance, streaming large blobs - (exceeding core.bigFileThreshold) passes a truncated size to - stream_blob(), causing the stream write to fail with an error. + - unpack_non_delta_entry() and get_data(), used for commits, trees, + tags and blobs below core.bigFileThreshold, and for all blobs with + "git unpack-objects -n", which never streams; - Use size_t across the board in builtin/unpack-objects.c and remove the - now-unnecessary size cast in unpack_delta_entry(). Downstream consumers - like odb_write_object() and fsck_object() are updated separately. + - unpack_delta_entry() and the functions that keep and resolve + deltas; - Triggering this requires processing 4GiB or more of input on a platform - where "long" is 32 bits, so no test is added. + - struct obj_buffer, which keeps objects around for --strict. + + On platforms where "long" is 32 bits while size_t is 64, such as 64-bit + Windows, the size of an object of 4GiB or more is therefore truncated: + + - For an object that is not streamed, get_data() allocates a buffer + of the truncated size, inflate runs out of room in it, and + unpack-objects fails with "inflate returned -5". Truncating the + size to 16 bits in get_data() and unpack_non_delta_entry() + reproduces this with a pack holding a 70000-byte blob, both with + and without -n. + + - unpack_delta_entry() reads the base of a delta through + cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even + when the result would be smaller. + + - The result of a delta reaches write_object() as a size_t, but + write_object() takes it as an "unsigned long". A result of 4GiB or + more is silently truncated there, and an object with the wrong + contents, hence the wrong name, is written. + + Use size_t across builtin/unpack-objects.c. The functions that + write_object() hands objects to, odb_write_object() and + parse_object_buffer(), still take an "unsigned long", as does + fsck_object(); they are updated separately. Until then, move the size + cast from unpack_delta_entry() to write_object(), where the size is + passed to them. fsck_object() only sees objects that went through + parse_object_buffer() first, and so does write_cached_object(), which + writes them out later; neither needs a cast of its own. + + As a result, on those platforms: + + - "git unpack-objects -n" handles objects of any size; + + - a delta with a base of 4GiB or more and a smaller result is + resolved; + + - an object of 4GiB or more that is not streamed and has to be + written, including the result of a delta, makes unpack-objects die + with "object too large to read on this platform" instead of failing + with "inflate returned -5" or being written truncated. + + Truncating the size to 16 bits where write_object() passes it on + confirms the last point for a 70000-byte blob: with the cast, + unpack-objects dies with "object too large to read on this platform: + 70000 is cut off to 4464"; without it, it exits with 0 and writes a + truncated blob under another name. + + Triggering this requires processing an object of 4GiB or more on a + platform where "long" is 32 bits while size_t is 64, so no test is + added. Signed-off-by: Elia Pinto @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type + void *buf, size_t size) { if (!strict) { - if (odb_write_object(the_repository->objects, buf, size, type, +- if (odb_write_object(the_repository->objects, buf, size, type, ++ if (odb_write_object(the_repository->objects, buf, ++ cast_size_t_to_ulong(size), type, + &obj_list[nr].oid) < 0) + die("failed to write object"); + added_object(nr, type, buf, size); +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type, + obj_list[nr].obj = NULL; + } else if (type == OBJ_BLOB) { + struct blob *blob; +- if (odb_write_object(the_repository->objects, buf, size, type, ++ if (odb_write_object(the_repository->objects, buf, ++ cast_size_t_to_ulong(size), type, + &obj_list[nr].oid) < 0) + die("failed to write object"); + added_object(nr, type, buf, size); +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type, + &obj_list[nr].oid); + added_object(nr, type, buf, size); + obj = parse_object_buffer(the_repository, &obj_list[nr].oid, +- type, size, buf, ++ type, cast_size_t_to_ulong(size), buf, + &eaten); + if (!obj) + die("invalid %s", type_name(type)); @@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type, } @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type unsigned nr) { void *buf = get_data(size); -@@ builtin/unpack-objects.c: static ssize_t zlib_stream_read(struct odb_stream *in_stream, - return buf_len - zstream->avail_out; - } - --static void stream_blob(unsigned long size, unsigned nr) -+static void stream_blob(size_t size, unsigned nr) - { - git_zstream zstream = { 0 }; - struct zlib_stream in_stream = { -@@ builtin/unpack-objects.c: static void stream_blob(unsigned long size, unsigned nr) +@@ builtin/unpack-objects.c: static void stream_blob(size_t size, unsigned nr) } static int resolve_against_held(unsigned nr, const struct object_id *base, 2: 438f7c6ba2 = 2: 9952f0fb10 index-pack: use size_t in write_compressed() 3: 0258282745 ! 3: e66860d274 index-pack: use size_t for object sizes @@ Commit message Functions like sha1_object() and append_obj_to_pack() in builtin/index-pack.c take object sizes as an "unsigned long", even though struct object_entry already tracks them as size_t. On platforms - where "long" is 32 bits, this truncates the size for objects of 4GiB or - more. + where "long" is 32 bits while size_t is 64, such as 64-bit Windows, + this truncates the size of objects of 4GiB or more. When receiving, on those platforms, a large object that already exists locally, sha1_object() compares the real size (size_t) against the @@ Commit message Use size_t for the size parameter across these functions. Triggering this requires processing 4GiB or more of input on a platform - where "long" is 32 bits, so no test is added. + where "long" is 32 bits while size_t is 64, so no test is added. Signed-off-by: Elia Pinto 4: 87b0d2bd96 ! 4: 00939c7f1b pack-check: use size_t for the size passed to verify callbacks @@ Commit message replicated in struct odb_fsck_options (odb.h) and implemented by fsck_obj_buffer() in builtin/fsck.c. - On platforms where "long" is 32 bits, running "git fsck" on packs with - objects of 4GiB or more passes a truncated size to fsck_obj_buffer(), - causing parse_object_buffer() and fsck_object() to inspect only a - prefix. + On platforms where "long" is 32 bits while size_t is 64, such as + 64-bit Windows, running "git fsck" on packs with objects of 4GiB or + more passes a truncated size to fsck_obj_buffer(), causing + parse_object_buffer() and fsck_object() to inspect only a prefix. Use size_t for the size parameter in verify_fn, struct odb_fsck_options.object_cb, and fsck_obj_buffer(). + parse_object_buffer() and fsck_object() themselves still take an + "unsigned long"; until they are updated separately, they keep + inspecting only a prefix. Triggering this requires processing objects of 4GiB or more on a - platform where "long" is 32 bits, so no test is added. + platform where "long" is 32 bits while size_t is 64, so no test is + added. Signed-off-by: Elia Pinto base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4 -- 2.53.0