git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/4] use size_t when receiving and verifying packs

From
Elia Pinto <gitter.spiros@gmail.com>
Date
Nov 18, 2026, 06:56 UTC
Message-ID
<cover.1794984184.git.gitter.spiros@gmail.com>
In-Reply-To
<cover.1793446357.git.gitter.spiros@gmail.com>

This series fixes object sizes that are truncated when receiving and verifying packs. Most of them are kept in an "unsigned long", which is a problem where "long" is 32 bits while size_t is 64, such as on 64-bit Windows; 2/4 is about an "unsigned int", which is 32 bits everywhere.

 1/4 unpack-objects fails with "inflate returned -5" on an object of
     4GiB or more that is not streamed, dies on a delta base that
     large, and silently writes a truncated object for a delta result
     that large. With 1/4, "unpack-objects -n" and large delta bases
     work, and writing such an object, unless it is streamed, dies
     with a clear error.
 2/4 "index-pack --fix-thin" with a local base of 4GiB or more
     compresses only part of it and dies later with "serious inflate
     inconsistency" (all platforms).
 3/4 index-pack reports a false "SHA1 COLLISION FOUND" for a large
     object that already exists locally.
 4/4 "git fsck" checks only a prefix of large objects in packs.

None of them comes with a test, as each needs an object of 4GiB or more; 1/4 and 2/4 describe how I checked them with the size truncated to 16 bits.

Changes since v1:
 - Rebased onto master at e1ec09b23c (The 22nd batch). v1's 1/4 no
   longer applied: e07ac16d1c (unpack-objects: pass the size of a
   streamed blob as size_t, 2026-11-06) by Kristofer Karlsson, now in
   master, makes the same change to stream_blob(). 1/4 drops that hunk;
   Kristofer is Cc'd.
 - The message of 1/4 used stream_blob() as its only example, so it is
   rewritten around the paths that are still affected: objects that
   are not streamed, all blobs with "git unpack-objects -n", and
   deltas.
 - 1/4 no longer drops the cast_size_t_to_ulong() in
   unpack_delta_entry(), but moves it to write_object(), where the size
   is passed to odb_write_object() and parse_object_buffer(). v1 let
   objects of 4GiB or more reach these functions and be written
   truncated.
 - 4/4 says that parse_object_buffer() and fsck_object() still see a
   truncated size until they take a size_t.
 - The messages of 1/4, 3/4 and 4/4 now say that the problem is "long"
   being 32 bits while size_t is 64. The code of 2/4, 3/4 and 4/4 is
   unchanged.
There were no comments on v1.

v1 left pack-objects for a later series. That fix has since been merged as ep/pack-objects-streamed-large-blobs-fix (ed0aed1e17); it covers the streamed path and explains why the other truncations in pack-objects cannot be reached.

The callees odb_write_object(), parse_object_buffer() and fsck_object() still take an "unsigned long" on master. They are changed by two other series of mine, which are not queued yet:

  https://lore.kernel.org/git/cover.1793444420.git.gitter.spiros@gmail.com/
  https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/
The series is based on master at e1ec09b23c.
Elia Pinto (4):
  unpack-objects: use size_t for object sizes
  index-pack: use size_t in write_compressed()
  index-pack: use size_t for object sizes
  pack-check: use size_t for the size passed to verify callbacks
 builtin/fsck.c           |  2 +-
 builtin/index-pack.c     | 14 +++++++-------
 builtin/unpack-objects.c | 42 ++++++++++++++++++++--------------------
 odb.h                    |  2 +-
 pack.h                   |  2 +-
 5 files changed, 31 insertions(+), 31 deletions(-)
Range-diff against v1:
1:  f259eeb6a6 ! 1:  b2c3d349a4 unpack-objects: use size_t for object sizes
    @@ Metadata
      ## Commit message ##
         unpack-objects: use size_t for object sizes
     
    -    builtin/unpack-objects.c uses "unsigned long" to track object sizes
    -    throughout the file, while sizes read from packs are handled as size_t.
    -    On platforms where "long" is 32 bits, this truncates the size for
    -    objects of 4GiB or more.
    +    Since f2063855fb (index-pack, unpack-objects: use size_t for object
    +    size, 2026-05-08), unpack_one() decodes the size of an object into a
    +    size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed
    +    blob as size_t, 2026-11-06) passes it on as such to stream_blob(),
    +    which handles blobs larger than core.bigFileThreshold. The rest of
    +    builtin/unpack-objects.c still takes the size as an "unsigned long":
     
    -    When handling large objects on those platforms, unpack-objects fails
    -    rather than corrupting data. For instance, streaming large blobs
    -    (exceeding core.bigFileThreshold) passes a truncated size to
    -    stream_blob(), causing the stream write to fail with an error.
    +     - unpack_non_delta_entry() and get_data(), used for commits, trees,
    +       tags and blobs below core.bigFileThreshold, and for all blobs with
    +       "git unpack-objects -n", which never streams;
     
    -    Use size_t across the board in builtin/unpack-objects.c and remove the
    -    now-unnecessary size cast in unpack_delta_entry(). Downstream consumers
    -    like odb_write_object() and fsck_object() are updated separately.
    +     - unpack_delta_entry() and the functions that keep and resolve
    +       deltas;
     
    -    Triggering this requires processing 4GiB or more of input on a platform
    -    where "long" is 32 bits, so no test is added.
    +     - struct obj_buffer, which keeps objects around for --strict.
    +
    +    On platforms where "long" is 32 bits while size_t is 64, such as 64-bit
    +    Windows, the size of an object of 4GiB or more is therefore truncated:
    +
    +     - For an object that is not streamed, get_data() allocates a buffer
    +       of the truncated size, inflate runs out of room in it, and
    +       unpack-objects fails with "inflate returned -5". Truncating the
    +       size to 16 bits in get_data() and unpack_non_delta_entry()
    +       reproduces this with a pack holding a 70000-byte blob, both with
    +       and without -n.
    +
    +     - unpack_delta_entry() reads the base of a delta through
    +       cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even
    +       when the result would be smaller.
    +
    +     - The result of a delta reaches write_object() as a size_t, but
    +       write_object() takes it as an "unsigned long". A result of 4GiB or
    +       more is silently truncated there, and an object with the wrong
    +       contents, hence the wrong name, is written.
    +
    +    Use size_t across builtin/unpack-objects.c. The functions that
    +    write_object() hands objects to, odb_write_object() and
    +    parse_object_buffer(), still take an "unsigned long", as does
    +    fsck_object(); they are updated separately. Until then, move the size
    +    cast from unpack_delta_entry() to write_object(), where the size is
    +    passed to them. fsck_object() only sees objects that went through
    +    parse_object_buffer() first, and so does write_cached_object(), which
    +    writes them out later; neither needs a cast of its own.
    +
    +    As a result, on those platforms:
    +
    +     - "git unpack-objects -n" handles objects of any size;
    +
    +     - a delta with a base of 4GiB or more and a smaller result is
    +       resolved;
    +
    +     - an object of 4GiB or more that is not streamed and has to be
    +       written, including the result of a delta, makes unpack-objects die
    +       with "object too large to read on this platform" instead of failing
    +       with "inflate returned -5" or being written truncated.
    +
    +    Truncating the size to 16 bits where write_object() passes it on
    +    confirms the last point for a 70000-byte blob: with the cast,
    +    unpack-objects dies with "object too large to read on this platform:
    +    70000 is cut off to 4464"; without it, it exits with 0 and writes a
    +    truncated blob under another name.
    +
    +    Triggering this requires processing an object of 4GiB or more on a
    +    platform where "long" is 32 bits while size_t is 64, so no test is
    +    added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
    @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
     +			 void *buf, size_t size)
      {
      	if (!strict) {
    - 		if (odb_write_object(the_repository->objects, buf, size, type,
    +-		if (odb_write_object(the_repository->objects, buf, size, type,
    ++		if (odb_write_object(the_repository->objects, buf,
    ++				     cast_size_t_to_ulong(size), type,
    + 				     &obj_list[nr].oid) < 0)
    + 			die("failed to write object");
    + 		added_object(nr, type, buf, size);
    +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
    + 		obj_list[nr].obj = NULL;
    + 	} else if (type == OBJ_BLOB) {
    + 		struct blob *blob;
    +-		if (odb_write_object(the_repository->objects, buf, size, type,
    ++		if (odb_write_object(the_repository->objects, buf,
    ++				     cast_size_t_to_ulong(size), type,
    + 				     &obj_list[nr].oid) < 0)
    + 			die("failed to write object");
    + 		added_object(nr, type, buf, size);
    +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
    + 				 &obj_list[nr].oid);
    + 		added_object(nr, type, buf, size);
    + 		obj = parse_object_buffer(the_repository, &obj_list[nr].oid,
    +-					  type, size, buf,
    ++					  type, cast_size_t_to_ulong(size), buf,
    + 					  &eaten);
    + 		if (!obj)
    + 			die("invalid %s", type_name(type));
     @@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
      }
      
    @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
      				   unsigned nr)
      {
      	void *buf = get_data(size);
    -@@ builtin/unpack-objects.c: static ssize_t zlib_stream_read(struct odb_stream *in_stream,
    - 	return buf_len - zstream->avail_out;
    - }
    - 
    --static void stream_blob(unsigned long size, unsigned nr)
    -+static void stream_blob(size_t size, unsigned nr)
    - {
    - 	git_zstream zstream = { 0 };
    - 	struct zlib_stream in_stream = {
    -@@ builtin/unpack-objects.c: static void stream_blob(unsigned long size, unsigned nr)
    +@@ builtin/unpack-objects.c: static void stream_blob(size_t size, unsigned nr)
      }
      
      static int resolve_against_held(unsigned nr, const struct object_id *base,
2:  438f7c6ba2 = 2:  9952f0fb10 index-pack: use size_t in write_compressed()
3:  0258282745 ! 3:  e66860d274 index-pack: use size_t for object sizes
    @@ Commit message
         Functions like sha1_object() and append_obj_to_pack() in
         builtin/index-pack.c take object sizes as an "unsigned long", even
         though struct object_entry already tracks them as size_t. On platforms
    -    where "long" is 32 bits, this truncates the size for objects of 4GiB or
    -    more.
    +    where "long" is 32 bits while size_t is 64, such as 64-bit Windows,
    +    this truncates the size of objects of 4GiB or more.
     
         When receiving, on those platforms, a large object that already exists
         locally, sha1_object() compares the real size (size_t) against the
    @@ Commit message
         Use size_t for the size parameter across these functions.
     
         Triggering this requires processing 4GiB or more of input on a platform
    -    where "long" is 32 bits, so no test is added.
    +    where "long" is 32 bits while size_t is 64, so no test is added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
4:  87b0d2bd96 ! 4:  00939c7f1b pack-check: use size_t for the size passed to verify callbacks
    @@ Commit message
         replicated in struct odb_fsck_options (odb.h) and implemented by
         fsck_obj_buffer() in builtin/fsck.c.
     
    -    On platforms where "long" is 32 bits, running "git fsck" on packs with
    -    objects of 4GiB or more passes a truncated size to fsck_obj_buffer(),
    -    causing parse_object_buffer() and fsck_object() to inspect only a
    -    prefix.
    +    On platforms where "long" is 32 bits while size_t is 64, such as
    +    64-bit Windows, running "git fsck" on packs with objects of 4GiB or
    +    more passes a truncated size to fsck_obj_buffer(), causing
    +    parse_object_buffer() and fsck_object() to inspect only a prefix.
     
         Use size_t for the size parameter in verify_fn,
         struct odb_fsck_options.object_cb, and fsck_obj_buffer().
    +    parse_object_buffer() and fsck_object() themselves still take an
    +    "unsigned long"; until they are updated separately, they keep
    +    inspecting only a prefix.
     
         Triggering this requires processing objects of 4GiB or more on a
    -    platform where "long" is 32 bits, so no test is added.
    +    platform where "long" is 32 bits while size_t is 64, so no test is
    +    added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
-- 
2.53.0
Previous: Elia PintoNext: Elia Pinto
Message 6 of 14 in “use size_t when receiving and verifying packs”
  1. 0/4 use size_t when receiving and verifying packsElia Pinto, Oct 31, 2026
  2. 1/4 unpack-objects: use size_t for object sizesElia Pinto, Oct 31, 2026
  3. 2/4 index-pack: use size_t in write_compressed()Elia Pinto, Oct 31, 2026
  4. 3/4 index-pack: use size_t for object sizesElia Pinto, Oct 31, 2026
  5. 4/4 pack-check: use size_t for the size passed to verify callbacksElia Pinto, Oct 31, 2026
  6. 0/4 use size_t when receiving and verifying packsElia Pinto, Nov 18, 2026
  7. 1/4 unpack-objects: use size_t for object sizesElia Pinto, Nov 18, 2026
  8. Patrick SteinhardtNov 18, 2026
  9. Junio C HamanoNov 18, 2026
  10. 2/4 index-pack: use size_t in write_compressed()Elia Pinto, Nov 18, 2026
  11. Patrick SteinhardtNov 18, 2026
  12. 3/4 index-pack: use size_t for object sizesElia Pinto, Nov 18, 2026
  13. 4/4 pack-check: use size_t for the size passed to verify callbacksElia Pinto, Nov 18, 2026
  14. Patrick SteinhardtNov 18, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.