git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3] pack-objects: use size_t for streamed large blobs

From
Junio C Hamano <gitster@pobox.com>
Date
Nov 3, 2026, 22:14 UTC
Message-ID
<xmqqo6c54l75.fsf@gitster.g>
In-Reply-To
<20261101072904.274912-1-gitter.spiros@gmail.com>
Elia Pinto <gitter.spiros@gmail.com> writes:
Show 6 quoted lines
> On platforms where "long" is 32 bits but size_t is 64 bits, like 64-bit
> Windows, handling a blob of 4GiB or more truncates the size in the pack
> header while the data decompresses to the full size, corrupting the
> object. If the compressed data also exceeds 4GiB, the length
> accumulation overflows "unsigned long", misaligning the offsets of all
> subsequent objects in the index.
Makes sense.
Will queue.  Thanks.
Show 61 quoted lines
> -static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,
> -					   const struct object_id *oid)
> +static off_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,
> +				   const struct object_id *oid)
>  {
>  	git_zstream stream;
>  	unsigned char ibuf[1024 * 16];
>  	unsigned char obuf[1024 * 16];
> -	unsigned long olen = 0;
> +	off_t olen = 0;
>  	struct repo_config_values *cfg = repo_config_values(the_repository);
>  
>  	git_deflate_init(&stream, cfg->pack_compression_level);
> @@ -513,10 +513,11 @@ static inline int oe_size_greater_than(struct packing_data *pack,
>  }
>  
>  /* Return 0 if we will bust the pack-size limit */
> -static unsigned long write_no_reuse_object(struct hashfile *f, struct object_entry *entry,
> -					   unsigned long limit, int usable_delta)
> +static off_t write_no_reuse_object(struct hashfile *f, struct object_entry *entry,
> +				   unsigned long limit, int usable_delta)
>  {
> -	unsigned long size, datalen;
> +	size_t size;
> +	off_t datalen;
>  	unsigned char header[MAX_PACK_OBJECT_HEADER],
>  		      dheader[MAX_PACK_OBJECT_HEADER];
>  	unsigned hdrlen;
>
> Range-diff against v2:
> 1:  185b9da891 ! 1:  02e682725b pack-objects: use size_t for streamed large blobs
>     @@ Commit message
>          write_large_blob_data() adds up the compressed length in an "unsigned
>          long" as well.
>      
>     -    On platforms where "long" is 32 bits, handling a blob of 4GiB or more
>     -    truncates the size in the pack header while the data decompresses to the
>     -    full size, corrupting the object. If the compressed data also exceeds
>     -    4GiB, the length accumulation overflows "unsigned long", misaligning the
>     -    offsets of all subsequent objects in the index.
>     +    On platforms where "long" is 32 bits but size_t is 64 bits, like 64-bit
>     +    Windows, handling a blob of 4GiB or more truncates the size in the pack
>     +    header while the data decompresses to the full size, corrupting the
>     +    object. If the compressed data also exceeds 4GiB, the length
>     +    accumulation overflows "unsigned long", misaligning the offsets of all
>     +    subsequent objects in the index.
>      
>          When running repack or gc on a loose large blob, the original loose copy
>          is pruned, leaving only the corrupted packed copy and resulting in data
>     @@ Commit message
>          because core.bigFileThreshold is itself an "unsigned long".
>      
>          Triggering this requires processing an object of 4GiB or more on a
>     -    platform where "long" is 32 bits, so no test is added.
>     +    platform where "long" is 32 bits but size_t is 64 bits, so no test is
>     +    added.
>      
>          Helped-by: Simon Richter <Simon.Richter@hogyros.de>
>          Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
>
> base-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7
Previous: Elia Pinto
Message 5 of 5 in “pack-objects: use size_t for streamed large blobs”
  1. pack-objects: use size_t for streamed large blobsElia Pinto, Oct 31, 2026
  2. pack-objects: use size_t for streamed large blobsElia Pinto, Oct 31, 2026
  3. Junio C HamanoNov 1, 2026
  4. pack-objects: use size_t for streamed large blobsElia Pinto, Nov 1, 2026
  5. Junio C HamanoNov 3, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.