git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 01/11] index-pack, unpack-objects: use size_t for object size

From
Torsten Bögershausen <tboegi@web.de>
Date
May 5, 2026, 19:11 UTC
Message-ID
<20260505191100.GA12275@tb-raspi4>
In-Reply-To
<dc660106ea8511e6adc44d2b70e9a4ae8b18090e.1777914508.git.gitgitgadget@gmail.com>
On Mon, May 04, 2026 at 05:08:18PM +0000, Johannes Schindelin via GitGitGadget wrote:
Show 52 quoted lines
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
> 
> When unpacking objects from a packfile, the object size is decoded
> from a variable-length encoding. On platforms where unsigned long is
> 32-bit (such as Windows, even in 64-bit builds), the shift operation
> overflows when decoding sizes larger than 4GB. The result is a
> truncated size value, causing the unpacked object to be corrupted or
> rejected.
> 
> Fix this by changing the size variable to size_t, which is 64-bit on
> 64-bit platforms, and ensuring the shift arithmetic occurs in 64-bit
> space.
> 
> This was originally authored by LordKiRon <https://github.com/LordKiRon>,
> who preferred not to reveal their real name and therefore agreed that I
> take over authorship.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  builtin/index-pack.c     | 9 +++++----
>  builtin/unpack-objects.c | 5 +++--
>  2 files changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index ca7784dc2c..cc660582e9 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -37,7 +37,7 @@ static const char index_pack_usage[] =
>  
>  struct object_entry {
>  	struct pack_idx_entry idx;
> -	unsigned long size;
> +	size_t size;
>  	unsigned char hdr_size;
>  	signed char type;
>  	signed char real_type;
> @@ -469,7 +469,7 @@ static int is_delta_type(enum object_type type)
>  	return (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA);
>  }
>  
> -static void *unpack_entry_data(off_t offset, unsigned long size,
> +static void *unpack_entry_data(off_t offset, size_t size,
>  			       enum object_type type, struct object_id *oid)
>  {
>  	static char fixed_buf[8192];
> @@ -524,7 +524,8 @@ static void *unpack_raw_entry(struct object_entry *obj,
>  			      struct object_id *oid)
>  {
>  	unsigned char *p;
> -	unsigned long size, c;
> +	size_t size;
> +	unsigned long c;

Does this look a little bit strange ? p points to an unsigned char (better would be *uint8_t) then it is dereferenced into an "unsigned long". Then it is masked with 0x7f In short: should "c" be declared as uint8_t ?

Show 39 quoted lines
>  	off_t base_offset;
>  	unsigned shift;
>  	void *data;
> @@ -542,7 +543,7 @@ static void *unpack_raw_entry(struct object_entry *obj,
>  		p = fill(1);
>  		c = *p;
>  		use(1);
> -		size += (c & 0x7f) << shift;
> +		size += ((size_t)c & 0x7f) << shift;
>  		shift += 7;
>  	}
>  	obj->size = size;
> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
> index e01cf6e360..59a36c2481 100644
> --- a/builtin/unpack-objects.c
> +++ b/builtin/unpack-objects.c
> @@ -533,7 +533,8 @@ static void unpack_one(unsigned nr)
>  {
>  	unsigned shift;
>  	unsigned char *pack;
> -	unsigned long size, c;
> +	size_t size;
> +	unsigned long c;
>  	enum object_type type;
>  
>  	obj_list[nr].offset = consumed_bytes;
> @@ -548,7 +549,7 @@ static void unpack_one(unsigned nr)
>  		pack = fill(1);
>  		c = *pack;
>  		use(1);
> -		size += (c & 0x7f) << shift;
> +		size += ((size_t)c & 0x7f) << shift;
>  		shift += 7;
>  	}
>  
> -- 
> gitgitgadget
> 
> 
Previous: Johannes Schindelin via GitGitGadgetNext: Johannes Schindelin
Message 19 of 60 in “Handle cloning of objects larger than 4GB on Windows”
  1. 0/6 Handle cloning of objects larger than 4GB on WindowsJohannes Schindelin via GitGitGadget, Apr 28, 2026
  2. 1/6 index-pack, unpack-objects: use size_t for object sizeJohannes Schindelin via GitGitGadget, Apr 28, 2026
  3. Torsten BögershausenApr 30, 2026
  4. Johannes SchindelinMay 3, 2026
  5. 2/6 git-zlib: handle data streams larger than 4GBJohannes Schindelin via GitGitGadget, Apr 28, 2026
  6. 3/6 odb, packfile: use size_t for streaming object sizesJohannes Schindelin via GitGitGadget, Apr 28, 2026
  7. 4/6 delta, packfile: use size_t for delta header sizesJohannes Schindelin via GitGitGadget, Apr 28, 2026
  8. Derrick StoleeApr 29, 2026
  9. Johannes SchindelinMay 3, 2026
  10. 5/6 test-tool: add a helper to synthesize large packfilesJohannes Schindelin via GitGitGadget, Apr 28, 2026
  11. 6/6 t5608: add regression test for >4GB object cloneJohannes Schindelin via GitGitGadget, Apr 28, 2026
  12. Derrick StoleeApr 29, 2026
  13. Jeff KingMay 1, 2026
  14. Derrick StoleeMay 1, 2026
  15. Johannes SchindelinMay 4, 2026
  16. Derrick StoleeApr 29, 2026
  17. 00/11 Handle cloning of objects larger than 4GB on WindowsJohannes Schindelin via GitGitGadget, May 4, 2026
  18. 01/11 index-pack, unpack-objects: use size_t for object sizeJohannes Schindelin via GitGitGadget, May 4, 2026
  19. Torsten BögershausenMay 5, 2026
  20. Johannes SchindelinMay 8, 2026
  21. Torsten BögershausenMay 8, 2026
  22. Junio C HamanoMay 10, 2026
  23. Torsten BögershausenMay 10, 2026
  24. 02/11 git-zlib: handle data streams larger than 4GBJohannes Schindelin via GitGitGadget, May 4, 2026
  25. 03/11 odb, packfile: use size_t for streaming object sizesJohannes Schindelin via GitGitGadget, May 4, 2026
  26. Torsten BögershausenMay 5, 2026
  27. Johannes SchindelinMay 8, 2026
  28. 04/11 delta, packfile: use size_t for delta header sizesJohannes Schindelin via GitGitGadget, May 4, 2026
  29. 05/11 test-tool: add a helper to synthesize large packfilesJohannes Schindelin via GitGitGadget, May 4, 2026
  30. 06/11 t5608: add regression test for >4GB object cloneJohannes Schindelin via GitGitGadget, May 4, 2026
  31. 07/11 test-tool synthesize: use the unsafe hash for speedJohannes Schindelin via GitGitGadget, May 4, 2026
  32. 08/11 test-tool synthesize: precompute pack for 4 GiB + 1Johannes Schindelin via GitGitGadget, May 4, 2026
  33. Derrick StoleeMay 4, 2026
  34. Johannes SchindelinMay 5, 2026
  35. 09/11 test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1Johannes Schindelin via GitGitGadget, May 4, 2026
  36. 10/11 t5608: mark >4GB tests as EXPENSIVEJohannes Schindelin via GitGitGadget, May 4, 2026
  37. 11/11 ci: run expensive tests on push builds to integration branchesJohannes Schindelin via GitGitGadget, May 4, 2026
  38. Derrick StoleeMay 4, 2026
  39. Junio C HamanoMay 5, 2026
  40. Junio C HamanoMay 5, 2026
  41. Johannes SchindelinMay 6, 2026
  42. Junio C HamanoMay 7, 2026
  43. Patrick SteinhardtMay 7, 2026
  44. Junio C HamanoMay 8, 2026
  45. 00/11 Handle cloning of objects larger than 4GB on WindowsJohannes Schindelin via GitGitGadget, May 8, 2026
  46. 01/11 index-pack, unpack-objects: use size_t for object sizeJohannes Schindelin via GitGitGadget, May 8, 2026
  47. 02/11 git-zlib: handle data streams larger than 4GBJohannes Schindelin via GitGitGadget, May 8, 2026
  48. 03/11 odb, packfile: use size_t for streaming object sizesJohannes Schindelin via GitGitGadget, May 8, 2026
  49. 04/11 delta, packfile: use size_t for delta header sizesJohannes Schindelin via GitGitGadget, May 8, 2026
  50. 05/11 test-tool: add a helper to synthesize large packfilesJohannes Schindelin via GitGitGadget, May 8, 2026
  51. 06/11 t5608: add regression test for >4GB object cloneJohannes Schindelin via GitGitGadget, May 8, 2026
  52. 07/11 test-tool synthesize: use the unsafe hash for speedJohannes Schindelin via GitGitGadget, May 8, 2026
  53. 08/11 test-tool synthesize: precompute pack for 4 GiB + 1Johannes Schindelin via GitGitGadget, May 8, 2026
  54. 09/11 test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1Johannes Schindelin via GitGitGadget, May 8, 2026
  55. 10/11 t5608: mark >4GB tests as EXPENSIVEJohannes Schindelin via GitGitGadget, May 8, 2026
  56. 11/11 ci: run expensive tests on push builds to integration branchesJohannes Schindelin via GitGitGadget, May 8, 2026
  57. ci: enable EXPENSIVE for contributor buildsJunio C Hamano, May 10, 2026
  58. Patrick SteinhardtMay 11, 2026
  59. Junio C HamanoMay 11, 2026
  60. Patrick SteinhardtMay 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.