{"thread":{"id":"66731","subject":"[PATCH] pack-objects: use size_t for streamed large blobs","startedAt":"2026-10-31T18:04:10Z","lastAt":"2026-11-03T22:14:06Z","messageCount":5,"participants":["Elia Pinto","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"557039","messageId":"20261031180411.257932-1-gitter.spiros@gmail.com","threadId":"66731","inReplyTo":null,"subject":"[PATCH] pack-objects: use size_t for streamed large blobs","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T18:04:10Z","receivedAt":"2026-10-31T18:04:10Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"In builtin/pack-objects.c, write_no_reuse_object() handles blobs\nexceeding core.bigFileThreshold by writing them in a streaming path. It\ntakes the size from st->size (size_t) and assigns it to an \"unsigned\nlong\" variable, which is then written to the pack object header, and\nwrite_large_blob_data() adds up the compressed length in an \"unsigned\nlong\" as well.\n\nOn platforms where \"long\" is 32 bits, handling a blob of 4GiB or more\ntruncates the size in the pack header while the data decompresses to the\nfull size, corrupting the object. If the compressed data also exceeds\n4GiB, the length accumulation overflows \"unsigned long\", misaligning the\noffsets of all subsequent objects in the index.\n\nWhen running repack or gc on a loose large blob, the original loose copy\nis pruned, leaving only the corrupted packed copy and resulting in data\nloss. Truncating st->size to 16 bits, with a 100KiB loose blob reachable\nfrom a commit and \"git -c core.bigFileThreshold=1k repack -a -d\", leaves\na corrupt packed object and no loose copy.\n\nSwitch the streamed path to use size_t for size and datalen, make\nwrite_large_blob_data() add up and return the length as a size_t, and\nuse off_t for write_no_reuse_object()'s return value. Other truncations\nin pack-objects are not reachable because core.bigFileThreshold is\nitself an \"unsigned long\".\n\nTriggering this requires processing an object of 4GiB or more on a\nplatform where \"long\" is 32 bits, so no test is added.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca3a891dfb..452e785888 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -412,13 +412,13 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \treturn stream.total_out;\n }\n \n-static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n-\t\t\t\t\t   const struct object_id *oid)\n+static size_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n+\t\t\t\t    const struct object_id *oid)\n {\n \tgit_zstream stream;\n \tunsigned char ibuf[1024 * 16];\n \tunsigned char obuf[1024 * 16];\n-\tunsigned long olen = 0;\n+\tsize_t olen = 0;\n \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n \n \tgit_deflate_init(&stream, cfg->pack_compression_level);\n@@ -513,10 +513,10 @@ static inline int oe_size_greater_than(struct packing_data *pack,\n }\n \n /* Return 0 if we will bust the pack-size limit */\n-static unsigned long write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n-\t\t\t\t\t   unsigned long limit, int usable_delta)\n+static off_t write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n+\t\t\t\t   unsigned long limit, int usable_delta)\n {\n-\tunsigned long size, datalen;\n+\tsize_t size, datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t      dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n\nbase-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7\n-- \n2.53.0\n\n\n"},{"id":"557041","messageId":"20261031183748.273957-1-gitter.spiros@gmail.com","threadId":"66731","inReplyTo":"20261031180411.257932-1-gitter.spiros@gmail.com","subject":"[PATCH v2] pack-objects: use size_t for streamed large blobs","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T18:37:48Z","receivedAt":"2026-10-31T18:37:48Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"In builtin/pack-objects.c, write_no_reuse_object() handles blobs\nexceeding core.bigFileThreshold by writing them in a streaming path. It\ntakes the size from st->size (size_t) and assigns it to an \"unsigned\nlong\" variable, which is then written to the pack object header, and\nwrite_large_blob_data() adds up the compressed length in an \"unsigned\nlong\" as well.\n\nOn platforms where \"long\" is 32 bits, handling a blob of 4GiB or more\ntruncates the size in the pack header while the data decompresses to the\nfull size, corrupting the object. If the compressed data also exceeds\n4GiB, the length accumulation overflows \"unsigned long\", misaligning the\noffsets of all subsequent objects in the index.\n\nWhen running repack or gc on a loose large blob, the original loose copy\nis pruned, leaving only the corrupted packed copy and resulting in data\nloss. Truncating st->size to 16 bits, with a 100KiB loose blob reachable\nfrom a commit and \"git -c core.bigFileThreshold=1k repack -a -d\", leaves\na corrupt packed object and no loose copy.\n\nSwitch the streamed path to use size_t for the object size, and off_t\nfor the length of the compressed data, which write_large_blob_data()\nadds up while it writes the stream piece by piece, and for\nwrite_no_reuse_object()'s return value, like write_reuse_object()\nalready does. Other truncations in pack-objects are not reachable\nbecause core.bigFileThreshold is itself an \"unsigned long\".\n\nTriggering this requires processing an object of 4GiB or more on a\nplatform where \"long\" is 32 bits, so no test is added.\n\nHelped-by: Simon Richter <Simon.Richter@hogyros.de>\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\nChanges since v1: use off_t for the length of the streamed data,\nfollowing Simon's comment.\n\n builtin/pack-objects.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca3a891dfb..24203b5359 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -412,13 +412,13 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \treturn stream.total_out;\n }\n \n-static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n-\t\t\t\t\t   const struct object_id *oid)\n+static off_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n+\t\t\t\t   const struct object_id *oid)\n {\n \tgit_zstream stream;\n \tunsigned char ibuf[1024 * 16];\n \tunsigned char obuf[1024 * 16];\n-\tunsigned long olen = 0;\n+\toff_t olen = 0;\n \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n \n \tgit_deflate_init(&stream, cfg->pack_compression_level);\n@@ -513,10 +513,11 @@ static inline int oe_size_greater_than(struct packing_data *pack,\n }\n \n /* Return 0 if we will bust the pack-size limit */\n-static unsigned long write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n-\t\t\t\t\t   unsigned long limit, int usable_delta)\n+static off_t write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n+\t\t\t\t   unsigned long limit, int usable_delta)\n {\n-\tunsigned long size, datalen;\n+\tsize_t size;\n+\toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t      dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n\nRange-diff against v1:\n1:  45e45c5b75 ! 1:  185b9da891 pack-objects: use size_t for streamed large blobs\n    @@ Commit message\n         from a commit and \"git -c core.bigFileThreshold=1k repack -a -d\", leaves\n         a corrupt packed object and no loose copy.\n     \n    -    Switch the streamed path to use size_t for size and datalen, make\n    -    write_large_blob_data() add up and return the length as a size_t, and\n    -    use off_t for write_no_reuse_object()'s return value. Other truncations\n    -    in pack-objects are not reachable because core.bigFileThreshold is\n    -    itself an \"unsigned long\".\n    +    Switch the streamed path to use size_t for the object size, and off_t\n    +    for the length of the compressed data, which write_large_blob_data()\n    +    adds up while it writes the stream piece by piece, and for\n    +    write_no_reuse_object()'s return value, like write_reuse_object()\n    +    already does. Other truncations in pack-objects are not reachable\n    +    because core.bigFileThreshold is itself an \"unsigned long\".\n     \n         Triggering this requires processing an object of 4GiB or more on a\n         platform where \"long\" is 32 bits, so no test is added.\n     \n    +    Helped-by: Simon Richter <Simon.Richter@hogyros.de>\n         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>\n     \n      ## builtin/pack-objects.c ##\n    @@ builtin/pack-objects.c: static unsigned long do_compress(void **pptr, unsigned l\n      \n     -static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n     -\t\t\t\t\t   const struct object_id *oid)\n    -+static size_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n    -+\t\t\t\t    const struct object_id *oid)\n    ++static off_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n    ++\t\t\t\t   const struct object_id *oid)\n      {\n      \tgit_zstream stream;\n      \tunsigned char ibuf[1024 * 16];\n      \tunsigned char obuf[1024 * 16];\n     -\tunsigned long olen = 0;\n    -+\tsize_t olen = 0;\n    ++\toff_t olen = 0;\n      \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n      \n      \tgit_deflate_init(&stream, cfg->pack_compression_level);\n    @@ builtin/pack-objects.c: static inline int oe_size_greater_than(struct packing_da\n     +\t\t\t\t   unsigned long limit, int usable_delta)\n      {\n     -\tunsigned long size, datalen;\n    -+\tsize_t size, datalen;\n    ++\tsize_t size;\n    ++\toff_t datalen;\n      \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n      \t\t      dheader[MAX_PACK_OBJECT_HEADER];\n      \tunsigned hdrlen;\n\nbase-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7\n-- \n2.53.0\n\n\n"},{"id":"557054","messageId":"xmqqse1l12hh.fsf@gitster.g","threadId":"66731","inReplyTo":"20261031183748.273957-1-gitter.spiros@gmail.com","subject":"Re: [PATCH v2] pack-objects: use size_t for streamed large blobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-01T06:37:30Z","receivedAt":"2026-11-01T06:37:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elia Pinto <gitter.spiros@gmail.com> writes:\n\n> In builtin/pack-objects.c, write_no_reuse_object() handles blobs\n> exceeding core.bigFileThreshold by writing them in a streaming path. It\n> takes the size from st->size (size_t) and assigns it to an \"unsigned\n> long\" variable, which is then written to the pack object header, and\n> write_large_blob_data() adds up the compressed length in an \"unsigned\n> long\" as well.\n>\n> On platforms where \"long\" is 32 bits, handling a blob of 4GiB or more\n\nOn 32-bit boxes, replacing ulong with size_t would not fix anything,\nwhich you already had an discussion.\n\nPlease rewrite the above to stress that you are talking about\nplatforms where \"long\" is 32 bits but \"size_t\" is larger than that,\nso that the next person do not have to be puzzled by thinking this\nis about 32-bit boxes.\n\n"},{"id":"557056","messageId":"20261101072904.274912-1-gitter.spiros@gmail.com","threadId":"66731","inReplyTo":"20261031183748.273957-1-gitter.spiros@gmail.com","subject":"[PATCH v3] pack-objects: use size_t for streamed large blobs","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T07:29:03Z","receivedAt":"2026-11-01T07:29:03Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"In builtin/pack-objects.c, write_no_reuse_object() handles blobs\nexceeding core.bigFileThreshold by writing them in a streaming path. It\ntakes the size from st->size (size_t) and assigns it to an \"unsigned\nlong\" variable, which is then written to the pack object header, and\nwrite_large_blob_data() adds up the compressed length in an \"unsigned\nlong\" as well.\n\nOn platforms where \"long\" is 32 bits but size_t is 64 bits, like 64-bit\nWindows, handling a blob of 4GiB or more truncates the size in the pack\nheader while the data decompresses to the full size, corrupting the\nobject. If the compressed data also exceeds 4GiB, the length\naccumulation overflows \"unsigned long\", misaligning the offsets of all\nsubsequent objects in the index.\n\nWhen running repack or gc on a loose large blob, the original loose copy\nis pruned, leaving only the corrupted packed copy and resulting in data\nloss. Truncating st->size to 16 bits, with a 100KiB loose blob reachable\nfrom a commit and \"git -c core.bigFileThreshold=1k repack -a -d\", leaves\na corrupt packed object and no loose copy.\n\nSwitch the streamed path to use size_t for the object size, and off_t\nfor the length of the compressed data, which write_large_blob_data()\nadds up while it writes the stream piece by piece, and for\nwrite_no_reuse_object()'s return value, like write_reuse_object()\nalready does. Other truncations in pack-objects are not reachable\nbecause core.bigFileThreshold is itself an \"unsigned long\".\n\nTriggering this requires processing an object of 4GiB or more on a\nplatform where \"long\" is 32 bits but size_t is 64 bits, so no test is\nadded.\n\nHelped-by: Simon Richter <Simon.Richter@hogyros.de>\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\nChanges since v2: say that this is about platforms where \"long\" is\n32 bits but size_t is 64 bits, as Junio asked.\n\n builtin/pack-objects.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca3a891dfb..24203b5359 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -412,13 +412,13 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \treturn stream.total_out;\n }\n \n-static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n-\t\t\t\t\t   const struct object_id *oid)\n+static off_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n+\t\t\t\t   const struct object_id *oid)\n {\n \tgit_zstream stream;\n \tunsigned char ibuf[1024 * 16];\n \tunsigned char obuf[1024 * 16];\n-\tunsigned long olen = 0;\n+\toff_t olen = 0;\n \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n \n \tgit_deflate_init(&stream, cfg->pack_compression_level);\n@@ -513,10 +513,11 @@ static inline int oe_size_greater_than(struct packing_data *pack,\n }\n \n /* Return 0 if we will bust the pack-size limit */\n-static unsigned long write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n-\t\t\t\t\t   unsigned long limit, int usable_delta)\n+static off_t write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n+\t\t\t\t   unsigned long limit, int usable_delta)\n {\n-\tunsigned long size, datalen;\n+\tsize_t size;\n+\toff_t datalen;\n \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n \t\t      dheader[MAX_PACK_OBJECT_HEADER];\n \tunsigned hdrlen;\n\nRange-diff against v2:\n1:  185b9da891 ! 1:  02e682725b pack-objects: use size_t for streamed large blobs\n    @@ Commit message\n         write_large_blob_data() adds up the compressed length in an \"unsigned\n         long\" as well.\n     \n    -    On platforms where \"long\" is 32 bits, handling a blob of 4GiB or more\n    -    truncates the size in the pack header while the data decompresses to the\n    -    full size, corrupting the object. If the compressed data also exceeds\n    -    4GiB, the length accumulation overflows \"unsigned long\", misaligning the\n    -    offsets of all subsequent objects in the index.\n    +    On platforms where \"long\" is 32 bits but size_t is 64 bits, like 64-bit\n    +    Windows, handling a blob of 4GiB or more truncates the size in the pack\n    +    header while the data decompresses to the full size, corrupting the\n    +    object. If the compressed data also exceeds 4GiB, the length\n    +    accumulation overflows \"unsigned long\", misaligning the offsets of all\n    +    subsequent objects in the index.\n     \n         When running repack or gc on a loose large blob, the original loose copy\n         is pruned, leaving only the corrupted packed copy and resulting in data\n    @@ Commit message\n         because core.bigFileThreshold is itself an \"unsigned long\".\n     \n         Triggering this requires processing an object of 4GiB or more on a\n    -    platform where \"long\" is 32 bits, so no test is added.\n    +    platform where \"long\" is 32 bits but size_t is 64 bits, so no test is\n    +    added.\n     \n         Helped-by: Simon Richter <Simon.Richter@hogyros.de>\n         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>\n\nbase-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7\n-- \n2.53.0\n\n\n"},{"id":"557385","messageId":"xmqqo6c54l75.fsf@gitster.g","threadId":"66731","inReplyTo":"20261101072904.274912-1-gitter.spiros@gmail.com","subject":"Re: [PATCH v3] pack-objects: use size_t for streamed large blobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-03T22:14:06Z","receivedAt":"2026-11-03T22:14:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elia Pinto <gitter.spiros@gmail.com> writes:\n\n> On platforms where \"long\" is 32 bits but size_t is 64 bits, like 64-bit\n> Windows, handling a blob of 4GiB or more truncates the size in the pack\n> header while the data decompresses to the full size, corrupting the\n> object. If the compressed data also exceeds 4GiB, the length\n> accumulation overflows \"unsigned long\", misaligning the offsets of all\n> subsequent objects in the index.\n\nMakes sense.\n\nWill queue.  Thanks.\n\n> -static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n> -\t\t\t\t\t   const struct object_id *oid)\n> +static off_t write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n> +\t\t\t\t   const struct object_id *oid)\n>  {\n>  \tgit_zstream stream;\n>  \tunsigned char ibuf[1024 * 16];\n>  \tunsigned char obuf[1024 * 16];\n> -\tunsigned long olen = 0;\n> +\toff_t olen = 0;\n>  \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n>  \n>  \tgit_deflate_init(&stream, cfg->pack_compression_level);\n> @@ -513,10 +513,11 @@ static inline int oe_size_greater_than(struct packing_data *pack,\n>  }\n>  \n>  /* Return 0 if we will bust the pack-size limit */\n> -static unsigned long write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n> -\t\t\t\t\t   unsigned long limit, int usable_delta)\n> +static off_t write_no_reuse_object(struct hashfile *f, struct object_entry *entry,\n> +\t\t\t\t   unsigned long limit, int usable_delta)\n>  {\n> -\tunsigned long size, datalen;\n> +\tsize_t size;\n> +\toff_t datalen;\n>  \tunsigned char header[MAX_PACK_OBJECT_HEADER],\n>  \t\t      dheader[MAX_PACK_OBJECT_HEADER];\n>  \tunsigned hdrlen;\n>\n> Range-diff against v2:\n> 1:  185b9da891 ! 1:  02e682725b pack-objects: use size_t for streamed large blobs\n>     @@ Commit message\n>          write_large_blob_data() adds up the compressed length in an \"unsigned\n>          long\" as well.\n>      \n>     -    On platforms where \"long\" is 32 bits, handling a blob of 4GiB or more\n>     -    truncates the size in the pack header while the data decompresses to the\n>     -    full size, corrupting the object. If the compressed data also exceeds\n>     -    4GiB, the length accumulation overflows \"unsigned long\", misaligning the\n>     -    offsets of all subsequent objects in the index.\n>     +    On platforms where \"long\" is 32 bits but size_t is 64 bits, like 64-bit\n>     +    Windows, handling a blob of 4GiB or more truncates the size in the pack\n>     +    header while the data decompresses to the full size, corrupting the\n>     +    object. If the compressed data also exceeds 4GiB, the length\n>     +    accumulation overflows \"unsigned long\", misaligning the offsets of all\n>     +    subsequent objects in the index.\n>      \n>          When running repack or gc on a loose large blob, the original loose copy\n>          is pruned, leaving only the corrupted packed copy and resulting in data\n>     @@ Commit message\n>          because core.bigFileThreshold is itself an \"unsigned long\".\n>      \n>          Triggering this requires processing an object of 4GiB or more on a\n>     -    platform where \"long\" is 32 bits, so no test is added.\n>     +    platform where \"long\" is 32 bits but size_t is 64 bits, so no test is\n>     +    added.\n>      \n>          Helped-by: Simon Richter <Simon.Richter@hogyros.de>\n>          Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>\n>\n> base-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7\n\n"}]}