{"thread":{"id":"66735","subject":"[PATCH 0/2] archive, apply: do not truncate files of 4GiB or more","startedAt":"2026-11-01T09:19:21Z","lastAt":"2026-11-19T17:48:38Z","messageCount":6,"participants":["Elia Pinto","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"557071","messageId":"cover.1793524457.git.gitter.spiros@gmail.com","threadId":"66735","inReplyTo":null,"subject":"[PATCH 0/2] archive, apply: do not truncate files of 4GiB or more","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T09:19:21Z","receivedAt":"2026-11-01T09:19:21Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Two more cases where, on platforms where \"long\" is 32 bits but size_t\nis 64 bits (64-bit Windows), a file of 4GiB or more is written truncated\nor corrupt without any error.\n\n- 1/2 archive: tar and zip of a file written by streaming come out\n  corrupt.\n- 2/2 apply: the file being patched is written truncated with exit 0\n  (also with --index and git am).\n\nNo new tests; both were verified by simulating the truncation at 16\nbits, as described in the messages.\n\nRegarding t5004, the existing test \"zip archive with files bigger than\n4GB\" has been excluded where \"long\" is 32 bits (LONG_IS_64BIT) since\n2017, because back then the object database couldn't read objects that\nlarge. Today on 64-bit Windows it can, and with this series zip works\ntoo. Thus, the guard could become SIZE_T_IS_64BIT to enable a real test\non Windows. Since I haven't run it on Windows, I'm offering this as a\nfollow-up or asking reviewers for an opinion.\n\nWith this series, the search for these truncations is finished. The\nremaining warnings are not reachable: a 4GiB index, sizes bounded by\nconfig or by the format, commits over 4GiB that already die.\n\nIt doesn't depend on the other series in flight and merges cleanly with\nthem. Based on master 50fe5cc63f.\n\nElia Pinto (2):\n  archive: use size_t for the size of archive entries\n  apply: use size_t for the size of the result written out\n\n apply.c       |  8 ++++----\n archive-tar.c | 10 +++++-----\n archive-zip.c | 16 ++++++++--------\n archive.h     |  2 +-\n 4 files changed, 18 insertions(+), 18 deletions(-)\n\n\nbase-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7\n-- \n2.53.0\n\n\n"},{"id":"557072","messageId":"bfd638278f13b1ca127b723e6af738784fc6a429.1793524457.git.gitter.spiros@gmail.com","threadId":"66735","inReplyTo":"cover.1793524457.git.gitter.spiros@gmail.com","subject":"[PATCH 1/2] archive: use size_t for the size of archive entries","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T09:19:22Z","receivedAt":"2026-11-01T09:19:22Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"The archive entry callback and format implementations use \"unsigned\nlong\" for the entry size, while blobs above core.bigFileThreshold\narrive as size_t.\n\nOn platforms where \"long\" is 32 bits but size_t is 64 bits, such as\n64-bit Windows, streaming a file of 4GiB or more causes silent\ncorruption: for tar, the truncated size goes into the ustar header while\nstream_blocked() writes the entire blob, corrupting the archive; for\nzip, write_zip_entry() overflows compressed_size, producing incorrect\nheaders and a corrupted archive.\n\nThis was verified using a 16-bit simulation, where tar extracted\ntruncated data and Python's zipfile rejected the output due to bad\nmagic numbers or CRC-32 errors.\n\nUse size_t in the callback, tar writer, and zip writer functions,\nallowing the existing zip64 helpers to handle large files correctly.\n\nTriggering this requires processing a file of 4GiB or more on a platform\nwhere \"long\" is 32 bits but size_t is 64 bits, so no test is added.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n archive-tar.c | 10 +++++-----\n archive-zip.c | 16 ++++++++--------\n archive.h     |  2 +-\n 3 files changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex a1c66024d4..31ab704886 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -66,7 +66,7 @@ static void write_if_needed(void)\n  * queues up writes, so that all our write(2) calls write exactly one\n  * full block; pads writes to RECORDSIZE\n  */\n-static void do_write_blocked(const void *data, unsigned long size)\n+static void do_write_blocked(const void *data, size_t size)\n {\n \tconst char *buf = data;\n \n@@ -102,7 +102,7 @@ static void finish_record(void)\n \twrite_if_needed();\n }\n \n-static void write_blocked(const void *data, unsigned long size)\n+static void write_blocked(const void *data, size_t size)\n {\n \tdo_write_blocked(data, size);\n \tfinish_record();\n@@ -218,7 +218,7 @@ static size_t get_path_prefix(const char *path, size_t pathlen, size_t maxlen)\n \n static void prepare_header(struct archiver_args *args,\n \t\t\t   struct ustar_header *header,\n-\t\t\t   unsigned int mode, unsigned long size)\n+\t\t\t   unsigned int mode, size_t size)\n {\n \txsnprintf(header->mode, sizeof(header->mode), \"%07o\", mode & 07777);\n \txsnprintf(header->size, sizeof(header->size), \"%011\"PRIoMAX , S_ISREG(mode) ? (uintmax_t)size : (uintmax_t)0);\n@@ -256,11 +256,11 @@ static int write_tar_entry(struct archiver_args *args,\n \t\t\t   const struct object_id *oid,\n \t\t\t   const char *path, size_t pathlen,\n \t\t\t   unsigned int mode,\n-\t\t\t   void *buffer, unsigned long size)\n+\t\t\t   void *buffer, size_t size)\n {\n \tstruct ustar_header header;\n \tstruct strbuf ext_header = STRBUF_INIT;\n-\tunsigned long size_in_header;\n+\tsize_t size_in_header;\n \tint err = 0;\n \n \tmemset(&header, 0, sizeof(header));\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 785689584c..2ca6dd956d 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -191,9 +191,9 @@ static uint32_t clamp32(uintmax_t n)\n \treturn (n < max) ? n : max;\n }\n \n-static void *zlib_deflate_raw(void *data, unsigned long size,\n+static void *zlib_deflate_raw(void *data, size_t size,\n \t\t\t      int compression_level,\n-\t\t\t      unsigned long *compressed_size)\n+\t\t\t      size_t *compressed_size)\n {\n \tgit_zstream stream;\n \tsize_t maxsize;\n@@ -224,8 +224,8 @@ static void *zlib_deflate_raw(void *data, unsigned long size,\n \treturn buffer;\n }\n \n-static void write_zip_data_desc(unsigned long size,\n-\t\t\t\tunsigned long compressed_size,\n+static void write_zip_data_desc(size_t size,\n+\t\t\t\tsize_t compressed_size,\n \t\t\t\tunsigned long crc)\n {\n \tif (size >= 0xffffffff || compressed_size >= 0xffffffff) {\n@@ -248,8 +248,8 @@ static void write_zip_data_desc(unsigned long size,\n }\n \n static void set_zip_header_data_desc(struct zip_local_header *header,\n-\t\t\t\t     unsigned long size,\n-\t\t\t\t     unsigned long compressed_size,\n+\t\t\t\t     size_t size,\n+\t\t\t\t     size_t compressed_size,\n \t\t\t\t     unsigned long crc)\n {\n \tcopy_le32(header->crc32, crc);\n@@ -285,7 +285,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\t   const struct object_id *oid,\n \t\t\t   const char *path, size_t pathlen,\n \t\t\t   unsigned int mode,\n-\t\t\t   void *buffer, unsigned long size)\n+\t\t\t   void *buffer, size_t size)\n {\n \tstruct zip_local_header header;\n \tuintmax_t offset = zip_offset;\n@@ -293,7 +293,7 @@ static int write_zip_entry(struct archiver_args *args,\n \tsize_t header_extra_size = zip_extra_mtime.len;\n \tint need_zip64_extra = 0;\n \tunsigned long attr2;\n-\tunsigned long compressed_size;\n+\tsize_t compressed_size;\n \tunsigned long crc;\n \tenum zip_method method;\n \tunsigned char *out;\ndiff --git a/archive.h b/archive.h\nindex bbe65ba0f9..e52328d639 100644\n--- a/archive.h\n+++ b/archive.h\n@@ -56,7 +56,7 @@ typedef int (*write_archive_entry_fn_t)(struct archiver_args *args,\n \t\t\t\t\tconst struct object_id *oid,\n \t\t\t\t\tconst char *path, size_t pathlen,\n \t\t\t\t\tunsigned int mode,\n-\t\t\t\t\tvoid *buffer, unsigned long size);\n+\t\t\t\t\tvoid *buffer, size_t size);\n \n int write_archive_entries(struct archiver_args *args, write_archive_entry_fn_t write_entry);\n \n-- \n2.53.0\n\n\n"},{"id":"557073","messageId":"6446c73595b4d2d9360a25f816b5d42ee66b5393.1793524457.git.gitter.spiros@gmail.com","threadId":"66735","inReplyTo":"cover.1793524457.git.gitter.spiros@gmail.com","subject":"[PATCH 2/2] apply: use size_t for the size of the result written out","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T09:19:23Z","receivedAt":"2026-11-01T09:19:23Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"While \"git apply\" limits the patch size, the result size is stored as\nsize_t but passed to functions using \"unsigned long size\".\n\nOn platforms where \"long\" is 32 bits but size_t is 64 bits, such as\n64-bit Windows, applying a patch to a file of 4GiB or more causes the\nresult to be written out truncated with exit 0, silently losing data;\nthis also affects \"git am\" and \"git apply --index\", which writes the\ntruncated content to the index.\n\nThis was verified using a 16-bit simulation, where \"git apply\" left\na 7901-line file truncated to 2859 lines.\n\nUse size_t in create_file(), create_one_file(), try_create_file(), and\nadd_index_file(). add_index_file() passes the size on to\nodb_write_object(), which takes a size_t with the \"use size_t for a few\nmore object and buffer sizes\" series.\n\nTriggering this requires processing a file of 4GiB or more on a platform\nwhere \"long\" is 32 bits but size_t is 64 bits, so no test is added.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n apply.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex a0b7119c63..02d6791ad7 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4458,7 +4458,7 @@ static int add_index_file(struct apply_state *state,\n \t\t\t  const char *path,\n \t\t\t  unsigned mode,\n \t\t\t  void *buf,\n-\t\t\t  unsigned long size)\n+\t\t\t  size_t size)\n {\n \tstruct stat st;\n \tstruct cache_entry *ce;\n@@ -4513,7 +4513,7 @@ static int add_index_file(struct apply_state *state,\n  */\n static int try_create_file(struct apply_state *state, const char *path,\n \t\t\t   unsigned int mode, const char *buf,\n-\t\t\t   unsigned long size)\n+\t\t\t   size_t size)\n {\n \tint fd, res;\n \tstruct strbuf nbuf = STRBUF_INIT;\n@@ -4564,7 +4564,7 @@ static int create_one_file(struct apply_state *state,\n \t\t\t   char *path,\n \t\t\t   unsigned mode,\n \t\t\t   const char *buf,\n-\t\t\t   unsigned long size)\n+\t\t\t   size_t size)\n {\n \tchar *newpath = NULL;\n \tint res;\n@@ -4686,7 +4686,7 @@ static int create_file(struct apply_state *state, struct patch *patch)\n {\n \tchar *path = patch->new_name;\n \tunsigned mode = patch->new_mode;\n-\tunsigned long size = patch->resultsize;\n+\tsize_t size = patch->resultsize;\n \tchar *buf = patch->result;\n \n \tif (!mode)\n-- \n2.53.0\n\n\n"},{"id":"558804","messageId":"20261119065711.827536-1-gitter.spiros@gmail.com","threadId":"66735","inReplyTo":"cover.1793524457.git.gitter.spiros@gmail.com","subject":"Re: [PATCH 0/2] archive, apply: do not truncate files of 4GiB or more","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-19T06:57:11Z","receivedAt":"2026-11-19T06:57:11Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Hi,\n\nA gentle ping on this two-patch series, which has had no comments so\nfar. It still applies cleanly to current master (e1ec09b23c) and\nmerges cleanly with 'seen'.\n\n1/2 (archive) stands on its own. 2/2 (apply) has a question I should\nhave raised in the cover letter. It fixes the file written to the\nworking tree, but with --index, and so with \"git am\", the result still\ngoes through odb_write_object(), whose length is an \"unsigned long\"\nuntil \"use size_t for a few more object and buffer sizes\" [1] lands.\nWhere \"long\" has 32 bits while size_t has 64, the blob written to the\nindex is then still truncated, while the working tree file is now\ncomplete.\n\nI see two ways to handle that:\n\n - pass the size through cast_size_t_to_ulong() in add_index_file(),\n   so that \"git apply --index\" dies instead, as 5/5 of the loose object\n   series [2] does for notes, and drop the cast once [1] is in;\n\n - hold back 2/2 until [1] lands, as Patrick suggested for a similar\n   case in the pack-receive series [3].\n\nWhich one would you prefer?\n\n[1] https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/\n[2] https://lore.kernel.org/git/cover.1795069177.git.gitter.spiros@gmail.com/\n[3] https://lore.kernel.org/git/av2pBh7vLj5DVDha@pks.im/\n\nThanks,\nElia\n\n"},{"id":"558809","messageId":"av7HMbnylHiz_MVz@pks.im","threadId":"66735","inReplyTo":"20261119065711.827536-1-gitter.spiros@gmail.com","subject":"Re: [PATCH 0/2] archive, apply: do not truncate files of 4GiB or more","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-19T09:53:53Z","receivedAt":"2026-11-19T09:53:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 19, 2026 at 07:57:11AM +0100, Elia Pinto wrote:\n> Hi,\n> \n> A gentle ping on this two-patch series, which has had no comments so\n> far. It still applies cleanly to current master (e1ec09b23c) and\n> merges cleanly with 'seen'.\n> \n> 1/2 (archive) stands on its own. 2/2 (apply) has a question I should\n> have raised in the cover letter. It fixes the file written to the\n> working tree, but with --index, and so with \"git am\", the result still\n> goes through odb_write_object(), whose length is an \"unsigned long\"\n> until \"use size_t for a few more object and buffer sizes\" [1] lands.\n> Where \"long\" has 32 bits while size_t has 64, the blob written to the\n> index is then still truncated, while the working tree file is now\n> complete.\n> \n> I see two ways to handle that:\n> \n>  - pass the size through cast_size_t_to_ulong() in add_index_file(),\n>    so that \"git apply --index\" dies instead, as 5/5 of the loose object\n>    series [2] does for notes, and drop the cast once [1] is in;\n> \n>  - hold back 2/2 until [1] lands, as Patrick suggested for a similar\n>    case in the pack-receive series [3].\n> \n> Which one would you prefer?\n\nI'd prefer the latter. So how about you send a new, rebased version of\nthis series once it has landed?\n\nPatrick\n\n"},{"id":"558869","messageId":"20261119174838.829420-1-gitter.spiros@gmail.com","threadId":"66735","inReplyTo":"av7HMbnylHiz_MVz@pks.im","subject":"Re: [PATCH 0/2] archive, apply: do not truncate files of 4GiB or more","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-19T17:48:38Z","receivedAt":"2026-11-19T17:48:38Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"On Thu, Nov 19, 2026 at 10:53:53AM +0100, Patrick Steinhardt wrote:\n> I'd prefer the latter. So how about you send a new, rebased version of\n> this series once it has landed?\n\nOK, I'll resend the series when needed. Thanks,\nElia\n\n"}]}