{"thread":{"id":"66617","subject":"[PATCH 0/3] use size_t for a few more object and buffer sizes","startedAt":"2026-10-21T10:16:36Z","lastAt":"2026-11-03T06:59:33Z","messageCount":24,"participants":["Elia Pinto","Patrick Steinhardt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"555708","messageId":"20261021101640.329990-1-gitter.spiros@gmail.com","threadId":"66617","inReplyTo":null,"subject":"[PATCH 0/3] use size_t for a few more object and buffer sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-21T10:16:36Z","receivedAt":"2026-10-21T10:16:36Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"On 64-bit Windows, \"long\" is still 32 bits wide. Whenever a size_t\ntravels through an \"unsigned long\", anything above 4 GiB is silently\nlost. Johannes's js/objects-larger-than-4gb-on-windows work fixed the\nread side of the object database, and Peff's jk/xdiff-size_t did the\nsame for the diff and merge machinery. This series picks up three\nplaces that were left narrow.\n\n - odb: the write API still takes the object length as unsigned long.\n A large file that is converted before being stored (for example\n with core.autocrlf, which Git for Windows enables by default) can\n reach odb_write_object() with a size_t length above 4 GiB. The\n blob is then written truncated, with no error, and the index\n happily records it.\n\n - diff: diff_words_fill() grows its buffer with a signed \"long\"\n allocation size. Its output can be twice its input, and on LLP64\n the growth computation overflows past about 700MiB, which is well\n within what xdiff accepts. In practice Git dies on a huge\n allocation request.\n\n - convert: the helpers that decide whether a file is binary, has\n CRLF line endings or \"$Id$\" keywords take an unsigned long length.\n For files over 4GiB they only look at a prefix. In the same area,\n read_blob_data_from_index() narrows its result with\n cast_size_t_to_ulong(), so \"git add\" with core.autocrlf dies when\n the staged version is that large.\n\nThere are no new tests. Each case needs files of several GiB and a\n32-bit \"long\", and the commit messages explain how to reproduce them.\n\nHow these were found\n--------------------\n\nI wanted a list of all the places where a 64-bit size is narrowed on\nWindows, rather than waiting for the next bug report. Linux compilers\ncannot see the problem, because there \"long\" and \"size_t\" have the\nsame width. So I analysed the tree as a Windows target instead.\n\nThe first step is a compilation database from a real MinGW\nconfiguration. Forcing --target onto a database generated on Linux\ndoes not work well: Linux-only defines leak in, regex.h is missing\nfrom MinGW, and the precompiled header is rejected for the wrong\ntarget. A Meson cross build avoids all of that:\n\n $ cat mingw64.cross\n [binaries]\n c = 'x86_64-w64-mingw32-gcc'\n ar = 'x86_64-w64-mingw32-ar'\n strip = 'x86_64-w64-mingw32-strip'\n windres = 'x86_64-w64-mingw32-windres'\n\n [host_machine]\n system = 'windows'\n cpu_family = 'x86_64'\n cpu = 'x86_64'\n endian = 'little'\n\n $ meson setup build-win --cross-file mingw64.cross \\\n\t-Db_pch=false -Dsane_tool_path=/usr/bin \\\n\t-Dcurl=disabled -Dexpat=disabled -Dpcre2=disabled \\\n\t-Dgettext=disabled -Diconv=disabled -Drust=disabled\n $ meson compile -C build-win\n\nTwo of these options are not obvious. Without -Dsane_tool_path,\nsetup stops because meson.build adds \"C:/Program Files/Git/bin\" to\nthe program search path, which is not absolute on a Linux host. And\n-Db_pch=false is needed because clang cannot read the precompiled\nheader that GCC produces. The build only has to get far enough to\ngenerate the headers; link errors do not matter here.\n\nThe second step runs clang-tidy over that database with\n-Wshorten-64-to-32, for the x86_64-w64-mingw32 target, skipping the\nbundled zlib and OpenSSL. One detail cost me a while: with\n\"-checks=-*\", clang-tidy drops compiler warnings unless\nclang-diagnostic-shorten-64-to-32 is enabled explicitly. I wrapped\nall this in a small script, which also refuses to report \"no\nfindings\" when some translation units failed to parse:\n\n https://github.com/devzero2000/toyscript/blob/main/check_llp64_portability.sh\n\nIt is not meant for inclusion. I mention it so that the numbers below\ncan be reproduced, and in case someone finds it useful.\n\nWhat the scan shows\n-------------------\n\nAll 569 translation units were analysed, none failed. There are 1432\n64-to-32 truncation warnings, but most of them are not specific to\nWindows: about 1200 narrow a size_t to an int, as they do on Linux.\nOnly 237 have \"long\" or \"unsigned long\" as their destination.\n\nMost of those 237 sit on API boundaries that are known and that\ncommit c6a4629e32 names explicitly: diff, blame, textconv and\nfast-import. Others are in tree walking, object parsing and the pack\ntools, where an oversized object makes Git die rather than misbehave.\nI would like to convert them in later small series, one area at a\ntime, in the same style as jk/xdiff-size_t.\n\nJohannes, the first patch continues your write-side work. I hope it\ndoes not step on a follow-up you already have in progress.\n\nElia Pinto (3):\n odb: use size_t for the length of objects to write\n diff: use size_t for the allocation size of word-diff buffers\n convert: use size_t for buffer sizes\n\n attr.c | 2 +-\n convert.c | 14 +++++++-------\n diff.c | 4 ++--\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/source.h | 2 +-\n read-cache-ll.h | 2 +-\n read-cache.c | 4 ++--\n 10 files changed, 19 insertions(+), 19 deletions(-)\n\n\nbase-commit: 47fdf2208c5b94c8cc01c106b8f7c85efb91c926\n--\n2.53.0\n\n\n"},{"id":"555709","messageId":"20261021101640.329990-2-gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"20261021101640.329990-1-gitter.spiros@gmail.com","subject":"[PATCH 1/3] odb: use size_t for the length of objects to write","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-21T10:16:37Z","receivedAt":"2026-10-21T10:16:37Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"c6a4629e32 (odb: use size_t for object_info.sizep and the size APIs,\n2026-06-15) widened the read side of the object database API, and the\nwrite_object() callback of odb_source already takes a size_t. But\nodb_write_object_ext(), odb_write_object() and odb_source_write_object()\nin front of it, as well as write_loose_object() behind it, still take\nan \"unsigned long\", which is only 32 bits wide on Windows.\n\nMost callers hand them a size_t from a strbuf or from odb_read_object(),\nwhich silently gets truncated. The most visible case is index_mem() and\nindex_stream_convert_blob(): when \"git add\" or \"git hash-object -w\"\nruns a file over 4GiB through a conversion (core.autocrlf, which is on\nby default in Git for Windows, an eol or ident attribute, or a clean\nfilter), we write a blob made of the first \"size mod 2^32\" bytes. The\nindex records it with the stat data of the real file, so \"git status\"\nstays clean and nobody notices until the file is checked out again.\n\"git hash-object\" without \"-w\" uses hash_object_file(), which takes a\nsize_t, so the two disagree on the object ID of the same file.\n\nUse size_t throughout, including for the size kept by the in-memory\nsource.\n\nThe existing \"hash filtered files over 4GB correctly\" test in t1007\nonly hashes. Running the same with \"git hash-object -w\" on Windows\nshows the problem, but needs another pass over a 5GiB file, so no new\ntest is added here.\n\nSigned-off-by: Elia Pinto \n---\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/source.h | 2 +-\n 5 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 98c5c862f5..6e9daf11e4 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -667,7 +667,7 @@ void odb_assert_oid_type(struct object_database *odb,\n }\n \n int odb_write_object_ext(struct object_database *odb,\n-\t\t\t const void *buf, unsigned long len,\n+\t\t\t const void *buf, size_t len,\n \t\t\t enum object_type type,\n \t\t\t struct object_id *oid,\n \t\t\t const struct object_id *compat_oid_in,\ndiff --git a/odb.h b/odb.h\nindex 0ccc47ee73..25d02fae75 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -633,14 +633,14 @@ enum odb_write_object_flags {\n * Returns 0 on success, a negative error code otherwise.\n */\n int odb_write_object_ext(struct object_database *odb,\n-\t\t\t const void *buf, unsigned long len,\n+\t\t\t const void *buf, size_t len,\n \t\t\t enum object_type type,\n \t\t\t struct object_id *oid,\n \t\t\t const struct object_id *compat_oid,\n \t\t\t enum odb_write_object_flags flags);\n \n static inline int odb_write_object(struct object_database *odb,\n-\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t const void *buf, size_t len,\n \t\t\t\t enum object_type type,\n \t\t\t\t struct object_id *oid)\n {\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 868e723c81..ab7173cd2d 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -10,7 +10,7 @@\n struct inmemory_object {\n \tenum object_type type;\n \tconst void *buf;\n-\tunsigned long size;\n+\tsize_t size;\n };\n \n static const struct inmemory_object *find_cached_object(struct odb_source_inmemory *source,\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 0664634938..fe1a298364 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -770,7 +770,7 @@ static int end_loose_object_common(struct odb_source_loose *loose,\n \n static int write_loose_object(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid, char *hdr,\n-\t\t\t int hdrlen, const void *buf, unsigned long len,\n+\t\t\t int hdrlen, const void *buf, size_t len,\n \t\t\t const time_t *mtime, unsigned flags)\n {\n \tint fd, ret;\ndiff --git a/odb/source.h b/odb/source.h\nindex 1867347dfc..8e6ed96399 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -469,7 +469,7 @@ static inline int odb_source_freshen_object(struct odb_source *source,\n * object ID and the compatibility object ID, if non-NULL.\n */\n static inline int odb_source_write_object(struct odb_source *source,\n-\t\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t\t const void *buf, size_t len,\n \t\t\t\t\t enum object_type type,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t const struct object_id *compat_oid,\n-- \n2.53.0\n\n\n"},{"id":"555710","messageId":"20261021101640.329990-3-gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"20261021101640.329990-1-gitter.spiros@gmail.com","subject":"[PATCH 2/3] diff: use size_t for the allocation size of word-diff buffers","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-21T10:16:38Z","receivedAt":"2026-10-21T10:16:38Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Since 3c7a31204f (xdiff: use size_t for buffer sizes, 2026-09-30) the\nsize of an mmfile_t is a size_t, but the word-diff code still grows\nthese buffers with ALLOC_GROW() using an \"unsigned long\" (in struct\ndiff_words_buffer) or a \"long\" (in diff_words_fill()) for the allocated\nsize.\n\nOn platforms where \"long\" is 32 bits wide this is not just cosmetic.\ndiff_words_fill() writes every word followed by a newline, so its\noutput can be up to twice as large as its input, which is limited by\nMAX_XDIFF_SIZE to just under 1GiB. alloc_nr() computes (alloc + 16) * 3\nin the type of \"alloc\", which overflows a signed 32-bit \"long\" as soon\nas the buffer grows past about 700MiB. The result is undefined behavior,\nin practice a negative size that REALLOC_ARRAY() turns into a huge\nallocation request and a die().\n\nThe buffer in diff_words_append() is bounded by MAX_XDIFF_SIZE, so\nthere it is only a matter of consistency, but if that limit ever went\naway, ALLOC_GROW() would truncate the requested size, and the memcpy()\nright after it would overflow the buffer.\n\nUse size_t for both.\n\nSigned-off-by: Elia Pinto \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex b4ac17f8ef..9483931c05 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1959,7 +1959,7 @@ static void emit_rewrite_diff(const char *name_a,\n \n struct diff_words_buffer {\n \tmmfile_t text;\n-\tunsigned long alloc;\n+\tsize_t alloc;\n \tstruct diff_words_orig {\n \t\tconst char *begin, *end;\n \t} *orig;\n@@ -2192,7 +2192,7 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,\n \t\tregex_t *word_regex)\n {\n \tint i, j;\n-\tlong alloc = 0;\n+\tsize_t alloc = 0;\n \n \tout->size = 0;\n \tout->ptr = NULL;\n-- \n2.53.0\n\n\n"},{"id":"555711","messageId":"20261021101640.329990-4-gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"20261021101640.329990-1-gitter.spiros@gmail.com","subject":"[PATCH 3/3] convert: use size_t for buffer sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-21T10:16:39Z","receivedAt":"2026-10-21T10:16:39Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"The conversion code receives its buffers with a size_t length, but the\nhelpers that scan them, gather_stats(), gather_convert_stats(),\ngather_convert_stats_ascii() and count_ident(), take an \"unsigned long\".\nOn platforms where that is 32 bits wide, like Windows, they only look\nat the first \"size mod 2^32\" bytes of a file over 4GiB. The decision\nwhether a file is binary, or contains CRLF line endings or \"$Id$\"\nkeywords, is then based on that prefix, and we may convert the line\nendings of a file that is binary further down, or not expand keywords\nthat appear after the first 4GiB.\n\nread_blob_data_from_index(), which feeds has_crlf_in_index(),\nget_cached_convert_stats_ascii() and the reading of .gitattributes from\nthe index, gets the size of the blob as a size_t, but hands it out\nthrough cast_size_t_to_ulong(), so on such platforms \"git add\" of a file\nwhose staged version is over 4GiB dies when core.autocrlf is in effect.\n\nUse size_t for all of them.\n\nSigned-off-by: Elia Pinto \n---\n attr.c | 2 +-\n convert.c | 14 +++++++-------\n read-cache-ll.h | 2 +-\n read-cache.c | 4 ++--\n 4 files changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 0e63f1b6de..1acb4ef430 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -793,7 +793,7 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n {\n \tstruct attr_stack *stack = NULL;\n \tchar *buf;\n-\tunsigned long size;\n+\tsize_t size;\n \tint sparse_dir_pos = -1;\n \n \tif (!istate)\ndiff --git a/convert.c b/convert.c\nindex 036506842c..ca5594cab0 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -42,9 +42,9 @@ struct text_stat {\n \tunsigned printable, nonprintable;\n };\n \n-static void gather_stats(const char *buf, unsigned long size, struct text_stat *stats)\n+static void gather_stats(const char *buf, size_t size, struct text_stat *stats)\n {\n-\tunsigned long i;\n+\tsize_t i;\n \n \tmemset(stats, 0, sizeof(*stats));\n \n@@ -102,7 +102,7 @@ static int convert_is_binary(const struct text_stat *stats)\n \treturn 0;\n }\n \n-static unsigned int gather_convert_stats(const char *data, unsigned long size)\n+static unsigned int gather_convert_stats(const char *data, size_t size)\n {\n \tstruct text_stat stats;\n \tint ret = 0;\n@@ -119,7 +119,7 @@ static unsigned int gather_convert_stats(const char *data, unsigned long size)\n \treturn ret;\n }\n \n-static const char *gather_convert_stats_ascii(const char *data, unsigned long size)\n+static const char *gather_convert_stats_ascii(const char *data, size_t size)\n {\n \tunsigned int convert_stats = gather_convert_stats(data, size);\n \n@@ -141,7 +141,7 @@ const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path)\n {\n \tconst char *ret;\n-\tunsigned long sz;\n+\tsize_t sz;\n \tvoid *data = read_blob_data_from_index(istate, path, &sz);\n \tret = gather_convert_stats_ascii(data, sz);\n \tfree(data);\n@@ -223,7 +223,7 @@ static void check_global_conv_flags_eol(const char *path,\n \n static int has_crlf_in_index(struct index_state *istate, const char *path)\n {\n-\tunsigned long sz;\n+\tsize_t sz;\n \tvoid *data;\n \tconst char *crp;\n \tint has_crlf = 0;\n@@ -1077,7 +1077,7 @@ static int read_convert_config(const char *var, const char *value,\n \treturn 0;\n }\n \n-static int count_ident(const char *cp, unsigned long size)\n+static int count_ident(const char *cp, size_t size)\n {\n \t/*\n \t * \"$Id: 0000000000000000000000000000000000000000 $\" <=> \"$Id$\"\ndiff --git a/read-cache-ll.h b/read-cache-ll.h\nindex 066dd8bc3b..d279059b5f 100644\n--- a/read-cache-ll.h\n+++ b/read-cache-ll.h\n@@ -416,7 +416,7 @@ int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\n int index_name_is_other(struct index_state *, const char *, int);\n-void *read_blob_data_from_index(struct index_state *, const char *, unsigned long *);\n+void *read_blob_data_from_index(struct index_state *, const char *, size_t *);\n \n /* do stat comparison even if CE_VALID is true */\n #define CE_MATCH_IGNORE_VALID\t\t01\ndiff --git a/read-cache.c b/read-cache.c\nindex c4cf08a3a3..635dad1ba3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3472,7 +3472,7 @@ int index_name_is_other(struct index_state *istate, const char *name,\n }\n \n void *read_blob_data_from_index(struct index_state *istate,\n-\t\t\t\tconst char *path, unsigned long *size)\n+\t\t\t\tconst char *path, size_t *size)\n {\n \tint pos, len;\n \tsize_t sz;\n@@ -3503,7 +3503,7 @@ void *read_blob_data_from_index(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \tif (size)\n-\t\t*size = cast_size_t_to_ulong(sz);\n+\t\t*size = sz;\n \treturn data;\n }\n \n-- \n2.53.0\n\n\n"},{"id":"555723","messageId":"atig6fUHbPqYdzXk@pks.im","threadId":"66617","inReplyTo":"20261021101640.329990-2-gitter.spiros@gmail.com","subject":"Re: [PATCH 1/3] odb: use size_t for the length of objects to write","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-21T11:24:25Z","receivedAt":"2026-10-21T11:24:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 21, 2026 at 12:16:37PM +0200, Elia Pinto wrote:\n> c6a4629e32 (odb: use size_t for object_info.sizep and the size APIs,\n> 2026-06-15) widened the read side of the object database API, and the\n> write_object() callback of odb_source already takes a size_t. But\n> odb_write_object_ext(), odb_write_object() and odb_source_write_object()\n> in front of it, as well as write_loose_object() behind it, still take\n> an \"unsigned long\", which is only 32 bits wide on Windows.\n\nSometimes I wish that C would give us an easier way to detect such\nmismatches. It of course does have optional warnings, but enabling those\nnow would probably lead to thousands of warnings.\n\n> Most callers hand them a size_t from a strbuf or from odb_read_object(),\n> which silently gets truncated. The most visible case is index_mem() and\n> index_stream_convert_blob(): when \"git add\" or \"git hash-object -w\"\n> runs a file over 4GiB through a conversion (core.autocrlf, which is on\n> by default in Git for Windows, an eol or ident attribute, or a clean\n> filter), we write a blob made of the first \"size mod 2^32\" bytes. The\n> index records it with the stat data of the real file, so \"git status\"\n> stays clean and nobody notices until the file is checked out again.\n> \"git hash-object\" without \"-w\" uses hash_object_file(), which takes a\n> size_t, so the two disagree on the object ID of the same file.\n\nAwesome.\n\n> Use size_t throughout, including for the size kept by the in-memory\n> source.\n> \n> The existing \"hash filtered files over 4GB correctly\" test in t1007\n> only hashes. Running the same with \"git hash-object -w\" on Windows\n> shows the problem, but needs another pass over a 5GiB file, so no new\n> test is added here.\n\nBut wouldn't it make sense to add a test? It's a rather serious bug, and\nthe existing tests are marked with EXPENSIVE already anyway. So I don't\nthink this is a good enough argument to not add another EXPENSIVE test.\n\n> diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\n> index 868e723c81..ab7173cd2d 100644\n> --- a/odb/source-inmemory.c\n> +++ b/odb/source-inmemory.c\n> @@ -10,7 +10,7 @@\n> struct inmemory_object {\n> \tenum object_type type;\n> \tconst void *buf;\n> -\tunsigned long size;\n> +\tsize_t size;\n> };\n\nI really hope nobody writes 4GB blobs into memory :) But despite that,\nthis fix is correct anyway.\n\nPatrick\n\n"},{"id":"555724","messageId":"atig9JAm_d-1CE0p@pks.im","threadId":"66617","inReplyTo":"20261021101640.329990-3-gitter.spiros@gmail.com","subject":"Re: [PATCH 2/3] diff: use size_t for the allocation size of word-diff buffers","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-21T11:24:36Z","receivedAt":"2026-10-21T11:24:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 21, 2026 at 12:16:38PM +0200, Elia Pinto wrote:\n> diff --git a/diff.c b/diff.c\n> index b4ac17f8ef..9483931c05 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1959,7 +1959,7 @@ static void emit_rewrite_diff(const char *name_a,\n> \n> struct diff_words_buffer {\n> \tmmfile_t text;\n> -\tunsigned long alloc;\n> +\tsize_t alloc;\n> \tstruct diff_words_orig {\n> \t\tconst char *begin, *end;\n> \t} *orig;\n> @@ -2192,7 +2192,7 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,\n> \t\tregex_t *word_regex)\n> {\n> \tint i, j;\n> -\tlong alloc = 0;\n> +\tsize_t alloc = 0;\n\nThis is especially offending. I double checked that we really don't make\nuse of a potential negative value, but all this is ever used for is\n`ALLOC_GROW()`.\n\nPatrick\n\n"},{"id":"555725","messageId":"atig-5kJzKHZS4jj@pks.im","threadId":"66617","inReplyTo":"20261021101640.329990-4-gitter.spiros@gmail.com","subject":"Re: [PATCH 3/3] convert: use size_t for buffer sizes","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-21T11:24:43Z","receivedAt":"2026-10-21T11:24:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 21, 2026 at 12:16:39PM +0200, Elia Pinto wrote:\n> diff --git a/convert.c b/convert.c\n> index 036506842c..ca5594cab0 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -42,9 +42,9 @@ struct text_stat {\n> \tunsigned printable, nonprintable;\n> };\n> \n> -static void gather_stats(const char *buf, unsigned long size, struct text_stat *stats)\n> +static void gather_stats(const char *buf, size_t size, struct text_stat *stats)\n> {\n> -\tunsigned long i;\n> +\tsize_t i;\n> \n> \tmemset(stats, 0, sizeof(*stats));\n> \n\nOne thing to note here is that `struct text_stat` still uses `unsigned`\ncounters. Is this something we should fix, as well?\n\n> @@ -1077,7 +1077,7 @@ static int read_convert_config(const char *var, const char *value,\n> \treturn 0;\n> }\n> \n> -static int count_ident(const char *cp, unsigned long size)\n> +static int count_ident(const char *cp, size_t size)\n> {\n> \t/*\n> \t * \"$Id: 0000000000000000000000000000000000000000 $\" <=> \"$Id$\"\n\nThis function is counting `$Id$` markers, so it's rather unlikely that\nit ever overflows. Theoretically it's possible though.\n\nPatrick\n\n"},{"id":"556871","messageId":"cover.1793352943.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"20261021101640.329990-1-gitter.spiros@gmail.com","subject":"[PATCH v2 0/4] use size_t for a few more object and buffer sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T09:58:49Z","receivedAt":"2026-10-30T09:58:49Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"On 64-bit Windows \"long\" is still 32 bits wide, so a size_t that goes\nthrough an \"unsigned long\" loses anything above 4 GiB. Like v1, this\nseries widens three such places: the length taken by the object\nwrite API (patch 1), the allocation size of the word-diff buffers\n(patch 2) and the buffer sizes in convert.c (patch 3). Patch 4 is\nnew. The cover letter of v1 explains how these were found:\n\n https://lore.kernel.org/git/20261021101640.329990-1-gitter.spiros@gmail.com/\n\nChanges since v1, all following Patrick's review (thanks!):\n\n - Patch 1 adds the test Patrick asked for. The existing \"hash\n filtered files over 4GB correctly\" test in t1007 only hashes; the\n new one writes the object with \"git hash-object -w\" and checks its\n object ID and size. It is EXPENSIVE and SIZE_T_IS_64BIT, like its\n neighbours. Truncating the length to 32 bits in\n odb_write_object_ext() on Linux, to mimic Windows, makes it fail.\n\n - Patch 4 is new and answers Patrick's question on patch 3: struct\n text_stat still counted in \"unsigned\", and count_ident() in an\n \"int\". Note that this one is not specific to LLP64, as \"unsigned\"\n is 32 bits everywhere; it is in this series because it came up\n in the review and touches the same code. A wrapped counter only\n leads to a wrong decision (e.g. no CRLF conversion for a file with\n exactly 2^32 lone LFs), not to a memory safety problem, because\n the counts are only size hints for strbuf_grow().\n\n - Rebased on current master; no conflicts.\n\nElia Pinto (4):\n odb: use size_t for the length of objects to write\n diff: use size_t for the allocation size of word-diff buffers\n convert: use size_t for buffer sizes\n convert: use size_t for text statistics and ident counts\n\n attr.c | 2 +-\n convert.c | 24 ++++++++++++------------\n diff.c | 4 ++--\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/source.h | 2 +-\n read-cache-ll.h | 2 +-\n read-cache.c | 4 ++--\n t/t1007-hash-object.sh | 13 +++++++++++++\n 11 files changed, 37 insertions(+), 24 deletions(-)\n\nRange-diff against v1:\n1: 0fff9532f9 ! 1: 8fcb9f10f1 odb: use size_t for the length of objects to write\n @@ Commit message\n source.\n \n The existing \"hash filtered files over 4GB correctly\" test in t1007\n - only hashes. Running the same with \"git hash-object -w\" on Windows\n - shows the problem, but needs another pass over a 5GiB file, so no new\n - test is added here.\n + only hashes, so add a variant that writes the object with \"git\n + hash-object -w\" and checks both its object ID and its size. Like the\n + other tests over 4GB, it needs EXPENSIVE and SIZE_T_IS_64BIT. Truncating\n + the length to 32 bits in odb_write_object_ext(), as happens on Windows\n + without this patch, makes it fail with a different object ID.\n \n Signed-off-by: Elia Pinto \n \n @@ odb/source.h: static inline int odb_source_freshen_object(struct odb_source *sou\n \t\t\t\t\t enum object_type type,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t const struct object_id *compat_oid,\n +\n + ## t/t1007-hash-object.sh ##\n +@@ t/t1007-hash-object.sh: test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n + \ttest_cmp expect actual\n + '\n + \n ++test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n ++\t\t'write filtered files over 4GB correctly' '\n ++\t{ test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } &&\n ++\ttest_oid large5GB >expect &&\n ++\ttest_config filter.null-filter.clean \"cat\" &&\n ++\techo \"big filter=null-filter\" >.gitattributes &&\n ++\tgit hash-object -w -- big >actual &&\n ++\ttest_cmp expect actual &&\n ++\techo $((5*1024*1024*1024)) >expect &&\n ++\tgit cat-file -s $(cat actual) >actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n + test_done\n2: 4ab9569f13 = 2: 42d4eae985 diff: use size_t for the allocation size of word-diff buffers\n3: 7ea1a566b5 = 3: 88ca2ceb80 convert: use size_t for buffer sizes\n-: ---------- > 4: 84e9e38a3e convert: use size_t for text statistics and ident counts\n\nbase-commit: b9e5cff1803ab24582201ef059516eb16014b9d9\n-- \n2.53.0\n\n\n"},{"id":"556872","messageId":"8fcb9f10f13962acd5cdac652bea8fc4164f2e78.1793352943.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793352943.git.gitter.spiros@gmail.com","subject":"[PATCH v2 1/4] odb: use size_t for the length of objects to write","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T09:58:50Z","receivedAt":"2026-10-30T09:58:50Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"c6a4629e32 (odb: use size_t for object_info.sizep and the size APIs,\n2026-06-15) widened the read side of the object database API, and the\nwrite_object() callback of odb_source already takes a size_t. But\nodb_write_object_ext(), odb_write_object() and odb_source_write_object()\nin front of it, as well as write_loose_object() behind it, still take\nan \"unsigned long\", which is only 32 bits wide on Windows.\n\nMost callers hand them a size_t from a strbuf or from odb_read_object(),\nwhich silently gets truncated. The most visible case is index_mem() and\nindex_stream_convert_blob(): when \"git add\" or \"git hash-object -w\"\nruns a file over 4GiB through a conversion (core.autocrlf, which is on\nby default in Git for Windows, an eol or ident attribute, or a clean\nfilter), we write a blob made of the first \"size mod 2^32\" bytes. The\nindex records it with the stat data of the real file, so \"git status\"\nstays clean and nobody notices until the file is checked out again.\n\"git hash-object\" without \"-w\" uses hash_object_file(), which takes a\nsize_t, so the two disagree on the object ID of the same file.\n\nUse size_t throughout, including for the size kept by the in-memory\nsource.\n\nThe existing \"hash filtered files over 4GB correctly\" test in t1007\nonly hashes, so add a variant that writes the object with \"git\nhash-object -w\" and checks both its object ID and its size. Like the\nother tests over 4GB, it needs EXPENSIVE and SIZE_T_IS_64BIT. Truncating\nthe length to 32 bits in odb_write_object_ext(), as happens on Windows\nwithout this patch, makes it fail with a different object ID.\n\nSigned-off-by: Elia Pinto \n---\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/source.h | 2 +-\n t/t1007-hash-object.sh | 13 +++++++++++++\n 6 files changed, 19 insertions(+), 6 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 2c93a9490c..1f173625b7 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -640,7 +640,7 @@ void odb_assert_oid_type(struct object_database *odb,\n }\n \n int odb_write_object_ext(struct object_database *odb,\n-\t\t\t const void *buf, unsigned long len,\n+\t\t\t const void *buf, size_t len,\n \t\t\t enum object_type type,\n \t\t\t struct object_id *oid,\n \t\t\t const struct object_id *compat_oid_in,\ndiff --git a/odb.h b/odb.h\nindex dd85b880a7..45182d28cb 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -621,14 +621,14 @@ enum odb_write_object_flags {\n * Returns 0 on success, a negative error code otherwise.\n */\n int odb_write_object_ext(struct object_database *odb,\n-\t\t\t const void *buf, unsigned long len,\n+\t\t\t const void *buf, size_t len,\n \t\t\t enum object_type type,\n \t\t\t struct object_id *oid,\n \t\t\t const struct object_id *compat_oid,\n \t\t\t enum odb_write_object_flags flags);\n \n static inline int odb_write_object(struct object_database *odb,\n-\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t const void *buf, size_t len,\n \t\t\t\t enum object_type type,\n \t\t\t\t struct object_id *oid)\n {\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 75cf29fa9d..c13cffc805 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -10,7 +10,7 @@\n struct inmemory_object {\n \tenum object_type type;\n \tconst void *buf;\n-\tunsigned long size;\n+\tsize_t size;\n };\n \n static const struct inmemory_object *find_cached_object(struct odb_source_inmemory *source,\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 50cb388c35..60801e70e8 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -758,7 +758,7 @@ static int end_loose_object_common(struct odb_source_loose *loose,\n \n static int write_loose_object(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid, char *hdr,\n-\t\t\t int hdrlen, const void *buf, unsigned long len,\n+\t\t\t int hdrlen, const void *buf, size_t len,\n \t\t\t const time_t *mtime, unsigned flags)\n {\n \tint fd, ret;\ndiff --git a/odb/source.h b/odb/source.h\nindex 8b839caa39..07d5019aa5 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -451,7 +451,7 @@ static inline int odb_source_freshen_object(struct odb_source *source,\n * object ID and the compatibility object ID, if non-NULL.\n */\n static inline int odb_source_write_object(struct odb_source *source,\n-\t\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t\t const void *buf, size_t len,\n \t\t\t\t\t enum object_type type,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t const struct object_id *compat_oid,\ndiff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh\nindex 463b38f990..0a89bbb455 100755\n--- a/t/t1007-hash-object.sh\n+++ b/t/t1007-hash-object.sh\n@@ -297,4 +297,17 @@ test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n \ttest_cmp expect actual\n '\n \n+test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n+\t\t'write filtered files over 4GB correctly' '\n+\t{ test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } &&\n+\ttest_oid large5GB >expect &&\n+\ttest_config filter.null-filter.clean \"cat\" &&\n+\techo \"big filter=null-filter\" >.gitattributes &&\n+\tgit hash-object -w -- big >actual &&\n+\ttest_cmp expect actual &&\n+\techo $((5*1024*1024*1024)) >expect &&\n+\tgit cat-file -s $(cat actual) >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.53.0\n\n\n"},{"id":"556873","messageId":"42d4eae98540028cbb1deba469d82d676642ca17.1793352943.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793352943.git.gitter.spiros@gmail.com","subject":"[PATCH v2 2/4] diff: use size_t for the allocation size of word-diff buffers","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T09:58:51Z","receivedAt":"2026-10-30T09:58:51Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Since 3c7a31204f (xdiff: use size_t for buffer sizes, 2026-09-30) the\nsize of an mmfile_t is a size_t, but the word-diff code still grows\nthese buffers with ALLOC_GROW() using an \"unsigned long\" (in struct\ndiff_words_buffer) or a \"long\" (in diff_words_fill()) for the allocated\nsize.\n\nOn platforms where \"long\" is 32 bits wide this is not just cosmetic.\ndiff_words_fill() writes every word followed by a newline, so its\noutput can be up to twice as large as its input, which is limited by\nMAX_XDIFF_SIZE to just under 1GiB. alloc_nr() computes (alloc + 16) * 3\nin the type of \"alloc\", which overflows a signed 32-bit \"long\" as soon\nas the buffer grows past about 700MiB. The result is undefined behavior,\nin practice a negative size that REALLOC_ARRAY() turns into a huge\nallocation request and a die().\n\nThe buffer in diff_words_append() is bounded by MAX_XDIFF_SIZE, so\nthere it is only a matter of consistency, but if that limit ever went\naway, ALLOC_GROW() would truncate the requested size, and the memcpy()\nright after it would overflow the buffer.\n\nUse size_t for both.\n\nSigned-off-by: Elia Pinto \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex b4ac17f8ef..9483931c05 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1959,7 +1959,7 @@ static void emit_rewrite_diff(const char *name_a,\n \n struct diff_words_buffer {\n \tmmfile_t text;\n-\tunsigned long alloc;\n+\tsize_t alloc;\n \tstruct diff_words_orig {\n \t\tconst char *begin, *end;\n \t} *orig;\n@@ -2192,7 +2192,7 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,\n \t\tregex_t *word_regex)\n {\n \tint i, j;\n-\tlong alloc = 0;\n+\tsize_t alloc = 0;\n \n \tout->size = 0;\n \tout->ptr = NULL;\n-- \n2.53.0\n\n\n"},{"id":"556874","messageId":"88ca2ceb801f93f39eabac3bf0b51f53fd04e4d0.1793352943.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793352943.git.gitter.spiros@gmail.com","subject":"[PATCH v2 3/4] convert: use size_t for buffer sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T09:58:52Z","receivedAt":"2026-10-30T09:58:52Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"The conversion code receives its buffers with a size_t length, but the\nhelpers that scan them, gather_stats(), gather_convert_stats(),\ngather_convert_stats_ascii() and count_ident(), take an \"unsigned long\".\nOn platforms where that is 32 bits wide, like Windows, they only look\nat the first \"size mod 2^32\" bytes of a file over 4GiB. The decision\nwhether a file is binary, or contains CRLF line endings or \"$Id$\"\nkeywords, is then based on that prefix, and we may convert the line\nendings of a file that is binary further down, or not expand keywords\nthat appear after the first 4GiB.\n\nread_blob_data_from_index(), which feeds has_crlf_in_index(),\nget_cached_convert_stats_ascii() and the reading of .gitattributes from\nthe index, gets the size of the blob as a size_t, but hands it out\nthrough cast_size_t_to_ulong(), so on such platforms \"git add\" of a file\nwhose staged version is over 4GiB dies when core.autocrlf is in effect.\n\nUse size_t for all of them.\n\nSigned-off-by: Elia Pinto \n---\n attr.c | 2 +-\n convert.c | 14 +++++++-------\n read-cache-ll.h | 2 +-\n read-cache.c | 4 ++--\n 4 files changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 0e63f1b6de..1acb4ef430 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -793,7 +793,7 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n {\n \tstruct attr_stack *stack = NULL;\n \tchar *buf;\n-\tunsigned long size;\n+\tsize_t size;\n \tint sparse_dir_pos = -1;\n \n \tif (!istate)\ndiff --git a/convert.c b/convert.c\nindex 036506842c..ca5594cab0 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -42,9 +42,9 @@ struct text_stat {\n \tunsigned printable, nonprintable;\n };\n \n-static void gather_stats(const char *buf, unsigned long size, struct text_stat *stats)\n+static void gather_stats(const char *buf, size_t size, struct text_stat *stats)\n {\n-\tunsigned long i;\n+\tsize_t i;\n \n \tmemset(stats, 0, sizeof(*stats));\n \n@@ -102,7 +102,7 @@ static int convert_is_binary(const struct text_stat *stats)\n \treturn 0;\n }\n \n-static unsigned int gather_convert_stats(const char *data, unsigned long size)\n+static unsigned int gather_convert_stats(const char *data, size_t size)\n {\n \tstruct text_stat stats;\n \tint ret = 0;\n@@ -119,7 +119,7 @@ static unsigned int gather_convert_stats(const char *data, unsigned long size)\n \treturn ret;\n }\n \n-static const char *gather_convert_stats_ascii(const char *data, unsigned long size)\n+static const char *gather_convert_stats_ascii(const char *data, size_t size)\n {\n \tunsigned int convert_stats = gather_convert_stats(data, size);\n \n@@ -141,7 +141,7 @@ const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path)\n {\n \tconst char *ret;\n-\tunsigned long sz;\n+\tsize_t sz;\n \tvoid *data = read_blob_data_from_index(istate, path, &sz);\n \tret = gather_convert_stats_ascii(data, sz);\n \tfree(data);\n@@ -223,7 +223,7 @@ static void check_global_conv_flags_eol(const char *path,\n \n static int has_crlf_in_index(struct index_state *istate, const char *path)\n {\n-\tunsigned long sz;\n+\tsize_t sz;\n \tvoid *data;\n \tconst char *crp;\n \tint has_crlf = 0;\n@@ -1077,7 +1077,7 @@ static int read_convert_config(const char *var, const char *value,\n \treturn 0;\n }\n \n-static int count_ident(const char *cp, unsigned long size)\n+static int count_ident(const char *cp, size_t size)\n {\n \t/*\n \t * \"$Id: 0000000000000000000000000000000000000000 $\" <=> \"$Id$\"\ndiff --git a/read-cache-ll.h b/read-cache-ll.h\nindex 066dd8bc3b..d279059b5f 100644\n--- a/read-cache-ll.h\n+++ b/read-cache-ll.h\n@@ -416,7 +416,7 @@ int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\n int index_name_is_other(struct index_state *, const char *, int);\n-void *read_blob_data_from_index(struct index_state *, const char *, unsigned long *);\n+void *read_blob_data_from_index(struct index_state *, const char *, size_t *);\n \n /* do stat comparison even if CE_VALID is true */\n #define CE_MATCH_IGNORE_VALID\t\t01\ndiff --git a/read-cache.c b/read-cache.c\nindex c4cf08a3a3..635dad1ba3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3472,7 +3472,7 @@ int index_name_is_other(struct index_state *istate, const char *name,\n }\n \n void *read_blob_data_from_index(struct index_state *istate,\n-\t\t\t\tconst char *path, unsigned long *size)\n+\t\t\t\tconst char *path, size_t *size)\n {\n \tint pos, len;\n \tsize_t sz;\n@@ -3503,7 +3503,7 @@ void *read_blob_data_from_index(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \tif (size)\n-\t\t*size = cast_size_t_to_ulong(sz);\n+\t\t*size = sz;\n \treturn data;\n }\n \n-- \n2.53.0\n\n\n"},{"id":"556875","messageId":"84e9e38a3e63a2b135847471a13ea36d37bd64d4.1793352943.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793352943.git.gitter.spiros@gmail.com","subject":"[PATCH v2 4/4] convert: use size_t for text statistics and ident counts","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T09:58:53Z","receivedAt":"2026-10-30T09:58:53Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"gather_stats() counts NULs, CRs, LFs, CRLFs and (non)printable\ncharacters in \"unsigned\" fields of struct text_stat, and\ncount_ident() counts \"$Id$\" markers in an \"int\". Now that the buffer\nsizes they work on are size_t, a large enough buffer overflows them.\nUnlike the previous patches, this is not specific to LLP64: \"unsigned\"\nis 32 bits wide on all the platforms we care about.\n\nThe counters only drive decisions, but a wrapped counter makes the\nwrong one. For example, a file with exactly 2^32 lone LFs gets\nlonelf == 0, so will_convert_lf_to_crlf() decides there is nothing\nto convert, and a wrapped \"printable\" or \"nonprintable\" can fool the\nheuristic in convert_is_binary(). In count_ident(), going past\nINT_MAX is undefined behaviour.\n\nThis is not a memory safety problem: the counts are only used as\nhints to strbuf_grow() in crlf_to_worktree() and ident_to_worktree(),\nand the buffers are then filled with strbuf_add(), which grows them\nas needed.\n\nUse size_t for all of them, and compute the size hint in\nident_to_worktree() with st_add() and st_mult(), so that it dies\ninstead of wrapping around.\n\nTriggering any of this needs more than 4GiB of input, so no test is\nadded.\n\nSuggested-by: Patrick Steinhardt \nSigned-off-by: Elia Pinto \n---\n convert.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex ca5594cab0..34fb4ce685 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -36,10 +36,10 @@\n \n struct text_stat {\n \t/* NUL, CR, LF and CRLF counts */\n-\tunsigned nul, lonecr, lonelf, crlf;\n+\tsize_t nul, lonecr, lonelf, crlf;\n \n \t/* These are just approximations! */\n-\tunsigned printable, nonprintable;\n+\tsize_t printable, nonprintable;\n };\n \n static void gather_stats(const char *buf, size_t size, struct text_stat *stats)\n@@ -1077,12 +1077,12 @@ static int read_convert_config(const char *var, const char *value,\n \treturn 0;\n }\n \n-static int count_ident(const char *cp, size_t size)\n+static size_t count_ident(const char *cp, size_t size)\n {\n \t/*\n \t * \"$Id: 0000000000000000000000000000000000000000 $\" <=> \"$Id$\"\n \t */\n-\tint cnt = 0;\n+\tsize_t cnt = 0;\n \tchar ch;\n \n \twhile (size) {\n@@ -1170,7 +1170,7 @@ static int ident_to_worktree(const char *src, size_t len,\n \tstruct object_id oid;\n \tchar *to_free = NULL;\n \tconst char *dollar, *spc;\n-\tint cnt;\n+\tsize_t cnt;\n \n \tif (!ident)\n \t\treturn 0;\n@@ -1184,7 +1184,7 @@ static int ident_to_worktree(const char *src, size_t len,\n \t\tto_free = strbuf_detach(buf, NULL);\n \thash_object_file(the_hash_algo, src, len, OBJ_BLOB, &oid);\n \n-\tstrbuf_grow(buf, len + cnt * (the_hash_algo->hexsz + 3));\n+\tstrbuf_grow(buf, st_add(len, st_mult(cnt, the_hash_algo->hexsz + 3)));\n \tfor (;;) {\n \t\t/* step 1: run to the next '$' */\n \t\tdollar = memchr(src, '$', len);\n-- \n2.53.0\n\n\n"},{"id":"556911","messageId":"auSDFRHpLy65PMF_@pks.im","threadId":"66617","inReplyTo":"8fcb9f10f13962acd5cdac652bea8fc4164f2e78.1793352943.git.gitter.spiros@gmail.com","subject":"Re: [PATCH v2 1/4] odb: use size_t for the length of objects to write","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-30T11:44:34Z","receivedAt":"2026-10-30T11:44:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 30, 2026 at 10:58:50AM +0100, Elia Pinto wrote:\n> diff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh\n> index 463b38f990..0a89bbb455 100755\n> --- a/t/t1007-hash-object.sh\n> +++ b/t/t1007-hash-object.sh\n> @@ -297,4 +297,17 @@ test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n> +\t\t'write filtered files over 4GB correctly' '\n> +\t{ test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } &&\n> +\ttest_oid large5GB >expect &&\n> +\ttest_config filter.null-filter.clean \"cat\" &&\n> +\techo \"big filter=null-filter\" >.gitattributes &&\n> +\tgit hash-object -w -- big >actual &&\n> +\ttest_cmp expect actual &&\n> +\techo $((5*1024*1024*1024)) >expect &&\n\nIsn't this the exact same as `big`? So...\n\n> +\tgit cat-file -s $(cat actual) >actual &&\n\nWe shouldn't read from and write to the same file. Maybe write it to\n\"actual.size\" instead?\n\n> +\ttest_cmp expect actual\n> +'\n\n... can't we `test_cmp big actual` here?\n\nAlso, it might make sense to use `test_cmp_bin ()` instead.\n\nPatrick\n\n"},{"id":"556912","messageId":"auSDKKHJONZjx2LX@pks.im","threadId":"66617","inReplyTo":"84e9e38a3e63a2b135847471a13ea36d37bd64d4.1793352943.git.gitter.spiros@gmail.com","subject":"Re: [PATCH v2 4/4] convert: use size_t for text statistics and ident counts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-30T11:44:40Z","receivedAt":"2026-10-30T11:44:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 30, 2026 at 10:58:53AM +0100, Elia Pinto wrote:\n> gather_stats() counts NULs, CRs, LFs, CRLFs and (non)printable\n> characters in \"unsigned\" fields of struct text_stat, and\n> count_ident() counts \"$Id$\" markers in an \"int\". Now that the buffer\n> sizes they work on are size_t, a large enough buffer overflows them.\n\nThis sounds a bit too deterministic: it doesn't necessarily overflow\nthem, but the possibility exists in theory.\n\n> Unlike the previous patches, this is not specific to LLP64: \"unsigned\"\n> is 32 bits wide on all the platforms we care about.\n> \n> The counters only drive decisions, but a wrapped counter makes the\n> wrong one. For example, a file with exactly 2^32 lone LFs gets\n> lonelf == 0, so will_convert_lf_to_crlf() decides there is nothing\n> to convert, and a wrapped \"printable\" or \"nonprintable\" can fool the\n> heuristic in convert_is_binary(). In count_ident(), going past\n> INT_MAX is undefined behaviour.\n> \n> This is not a memory safety problem: the counts are only used as\n> hints to strbuf_grow() in crlf_to_worktree() and ident_to_worktree(),\n> and the buffers are then filled with strbuf_add(), which grows them\n> as needed.\n\nThese paragraphs very much read like AI to me. I would appreciate a\nhuman touch, also to demonstrate that a human was involved between us\nand the AI that tries to understand what was done.\n\n> Use size_t for all of them, and compute the size hint in\n> ident_to_worktree() with st_add() and st_mult(), so that it dies\n> instead of wrapping around.\n\nThat's awfully specific.\n\nThe patch itself looks good to me, thanks.\n\nPatrick\n\n"},{"id":"557002","messageId":"20261031101619.172770-1-gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"auSDFRHpLy65PMF_@pks.im","subject":"Re: [PATCH v2 1/4] odb: use size_t for the length of objects to write","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:18Z","receivedAt":"2026-10-31T10:16:18Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"On Fri, Oct 30, 2026 at 12:44:34PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh\n> > index 463b38f990..0a89bbb455 100755\n> > --- a/t/t1007-hash-object.sh\n> > +++ b/t/t1007-hash-object.sh\n> > @@ -297,4 +297,17 @@ test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n> > \ttest_cmp expect actual\n> > '\n> > \n> > +test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n> > +\t\t'write filtered files over 4GB correctly' '\n> > +\t{ test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } &&\n> > +\ttest_oid large5GB >expect &&\n> > +\ttest_config filter.null-filter.clean \"cat\" &&\n> > +\techo \"big filter=null-filter\" >.gitattributes &&\n> > +\tgit hash-object -w -- big >actual &&\n> > +\ttest_cmp expect actual &&\n> > +\techo $((5*1024*1024*1024)) >expect &&\n>\n> Isn't this the exact same as `big`? So...\n\nAh, there was a slight misunderstanding here: `expect` didn't contain\nthe content of `big`, but its expected size in bytes (`5368709120`),\nwhich was then compared against the output of `cat-file -s`.\n\nThat said, your point stands: checking only the size was a weak and\nredundant check. The OID printed by `hash-object -w` is computed\ndirectly from the object content that was written. If the length had\nbeen truncated (like on Windows), the written object would be different\nand its OID wouldn't match `large5GB`. I verified this by simulating the\ntruncation: forcing the length to 32 bits in `odb_write_object_ext()`\ncauses the test to fail right there on the OID comparison.\n\nBecause of this, `test_cmp big actual` isn't needed, and running it\nwould unnecessarily cost another expensive read of 5 GiB from the object\ndatabase.\n\n>\n> > +\tgit cat-file -s $(cat actual) >actual &&\n>\n> We shouldn't read from and write to the same file. Maybe write it to\n> \"actual.size\" instead?\n\nAgreed, reading from and writing to the same file was fragile, even\nthough it worked because shell redirection happens after expansion. In\nv3 this goes away completely since we dropped the size line.\n\n>\n> Also, it might make sense to use `test_cmp_bin ()` instead.\n\nSince the size check is gone, we only compare the OID, which is text, so\n`test_cmp` remains the correct tool. `test_cmp_bin` would only make\nsense for comparing binary content, which we no longer do for the\nreasons mentioned above.\n\nInstead, v3 checks that the object actually exists in the database by\nrunning `git cat-file -e \"$(cat actual)\"`. This is a cheap check that\nconfirms `-w` wrote it successfully without needing to read back the\n5 GiB content.\n\nThanks for the review! v3 will follow shortly in the same thread.\n\nElia\n\n"},{"id":"557003","messageId":"20261031101619.172770-2-gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"auSDKKHJONZjx2LX@pks.im","subject":"Re: [PATCH v2 4/4] convert: use size_t for text statistics and ident counts","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:19Z","receivedAt":"2026-10-31T10:16:19Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"On Fri, Oct 30, 2026 at 12:44:40PM +0100, Patrick Steinhardt wrote:\n> > gather_stats() counts NULs, CRs, LFs, CRLFs and (non)printable\n> > characters in \"unsigned\" fields of struct text_stat, and\n> > count_ident() counts \"$Id$\" markers in an \"int\". Now that the buffer\n> > sizes they work on are size_t, a large enough buffer overflows them.\n>\n> This sounds a bit too deterministic: it doesn't necessarily overflow\n> them, but the possibility exists in theory.\n\nIndeed, it's a theoretical possibility rather than something that\nautomatically happens. In v3 I've adjusted it to say \"large inputs can\noverflow these counters\".\n\n>\n> > Unlike the previous patches, this is not specific to LLP64: \"unsigned\"\n> > [...]\n> > as needed.\n>\n> These paragraphs very much read like AI to me. I would appreciate a\n> human touch, also to demonstrate that a human was involved between us\n> and the AI that tries to understand what was done.\n\nYou're completely right. I used AI tooling to help draft that commit\nmessage, and it definitely shows. Fair cop.\n\nTo clarify my involvement: I review the code, check the cases, run the\ntests, and take full responsibility for what gets submitted. But the\nwording here missed the mark. For v3 I rewrote the message myself,\nkeeping it much shorter, focusing on a single example (lone LFs), and\ndropping the heavy-handed explanations.\n\n>\n> > Use size_t for all of them, and compute the size hint in\n> > ident_to_worktree() with st_add() and st_mult(), so that it dies\n> > instead of wrapping around.\n>\n> That's awfully specific.\n\nAgreed, it spells out the implementation details which are already\nobvious from the diff. In v3 I've simplified it to just state that the\nsize hint is checked for overflow.\n\n>\n> The patch itself looks good to me, thanks.\n\nThanks for the careful review! v3 will follow shortly in the same\nthread.\n\nElia\n\n"},{"id":"557004","messageId":"cover.1793439137.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793352943.git.gitter.spiros@gmail.com","subject":"[PATCH v3 0/4] use size_t for a few more object and buffer sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:31Z","receivedAt":"2026-10-31T10:16:31Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Thanks to Patrick Steinhardt for reviewing v2.\n\nChanges since v2:\n\n - Patch 1: drop the redundant file size check since the OID printed by\n `hash-object -w` already proves the content is complete, using\n `cat-file -e` just to verify the object exists. This also avoids\n reading and writing the same file unnecessarily.\n - Patch 2 & 3: unchanged.\n - Patch 4: shortened commit message following Patrick's comments.\n\nLink to v2:\n https://lore.kernel.org/git/cover.1793352943.git.gitter.spiros@gmail.com/\n\nElia Pinto (4):\n odb: use size_t for the length of objects to write\n diff: use size_t for the allocation size of word-diff buffers\n convert: use size_t for buffer sizes\n convert: use size_t for text statistics and ident counts\n\n attr.c | 2 +-\n convert.c | 24 ++++++++++++------------\n diff.c | 4 ++--\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/source.h | 2 +-\n read-cache-ll.h | 2 +-\n read-cache.c | 4 ++--\n t/t1007-hash-object.sh | 11 +++++++++++\n 11 files changed, 35 insertions(+), 24 deletions(-)\n\nRange-diff against v2:\n1: 8fcb9f10f1 ! 1: 4961a1824a odb: use size_t for the length of objects to write\n @@ Commit message\n \n The existing \"hash filtered files over 4GB correctly\" test in t1007\n only hashes, so add a variant that writes the object with \"git\n - hash-object -w\" and checks both its object ID and its size. Like the\n + hash-object -w\" and checks its object ID and that it exists. Like the\n other tests over 4GB, it needs EXPENSIVE and SIZE_T_IS_64BIT. Truncating\n the length to 32 bits in odb_write_object_ext(), as happens on Windows\n without this patch, makes it fail with a different object ID.\n @@ t/t1007-hash-object.sh: test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n +\techo \"big filter=null-filter\" >.gitattributes &&\n +\tgit hash-object -w -- big >actual &&\n +\ttest_cmp expect actual &&\n -+\techo $((5*1024*1024*1024)) >expect &&\n -+\tgit cat-file -s $(cat actual) >actual &&\n -+\ttest_cmp expect actual\n ++\tgit cat-file -e \"$(cat actual)\"\n +'\n +\n test_done\n2: 42d4eae985 = 2: b27ef2a888 diff: use size_t for the allocation size of word-diff buffers\n3: 88ca2ceb80 = 3: 1617177880 convert: use size_t for buffer sizes\n4: 84e9e38a3e ! 4: 153463b531 convert: use size_t for text statistics and ident counts\n @@ Metadata\n ## Commit message ##\n convert: use size_t for text statistics and ident counts\n \n - gather_stats() counts NULs, CRs, LFs, CRLFs and (non)printable\n - characters in \"unsigned\" fields of struct text_stat, and\n - count_ident() counts \"$Id$\" markers in an \"int\". Now that the buffer\n - sizes they work on are size_t, a large enough buffer overflows them.\n - Unlike the previous patches, this is not specific to LLP64: \"unsigned\"\n - is 32 bits wide on all the platforms we care about.\n + gather_stats() and count_ident() use \"unsigned\" and \"int\" counters to\n + count characters and markers in buffers whose sizes are measured in\n + size_t. On platforms where size_t can exceed 32 bits, large inputs can\n + overflow these counters.\n \n - The counters only drive decisions, but a wrapped counter makes the\n - wrong one. For example, a file with exactly 2^32 lone LFs gets\n - lonelf == 0, so will_convert_lf_to_crlf() decides there is nothing\n - to convert, and a wrapped \"printable\" or \"nonprintable\" can fool the\n - heuristic in convert_is_binary(). In count_ident(), going past\n - INT_MAX is undefined behaviour.\n + For example, a file with exactly 2^32 lone LFs wraps lonelf back\n + to zero, causing will_convert_lf_to_crlf() to miss the conversion.\n \n - This is not a memory safety problem: the counts are only used as\n - hints to strbuf_grow() in crlf_to_worktree() and ident_to_worktree(),\n - and the buffers are then filled with strbuf_add(), which grows them\n - as needed.\n + Switch the counters to size_t and check the size hint for overflow.\n \n - Use size_t for all of them, and compute the size hint in\n - ident_to_worktree() with st_add() and st_mult(), so that it dies\n - instead of wrapping around.\n -\n - Triggering any of this needs more than 4GiB of input, so no test is\n - added.\n + Triggering this requires processing over 4GiB of input, so no test\n + is added.\n \n Suggested-by: Patrick Steinhardt \n Signed-off-by: Elia Pinto \n\nbase-commit: b9e5cff1803ab24582201ef059516eb16014b9d9\n-- \n2.53.0\n\n\n"},{"id":"557005","messageId":"4961a1824ad75b6cfa81ab89af650cfc440cff81.1793439137.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793439137.git.gitter.spiros@gmail.com","subject":"[PATCH v3 1/4] odb: use size_t for the length of objects to write","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:32Z","receivedAt":"2026-10-31T10:16:32Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"c6a4629e32 (odb: use size_t for object_info.sizep and the size APIs,\n2026-06-15) widened the read side of the object database API, and the\nwrite_object() callback of odb_source already takes a size_t. But\nodb_write_object_ext(), odb_write_object() and odb_source_write_object()\nin front of it, as well as write_loose_object() behind it, still take\nan \"unsigned long\", which is only 32 bits wide on Windows.\n\nMost callers hand them a size_t from a strbuf or from odb_read_object(),\nwhich silently gets truncated. The most visible case is index_mem() and\nindex_stream_convert_blob(): when \"git add\" or \"git hash-object -w\"\nruns a file over 4GiB through a conversion (core.autocrlf, which is on\nby default in Git for Windows, an eol or ident attribute, or a clean\nfilter), we write a blob made of the first \"size mod 2^32\" bytes. The\nindex records it with the stat data of the real file, so \"git status\"\nstays clean and nobody notices until the file is checked out again.\n\"git hash-object\" without \"-w\" uses hash_object_file(), which takes a\nsize_t, so the two disagree on the object ID of the same file.\n\nUse size_t throughout, including for the size kept by the in-memory\nsource.\n\nThe existing \"hash filtered files over 4GB correctly\" test in t1007\nonly hashes, so add a variant that writes the object with \"git\nhash-object -w\" and checks its object ID and that it exists. Like the\nother tests over 4GB, it needs EXPENSIVE and SIZE_T_IS_64BIT. Truncating\nthe length to 32 bits in odb_write_object_ext(), as happens on Windows\nwithout this patch, makes it fail with a different object ID.\n\nSigned-off-by: Elia Pinto \n---\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/source.h | 2 +-\n t/t1007-hash-object.sh | 11 +++++++++++\n 6 files changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 2c93a9490c..1f173625b7 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -640,7 +640,7 @@ void odb_assert_oid_type(struct object_database *odb,\n }\n \n int odb_write_object_ext(struct object_database *odb,\n-\t\t\t const void *buf, unsigned long len,\n+\t\t\t const void *buf, size_t len,\n \t\t\t enum object_type type,\n \t\t\t struct object_id *oid,\n \t\t\t const struct object_id *compat_oid_in,\ndiff --git a/odb.h b/odb.h\nindex dd85b880a7..45182d28cb 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -621,14 +621,14 @@ enum odb_write_object_flags {\n * Returns 0 on success, a negative error code otherwise.\n */\n int odb_write_object_ext(struct object_database *odb,\n-\t\t\t const void *buf, unsigned long len,\n+\t\t\t const void *buf, size_t len,\n \t\t\t enum object_type type,\n \t\t\t struct object_id *oid,\n \t\t\t const struct object_id *compat_oid,\n \t\t\t enum odb_write_object_flags flags);\n \n static inline int odb_write_object(struct object_database *odb,\n-\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t const void *buf, size_t len,\n \t\t\t\t enum object_type type,\n \t\t\t\t struct object_id *oid)\n {\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 75cf29fa9d..c13cffc805 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -10,7 +10,7 @@\n struct inmemory_object {\n \tenum object_type type;\n \tconst void *buf;\n-\tunsigned long size;\n+\tsize_t size;\n };\n \n static const struct inmemory_object *find_cached_object(struct odb_source_inmemory *source,\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 50cb388c35..60801e70e8 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -758,7 +758,7 @@ static int end_loose_object_common(struct odb_source_loose *loose,\n \n static int write_loose_object(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid, char *hdr,\n-\t\t\t int hdrlen, const void *buf, unsigned long len,\n+\t\t\t int hdrlen, const void *buf, size_t len,\n \t\t\t const time_t *mtime, unsigned flags)\n {\n \tint fd, ret;\ndiff --git a/odb/source.h b/odb/source.h\nindex 8b839caa39..07d5019aa5 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -451,7 +451,7 @@ static inline int odb_source_freshen_object(struct odb_source *source,\n * object ID and the compatibility object ID, if non-NULL.\n */\n static inline int odb_source_write_object(struct odb_source *source,\n-\t\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t\t const void *buf, size_t len,\n \t\t\t\t\t enum object_type type,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t const struct object_id *compat_oid,\ndiff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh\nindex 463b38f990..e76557981d 100755\n--- a/t/t1007-hash-object.sh\n+++ b/t/t1007-hash-object.sh\n@@ -297,4 +297,15 @@ test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n \ttest_cmp expect actual\n '\n \n+test_expect_success EXPENSIVE,SIZE_T_IS_64BIT \\\n+\t\t'write filtered files over 4GB correctly' '\n+\t{ test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } &&\n+\ttest_oid large5GB >expect &&\n+\ttest_config filter.null-filter.clean \"cat\" &&\n+\techo \"big filter=null-filter\" >.gitattributes &&\n+\tgit hash-object -w -- big >actual &&\n+\ttest_cmp expect actual &&\n+\tgit cat-file -e \"$(cat actual)\"\n+'\n+\n test_done\n-- \n2.53.0\n\n\n"},{"id":"557006","messageId":"b27ef2a8882c2badb88d257ca28ed38b67c13c1b.1793439137.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793439137.git.gitter.spiros@gmail.com","subject":"[PATCH v3 2/4] diff: use size_t for the allocation size of word-diff buffers","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:33Z","receivedAt":"2026-10-31T10:16:33Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Since 3c7a31204f (xdiff: use size_t for buffer sizes, 2026-09-30) the\nsize of an mmfile_t is a size_t, but the word-diff code still grows\nthese buffers with ALLOC_GROW() using an \"unsigned long\" (in struct\ndiff_words_buffer) or a \"long\" (in diff_words_fill()) for the allocated\nsize.\n\nOn platforms where \"long\" is 32 bits wide this is not just cosmetic.\ndiff_words_fill() writes every word followed by a newline, so its\noutput can be up to twice as large as its input, which is limited by\nMAX_XDIFF_SIZE to just under 1GiB. alloc_nr() computes (alloc + 16) * 3\nin the type of \"alloc\", which overflows a signed 32-bit \"long\" as soon\nas the buffer grows past about 700MiB. The result is undefined behavior,\nin practice a negative size that REALLOC_ARRAY() turns into a huge\nallocation request and a die().\n\nThe buffer in diff_words_append() is bounded by MAX_XDIFF_SIZE, so\nthere it is only a matter of consistency, but if that limit ever went\naway, ALLOC_GROW() would truncate the requested size, and the memcpy()\nright after it would overflow the buffer.\n\nUse size_t for both.\n\nSigned-off-by: Elia Pinto \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex b4ac17f8ef..9483931c05 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1959,7 +1959,7 @@ static void emit_rewrite_diff(const char *name_a,\n \n struct diff_words_buffer {\n \tmmfile_t text;\n-\tunsigned long alloc;\n+\tsize_t alloc;\n \tstruct diff_words_orig {\n \t\tconst char *begin, *end;\n \t} *orig;\n@@ -2192,7 +2192,7 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,\n \t\tregex_t *word_regex)\n {\n \tint i, j;\n-\tlong alloc = 0;\n+\tsize_t alloc = 0;\n \n \tout->size = 0;\n \tout->ptr = NULL;\n-- \n2.53.0\n\n\n"},{"id":"557007","messageId":"1617177880b7cbde051bc4295f1fbd6ea2be457b.1793439137.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793439137.git.gitter.spiros@gmail.com","subject":"[PATCH v3 3/4] convert: use size_t for buffer sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:34Z","receivedAt":"2026-10-31T10:16:34Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"The conversion code receives its buffers with a size_t length, but the\nhelpers that scan them, gather_stats(), gather_convert_stats(),\ngather_convert_stats_ascii() and count_ident(), take an \"unsigned long\".\nOn platforms where that is 32 bits wide, like Windows, they only look\nat the first \"size mod 2^32\" bytes of a file over 4GiB. The decision\nwhether a file is binary, or contains CRLF line endings or \"$Id$\"\nkeywords, is then based on that prefix, and we may convert the line\nendings of a file that is binary further down, or not expand keywords\nthat appear after the first 4GiB.\n\nread_blob_data_from_index(), which feeds has_crlf_in_index(),\nget_cached_convert_stats_ascii() and the reading of .gitattributes from\nthe index, gets the size of the blob as a size_t, but hands it out\nthrough cast_size_t_to_ulong(), so on such platforms \"git add\" of a file\nwhose staged version is over 4GiB dies when core.autocrlf is in effect.\n\nUse size_t for all of them.\n\nSigned-off-by: Elia Pinto \n---\n attr.c | 2 +-\n convert.c | 14 +++++++-------\n read-cache-ll.h | 2 +-\n read-cache.c | 4 ++--\n 4 files changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 0e63f1b6de..1acb4ef430 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -793,7 +793,7 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n {\n \tstruct attr_stack *stack = NULL;\n \tchar *buf;\n-\tunsigned long size;\n+\tsize_t size;\n \tint sparse_dir_pos = -1;\n \n \tif (!istate)\ndiff --git a/convert.c b/convert.c\nindex 036506842c..ca5594cab0 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -42,9 +42,9 @@ struct text_stat {\n \tunsigned printable, nonprintable;\n };\n \n-static void gather_stats(const char *buf, unsigned long size, struct text_stat *stats)\n+static void gather_stats(const char *buf, size_t size, struct text_stat *stats)\n {\n-\tunsigned long i;\n+\tsize_t i;\n \n \tmemset(stats, 0, sizeof(*stats));\n \n@@ -102,7 +102,7 @@ static int convert_is_binary(const struct text_stat *stats)\n \treturn 0;\n }\n \n-static unsigned int gather_convert_stats(const char *data, unsigned long size)\n+static unsigned int gather_convert_stats(const char *data, size_t size)\n {\n \tstruct text_stat stats;\n \tint ret = 0;\n@@ -119,7 +119,7 @@ static unsigned int gather_convert_stats(const char *data, unsigned long size)\n \treturn ret;\n }\n \n-static const char *gather_convert_stats_ascii(const char *data, unsigned long size)\n+static const char *gather_convert_stats_ascii(const char *data, size_t size)\n {\n \tunsigned int convert_stats = gather_convert_stats(data, size);\n \n@@ -141,7 +141,7 @@ const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path)\n {\n \tconst char *ret;\n-\tunsigned long sz;\n+\tsize_t sz;\n \tvoid *data = read_blob_data_from_index(istate, path, &sz);\n \tret = gather_convert_stats_ascii(data, sz);\n \tfree(data);\n@@ -223,7 +223,7 @@ static void check_global_conv_flags_eol(const char *path,\n \n static int has_crlf_in_index(struct index_state *istate, const char *path)\n {\n-\tunsigned long sz;\n+\tsize_t sz;\n \tvoid *data;\n \tconst char *crp;\n \tint has_crlf = 0;\n@@ -1077,7 +1077,7 @@ static int read_convert_config(const char *var, const char *value,\n \treturn 0;\n }\n \n-static int count_ident(const char *cp, unsigned long size)\n+static int count_ident(const char *cp, size_t size)\n {\n \t/*\n \t * \"$Id: 0000000000000000000000000000000000000000 $\" <=> \"$Id$\"\ndiff --git a/read-cache-ll.h b/read-cache-ll.h\nindex 066dd8bc3b..d279059b5f 100644\n--- a/read-cache-ll.h\n+++ b/read-cache-ll.h\n@@ -416,7 +416,7 @@ int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\n int index_name_is_other(struct index_state *, const char *, int);\n-void *read_blob_data_from_index(struct index_state *, const char *, unsigned long *);\n+void *read_blob_data_from_index(struct index_state *, const char *, size_t *);\n \n /* do stat comparison even if CE_VALID is true */\n #define CE_MATCH_IGNORE_VALID\t\t01\ndiff --git a/read-cache.c b/read-cache.c\nindex c4cf08a3a3..635dad1ba3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3472,7 +3472,7 @@ int index_name_is_other(struct index_state *istate, const char *name,\n }\n \n void *read_blob_data_from_index(struct index_state *istate,\n-\t\t\t\tconst char *path, unsigned long *size)\n+\t\t\t\tconst char *path, size_t *size)\n {\n \tint pos, len;\n \tsize_t sz;\n@@ -3503,7 +3503,7 @@ void *read_blob_data_from_index(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \tif (size)\n-\t\t*size = cast_size_t_to_ulong(sz);\n+\t\t*size = sz;\n \treturn data;\n }\n \n-- \n2.53.0\n\n\n"},{"id":"557008","messageId":"153463b531b03e942247c0f6f9b3d89b8e55ac7c.1793439137.git.gitter.spiros@gmail.com","threadId":"66617","inReplyTo":"cover.1793439137.git.gitter.spiros@gmail.com","subject":"[PATCH v3 4/4] convert: use size_t for text statistics and ident counts","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-31T10:16:35Z","receivedAt":"2026-10-31T10:16:35Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"gather_stats() and count_ident() use \"unsigned\" and \"int\" counters to\ncount characters and markers in buffers whose sizes are measured in\nsize_t. On platforms where size_t can exceed 32 bits, large inputs can\noverflow these counters.\n\nFor example, a file with exactly 2^32 lone LFs wraps lonelf back\nto zero, causing will_convert_lf_to_crlf() to miss the conversion.\n\nSwitch the counters to size_t and check the size hint for overflow.\n\nTriggering this requires processing over 4GiB of input, so no test\nis added.\n\nSuggested-by: Patrick Steinhardt \nSigned-off-by: Elia Pinto \n---\n convert.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex ca5594cab0..34fb4ce685 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -36,10 +36,10 @@\n \n struct text_stat {\n \t/* NUL, CR, LF and CRLF counts */\n-\tunsigned nul, lonecr, lonelf, crlf;\n+\tsize_t nul, lonecr, lonelf, crlf;\n \n \t/* These are just approximations! */\n-\tunsigned printable, nonprintable;\n+\tsize_t printable, nonprintable;\n };\n \n static void gather_stats(const char *buf, size_t size, struct text_stat *stats)\n@@ -1077,12 +1077,12 @@ static int read_convert_config(const char *var, const char *value,\n \treturn 0;\n }\n \n-static int count_ident(const char *cp, size_t size)\n+static size_t count_ident(const char *cp, size_t size)\n {\n \t/*\n \t * \"$Id: 0000000000000000000000000000000000000000 $\" <=> \"$Id$\"\n \t */\n-\tint cnt = 0;\n+\tsize_t cnt = 0;\n \tchar ch;\n \n \twhile (size) {\n@@ -1170,7 +1170,7 @@ static int ident_to_worktree(const char *src, size_t len,\n \tstruct object_id oid;\n \tchar *to_free = NULL;\n \tconst char *dollar, *spc;\n-\tint cnt;\n+\tsize_t cnt;\n \n \tif (!ident)\n \t\treturn 0;\n@@ -1184,7 +1184,7 @@ static int ident_to_worktree(const char *src, size_t len,\n \t\tto_free = strbuf_detach(buf, NULL);\n \thash_object_file(the_hash_algo, src, len, OBJ_BLOB, &oid);\n \n-\tstrbuf_grow(buf, len + cnt * (the_hash_algo->hexsz + 3));\n+\tstrbuf_grow(buf, st_add(len, st_mult(cnt, the_hash_algo->hexsz + 3)));\n \tfor (;;) {\n \t\t/* step 1: run to the next '$' */\n \t\tdollar = memchr(src, '$', len);\n-- \n2.53.0\n\n\n"},{"id":"557147","messageId":"auhvUm4VwXyWNhsE@pks.im","threadId":"66617","inReplyTo":"20261031101619.172770-2-gitter.spiros@gmail.com","subject":"Re: [PATCH v2 4/4] convert: use size_t for text statistics and ident counts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-02T11:09:06Z","receivedAt":"2026-11-02T11:09:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Oct 31, 2026 at 11:16:19AM +0100, Elia Pinto wrote:\n> On Fri, Oct 30, 2026 at 12:44:40PM +0100, Patrick Steinhardt wrote:\n[snip]\n> > These paragraphs very much read like AI to me. I would appreciate a\n> > human touch, also to demonstrate that a human was involved between us\n> > and the AI that tries to understand what was done.\n> \n> You're completely right. I used AI tooling to help draft that commit\n> message, and it definitely shows. Fair cop.\n> \n> To clarify my involvement: I review the code, check the cases, run the\n> tests, and take full responsibility for what gets submitted. But the\n> wording here missed the mark. For v3 I rewrote the message myself,\n> keeping it much shorter, focusing on a single example (lone LFs), and\n> dropping the heavy-handed explanations.\n\nSorry, but is this reply to my ask to proof read what AI generates and\nmake it sound less AI generated repeating the _exact same thing_ that I\njust asked you to change?\n\nTo put it bluntly: behaviour like this is what will make me ignore your\npatch series altogether going forward. It is frustrating to me, and I\nonly have limited time to spend on reviews. So I will much rather spend\nmy own time on patches where I don't feel like I'm talking to a meat\nproxy, because I'm a (somewhat) social creature that prefers to talk to\nhumans.\n\nSo please reevaluate what you are doing here.\n\nPatrick\n\n"},{"id":"557232","messageId":"xmqqzevrgfb0.fsf@gitster.g","threadId":"66617","inReplyTo":"auhvUm4VwXyWNhsE@pks.im","subject":"Re: [PATCH v2 4/4] convert: use size_t for text statistics and ident counts","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-02T20:16:03Z","receivedAt":"2026-11-02T20:16:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> You're completely right. I used AI tooling to help draft that commit\n>> message, and it definitely shows. Fair cop.\n>> \n>> To clarify my involvement: I review the code, check the cases, run the\n>> tests, and take full responsibility for what gets submitted. But the\n>> wording here missed the mark. For v3 I rewrote the message myself,\n>> keeping it much shorter, focusing on a single example (lone LFs), and\n>> dropping the heavy-handed explanations.\n>\n> Sorry, but is this reply to my ask to proof read what AI generates and\n> make it sound less AI generated repeating the _exact same thing_ that I\n> just asked you to change?\n\n[*]\n\n> To put it bluntly: behaviour like this is what will make me ignore your\n> patch series altogether going forward. It is frustrating to me, and I\n> only have limited time to spend on reviews. So I will much rather spend\n> my own time on patches where I don't feel like I'm talking to a meat\n> proxy, because I'm a (somewhat) social creature that prefers to talk to\n> humans.\n>\n> So please reevaluate what you are doing here.\n\nSorry to interject but recognising the name as a rather long-time\ncontributor from the pre-AI era, I am not sure what answer from Elia\nwould have pleased you. Instead of saying that the initial work was\ndone by AI, and that the human involvement was to sanity-check the\nproduct of the machine and tweak the messaging to reduce the\nmachine-generated look, what do you want to see in contributors who\nwork with AI?\n\nMy question is not specifically about Elia's recent topics, but\nwould apply more widely.\n\nAlso, I had hard time to parse and understand [*] above. Did you\nmean \"I asked you to proofread what AI generates so it sounds less\nAI-generated, and yet this very e-mail reply of yours is doing the\nexact same thing I just asked you to change\", meaning \"the message\nI am replying still sounds AI generated\", or something?\n\nThinking aloud and letting my thoughts wander a bit.\n\nA contributor lets a machine generate a solution. Then he reads and\nunderstands every bit of it, writes a description of the solution as\na commit log message from scratch without looking at the machine\noutput, and from that message writes the code change from scratch.\nIt is more like using the machine output as mere inspiration for his\nown iteration. I would imagine you do not have an issue with that.\n\nNow, which part would we find problematic in the above sequence if\nthe contributor \"cheats\"? Perhaps he scans the machine output and,\nwithout understanding the most crucial bits, can still describe the\noutline of the solution from a superficial reading. He passes the\ncode along as-is, together with the description he wrote with only a\nsuperficial understanding of the problem area. When the patch gets\nsubmitted, it would not look any different from the work of the\nearlier contributor who used the machine output as inspiration to\nwrite his own solution. However, if we later ask deeper questions\nwhen problems arise from the change, the latter contributor may not\nbe able to provide as useful an answer as the former one who\nunderstood it deeply himself.\n\nBut that is something we cannot easily tell just by looking at the\nsubmission, so ...\n\n"},{"id":"557265","messageId":"aumGVZKfsCvdZ1T4@pks.im","threadId":"66617","inReplyTo":"xmqqzevrgfb0.fsf@gitster.g","subject":"Re: [PATCH v2 4/4] convert: use size_t for text statistics and ident counts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-03T06:59:33Z","receivedAt":"2026-11-03T06:59:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 02, 2026 at 12:16:03PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> You're completely right. I used AI tooling to help draft that commit\n> >> message, and it definitely shows. Fair cop.\n> >> \n> >> To clarify my involvement: I review the code, check the cases, run the\n> >> tests, and take full responsibility for what gets submitted. But the\n> >> wording here missed the mark. For v3 I rewrote the message myself,\n> >> keeping it much shorter, focusing on a single example (lone LFs), and\n> >> dropping the heavy-handed explanations.\n> >\n> > Sorry, but is this reply to my ask to proof read what AI generates and\n> > make it sound less AI generated repeating the _exact same thing_ that I\n> > just asked you to change?\n> \n> [*]\n> \n> > To put it bluntly: behaviour like this is what will make me ignore your\n> > patch series altogether going forward. It is frustrating to me, and I\n> > only have limited time to spend on reviews. So I will much rather spend\n> > my own time on patches where I don't feel like I'm talking to a meat\n> > proxy, because I'm a (somewhat) social creature that prefers to talk to\n> > humans.\n> >\n> > So please reevaluate what you are doing here.\n> \n> Sorry to interject but recognising the name as a rather long-time\n> contributor from the pre-AI era, I am not sure what answer from Elia\n> would have pleased you. Instead of saying that the initial work was\n> done by AI, and that the human involvement was to sanity-check the\n> product of the machine and tweak the messaging to reduce the\n> machine-generated look, what do you want to see in contributors who\n> work with AI?\n> \n> My question is not specifically about Elia's recent topics, but\n> would apply more widely.\n> \n> Also, I had hard time to parse and understand [*] above. Did you\n> mean \"I asked you to proofread what AI generates so it sounds less\n> AI-generated, and yet this very e-mail reply of yours is doing the\n> exact same thing I just asked you to change\", meaning \"the message\n> I am replying still sounds AI generated\", or something?\n\nYes, my issue is that it is still obviously AI generated.\n\n> Thinking aloud and letting my thoughts wander a bit.\n> \n> A contributor lets a machine generate a solution. Then he reads and\n> understands every bit of it, writes a description of the solution as\n> a commit log message from scratch without looking at the machine\n> output, and from that message writes the code change from scratch.\n> It is more like using the machine output as mere inspiration for his\n> own iteration. I would imagine you do not have an issue with that.\n\nYes, that's totally fine. I wouldn't even mind if they didn't rewrite\neverything from scratch. The important part to me is that I need to be\nable to recognize that a human is involved. But especially AI-generated\nmessages have a very uncanny feeling to them, and one is very often able\nto tell them apart from a message that is written by a human.\n\nFor code it's a different story: AI is very good at imitating the style\nthat we use in our code base, so my take is that you often wouldn't be\nable to tell apart code written by AI and code written by humans. So the\nonly useful signal that I have left to identify whether an author has\ninvested time thinking about their changes is the prose that encompasses\nthe code change.\n\nTo me, this means that commit messages are becoming more important these\ndays with the current capabilities that AI has. Commit messages are the\none place where the author can demonstrate that they understand the\nchange they have submitted. So when the author submits a message that\n\"looks AI generated, that sounds overly formal or bloated, that looks\nlike AI slop, that looks good on the surface but makes no sense\", then I\nwill push back because I get the sense that there was no human involved\nthat understands the changes.\n\nAnd that extends to messages that get sent on the mailing list, too.\n\n> Now, which part would we find problematic in the above sequence if\n> the contributor \"cheats\"? Perhaps he scans the machine output and,\n> without understanding the most crucial bits, can still describe the\n> outline of the solution from a superficial reading. He passes the\n> code along as-is, together with the description he wrote with only a\n> superficial understanding of the problem area. When the patch gets\n> submitted, it would not look any different from the work of the\n> earlier contributor who used the machine output as inspiration to\n> write his own solution.\n\nThe pragmatic stance here is that I find none of that problematic: the\nmoment the author can prove that they understand their submission by\nproperly describing and explaining it I'm okay-ish (well, assuming the\ncode itself is of course fine). As said above, for many submissions it's\nalmost impossible to tell apart code written by AI and humans because AI\nis able to imitate coding style quite well, and consequently I won't be\nable to tell them apart anyway.\n\nNow there's of course nuances here, and I'm still quite concerned about\ncopyright for AI-generated code. But especially for small contributions\nI hope that's not going to be a huge liability going forward.\n\n> However, if we later ask deeper questions when problems arise from the\n> change, the latter contributor may not be able to provide as useful an\n> answer as the former one who understood it deeply himself.\n> \n> But that is something we cannot easily tell just by looking at the\n> submission, so ...\n\nYes. The moment they cannot answer questions or where those answers or\nthe descriptions of the change themselves are very likely written by AI,\nthat's where I start to question the submission.\n\nPatrick\n\n"}]}