{"thread":{"id":"66734","subject":"[PATCH 0/2] grep, diff: do not silently truncate sizes of 4GiB or more","startedAt":"2026-11-01T08:44:55Z","lastAt":"2026-11-19T06:53:07Z","messageCount":4,"participants":["Elia Pinto"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"557068","messageId":"cover.1793521916.git.gitter.spiros@gmail.com","threadId":"66734","inReplyTo":null,"subject":"[PATCH 0/2] grep, diff: do not silently truncate sizes of 4GiB or more","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T08:44:55Z","receivedAt":"2026-11-01T08:44:55Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"This series continues the search for truncated sizes on platforms where\n\"long\" is 32 bits but size_t is 64 bits (64-bit Windows). This time it\ncovers grep and diff, where a file of 4GiB or more produced wrong\nresults without any error.\n\n- 1/2 grep: on the working tree and --no-index, it silently missed\n  matches beyond the truncated size. Now it uses size_t and works,\n  for blobs too.\n- 2/2 diff: on the working tree, textconv, --no-index and combine-diff,\n  it produced a diff of a prefix, and a patch that truncates the file\n  when applied (as git add -p does). Now it dies with a clear error,\n  like the blob side already did.\n\nOf the remaining -Wshorten-64-to-32 warnings in the diff and grep code,\nthese are the only reachable ones. The others are in xdiff internals\n(inputs limited to MAX_XDIFF_SIZE, 1023MiB), in buffer_is_binary()\n(which looks at 8000 bytes), or in places where the size can't exceed\n4GiB. Widening diff_filespec.size to size_t, so that diff actually works\nabove 4GiB, is a much bigger change, left for later.\n\nNo tests, as they would need 4GiB of input on such a platform. Both\neffects were verified by simulating the truncation at 16 bits, and the\ncommit messages give the details.\n\nIt doesn't depend on the other series in flight and merges cleanly with\nthem. Based on master 50fe5cc63f.\n\nElia Pinto (2):\n  grep: use size_t for the size of buffers to search\n  diff: die instead of truncating sizes of 4GiB or more\n\n combine-diff.c  | 11 +++++++----\n diff-no-index.c |  2 +-\n diff.c          |  8 ++++----\n grep.c          | 16 ++++++++--------\n grep.h          |  4 ++--\n 5 files changed, 22 insertions(+), 19 deletions(-)\n\n\nbase-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7\n-- \n2.53.0\n\n\n"},{"id":"557069","messageId":"bdf2f51187093544959ba7028b86187b56b2dcaa.1793521916.git.gitter.spiros@gmail.com","threadId":"66734","inReplyTo":"cover.1793521916.git.gitter.spiros@gmail.com","subject":"[PATCH 1/2] grep: use size_t for the size of buffers to search","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T08:44:56Z","receivedAt":"2026-11-01T08:44:56Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"struct grep_source tracks the size of the search buffer in an \"unsigned\nlong\", while grep_source_load_file() reads worktree files into memory\nusing size_t. On platforms where \"long\" is 32 bits but size_t is 64 bits\n(such as 64-bit Windows), processing a file of 4GiB or more truncates\nthe size, causing \"git grep\" to search only the first \"size mod 2^32\"\nbytes and silently miss matches beyond that threshold. With gs->size\ntruncated to 16 bits, \"git grep\" and \"git grep --no-index\" find nothing\nin a 100KiB file whose last line matches, while grepping the same file\nas a blob finds it.\n\nUse size_t for struct grep_source.size and across the internal matching\nengine functions in grep.c and grep.h. Additionally, remove the\ncast_size_t_to_ulong() call in grep_source_load_oid(), allowing grepping\nblobs of 4GiB or more to succeed instead of aborting.\n\nTriggering this requires processing 4GiB or more of input on a platform\nwhere \"long\" is 32 bits but size_t is 64 bits, so no test is added.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n grep.c | 16 ++++++++--------\n grep.h |  4 ++--\n 2 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1d75d31421..a8c01b4737 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -864,9 +864,9 @@ void free_grep_patterns(struct grep_opt *opt)\n \t\tfree_pattern_expr(opt->pattern_expression);\n }\n \n-static const char *end_of_line(const char *cp, unsigned long *left)\n+static const char *end_of_line(const char *cp, size_t *left)\n {\n-\tunsigned long l = *left;\n+\tsize_t l = *left;\n \twhile (l && *cp != '\\n') {\n \t\tl--;\n \t\tcp++;\n@@ -1454,7 +1454,7 @@ static int should_lookahead(struct grep_opt *opt)\n }\n \n static int look_ahead(struct grep_opt *opt,\n-\t\t      unsigned long *left_p,\n+\t\t      size_t *left_p,\n \t\t      unsigned *lno_p,\n \t\t      const char **bol_p)\n {\n@@ -1567,7 +1567,7 @@ static int grep_source_1(struct grep_opt *opt, struct grep_source *gs, int colle\n {\n \tconst char *bol;\n \tconst char *peek_bol = NULL;\n-\tunsigned long left;\n+\tsize_t left;\n \tunsigned lno = 1;\n \tunsigned last_hit = 0;\n \tint binary_match_only = 0;\n@@ -1735,7 +1735,7 @@ static int grep_source_1(struct grep_opt *opt, struct grep_source *gs, int colle\n \t\t\tgoto next_line;\n \t\t}\n \t\tif (show_function && (!peek_bol || peek_bol < bol)) {\n-\t\t\tunsigned long peek_left = left;\n+\t\t\tsize_t peek_left = left;\n \t\t\tconst char *peek_eol = eol;\n \n \t\t\t/*\n@@ -1854,7 +1854,7 @@ int grep_source(struct grep_opt *opt, struct grep_source *gs)\n \n static void grep_source_init_buf(struct grep_source *gs,\n \t\t\t\t const char *buf,\n-\t\t\t\t unsigned long size)\n+\t\t\t\t size_t size)\n {\n \tgs->type = GREP_SOURCE_BUF;\n \tgs->name = NULL;\n@@ -1865,7 +1865,7 @@ static void grep_source_init_buf(struct grep_source *gs,\n \tgs->identifier = NULL;\n }\n \n-int grep_buffer(struct grep_opt *opt, const char *buf, unsigned long size)\n+int grep_buffer(struct grep_opt *opt, const char *buf, size_t size)\n {\n \tstruct grep_source gs;\n \tint r;\n@@ -1935,7 +1935,7 @@ static int grep_source_load_oid(struct grep_source *gs)\n \n \tgs->buf = odb_read_object(gs->repo->objects, gs->identifier,\n \t\t\t\t  &type, &size_st);\n-\tgs->size = cast_size_t_to_ulong(size_st);\n+\tgs->size = size_st;\n \tif (!gs->buf)\n \t\treturn error(_(\"'%s': unable to read %s\"),\n \t\t\t     gs->name,\ndiff --git a/grep.h b/grep.h\nindex 13e26a9318..0bd705dfc0 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -212,7 +212,7 @@ void append_grep_pattern(struct grep_opt *opt, const char *pat, const char *orig\n void append_header_grep_pattern(struct grep_opt *, enum grep_header_field, const char *);\n void compile_grep_patterns(struct grep_opt *opt);\n void free_grep_patterns(struct grep_opt *opt);\n-int grep_buffer(struct grep_opt *opt, const char *buf, unsigned long size);\n+int grep_buffer(struct grep_opt *opt, const char *buf, size_t size);\n \n /* The field parameter is only used to filter header patterns\n  * (where appropriate). If filtering isn't desirable\n@@ -235,7 +235,7 @@ struct grep_source {\n \tstruct repository *repo; /* if GREP_SOURCE_OID */\n \n \tconst char *buf;\n-\tunsigned long size;\n+\tsize_t size;\n \n \tchar *path; /* for attribute lookups */\n \tstruct userdiff_driver *driver;\n-- \n2.53.0\n\n\n"},{"id":"557070","messageId":"93415b30f2ffdafbd8ce0e615b1ef1aa8bb0ad7d.1793521916.git.gitter.spiros@gmail.com","threadId":"66734","inReplyTo":"cover.1793521916.git.gitter.spiros@gmail.com","subject":"[PATCH 2/2] diff: die instead of truncating sizes of 4GiB or more","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-01T08:44:57Z","receivedAt":"2026-11-01T08:44:57Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"On platforms where \"long\" is 32 bits but size_t is 64 bits, such as\n64-bit Windows, worktree files, textconv outputs, convert_to_git()\nresults, and temporary diff files of 4GiB or larger have their sizes\nsilently truncated when they are stored in an \"unsigned long\", such as\nthe size field of struct diff_filespec.\n\nThis causes \"git diff\" to produce a silent incorrect diff based on a\nprefix, and commands like \"git add -p\" to apply a patch that truncates\nthe file. With the worktree size truncated to 16 bits, \"git diff --stat\"\nreports 1 insertion and 5042 deletions for a one-line change in a\n7901-line file, and applying the resulting patch leaves 2859 lines.\n\nUse cast_size_t_to_ulong() in these code paths, matching the blob\nside, so that Git dies with a clear error instead of producing wrong\nresults or corrupted patches. Widening diff_filespec.size is a much\nlarger change left for a separate patch.\n\nTriggering this requires processing 4GiB or more of input on a platform\nwhere \"long\" is 32 bits but size_t is 64 bits, so no test is added.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n combine-diff.c  | 11 +++++++----\n diff-no-index.c |  2 +-\n diff.c          |  8 ++++----\n 3 files changed, 12 insertions(+), 9 deletions(-)\n\ndiff --git a/combine-diff.c b/combine-diff.c\nindex 717d537688..edf0427a80 100644\n--- a/combine-diff.c\n+++ b/combine-diff.c\n@@ -322,7 +322,8 @@ static char *grab_blob(struct repository *r,\n \t} else if (textconv) {\n \t\tstruct diff_filespec *df = alloc_filespec(path);\n \t\tfill_filespec(df, oid, 1, mode);\n-\t\t*size = fill_textconv(r, textconv, df, &blob);\n+\t\t*size = cast_size_t_to_ulong(fill_textconv(r, textconv,\n+\t\t\t\t\t\t\t   df, &blob));\n \t\tfree_filespec(df);\n \t} else {\n \t\tsize_t size_st = 0;\n@@ -1069,7 +1070,9 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t} else if (textconv) {\n \t\t\tstruct diff_filespec *df = alloc_filespec(elem->path);\n \t\t\tfill_filespec(df, null_oid(the_hash_algo), 0, st.st_mode);\n-\t\t\tresult_size = fill_textconv(opt->repo, textconv, df, &result);\n+\t\t\tresult_size = cast_size_t_to_ulong(\n+\t\t\t\tfill_textconv(opt->repo, textconv,\n+\t\t\t\t\t      df, &result));\n \t\t\tfree_filespec(df);\n \t\t} else if (0 <= (fd = open(elem->path, O_RDONLY))) {\n \t\t\tsize_t len = xsize_t(st.st_size);\n@@ -1086,7 +1089,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\tif (!is_file)\n \t\t\t\telem->mode = canon_mode(S_IFLNK);\n \n-\t\t\tresult_size = len;\n+\t\t\tresult_size = cast_size_t_to_ulong(len);\n \t\t\tresult = xmallocz(len);\n \n \t\t\tdone = read_in_full(fd, result, len);\n@@ -1103,7 +1106,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\t\t\t\t   elem->path, result, len, &buf, global_conv_flags_eol)) {\n \t\t\t\t\tfree(result);\n \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n-\t\t\t\t\tresult_size = len;\n+\t\t\t\t\tresult_size = cast_size_t_to_ulong(len);\n \t\t\t\t}\n \t\t\t}\n \t\t}\ndiff --git a/diff-no-index.c b/diff-no-index.c\nindex a9151cc27a..dee80eb4ed 100644\n--- a/diff-no-index.c\n+++ b/diff-no-index.c\n@@ -116,7 +116,7 @@ static void populate_common(struct diff_filespec *s, struct strbuf *buf)\n \n \ts->should_munmap = 0;\n \ts->data = strbuf_detach(buf, &size);\n-\ts->size = size;\n+\ts->size = cast_size_t_to_ulong(size);\n \ts->should_free = 1;\n \ts->is_stdin = 1;\n }\ndiff --git a/diff.c b/diff.c\nindex b4ac17f8ef..9b1c69b378 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4516,7 +4516,7 @@ int diff_populate_filespec(struct repository *r,\n \t\t\ts->size = 0;\n \t\t\treturn err;\n \t\t}\n-\t\ts->size = xsize_t(st.st_size);\n+\t\ts->size = cast_size_t_to_ulong(xsize_t(st.st_size));\n \t\tif (!s->size)\n \t\t\tgoto empty;\n \t\tif (S_ISLNK(st.st_mode)) {\n@@ -4569,7 +4569,7 @@ int diff_populate_filespec(struct repository *r,\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\n \t\t\ts->data = strbuf_detach(&buf, &size);\n-\t\t\ts->size = size;\n+\t\t\ts->size = cast_size_t_to_ulong(size);\n \t\t\ts->should_free = 1;\n \t\t}\n \t}\n@@ -4662,7 +4662,7 @@ static void prep_temp_blob(struct index_state *istate,\n \tif (convert_to_working_tree(istate, path,\n \t\t\t(const char *)blob, (size_t)size, &buf, &meta)) {\n \t\tblob = buf.buf;\n-\t\tsize = buf.len;\n+\t\tsize = cast_size_t_to_ulong(buf.len);\n \t}\n \tif (write_in_full(temp->tempfile->fd, blob, size) < 0 ||\n \t    close_tempfile_gently(temp->tempfile))\n@@ -7838,7 +7838,7 @@ int textconv_object(struct repository *r,\n \t\treturn 0;\n \t}\n \n-\t*buf_size = fill_textconv(r, textconv, df, buf);\n+\t*buf_size = cast_size_t_to_ulong(fill_textconv(r, textconv, df, buf));\n \tfree_filespec(df);\n \treturn 1;\n }\n-- \n2.53.0\n\n\n"},{"id":"558803","messageId":"20261119065307.826988-1-gitter.spiros@gmail.com","threadId":"66734","inReplyTo":"cover.1793521916.git.gitter.spiros@gmail.com","subject":"Re: [PATCH 0/2] grep, diff: do not silently truncate sizes of 4GiB or more","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-19T06:53:07Z","receivedAt":"2026-11-19T06:53:07Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Hi,\n\nA gentle ping on this two-patch series, which has had no comments so\nfar. It still applies cleanly to current master (e1ec09b23c) and\nmerges cleanly with 'seen'.\n\nThe question I would most like an opinion on is the approach of 2/2.\nWhere \"long\" has 32 bits while size_t has 64, it makes \"git diff\" die\non a worktree file, textconv output and the like of 4GiB or more, as\nit already does on the blob side, instead of silently diffing a\nprefix. Making diff work for such files means widening\ndiff_filespec.size, which touches much more code. Is dying acceptable\nas a first step, or would you rather have the widening instead?\n\nThanks,\nElia\n\n"}]}