# [PATCH 0/2] status: agree with diff and add when conversion is active 4 messages from 2026-10-08 to 2026-10-09. Participants: Curtis Allen Smith, Junio C Hamano. Thread: https://gitlist.dev/t/66492 ## Curtis Allen Smith, 2026-10-08 20:45 Subject: [PATCH 0/2] status: agree with diff and add when conversion is active Message-ID: <20261008204603.1988-1-curtis.allen.smith@gmail.com> URL: https://gitlist.dev/e/20261008204603.1988-1-curtis.allen.smith%40gmail.com ``` "git status", "git diff" and "git add" can disagree about whether a file has been modified. Under * text eol=lf a tool that rewrites an otherwise unchanged file with CRLF endings makes "git status" report it as modified, while "git diff" shows nothing and "git add" stages nothing: the two commands that actually run the clean filter both conclude that the contents did not change. That contradiction, rather than the line endings as such, is what this series is about. The cause is the size comparison in ie_match_stat(). ie_modified() takes a difference between the file's size and the size recorded in the index as proof of a content change and returns without reading the file. That was sound in 2005, when the working tree file and the blob were the same bytes. Conversion made it unsound -- the whole point of a clean filter is that the two representations differ in their bytes and agree on their content -- and the shortcut was never revisited. The mtime branch of the very same function already reads the file and applies the conversion before deciding, so Git pays for the conversion-aware check in one branch and refuses to in the other. Patch 1 makes the size branch behave like the mtime branch whenever the path is subject to conversion. Paths with no conversion take the existing early return untouched. It also stops ce_compare_data() hashing a file whose converted length already differs from the size of its blob, since equal contents must have equal length; that is where most of the cost of the new check would otherwise go, and it helps the pre-existing mtime path as well. When the end-of-line conversion is the only one that applies, that length comes from a scan for CR, and the file is not converted either. Patch 2 adds core.convertAwareStatus for people who would rather keep the old shortcut, either everywhere (false) or only for paths with an expensive clean filter such as Git LFS (no-filter). I defaulted it to on, including filters: the measurements below say reading is not what costs, and the 2005 performance argument should not be re-applied in 2026 without evidence. Being ordinary configuration it also works per command, as "git -c core.convertAwareStatus=no-filter status". Note that there is no way to opt out of the size comparison today -- core.checkStat=minimal drops ctime, uid/gid and inode but still compares the size -- which is why patch 2 adds a variable instead of extending an existing one. Numbers ------- Linux (WSL2, ext4) on an i9-14900K, warm page cache, fastest of 5 runs, "status -uno" to separate the refresh from untracked scanning. One binary for both columns with core.convertAwareStatus flipped; "false" is the pre-series code path. 10000 files x 2.6 KB, "* text=auto eol=lf" false true clean tree 4 ms 4 ms 10000 genuinely modified 19 ms 49 ms 10000 CRLF-rewritten, 1st run 20 ms 148 ms the same, steady state 20 ms 6 ms 200 files x 1 MB, same attributes false true clean tree 2 ms 2 ms 200 genuinely modified 2 ms 26 ms 200 CRLF-rewritten, 1st run 2 ms 611 ms the same, steady state 2 ms 3 ms 10000 files x 2.6 KB, no conversion configured clean tree 4 ms 4 ms 10000 genuinely modified 19 ms 20 ms A clean tree and a repository without conversion are unaffected. What is paid for is stat-dirty converted paths. A file that was really edited is read and scanned for CR, but neither converted nor hashed, because its length already rules out a match. Without that the "genuinely modified" rows read 142 ms and 517 ms rather than 49 ms and 26 ms. Of the 214 MB in the second corpus, reading costs 6 ms from page cache, the conversion about 180 ms, and SHA-1 about 280 ms. The last row of the first block is the case the series exists for: the patched build settles at 6 ms where the unpatched one pays 20 ms on every invocation and still reports the files as modified, because it never refreshes their recorded sizes. The 1st-run rows are the one-time cost of discovering that. Their lengths match, so they are converted and hashed in full. This was reported against Git for Windows [1], where Torsten suggested bringing it to the list. It is not Windows-specific; anything with a clean filter runs into it, and Git LFS users on Linux see the same contradiction. Built with gcc 15.2 on top of 6de20f6; each commit builds and passes on its own. t0020 (with the new tests), t0021, t0026, t0027, t1300, t2106, t2200, t3700, t7508 and t0008 pass. [1] https://github.com/git-for-windows/git/issues/6410 Curtis Allen Smith (2): read-cache: do not trust a size change when conversion is active core: add core.convertAwareStatus to opt out of the content check Documentation/config/core.adoc | 22 +++++ environment.c | 14 ++++ environment.h | 7 ++ read-cache.c | 141 ++++++++++++++++++++++++++++++++- t/t0020-crlf.sh | 92 +++++++++++++++++++++ 5 files changed, 273 insertions(+), 3 deletions(-) -- 2.53.0 ``` ## Curtis Allen Smith, 2026-10-08 20:45 Subject: [PATCH 1/2] read-cache: do not trust a size change when conversion is active Message-ID: <20261008204603.1988-2-curtis.allen.smith@gmail.com> URL: https://gitlist.dev/e/20261008204603.1988-2-curtis.allen.smith%40gmail.com In-Reply-To: <20261008204603.1988-1-curtis.allen.smith@gmail.com> ``` "git status" can report a file as modified while "git diff" and "git add", which both run the clean filter, agree its contents are unchanged: git init t && cd t printf '* text eol=lf\n' >.gitattributes printf 'one\ntwo\nthree\n' >file.txt git add . && git commit -m init printf 'one\r\ntwo\r\nthree\r\n' >file.txt git status --short # -> M file.txt git diff # -> empty git add file.txt # -> stages nothing Three commands that answer the same question answer it differently, and the two that consult the conversion are the ones that get it right. ie_modified() declares a path modified as soon as ie_match_stat() reports DATA_CHANGED, which it does whenever the size of the file in the working tree differs from the size recorded in the index, and it returns without ever reading the file. That shortcut is sound only while the working tree file and the blob are the same bytes. When it was written in 2005 they were, and a size mismatch really was a proof of a content change. Conversion removed that premise: core.autocrlf arrived in 2007 and the "text" and "eol" attributes in 2010, and making the two representations differ in their bytes while agreeing on their content is precisely what they are for. The shortcut was never re-examined against the feature layered on top of it. The same function already handles the comparable case correctly. When only the mtime changed, it falls through to ce_modified_check_fs(), reads the file, applies the conversion, and answers "unchanged" when that is the truth. Git is therefore already willing to pay for a conversion-aware check here; the size branch is the only place where a difference in the bytes on disk is taken to be a difference in content. So fall through to that same check when the size changed and the path is subject to conversion. Paths without conversion take the early return exactly as before, and a repository that uses no conversion is unaffected. Hashing is the expensive part of that check -- Git's collision detecting SHA-1 runs at about 800 MB/s on the machine used below -- and it is avoidable most of the time. Contents that are equal necessarily have equal length, so ce_compare_data() now compares the length of the converted file against the size of the blob, which costs an object header lookup, and hashes only when the two agree. A file that was really edited almost always changes length and is rejected without being hashed. The file this commit is about has exactly the length of its blob, so it is hashed, found equal, and the index then records its new size, after which it is not read again. When the end-of-line conversion is the only one that applies, even the conversion can be skipped. "git add" either keeps such a file as it is or turns each CRLF into LF, so the converted length is one of two numbers, and a scan for CR gives both. If neither is the size of the blob, the file is modified, and it is neither converted nor hashed. In a repository of 200 files of 1 MB each under "* text=auto" with every file modified, these take "git status" from 517ms to 26ms; with 10000 files of 2.6 KB, from 142ms to 49ms. The inconsistency is a chronic annoyance for anyone sharing a tree between Windows and Unix with normalized line endings. Any tool that rewrites unchanged files in native line endings -- javadoc, code generators, formatters, a good number of editors -- changes the size of every file it touches and flags the whole output tree as modified with empty diffs. The documented remedy, "git add --renormalize", does not stick: the next checkout, stash or branch switch rewrites the checkout-form bytes and the recorded sizes along with them, and the next run of the tool flags everything again. Signed-off-by: Curtis Allen Smith --- read-cache.c | 129 ++++++++++++++++++++++++++++++++++++++++++++++-- t/t0020-crlf.sh | 45 +++++++++++++++++ 2 files changed, 171 insertions(+), 3 deletions(-) diff --git a/read-cache.c b/read-cache.c index c4cf08a3a..8875706d8 100644 --- a/read-cache.c +++ b/read-cache.c @@ -9,6 +9,7 @@ #include "git-compat-util.h" #include "config.h" +#include "convert.h" #include "date.h" #include "diff.h" #include "diffcore.h" @@ -228,6 +229,99 @@ int fake_lstat(const struct cache_entry *ce, struct stat *st) return 0; } +/* + * Count the CRs in "buf" that are immediately followed by an LF. + */ +static size_t count_crlf(const char *buf, size_t len) +{ + const char *end = buf + len; + size_t n = 0; + + while ((buf = memchr(buf, '\r', end - buf))) { + if (++buf < end && *buf == '\n') + n++; + } + return n; +} + +/* + * Compare an open file to the blob recorded for it, converting the file + * the way "git add" would. Contents that are equal necessarily have + * equal length, so when the converted length differs from the size of + * the blob the file is modified and there is no need to hash it, and + * hashing is by far the most expensive part of this comparison. + * + * Only the case this can help is handled here: a regular file whose + * conversion Git performs itself. A path driven by an external filter + * is left to index_fd(), which streams it into the filter. + * + * Returns 1 if the file differs, 0 if it matches, -1 if it could not be + * read. Does not close "fd". + */ +static int ce_compare_converted_data(struct index_state *istate, + const struct cache_entry *ce, + struct stat *st, int fd) +{ + struct strbuf raw = STRBUF_INIT; + struct strbuf converted = STRBUF_INIT; + struct object_info oi = OBJECT_INFO_INIT; + struct object_id oid; + struct conv_attrs ca; + enum object_type type; + size_t blob_size; + const char *buf; + size_t len; + int have_size, match = -1; + + oi.typep = &type; + oi.sizep = &blob_size; + have_size = (odb_read_object_info_extended(istate->repo->objects, + &ce->oid, &oi, + OBJECT_INFO_SKIP_FETCH_OBJECT | + OBJECT_INFO_QUICK) == ODB_READ_OK && + type == OBJ_BLOB); + + if (strbuf_read(&raw, fd, st->st_size) < 0) + goto out; + + /* + * When the end-of-line conversion is the only one, "git add" + * either keeps the file as it is or turns every CRLF into LF + * ("text=auto" refuses to convert a file with a lone CR, so + * stripping all CRs comes to the same thing). The converted + * length is therefore one of two values, and if neither is the + * size of the blob the file is modified without converting it. + */ + convert_attrs(istate, &ca, ce->name); + if (have_size && !ca.drv && !ca.ident && + !ca.working_tree_encoding && + raw.len != blob_size && + raw.len - count_crlf(raw.buf, raw.len) != blob_size) { + match = 1; + goto out; + } + + buf = raw.buf; + len = raw.len; + if (convert_to_git(istate, ce->name, raw.buf, raw.len, &converted, 0)) { + buf = converted.buf; + len = converted.len; + } + + if (have_size && len != blob_size) { + match = 1; + goto out; + } + + hash_object_file(istate->repo->hash_algo, buf, len, OBJ_BLOB, &oid); + match = !oideq(&oid, &ce->oid); + +out: + strbuf_release(&raw); + strbuf_release(&converted); + return match; +} + static int ce_compare_data(struct index_state *istate, const struct cache_entry *ce, struct stat *st) @@ -237,9 +331,16 @@ static int ce_compare_data(struct index_state *istate, if (fd >= 0) { struct object_id oid; - if (!index_fd(istate, &oid, fd, st, OBJ_BLOB, ce->name, 0)) + + if (S_ISREG(st->st_mode) && + would_convert_to_git(istate, ce->name) && + !would_convert_to_git_filter_fd(istate, ce->name)) { + match = ce_compare_converted_data(istate, ce, st, fd); + close(fd); + } else if (!index_fd(istate, &oid, fd, st, OBJ_BLOB, ce->name, 0)) { match = !oideq(&oid, &ce->oid); - /* index_fd() closed the file descriptor already */ + /* index_fd() closed the file descriptor already */ + } } return match; } @@ -438,6 +539,27 @@ int ie_match_stat(struct index_state *istate, return changed; } +/* + * A difference between the size of the file in the working tree and the + * size recorded for it in the index proves that the contents changed + * only as long as the two are byte-for-byte comparable. That stops + * being true as soon as the path is run through a clean filter: + * rewriting a file with CRLF endings under "text eol=lf", for example, + * changes its size in the working tree without changing the blob Git + * would record for it. For such a path the only way to tell is to read + * the contents and convert them, which is what we already do when only + * the mtime changed. + */ +static int size_change_is_conclusive(struct index_state *istate, + const struct cache_entry *ce, + struct stat *st) +{ + if (!S_ISREG(st->st_mode)) + return 1; + + return !would_convert_to_git(istate, ce->name); +} + int ie_modified(struct index_state *istate, const struct cache_entry *ce, struct stat *st, unsigned int options) @@ -480,7 +602,8 @@ int ie_modified(struct index_state *istate, */ (!S_ISLNK(st->st_mode) || ce->ce_stat_data.sd_size != MAX_PATH) && #endif - (S_ISGITLINK(ce->ce_mode) || ce->ce_stat_data.sd_size != 0)) + (S_ISGITLINK(ce->ce_mode) || ce->ce_stat_data.sd_size != 0) && + size_change_is_conclusive(istate, ce, st)) return changed; changed_fs = ce_modified_check_fs(istate, ce, st); diff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh index fd1cae09e..88b728d50 100755 --- a/t/t0020-crlf.sh +++ b/t/t0020-crlf.sh @@ -397,4 +397,49 @@ test_expect_success 'New CRLF file gets LF in repo' ' test_cmp alllf alllf2 ' +test_expect_success 'status does not report a CRLF-only rewrite as modified' ' + git init eol-status && + ( + cd eol-status && + echo "* text eol=lf" >.gitattributes && + printf "one\ntwo\nthree\n" >file.txt && + git add .gitattributes file.txt && + git commit -m initial && + + # a generator rewrites the file with CRLF, same content + printf "one\r\ntwo\r\nthree\r\n" >file.txt && + git status --porcelain -uno >actual && + test_must_be_empty actual && + git diff --exit-code && + + # a real change is still reported + printf "one\r\ntwo\r\nfour\r\n" >file.txt && + git status --porcelain -uno >actual && + echo " M file.txt" >expect && + test_cmp expect actual + ) +' + +test_expect_success 'status sizes a text file by its CRLF pairs, not its CRs' ' + git init eol-status-lone-cr && + ( + cd eol-status-lone-cr && + echo "* text eol=lf" >.gitattributes && + printf "one\rtwo\nthree\n" >file.txt && + git add .gitattributes file.txt && + git commit -m initial && + + # "git add" keeps the lone CR and drops the others + printf "one\rtwo\r\nthree\r\n" >file.txt && + git status --porcelain -uno >actual && + test_must_be_empty actual && + + # the converted length matches the blob, the content does not + printf "one\rtwo\nthrEE\r\n" >file.txt && + git status --porcelain -uno >actual && + echo " M file.txt" >expect && + test_cmp expect actual + ) +' + test_done -- 2.53.0 ``` ## Curtis Allen Smith, 2026-10-08 20:45 Subject: [PATCH 2/2] core: add core.convertAwareStatus to opt out of the content check Message-ID: <20261008204603.1988-3-curtis.allen.smith@gmail.com> URL: https://gitlist.dev/e/20261008204603.1988-3-curtis.allen.smith%40gmail.com In-Reply-To: <20261008204603.1988-1-curtis.allen.smith@gmail.com> ``` The previous commit makes an index refresh read and convert a path whose size changed when conversion is active for it, so that "git status" agrees with "git diff" and "git add". Reading costs more than trusting the size, and when the path has a clean filter configured the cost includes running that filter -- Git LFS on a large file, say. Reading is cheap enough on current hardware that agreeing with "git diff" is the better default, but nobody should be stuck with it if their filters are expensive. Add core.convertAwareStatus: true (default) consult the conversion for any path that has one, including paths with a clean filter no-filter consult only the conversions Git performs itself, and decide a path with a clean filter on its size false always treat a size change as a modification, as Git did before Being ordinary configuration, it can equally be given for a single command: git -c core.convertAwareStatus=no-filter status Paths that are not subject to conversion are decided on their size alone in every mode, so this costs nothing in a repository that does not use conversion. Signed-off-by: Curtis Allen Smith --- Documentation/config/core.adoc | 22 ++++++++++++++++ environment.c | 14 ++++++++++ environment.h | 7 +++++ read-cache.c | 12 +++++++++ t/t0020-crlf.sh | 47 ++++++++++++++++++++++++++++++++++ 5 files changed, 102 insertions(+) diff --git a/Documentation/config/core.adoc b/Documentation/config/core.adoc index 0b697f53f..9737c804f 100644 --- a/Documentation/config/core.adoc +++ b/Documentation/config/core.adoc @@ -156,6 +156,28 @@ some fields (e.g. JGit); by excluding these fields from the comparison, the `minimal` mode may help interoperability when the same repository is used by these other systems at the same time. +core.convertAwareStatus:: + When a path is subject to content conversion -- the `text` and + `eol` attributes, `core.autocrlf`, a `working-tree-encoding`, + or a clean filter -- the size of the file in the working tree + is not determined by its contents alone, so a change in size + does not prove that the contents changed. When this variable + is missing or set to `true`, Git reads and converts such a + file before reporting it as modified, which keeps 'git status' + in agreement with 'git diff' and 'git add'. When set to + `no-filter`, Git does this only for the conversions it + performs itself, and a path with a clean filter configured + (Git LFS, for example) is reported as modified on a size + change without running the filter. When set to `false`, a + size change is always taken as a modification, which is what + Git did before this variable existed. ++ +Reading the file costs more than trusting its size, so `no-filter` +and `false` trade this consistency for speed in repositories where +running the filter, or reading the file at all, is too expensive. +Paths that are not subject to conversion are decided on the size +alone in every mode. + core.quotePath:: Commands that output paths (e.g. 'ls-files', 'diff'), will quote "unusual" characters in the pathname by enclosing the diff --git a/environment.c b/environment.c index c83cf4483..b079262c8 100644 --- a/environment.c +++ b/environment.c @@ -344,6 +344,19 @@ int git_default_core_config(const char *var, const char *value, var, value); } + if (!strcmp(var, "core.convertawarestatus")) { + int b = git_parse_maybe_bool(value); + if (0 <= b) + cfg->convert_aware_status = b ? CONVERT_AWARE_STATUS_ALL + : CONVERT_AWARE_STATUS_NEVER; + else if (value && !strcasecmp(value, "no-filter")) + cfg->convert_aware_status = CONVERT_AWARE_STATUS_IN_PROCESS; + else + return error(_("invalid value for '%s': '%s'"), + var, value); + return 0; + } + if (!strcmp(var, "core.quotepath")) { quote_path_fully = git_config_bool(var, value); return 0; @@ -766,6 +779,7 @@ void repo_config_values_init(struct repo_config_values *cfg) cfg->apply_sparse_checkout = 0; cfg->trust_ctime = 1; cfg->check_stat = 1; + cfg->convert_aware_status = CONVERT_AWARE_STATUS_ALL; cfg->zlib_compression_level = Z_BEST_SPEED; cfg->pack_compression_level = Z_DEFAULT_COMPRESSION; cfg->precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */ diff --git a/environment.h b/environment.h index b336459e9..49a88de27 100644 --- a/environment.h +++ b/environment.h @@ -115,6 +115,12 @@ enum object_creation_mode { OBJECT_CREATION_USES_RENAMES = 1 }; +enum convert_aware_status { + CONVERT_AWARE_STATUS_NEVER = 0, + CONVERT_AWARE_STATUS_IN_PROCESS, + CONVERT_AWARE_STATUS_ALL +}; + struct repo_config_values { /* section "core" config values */ char *attributes_file; @@ -130,6 +136,7 @@ struct repo_config_values { int apply_sparse_checkout; int trust_ctime; int check_stat; + enum convert_aware_status convert_aware_status; int zlib_compression_level; int pack_compression_level; int precomposed_unicode; diff --git a/read-cache.c b/read-cache.c index 8875706d8..2bb27a388 100644 --- a/read-cache.c +++ b/read-cache.c @@ -554,9 +554,21 @@ static int size_change_is_conclusive(struct index_state *istate, const struct cache_entry *ce, struct stat *st) { + struct repo_config_values *cfg = repo_config_values(the_repository); + struct conv_attrs ca; + + if (cfg->convert_aware_status == CONVERT_AWARE_STATUS_NEVER) + return 1; + if (!S_ISREG(st->st_mode)) return 1; + if (cfg->convert_aware_status == CONVERT_AWARE_STATUS_IN_PROCESS) { + convert_attrs(istate, &ca, ce->name); + if (ca.drv) + return 1; + } + return !would_convert_to_git(istate, ce->name); } diff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh index 88b728d50..4c127207e 100755 --- a/t/t0020-crlf.sh +++ b/t/t0020-crlf.sh @@ -442,4 +442,51 @@ test_expect_success 'status sizes a text file by its CRLF pairs, not its CRs' ' ) ' +test_expect_success 'core.convertAwareStatus=false restores the size shortcut' ' + git init convert-aware && + ( + cd convert-aware && + echo "* text eol=lf" >.gitattributes && + printf "one\ntwo\nthree\n" >file.txt && + git add .gitattributes file.txt && + git commit -m initial && + printf "one\r\ntwo\r\nthree\r\n" >file.txt && + + git -c core.convertAwareStatus=false status --porcelain -uno >actual && + echo " M file.txt" >expect && + test_cmp expect actual && + + git -c core.convertAwareStatus=true status --porcelain -uno >actual && + test_must_be_empty actual + ) +' + +test_expect_success 'core.convertAwareStatus=no-filter leaves clean filters alone' ' + git init convert-aware-filter && + ( + cd convert-aware-filter && + write_script stripcr <<-\EOF && + tr -d "\015" + EOF + echo "file.txt filter=stripcr" >.gitattributes && + git config filter.stripcr.clean ./stripcr && + printf "one\ntwo\nthree\n" >file.txt && + git add .gitattributes file.txt && + git commit -m initial && + printf "one\r\ntwo\r\nthree\r\n" >file.txt && + + git -c core.convertAwareStatus=no-filter status --porcelain -uno >actual && + echo " M file.txt" >expect && + test_cmp expect actual && + + git status --porcelain -uno >actual && + test_must_be_empty actual + ) +' + +test_expect_success 'core.convertAwareStatus rejects an unknown value' ' + test_must_fail git -c core.convertAwareStatus=bogus status 2>err && + test_grep "invalid value" err +' + test_done -- 2.53.0 ``` ## Junio C Hamano, 2026-10-09 05:37 Subject: Re: [PATCH 1/2] read-cache: do not trust a size change when conversion is active Message-ID: URL: https://gitlist.dev/e/xmqqfqyfwi20.fsf%40gitster.g In-Reply-To: <20261008204603.1988-2-curtis.allen.smith@gmail.com> ``` Curtis Allen Smith writes: > "git status" can report a file as modified while "git diff" and > ... > next run of the tool flags everything again. > > Signed-off-by: Curtis Allen Smith > --- That's overly verbose. > read-cache.c | 129 ++++++++++++++++++++++++++++++++++++++++++++++-- > t/t0020-crlf.sh | 45 +++++++++++++++++ > 2 files changed, 171 insertions(+), 3 deletions(-) And it is curious why we need so much new code, especially after reading an explaination in the proposed log message that makes it sound as if "we let ce_modified_check_fs() to compare converted result already when timestamps differ, and it is just the matter of doing the same when sizes are the same" is what is happening in the patch. Why do we need to add a new function that compares converted data? A new function is not automatically a bad thing. If there is already an existing code path that does the same thing, a new function may be a good way to replace that code path with a more generic code and apply essentially the same logic implemented by that new more generic code to a new code path. But in such a refactoring patch, we usually see a comparable number of removed lines, which is not what we see in the diffstat above. ```