From: D. Ben Knoble Date: Fri, 03 Jul 2026 15:57:21 GMT Subject: Re: git-diff in a worktree is an order of magnitude slower? Message-ID: In-Reply-To: <20260621174518.GB2206349@coredump.intra.peff.net> On Sun, Jun 21, 2026 at 1:45 PM Jeff King wrote: > > On Sun, Jun 21, 2026 at 01:24:32PM -0400, Jeff King wrote: > > > I think this is the core of the issue. These entries are "racy git > > dirty" in the sense that their mtimes are the same as the index mtime, > > and so we double-check the contents. This is the first bullet point > > under the "Racy Git" section of Documentation/technical/racy-git.adoc. > > > > But diffcore_skip_stat_unmatch() doesn't count them as dirty, so we > > don't increment the counter, and thus top-level git-diff won't write out > > the new index. And thus every subsequent diff repeats the same > > expensive double-check. > > > > But I'm not sure where the blame lies. Either: > > > > 1. diffcore_skip_stat_unmatch() should be counting these in its > > "dirty" counter; or > > BTW, I don't think diffcore actually has the information it would need > to do so. The racy stuff is handled under the hood in ie_match_stat(), > which returns only a set of "changed" flags. So the caller cannot tell > the difference between the two cases: > > 1. We checked ce_match_stat_basic() which said "no change", and then > is_racy_timestamp() was false, so that was good enough. > > 2. is_racy_timestamp() is true, so we further did a content check, > found nothing, and returned the same "no change" > > Obviously we could pass back another flag, but that would disrupt the > other callers. Hmm. It looks like we could pass in a flag to say "assume > racy entries are modified". And then they come back to the diff code, > diffcore_skip_stat_unmatch() sees they're not real diffs and suppresses > them, but we _do_ count them as stat-dirty. > > Like this: > > diff --git a/builtin/diff.c b/builtin/diff.c > index 4b46e394ce..4d36b5c1e0 100644 > --- a/builtin/diff.c > +++ b/builtin/diff.c > @@ -271,6 +271,9 @@ static void builtin_diff_files(struct rev_info *revs, int argc, const char **arg > argv++; argc--; > } > > + if (revs->diffopt.skip_stat_unmatch) > + options |= DIFF_RACY_IS_MODIFIED; > + > /* > * "diff --base" should not combine merges because it was not > * asked to. "diff -c" should not densify (if the user wants > > That seems to work, in the sense that "git diff" does refresh the index > afterwards. But the timings are a bit funny. > > In my working tree of linux.git with many racy entries it was ~500ms to > do the first diff (and the second, and so on, because we never updated > the index). After the patch above, it is 1800ms to do the first diff, > and then fast (~30ms) after. > > I could believe it takes twice as long when we refresh the index > (because I don't think we use the stat-cleanliness we collected from the > diff, but rather just do a from-scratch index refresh). But that would > imply it should take ~1000ms. Where does the extra 800ms go? I guess > that somehow the content-check done by diffcore_skip_stat_unmatch() is > slower than the one done by ie_match_stat(). I think the individual > functions are respectively diff_filespec_check_stat_unmatch() and > ce_modified_check_fs(). > > I don't know if any of this is really worth digging too far. This feels > like a case we could do a bit better at, but I wonder how much it > matters in practice. As soon as you do any index-refresh (including "git > status"), the racy entries are cleared and everything is faster. It > just seems kind of lame that we write out the initial working tree with > so many racy entries. > > -Peff I'd like to dig into this some more, personally, but I'm not sure when I'll have the time (and we're deep in the guts of 2 systems whose implementation are quite foreign to me---the index and the diff machinery). The main reason is that I noticed this all when trying to figure out why my shell prompt was slow :) I'm willing to pay a slow first prompt for all subsequent prompts to be faster without having to remember (and alert others) "oh, this is racy git, just run 'git status' to fix it" or something. Obviously it's even better if that first racy diff + index update is not so slow, though. I think I saw 2 potential areas to dig? 1. The time spent on that refresh index diff mentioned above 2. Limiting racy entries on initial write. The latter was, I think, dismissed down-thread if I understood? It's not so nice to stall for a full second just to avoid raciness, and USE_NSEC alleviates the problem, too. (If that became available to more folks, see USE_NSEC meson thread, then I suppose I would be less likely to dig into (1) even though it sounds like an interesting puzzle.) So maybe instead of "dismissed" I mean "we decided to keep the USE_NSEC gate." The former I saw some interesting discussion about how to communicate bits to code, but no hint as to whether that changed your initial measurements. I suppose I could try for myself, but it will take me some time to process Junio's suggestions there into actual code. -- D. Ben Knoble