git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: git-diff in a worktree is an order of magnitude slower?

From
D. Ben Knoble <ben.knoble@gmail.com>
Date
Jul 3, 2026, 15:57 UTC
Message-ID
<CALnO6CBC+SK=ycHn4xgzoZAud5ZWpqSp1NSe5maKPhp9+f=LgQ@mail.gmail.com>
In-Reply-To
<20260621174518.GB2206349@coredump.intra.peff.net>
On Sun, Jun 21, 2026 at 1:45 PM Jeff King <peff@peff.net> wrote:
Show 77 quoted lines
>
> On Sun, Jun 21, 2026 at 01:24:32PM -0400, Jeff King wrote:
>
> > I think this is the core of the issue. These entries are "racy git
> > dirty" in the sense that their mtimes are the same as the index mtime,
> > and so we double-check the contents. This is the first bullet point
> > under the "Racy Git" section of Documentation/technical/racy-git.adoc.
> >
> > But diffcore_skip_stat_unmatch() doesn't count them as dirty, so we
> > don't increment the counter, and thus top-level git-diff won't write out
> > the new index. And thus every subsequent diff repeats the same
> > expensive double-check.
> >
> > But I'm not sure where the blame lies. Either:
> >
> >   1. diffcore_skip_stat_unmatch() should be counting these in its
> >      "dirty" counter; or
>
> BTW, I don't think diffcore actually has the information it would need
> to do so. The racy stuff is handled under the hood in ie_match_stat(),
> which returns only a set of "changed" flags. So the caller cannot tell
> the difference between the two cases:
>
>   1. We checked ce_match_stat_basic() which said "no change", and then
>      is_racy_timestamp() was false, so that was good enough.
>
>   2. is_racy_timestamp() is true, so we further did a content check,
>      found nothing, and returned the same "no change"
>
> Obviously we could pass back another flag, but that would disrupt the
> other callers. Hmm. It looks like we could pass in a flag to say "assume
> racy entries are modified". And then they come back to the diff code,
> diffcore_skip_stat_unmatch() sees they're not real diffs and suppresses
> them, but we _do_ count them as stat-dirty.
>
> Like this:
>
> diff --git a/builtin/diff.c b/builtin/diff.c
> index 4b46e394ce..4d36b5c1e0 100644
> --- a/builtin/diff.c
> +++ b/builtin/diff.c
> @@ -271,6 +271,9 @@ static void builtin_diff_files(struct rev_info *revs, int argc, const char **arg
>                 argv++; argc--;
>         }
>
> +       if (revs->diffopt.skip_stat_unmatch)
> +               options |= DIFF_RACY_IS_MODIFIED;
> +
>         /*
>          * "diff --base" should not combine merges because it was not
>          * asked to.  "diff -c" should not densify (if the user wants
>
> That seems to work, in the sense that "git diff" does refresh the index
> afterwards. But the timings are a bit funny.
>
> In my working tree of linux.git with many racy entries it was ~500ms to
> do the first diff (and the second, and so on, because we never updated
> the index). After the patch above, it is 1800ms to do the first diff,
> and then fast (~30ms) after.
>
> I could believe it takes twice as long when we refresh the index
> (because I don't think we use the stat-cleanliness we collected from the
> diff, but rather just do a from-scratch index refresh). But that would
> imply it should take ~1000ms. Where does the extra 800ms go? I guess
> that somehow the content-check done by diffcore_skip_stat_unmatch() is
> slower than the one done by ie_match_stat(). I think the individual
> functions are respectively diff_filespec_check_stat_unmatch() and
> ce_modified_check_fs().
>
> I don't know if any of this is really worth digging too far. This feels
> like a case we could do a bit better at, but I wonder how much it
> matters in practice. As soon as you do any index-refresh (including "git
> status"), the racy entries are cleared and everything is faster. It
> just seems kind of lame that we write out the initial working tree with
> so many racy entries.
>
> -Peff

I'd like to dig into this some more, personally, but I'm not sure when I'll have the time (and we're deep in the guts of 2 systems whose implementation are quite foreign to me---the index and the diff machinery). The main reason is that I noticed this all when trying to figure out why my shell prompt was slow :) I'm willing to pay a slow first prompt for all subsequent prompts to be faster without having to remember (and alert others) "oh, this is racy git, just run 'git status' to fix it" or something. Obviously it's even better if that first racy diff + index update is not so slow, though.

I think I saw 2 potential areas to dig?
1. The time spent on that refresh index diff mentioned above
2. Limiting racy entries on initial write.

The latter was, I think, dismissed down-thread if I understood? It's not so nice to stall for a full second just to avoid raciness, and USE_NSEC alleviates the problem, too. (If that became available to more folks, see USE_NSEC meson thread, then I suppose I would be less likely to dig into (1) even though it sounds like an interesting puzzle.) So maybe instead of "dismissed" I mean "we decided to keep the USE_NSEC gate."

The former I saw some interesting discussion about how to communicate bits to code, but no hint as to whether that changed your initial measurements. I suppose I could try for myself, but it will take me some time to process Junio's suggestions there into actual code.

-- 
D. Ben Knoble
Previous: Jeff King
Message 19 of 19 in “git-diff in a worktree is an order of magnitude slower?”
  1. D. Ben KnobleJun 8, 2026
  2. Jeff KingJun 9, 2026
  3. D. Ben KnobleJun 9, 2026
  4. Jeff KingJun 11, 2026
  5. Junio C HamanoJun 11, 2026
  6. brian m. carlsonJun 11, 2026
  7. D. Ben KnobleJun 20, 2026
  8. Junio C HamanoJun 21, 2026
  9. Junio C HamanoJun 21, 2026
  10. Jeff KingJun 21, 2026
  11. Jeff KingJun 21, 2026
  12. Junio C HamanoJun 21, 2026
  13. Jeff KingJun 21, 2026
  14. Junio C HamanoJun 21, 2026
  15. Junio C HamanoJun 22, 2026
  16. Jeff KingJun 28, 2026
  17. Junio C HamanoJun 21, 2026
  18. Jeff KingJun 21, 2026
  19. D. Ben KnobleJul 3, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.