From: D. Ben Knoble Date: Mon, 08 Jun 2026 23:36:45 GMT Subject: git-diff in a worktree is an order of magnitude slower? Message-ID: Hello all, I'd like to report and offer to help fix what I view as a serious performance bug: "git diff --no-ext-diff --quiet" performs about ~10x slower in a secondary worktree than in the main worktree. Fortunately, this doesn't seem to extend to "--cached" (and "--no-ext-diff" and "--quiet" are probably both red-herrings, since it _does_ extend to plain "git diff"). Here's a short demo in Git: # git switch -d v2.54.0 # ninja -C build # where my meson-generated build dir is # git worktree add --detach ../perf-test v2.54.0 # hyperfine -N --warmup 10 './build/bin-wrappers/git diff' Benchmark 1: ./build/bin-wrappers/git diff Time (mean ± σ): 3.4 ms ± 0.5 ms [User: 4.2 ms, System: 3.9 ms] Range (min … max): 2.5 ms … 5.8 ms 677 runs # pushd ../perf-test # hyperfine -N --warmup 10 '../git/build/bin-wrappers/git diff' Benchmark 1: ../git/build/bin-wrappers/git diff Time (mean ± σ): 223.3 ms ± 10.5 ms [User: 210.4 ms, System: 19.1 ms] Range (min … max): 213.5 ms … 243.9 ms 13 runs I've had a similar experience at $DAYJOB, where a large repo takes ~6ms for the former and ~650ms for the latter. I noticed because the Bash prompt functions execute "git diff --no-ext-diff --quiet", and that was (AFAICT) the largest culprit for a slow shell prompt in a worktree. To squelch that from the prompt, I have to go down the rabbit hole of the worktree config extension, so I figured better to fix the slow diff if possible anyway. 2 questions: 1. Is this known, and if so is anybody working on it? 2. How can I help identify problem areas? A little more ------------- I've reproduced this as far back as v2.50.0, which is as far back as I could get the meson build to work with little effort (so I can't rule out that this is an old regression). Using "perf record -F 99 -g -- diff" in both trees and then "perf report": - it looks like the main worktree spends most of it's time in preload_thread, threaded_has_symlink_leading_path, lstat_cache… - the worktree spends a lot more time in ie_match_stat, ce_modified_check_fs, ce_compare_data, index_fd, would_convert_to_git_filter_fd… Here's the relevant "perf stat": main tree: Performance counter stats for './build/bin-wrappers/git diff': 0 context-switches:u # 0,0 cs/sec cs_per_second 0 cpu-migrations:u # 0,0 migrations/sec migrations_per_second 967 page-faults:u # 65036,4 faults/sec page_faults_per_second 14,87 msec task-clock:u # 0,3 CPUs CPUs_utilized 48 616 branch-misses:u # 3,2 % branch_miss_rate (57,19%) 3 571 630 branches:u # 240,2 M/sec branch_frequency 13 635 411 cpu-cycles:u # 0,9 GHz cycles_frequency 22 120 068 instructions:u # 1,9 instructions insn_per_cycle (85,61%) 3 634 065 stalled-cycles-frontend:u # 0,28 frontend_cycles_idle (9,56%) 0,006860098 seconds time elapsed 0,001364000 seconds user 0,015157000 seconds sys worktree: Performance counter stats for '../git/build/bin-wrappers/git diff': 0 context-switches:u # 0,0 cs/sec cs_per_second 0 cpu-migrations:u # 0,0 migrations/sec migrations_per_second 1 585 page-faults:u # 5058,0 faults/sec page_faults_per_second 313,37 msec task-clock:u # 0,9 CPUs CPUs_utilized 2 481 188 branch-misses:u # 1,5 % branch_miss_rate (48,94%) 168 664 155 branches:u # 538,2 M/sec branch_frequency (51,21%) 1 004 095 217 cpu-cycles:u # 3,2 GHz cycles_frequency (67,74%) 3 864 851 223 instructions:u # 3,9 instructions insn_per_cycle (52,73%) 70 755 234 stalled-cycles-frontend:u # 0,07 frontend_cycles_idle (49,29%) 0,306707634 seconds time elapsed 0,269027000 seconds user 0,045512000 seconds sys My observations: - the worktree has ~twice as many page faults and - executes ~150 times as many instructions (3.8b compared to 23m). (When I try to run some "perf" stats as root to access other counters, like syscalls, "git diff" in the worktree says "not a git repository", so I'm not counting the actual behavior. Ditto with DTrace.) PS I almost CC'd Peff and Patrick, whose names stood out in "git shortlog builtin/{worktree,diff}* object-file* | sort -t\( -k2 -g", but decided they'd be their own best judge of whether they can understand what's going on? :) -- D. Ben Knoble