git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Performance regression in connectivity check during receive-pack (git 2.54)

From
Jeff King <peff@peff.net>
Date
Jul 21, 2026, 03:57 UTC
Message-ID
<20260721035733.GA581473@coredump.intra.peff.net>
In-Reply-To
<CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>
On Tue, Jul 21, 2026 at 03:17:23PM +1200, Wolfgang Kritzinger wrote:
Show 7 quoted lines
> `strace` shows that after a push, 2.54 does a failing open() of
> of numerous loose objects -- once in the quarantine (incoming)
> directory and once in the main object store -- before finding it
> in a pack:
> 
> openat(".../objects/tmp_objdir-incoming-XXXX/ed/58..", O_RDONLY) = ENOENT
> openat(".../objects/ed/58..", O_RDONLY) = ENOENT
Interesting. Here's a smaller reproduction recipe that shows the issue:
  # clone of git.git, or any other non-trivial repo; it should be mostly
  # packed
  src=/path/to/git
  git init empty
  export GIT_ALTERNATE_OBJECT_DIRECTORIES=$src/.git/objects
  strace -fe openat \
    git -C empty rev-list --objects $(git -C $src rev-parse HEAD) >/dev/null

In v2.50, we see almost no loose object open calls, because we check the pack first. But in v2.54, we see tons of them.

> 2.50 does not do this. In most customer deployments of Bitbucket,
> the Git data lives on an NFS share. The extra latency on NFS makes
> this process of checking for non-existent loose objects take too
> long, the push essentially hangs at the "Checking connectivity" step.

Yeah, I can imagine. But even on a fast filesystem, we definitely want to avoid all of those syscalls. Replacing "strace" above with a timing harness, even on a system with fast syscalls and a warm cache, the v2.54 version is ~12% slower.

> I believe this new behavior was introduced in the recent object
> database rework. After using bisect, I belive the problem can be
> traced back to commit 8384cbcb4c.

Hmm, my bisect ended up at a593373b09 (packfile: refactor `find_pack_entry()` to work on the packfile store, 2026-01-09), which is nearby. I'm not sure if it might depend on other factors (e.g., presence of commit graphs, midx, etc) or my reproduction is not exactly like yours, or if one of us messed up bisection.

+cc Patrick as the author of both commits.
Show 22 quoted lines
> I don't know the codebase well, but from what I can see is that
> the order in which objects are looked up in object databases
> changed.
> 
> Assuming there are two object databases configured (Main repo,
> and the quarantine directory, for example), the lookup order used
> to be:
> 
> 1. _quarantine dir_ packs
> 2. _main dir_ packs
> 3. _quarantine dir_ loose objects
> 4. _main dir_ loose objects
> 
> With Git 2.54, the order appears to have changed to:
> 
> 1. _quarantine dir_ packs
> 2. _quarantine dir_ loose objects
> 3. _main dir_ packs
> 4. _main dir_ loose objects
> 
> In my testing, within a well-packed repo, Git 2.50 actually never
> performed a loose object lookup.

Yeah, and that type of regression makes sense for what a593373b09 was trying to do. But I think the v2.54 behavior is wrong. We should check all packs before any loose objects.

I'm not sure of the correct fix. This is working against the whole "odb sources are independent and abstract" refactoring that a593373b09 was going for. But I think it's an important optimization. I guess the abstract version would be that each source has "fast" and "slow" lookups or something like that, and we check all fast ones before slow ones. But that is pretty gross.

I'll leave it to Patrick to ponder further. I haven't really been paying a lot of attention to the odb refactoring.

-Peff
Previous: Wolfgang KritzingerNext: Taylor Blau
Message 2 of 13 in “Performance regression in connectivity check during receive-pack (git 2.54)”
  1. Wolfgang KritzingerJul 21, 2026
  2. Jeff KingJul 21, 2026
  3. Taylor BlauJul 21, 2026
  4. Junio C HamanoJul 21, 2026
  5. Patrick SteinhardtJul 22, 2026
  6. Junio C HamanoJul 22, 2026
  7. Jeff KingJul 23, 2026
  8. Patrick SteinhardtJul 24, 2026
  9. Jeff KingJul 26, 2026
  10. Justin ToblerJul 27, 2026
  11. Patrick SteinhardtAug 4, 2026
  12. Jeff KingJul 23, 2026
  13. Jeff KingJul 23, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.