git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 1/6] odb/source-files: fix performance when reading objects with alternates

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 14, 2026, 13:44 UTC
Message-ID
<as-HRIokfGK555yo@pks.im>
In-Reply-To
<332d341e-0a1b-790f-4b0f-55a7ebb80f3f@gmx.de>
On Wed, Oct 14, 2026 at 03:35:56PM +0200, Johannes Schindelin wrote:
Show 64 quoted lines
> Hi Patrick,
> 
> On Wed, 14 Oct 2026, Patrick Steinhardt wrote:
> 
> > With the move towards pluggable object databases we reorganized how we
> > access alternates. Before the introduction of pluggability, we were
> > reading objects by first scanning through packfiles from all alternates
> > before scanning through all loose object directories. With the change
> > though, we are now iterating through the sources one by one, checking
> > each of them for whether they contain the object either as a packed or
> > loose object.
> > 
> > Checking whether an object exists in packfiles is comparatively cheap
> > because the list of packs is cached, and the pack indices are likely to
> > be in the page cache. In contrast to that, checking whether an object
> > exists in loose format always incurs one stat(3p) syscall. So scanning
> > for objects one alternate at a time led to a performance regression in
> > in the case where most of the objects are contained in a packfile of an
> > alternate, as reported in [1].
> > 
> > We could fix that issue by introducing another flag that tells the
> > backends to only search the optimized storage backend. But that is
> > adding more complexity to the object database layers that is ultimately
> > only really required because of the way the "files" backend stores its
> > data. So for an alternate backend, such new infrastructure would not
> > really be a good fit.
> > 
> > But since then we have moved handling of alternates into the "files"
> > backend itself. One of the consequences of this is that this backend is
> > now having full control over all of its sub-sources, and that also makes
> > it way easier for us to optimize for this access pattern again.
> > 
> > Do so by first trying to read objects from packfiles located in any of
> > its object directories before trying to read loose objects.
> > 
> > The following benchmark demonstrates that this significantly improves
> > performance in some cases. The benchmark uses the Git repository itself
> > and sets up its object directory as an alternate, while the primary
> > object directory is simply an empty object directory. It then reads all
> > commits enumerated via `git rev-list HEAD` with git-cat-file(1):
> > 
> >   Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)
> >     Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]
> >     Range (min … max):   177.1 ms … 187.7 ms    16 runs
> > 
> >   Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)
> >     Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]
> >     Range (min … max):    93.0 ms …  97.9 ms    30 runs
> > 
> >   Summary
> >     git cat-file --batch-check --unordered (rev = HEAD) ran
> >       1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)
> > 
> > [1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>
> > 
> > Reported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> 
> Thank you so much for this! This fixes the bug we had to patch
> independently in dc305fc33833 (odb: scan all sources’ packfiles before
> loose objects), via https://github.com/microsoft/git/pull/975, first
> contained in Microsoft Git v2.55.0.vfs.0.7. Would you kindly include (my
> colleague) Tyrie Vella's regression test in the next iteration of this
> patch series?

Yeah, makes sense. I'll fold the test into this patch with a Helped-by trailer. Thanks!

Patrick
Previous: Johannes SchindelinNext: Patrick Steinhardt
Message 22 of 37 in “odb/files: cleanups and improvements for alternates”
  1. 0/5 odb/files: cleanups and improvements for alternatesPatrick Steinhardt, Oct 13, 2026
  2. 1/5 odb/source-files: fix performance when reading objects with alternatesPatrick Steinhardt, Oct 13, 2026
  3. Justin ToblerOct 13, 2026
  4. 2/5 odb/source-files: fix performance when streaming objects with alternatesPatrick Steinhardt, Oct 13, 2026
  5. Justin ToblerOct 13, 2026
  6. 3/5 odb/source-files: make the second read an implementation detailPatrick Steinhardt, Oct 13, 2026
  7. Justin ToblerOct 13, 2026
  8. Patrick SteinhardtOct 14, 2026
  9. Justin ToblerOct 14, 2026
  10. 4/5 odb: remove read loop for found objectsPatrick Steinhardt, Oct 13, 2026
  11. Justin ToblerOct 13, 2026
  12. Patrick SteinhardtOct 14, 2026
  13. Justin ToblerOct 14, 2026
  14. 5/5 odb/source: drop `local` flagPatrick Steinhardt, Oct 13, 2026
  15. Justin ToblerOct 13, 2026
  16. Patrick SteinhardtOct 14, 2026
  17. Justin ToblerOct 14, 2026
  18. Patrick SteinhardtOct 15, 2026
  19. 0/6 odb/files: cleanups and improvements for alternatesPatrick Steinhardt, Oct 14, 2026
  20. 1/6 odb/source-files: fix performance when reading objects with alternatesPatrick Steinhardt, Oct 14, 2026
  21. Johannes SchindelinOct 14, 2026
  22. Patrick SteinhardtOct 14, 2026
  23. 2/6 odb/source-files: fix performance when streaming objects with alternatesPatrick Steinhardt, Oct 14, 2026
  24. 3/6 odb/source-files: make the second read an implementation detailPatrick Steinhardt, Oct 14, 2026
  25. 4/6 odb: remove read loop for found objectsPatrick Steinhardt, Oct 14, 2026
  26. 5/6 odb: don't try re-fetching corrupt objectsPatrick Steinhardt, Oct 14, 2026
  27. Justin ToblerOct 14, 2026
  28. 6/6 odb/source: drop `local` flagPatrick Steinhardt, Oct 14, 2026
  29. 0/6 odb/files: cleanups and improvements for alternatesPatrick Steinhardt, Oct 15, 2026
  30. 1/6 odb/source-files: fix performance when reading objects with alternatesPatrick Steinhardt, Oct 15, 2026
  31. 2/6 odb/source-files: fix performance when streaming objects with alternatesPatrick Steinhardt, Oct 15, 2026
  32. 3/6 odb/source-files: make the second read an implementation detailPatrick Steinhardt, Oct 15, 2026
  33. 4/6 odb: remove read loop for found objectsPatrick Steinhardt, Oct 15, 2026
  34. 5/6 odb: don't try re-fetching corrupt objectsPatrick Steinhardt, Oct 15, 2026
  35. 6/6 odb/source: drop `local` flagPatrick Steinhardt, Oct 15, 2026
  36. Justin ToblerOct 15, 2026
  37. Junio C HamanoOct 16, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.