Re: [PATCH v2 1/6] odb/source-files: fix performance when reading objects with alternates
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Oct 14, 2026, 13:44 UTC
- Message-ID
- <as-HRIokfGK555yo@pks.im>
- In-Reply-To
- <332d341e-0a1b-790f-4b0f-55a7ebb80f3f@gmx.de>
On Wed, Oct 14, 2026 at 03:35:56PM +0200, Johannes Schindelin wrote:
Show 64 quoted lines
> Hi Patrick, > > On Wed, 14 Oct 2026, Patrick Steinhardt wrote: > > > With the move towards pluggable object databases we reorganized how we > > access alternates. Before the introduction of pluggability, we were > > reading objects by first scanning through packfiles from all alternates > > before scanning through all loose object directories. With the change > > though, we are now iterating through the sources one by one, checking > > each of them for whether they contain the object either as a packed or > > loose object. > > > > Checking whether an object exists in packfiles is comparatively cheap > > because the list of packs is cached, and the pack indices are likely to > > be in the page cache. In contrast to that, checking whether an object > > exists in loose format always incurs one stat(3p) syscall. So scanning > > for objects one alternate at a time led to a performance regression in > > in the case where most of the objects are contained in a packfile of an > > alternate, as reported in [1]. > > > > We could fix that issue by introducing another flag that tells the > > backends to only search the optimized storage backend. But that is > > adding more complexity to the object database layers that is ultimately > > only really required because of the way the "files" backend stores its > > data. So for an alternate backend, such new infrastructure would not > > really be a good fit. > > > > But since then we have moved handling of alternates into the "files" > > backend itself. One of the consequences of this is that this backend is > > now having full control over all of its sub-sources, and that also makes > > it way easier for us to optimize for this access pattern again. > > > > Do so by first trying to read objects from packfiles located in any of > > its object directories before trying to read loose objects. > > > > The following benchmark demonstrates that this significantly improves > > performance in some cases. The benchmark uses the Git repository itself > > and sets up its object directory as an alternate, while the primary > > object directory is simply an empty object directory. It then reads all > > commits enumerated via `git rev-list HEAD` with git-cat-file(1): > > > > Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~) > > Time (mean ± σ): 179.4 ms ± 2.6 ms [User: 114.6 ms, System: 61.4 ms] > > Range (min … max): 177.1 ms … 187.7 ms 16 runs > > > > Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD) > > Time (mean ± σ): 95.0 ms ± 1.5 ms [User: 79.8 ms, System: 15.0 ms] > > Range (min … max): 93.0 ms … 97.9 ms 30 runs > > > > Summary > > git cat-file --batch-check --unordered (rev = HEAD) ran > > 1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~) > > > > [1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com> > > > > Reported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com> > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > > Thank you so much for this! This fixes the bug we had to patch > independently in dc305fc33833 (odb: scan all sources’ packfiles before > loose objects), via https://github.com/microsoft/git/pull/975, first > contained in Microsoft Git v2.55.0.vfs.0.7. Would you kindly include (my > colleague) Tyrie Vella's regression test in the next iteration of this > patch series?
Yeah, makes sense. I'll fold the test into this patch with a Helped-by trailer. Thanks!
Patrick