git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] packfile: fix perf regression with many packs

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 17, 2026, 05:18 UTC
Message-ID
<aoKZvxE8oP5B6O_4@pks.im>
In-Reply-To
<20260814165546.GA2563235@coredump.intra.peff.net>
On Fri, Aug 14, 2026 at 12:55:46PM -0400, Jeff King wrote:
Show 31 quoted lines
> On Fri, Aug 14, 2026 at 09:49:26AM +0200, Patrick Steinhardt wrote:
> 
> > On Thu, Aug 13, 2026 at 12:15:25PM -0400, Jeff King wrote:
> > > On Thu, Aug 13, 2026 at 09:35:49AM +0200, Patrick Steinhardt wrote:
> > > 
> > > > An alternative would be to use a hashmap here that tracks the packs that
> > > > have already been added. It has the advantage that it also covers the
> > > > `prepend()` operation and that callers don't have to be aware of this
> > > > mechanism at all. Furthermore, moving preexisting entries to the back or
> > > > front could become O(logn) if the list was doubly-linked. We do this
> > > > operation quite often to re-sort entries in the list when looking up
> > > > objects.
> > > 
> > > Don't we already use such a hashmap via packfile_store_add_pack() and
> > > packfile_store_load_pack()? That comes from ec48540fe8 (packfile.c:
> > > speed up loading lots of packfiles, 2019-11-27) and is how we know that
> > > this "is_new" flag is true (otherwise we'd get duplicates during
> > > "reprepare" operations).
> > 
> > That's a good point, we indeed do! Maybe it would make sense then to
> > remove that map from the packfile store and instead move it into the
> > packfile list to make it more generally useful.
> 
> The map protects more than just adding to the list; it avoids all of
> add_packed_git(), which allocates and does a bunch of stat() calls.  So
> it couldn't just be a check in packfile_list_append(), but would have to
> be a separate existence check well before that.
> 
> That's not impossible, but it would be a lot easier to see what
> generalized pattern would be most useful if there were more than one
> caller of packfile_list_append(). ;)

We only have a single caller that appends, but we have some more that use `packfile_list_prepend()`. And there we basically have the same problem.

Patrick
Previous: Jeff KingNext: Jeff King
Message 20 of 24 in “packfile: fix perf regression with many packs”
  1. packfile: fix perf regression with many packsJohannes Schindelin via GitGitGadget, Aug 12, 2026
  2. Junio C HamanoAug 12, 2026
  3. Jeff KingAug 12, 2026
  4. Patrick SteinhardtAug 13, 2026
  5. Johannes SchindelinAug 13, 2026
  6. Jeff KingAug 13, 2026
  7. Johannes SchindelinAug 13, 2026
  8. Ben KnobleAug 12, 2026
  9. Johannes SchindelinAug 13, 2026
  10. Ben KnobleAug 13, 2026
  11. Patrick SteinhardtAug 13, 2026
  12. Johannes SchindelinAug 13, 2026
  13. Patrick SteinhardtAug 13, 2026
  14. Johannes SchindelinAug 13, 2026
  15. Patrick SteinhardtAug 13, 2026
  16. Junio C HamanoAug 13, 2026
  17. Jeff KingAug 13, 2026
  18. Patrick SteinhardtAug 14, 2026
  19. Jeff KingAug 14, 2026
  20. Patrick SteinhardtAug 17, 2026
  21. Jeff KingAug 17, 2026
  22. packfile: fix perf regression with many packsJohannes Schindelin via GitGitGadget, Aug 13, 2026
  23. Patrick SteinhardtAug 14, 2026
  24. Junio C HamanoAug 14, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.