git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] packfile: fix perf regression with many packs

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 13, 2026, 10:01 UTC
Message-ID
<an2V7S-DkdypsGIE@pks.im>
In-Reply-To
<b4860540-6114-2a7b-e266-d1fc2f0041b9@gmx.de>
On Thu, Aug 13, 2026 at 11:20:11AM +0200, Johannes Schindelin wrote:
Show 8 quoted lines
> On Thu, 13 Aug 2026, Patrick Steinhardt wrote:
> > I wonder whether we should slightly reformulate this and rename `is_new`
> > to `accept_duplicates`. Because ultimately, that is what we're doing
> > now: instead of ensuring that the packfile is unique in the list, we
> > just don't care and just append the entry to the list.
> 
> Hmm. I don't quite agree, we're _not_ accepting duplicates. We know that
> those packfiles _cannot_ be duplicates.

I know that we're not, but this is only because the caller knows that the packs are new. Seen outside that context though the new parameter really just tells us whether or not we want to deduplicate packs or not.

Anyway, I'm splitting hairs and I won't insist on a change here.
Show 17 quoted lines
> > An alternative would be to use a hashmap here that tracks the packs that
> > have already been added. It has the advantage that it also covers the
> > `prepend()` operation and that callers don't have to be aware of this
> > mechanism at all. Furthermore, moving preexisting entries to the back or
> > front could become O(logn) if the list was doubly-linked. We do this
> > operation quite often to re-sort entries in the list when looking up
> > objects.
> 
> Indeed, that was my initial reaction, too. I was well on my way to start
> writing a hashmap-based fix when the AI assistant pointed out that no
> duplicates could possibly exist yet.
> 
> > Overall though I'm not quite sure whether the added complexity would be
> > worth it, see below patch.
> 
> Wow, you got a lot further than I did! And yes, I agree that we do not
> (yet?) need to deal with the added complexity.

I may want to pursue this patch anyway, as I think that the reordering would be sped up by that change quite signifcantly. And that would make a difference indeed when you have 38k packfiles, at least when you assume that objects are evenly distributed across all of those and that we perform reads of random objects.

I could do that tomorrow, and in that case it'd supersede your patch. But I'm also happy to have this improvement here land first and then I'll pursue this change eventually.

Thanks!
Patrick
Previous: Johannes SchindelinNext: Johannes Schindelin
Message 13 of 24 in “packfile: fix perf regression with many packs”
  1. packfile: fix perf regression with many packsJohannes Schindelin via GitGitGadget, Aug 12, 2026
  2. Junio C HamanoAug 12, 2026
  3. Jeff KingAug 12, 2026
  4. Patrick SteinhardtAug 13, 2026
  5. Johannes SchindelinAug 13, 2026
  6. Jeff KingAug 13, 2026
  7. Johannes SchindelinAug 13, 2026
  8. Ben KnobleAug 12, 2026
  9. Johannes SchindelinAug 13, 2026
  10. Ben KnobleAug 13, 2026
  11. Patrick SteinhardtAug 13, 2026
  12. Johannes SchindelinAug 13, 2026
  13. Patrick SteinhardtAug 13, 2026
  14. Johannes SchindelinAug 13, 2026
  15. Patrick SteinhardtAug 13, 2026
  16. Junio C HamanoAug 13, 2026
  17. Jeff KingAug 13, 2026
  18. Patrick SteinhardtAug 14, 2026
  19. Jeff KingAug 14, 2026
  20. Patrick SteinhardtAug 17, 2026
  21. Jeff KingAug 17, 2026
  22. packfile: fix perf regression with many packsJohannes Schindelin via GitGitGadget, Aug 13, 2026
  23. Patrick SteinhardtAug 14, 2026
  24. Junio C HamanoAug 14, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.