git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: upload-pack is slow with lots of refs

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Oct 3, 2012, 22:32 UTC
Message-ID
<CACBZZX4Fb0OCkh5kwKvLC+_0xb7q-UB7LH2_WY=dFN5SYUeezQ@mail.gmail.com>
In-Reply-To
<20121003180324.GB27446@sigill.intra.peff.net>
On Wed, Oct 3, 2012 at 8:03 PM, Jeff King <peff@peff.net> wrote:
Show 20 quoted lines
> What version of git are you using?  In the past year or so, I've made
> several tweaks to speed up large numbers of refs, including:
>
>   - cff38a5 (receive-pack: eliminate duplicate .have refs, v1.7.6); note
>     that this only helps if they are being pulled in by an alternates
>     repo. And even then, it only helps if they are mostly duplicates;
>     distinct ones are still O(n^2).
>
>   - 7db8d53 (fetch-pack: avoid quadratic behavior in remove_duplicates)
>     a0de288 (fetch-pack: avoid quadratic loop in filter_refs)
>     Both in v1.7.11. I think there is still a potential quadratic loop
>     in mark_complete()
>
>   - 90108a2 (upload-pack: avoid parsing tag destinations)
>     926f1dd (upload-pack: avoid parsing objects during ref advertisement)
>     Both in v1.7.10. Note that tag objects are more expensive to
>     advertise than commits, because we have to load and peel them.
>
> Even with those patches, though, I found that it was something like ~2s
> to advertise 100,000 refs.

FWIW I bisected between 1.7.9 and 1.7.10 and found that the point at which it went from 1.5/s to 2.5/s upload-pack runs on the pathological git.git repository was none of those, but:

    ccdc6037fe - parse_object: try internal cache before reading object db
Previous: Ævar Arnfjörð BjarmasonNext: Jeff King
Message 21 of 24 in “upload-pack is slow with lots of refs”
  1. Ævar Arnfjörð BjarmasonOct 3, 2012
  2. Nguyen Thai Ngoc DuyOct 3, 2012
  3. Jeff KingOct 3, 2012
  4. Junio C HamanoOct 3, 2012
  5. Jeff KingOct 3, 2012
  6. Shawn PearceOct 3, 2012
  7. Jeff KingOct 3, 2012
  8. Sascha CunzOct 4, 2012
  9. Jeff KingOct 5, 2012
  10. Johannes SixtOct 5, 2012
  11. Shawn PearceOct 5, 2012
  12. Johannes SixtOct 8, 2012
  13. Shawn PearceOct 9, 2012
  14. Johannes SixtOct 9, 2012
  15. Johannes SixtOct 9, 2012
  16. Ævar Arnfjörð BjarmasonOct 3, 2012
  17. Jeff KingOct 3, 2012
  18. Ævar Arnfjörð BjarmasonOct 3, 2012
  19. Jeff KingOct 3, 2012
  20. Ævar Arnfjörð BjarmasonOct 3, 2012
  21. Ævar Arnfjörð BjarmasonOct 3, 2012
  22. Jeff KingOct 3, 2012
  23. Ævar Arnfjörð BjarmasonOct 3, 2012
  24. Junio C HamanoOct 3, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.