git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/6] fetch: speed up lookup of want refs via commit-graph

From
Derrick Stolee <stolee@gmail.com>
Date
Aug 20, 2021, 14:27 UTC
Message-ID
<fce98460-eed6-c874-8c42-bf3a6f5f93a8@gmail.com>
In-Reply-To
<6872979c4557204821d788dc3f5e1c8bef0a773c.1629452412.git.ps@pks.im>
On 8/20/2021 6:08 AM, Patrick Steinhardt wrote:
Show 26 quoted lines
> When updating our local refs based on the refs fetched from the remote,
> we need to iterate through all requested refs and load their respective
> commits such that we can determine whether they need to be appended to
> FETCH_HEAD or not. In cases where we're fetching from a remote with
> exceedingly many refs, resolving these refs can be quite expensive given
> that we repeatedly need to unpack object headers for each of the
> referenced objects.
> 
> Speed this up by opportunistcally trying to resolve object IDs via the
> commit graph: more likely than not, they're going to be a commit anyway,
> and this lets us avoid having to unpack object headers completely in
> case the object is a commit that is part of the commit-graph. This
> significantly speeds up mirror-fetches in a real-world repository with
> 2.3M refs:
> 
>     Benchmark #1: HEAD~: git-fetch
>       Time (mean ± σ):     56.942 s ±  0.449 s    [User: 53.360 s, System: 5.356 s]
>       Range (min … max):   56.372 s … 57.533 s    5 runs
> 
>     Benchmark #2: HEAD: git-fetch
>       Time (mean ± σ):     33.657 s ±  0.167 s    [User: 30.302 s, System: 5.181 s]
>       Range (min … max):   33.454 s … 33.844 s    5 runs
> 
>     Summary
>       'HEAD: git-fetch' ran
>         1.69 ± 0.02 times faster than 'HEAD~: git-fetch'

These numbers are impressive, and it makes sense that performing a binary search on the OID lookup chunk of the commit-graph is faster than doing a binary search on the OIDs across the pack-index(es).

I do worry about the case where annotated tags greatly outnumber branches, so this binary search is extra overhead and the performance may degrade. Would it be worth checking the ref to see if it lies within "refs/heads/" (or even _not_ in "refs/tags/") before doing this commit-graph check?

Show 12 quoted lines
> -			commit = lookup_commit_reference_gently(the_repository,
> -								&rm->old_oid,
> -								1);
> -			if (!commit)
> -				rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE;
> +			commit = lookup_commit_in_graph(the_repository, &rm->old_oid);
> +			if (!commit) {
> +				commit = lookup_commit_reference_gently(the_repository,
> +									&rm->old_oid,
> +									1);
> +				if (!commit)
> +					rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE;
nit: I wouldn't nest this last "if (!commit)".
Code will work as advertised.

Thanks, -Stolee

Previous: Patrick SteinhardtNext: Junio C Hamano
Message 3 of 48 in “Speed up mirror-fetches with many refs”
  1. 0/6 Speed up mirror-fetches with many refsPatrick Steinhardt, Aug 20, 2021
  2. 1/6 fetch: speed up lookup of want refs via commit-graphPatrick Steinhardt, Aug 20, 2021
  3. Derrick StoleeAug 20, 2021
  4. Junio C HamanoAug 20, 2021
  5. Patrick SteinhardtAug 23, 2021
  6. Derrick StoleeAug 25, 2021
  7. 2/6 fetch: avoid unpacking headers in object existence checkPatrick Steinhardt, Aug 20, 2021
  8. Ævar Arnfjörð BjarmasonAug 25, 2021
  9. 3/6 connected: refactor iterator to return next object ID directlyPatrick Steinhardt, Aug 20, 2021
  10. Derrick StoleeAug 20, 2021
  11. Junio C HamanoAug 20, 2021
  12. René ScharfeAug 20, 2021
  13. Patrick SteinhardtAug 23, 2021
  14. 4/6 fetch-pack: optimize loading of refs via commit graphPatrick Steinhardt, Aug 20, 2021
  15. Derrick StoleeAug 20, 2021
  16. 5/6 fetch: refactor fetch refs to be more extendablePatrick Steinhardt, Aug 20, 2021
  17. Derrick StoleeAug 20, 2021
  18. 6/6 fetch: avoid second connectivity check if we already have all objectsPatrick Steinhardt, Aug 20, 2021
  19. Derrick StoleeAug 20, 2021
  20. Patrick SteinhardtAug 23, 2021
  21. Derrick StoleeAug 20, 2021
  22. Junio C HamanoAug 21, 2021
  23. 0/7 Speed up mirror-fetches with many refsPatrick Steinhardt, Aug 24, 2021
  24. 1/7 fetch: speed up lookup of want refs via commit-graphPatrick Steinhardt, Aug 24, 2021
  25. Derrick StoleeAug 25, 2021
  26. 2/7 fetch: avoid unpacking headers in object existence checkPatrick Steinhardt, Aug 24, 2021
  27. 3/7 connected: refactor iterator to return next object ID directlyPatrick Steinhardt, Aug 24, 2021
  28. 4/7 fetch-pack: optimize loading of refs via commit graphPatrick Steinhardt, Aug 24, 2021
  29. 5/7 fetch: refactor fetch refs to be more extendablePatrick Steinhardt, Aug 24, 2021
  30. Derrick StoleeAug 25, 2021
  31. Patrick SteinhardtSep 1, 2021
  32. 7/7 fetch: avoid second connectivity check if we already have all objectsPatrick Steinhardt, Aug 24, 2021
  33. 6/7 fetch: merge fetching and consuming refsPatrick Steinhardt, Aug 24, 2021
  34. Derrick StoleeAug 25, 2021
  35. Patrick SteinhardtSep 1, 2021
  36. Junio C HamanoAug 24, 2021
  37. Patrick SteinhardtAug 25, 2021
  38. Derrick StoleeAug 25, 2021
  39. 0/7 Speed up mirror-fetches with many refsPatrick Steinhardt, Sep 1, 2021
  40. 1/7 fetch: speed up lookup of want refs via commit-graphPatrick Steinhardt, Sep 1, 2021
  41. 2/7 fetch: avoid unpacking headers in object existence checkPatrick Steinhardt, Sep 1, 2021
  42. 3/7 connected: refactor iterator to return next object ID directlyPatrick Steinhardt, Sep 1, 2021
  43. 4/7 fetch-pack: optimize loading of refs via commit graphPatrick Steinhardt, Sep 1, 2021
  44. 5/7 fetch: refactor fetch refs to be more extendablePatrick Steinhardt, Sep 1, 2021
  45. 6/7 fetch: merge fetching and consuming refsPatrick Steinhardt, Sep 1, 2021
  46. 7/7 fetch: avoid second connectivity check if we already have all objectsPatrick Steinhardt, Sep 1, 2021
  47. Junio C HamanoSep 1, 2021
  48. Junio C HamanoSep 8, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.