git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 6/7] config: enable `pack.writeReverseIndex` by default

From
Junio C Hamano <gitster@pobox.com>
Date
Apr 13, 2023, 16:14 UTC
Message-ID
<xmqqttxjyd2r.fsf@gitster.g>
In-Reply-To
<56a0fc0098e0b0551e01414c8e67c17fb1ba3054.1681166596.git.me@ttaylorr.com>
Taylor Blau <me@ttaylorr.com> writes:
Show 24 quoted lines
> They are useful in GitHub's infrastructure, where we have seen a
> dramatic increase in performance when writing ".rev" files[1]. In
> particular:
>
>   - an ~80% reduction in the time it takes to serve fetches on a popular
>     repository, Homebrew/homebrew-core.
>
>   - a ~60% reduction in the peak memory usage to serve fetches on that
>     same repository.
>
>   - a collective savings of ~35% in CPU time across all pack-objects
>     invocations serving fetches across all repositories in a single
>     datacenter.
>
> Reverse indexes are also beneficial to end-users as well as forges. For
> example, the time it takes to generate a pack containing the objects for
> the 10 most recent commits in linux.git (representing a typical push) is
> significantly faster when on-disk reverse indexes are available:
> ...
> In the more than two years since e37d0b8730 was merged, Git's
> implementation of on-disk reverse indexes has been thoroughly tested,
> both from users enabling `pack.writeReverseIndexes`, and from GitHub's
> deployment of the feature. The latter has been running without incident
> for more than two years.

What is missing is what extra overhead this adds to "git gc". Does it add 5%? 15%? How big an overhead do we find reasonable?

Other than that, it looks like that the series is very well crafted. It was somewhat surprising that we still need to add a few new end-user facing or meant-for-testing knobs for a feature that we claim is well battle tested. It is understandable that we benefit by having knobs to selectively _disable_ a part of the feature now it will be on by default.

Will queue.  Thanks.
Previous: Taylor BlauNext: Taylor Blau
Message 13 of 29 in “pack-revindex: enable on-disk reverse indexes by default”
  1. 0/7 pack-revindex: enable on-disk reverse indexes by defaultTaylor Blau, Apr 10, 2023
  2. 1/7 pack-write.c: plug a leak in stage_tmp_packfiles()Taylor Blau, Apr 10, 2023
  3. Derrick StoleeApr 11, 2023
  4. Taylor BlauApr 11, 2023
  5. 2/7 t5325: mark as leak-freeTaylor Blau, Apr 10, 2023
  6. 3/7 pack-revindex: make `load_pack_revindex` take a repositoryTaylor Blau, Apr 10, 2023
  7. Derrick StoleeApr 11, 2023
  8. Taylor BlauApr 11, 2023
  9. Derrick StoleeApr 12, 2023
  10. 4/7 pack-revindex: introduce GIT_TEST_REV_INDEX_DIE_ON_DISKTaylor Blau, Apr 10, 2023
  11. 5/7 pack-revindex: introduce `pack.readReverseIndex`Taylor Blau, Apr 10, 2023
  12. 6/7 config: enable `pack.writeReverseIndex` by defaultTaylor Blau, Apr 10, 2023
  13. Junio C HamanoApr 13, 2023
  14. 7/7 t: invert `GIT_TEST_WRITE_REV_INDEX`Taylor Blau, Apr 10, 2023
  15. Derrick StoleeApr 11, 2023
  16. Taylor BlauApr 11, 2023
  17. Derrick StoleeApr 12, 2023
  18. Derrick StoleeApr 11, 2023
  19. Taylor BlauApr 11, 2023
  20. Derrick StoleeApr 12, 2023
  21. 0/7 pack-revindex: enable on-disk reverse indexes by defaultTaylor Blau, Apr 12, 2023
  22. 1/7 pack-write.c: plug a leak in stage_tmp_packfiles()Taylor Blau, Apr 12, 2023
  23. 2/7 t5325: mark as leak-freeTaylor Blau, Apr 12, 2023
  24. 3/7 pack-revindex: make `load_pack_revindex` take a repositoryTaylor Blau, Apr 12, 2023
  25. 4/7 pack-revindex: introduce GIT_TEST_REV_INDEX_DIE_ON_DISKTaylor Blau, Apr 12, 2023
  26. 5/7 pack-revindex: introduce `pack.readReverseIndex`Taylor Blau, Apr 12, 2023
  27. 7/7 t: invert `GIT_TEST_WRITE_REV_INDEX`Taylor Blau, Apr 12, 2023
  28. 6/7 config: enable `pack.writeReverseIndex` by defaultTaylor Blau, Apr 12, 2023
  29. Derrick StoleeApr 13, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.