git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 0/4] multi-pack-index: fix verify on large repos

From
Junio C Hamano <gitster@pobox.com>
Date
Mar 22, 2019, 05:37 UTC
Message-ID
<xmqqy357pizb.fsf@gitster-ct.c.googlers.com>
In-Reply-To
<pull.166.v3.git.gitgitgadget@gmail.com>
"Jeff Hostetler via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 13 quoted lines
> Teach "multi-pack-index verify" to handle cases where the number of
> packfiles exceeds the open file handle limit.
>
> The first commit fixes a problem that prevented the LRU-style
> close_one_pack() mechanism from working which caused midx verify to run out
> of file descriptors.
>
> The second commit teaches midx verify to sort the set of objects to verify
> by packfile rather than verifying them in OID order. This eliminates the
> need to have more than one packfile/idx open at the same time.
>
> With the second commit, runtime on 3600 packfiles went from 12 minutes to 25
> seconds.

These reference to the first and second commit might have become stale across interations, but logically it makes sense---the first point is about correctness (i.e. do not die by running out of fds) and the second one is about usable-performance.

But in this round (possibly in the previous one, too?) the "group objects by packfile" one addresses both points?

Previous: Jeff Hostetler via GitGitGadgetNext: Jeff Hostetler
Message 25 of 26 in “multi-pack-index: fix verify on large repos”
  1. 0/3 multi-pack-index: fix verify on large reposJeff Hostetler via GitGitGadget, Mar 18, 2019
  2. 1/3 midx: verify: add midx packfiles to the packed_git listJeff Hostetler via GitGitGadget, Mar 18, 2019
  3. 3/3 trace2:data: add trace2 data to midxJeff Hostetler via GitGitGadget, Mar 18, 2019
  4. 2/3 midx: verify: group objects by packfile to speed up object verificationJeff Hostetler via GitGitGadget, Mar 18, 2019
  5. Ævar Arnfjörð BjarmasonMar 18, 2019
  6. Jeff HostetlerMar 18, 2019
  7. Ævar Arnfjörð BjarmasonMar 18, 2019
  8. Junio C HamanoMar 19, 2019
  9. Jeff HostetlerMar 19, 2019
  10. Ævar Arnfjörð BjarmasonMar 19, 2019
  11. 0/4 multi-pack-index: fix verify on large reposJeff Hostetler via GitGitGadget, Mar 19, 2019
  12. 1/4 progress: add sparse mode to force 100% complete messageJeff Hostetler via GitGitGadget, Mar 19, 2019
  13. Eric SunshineMar 19, 2019
  14. Jeff HostetlerMar 19, 2019
  15. Eric SunshineMar 19, 2019
  16. 3/4 midx: verify: add midx packfiles to the packed_git listJeff Hostetler via GitGitGadget, Mar 19, 2019
  17. Jeff HostetlerMar 19, 2019
  18. 2/4 trace2:data: add trace2 data to midxJeff Hostetler via GitGitGadget, Mar 19, 2019
  19. 4/4 midx: verify: group objects by packfile to speed up object verificationJeff Hostetler via GitGitGadget, Mar 19, 2019
  20. 0/4 multi-pack-index: fix verify on large reposJeff Hostetler via GitGitGadget, Mar 21, 2019
  21. 2/4 trace2:data: add trace2 data to midxJeff Hostetler via GitGitGadget, Mar 21, 2019
  22. 3/4 midx: add progress indicators in multi-pack-index verifyJeff Hostetler via GitGitGadget, Mar 21, 2019
  23. 1/4 progress: add sparse mode to force 100% complete messageJeff Hostetler via GitGitGadget, Mar 21, 2019
  24. 4/4 midx: during verify group objects by packfile to speed verificationJeff Hostetler via GitGitGadget, Mar 21, 2019
  25. Junio C HamanoMar 22, 2019
  26. Jeff HostetlerMar 25, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.