git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 00/18] Multi-pack index (MIDX)

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jan 7, 2018, 22:42 UTC
Message-ID
<87k1wtb8a4.fsf@evledraar.gmail.com>
In-Reply-To
<20180107181459.222909-1-dstolee@microsoft.com>
On Sun, Jan 07 2018, Derrick Stolee jotted:
Show 30 quoted lines
>     git log --oneline --raw --parents
>
> Num Packs | Before MIDX | After MIDX |  Rel % | 1 pack %
> ----------+-------------+------------+--------+----------
>         1 |     35.64 s |    35.28 s |  -1.0% |   -1.0%
>        24 |     90.81 s |    40.06 s | -55.9% |  +12.4%
>       127 |    257.97 s |    42.25 s | -83.6% |  +18.6%
>
> The last column is the relative difference between the MIDX-enabled repo
> and the single-pack repo. The goal of the MIDX feature is to present the
> ODB as if it was fully repacked, so there is still room for improvement.
>
> Changing the command to
>
>     git log --oneline --raw --parents --abbrev=40
>
> has no observable difference (sub 1% change in all cases). This is likely
> due to the repack I used putting commits and trees in a small number of
> packfiles so the MRU cache workes very well. On more naturally-created
> lists of packfiles, there can be up to 20% improvement on this command.
>
> We are using a version of this patch with an upcoming release of GVFS.
> This feature is particularly important in that space since GVFS performs
> a "prefetch" step that downloads a pack of commits and trees on a daily
> basis. These packfiles are placed in an alternate that is shared by all
> enlistments. Some users have 150+ packfiles and the MRU misses and
> abbreviation computations are significant. Now, GVFS manages the MIDX file
> after adding new prefetch packfiles using the following command:
>
>     git midx --write --update-head --delete-expired --pack-dir=<alt>
(Not a critique of this, just a (stupid) question)

What's the practical use-case for this feature? Since it doesn't help with --abbrev=40 the speedup is all in the part that ensures we don't show an ambiguous SHA-1.

The reason we do that at all is because it makes for a prettier UI.

Are there things that both want the pretty SHA-1 and also care about the throughput? I'd have expected machine parsing to just use --no-abbrev-commit.

If something cares about both throughput and e.g. is saving the abbreviated SHA-1s isn't it better off picking some arbitrary size (e.g. --abbrev=20), after all the default abbreviation is going to show something as small as possible, which may soon become ambigous after the next commit.

Previous: Derrick StoleeNext: Derrick Stolee
Message 23 of 48 in “Multi-pack index (MIDX)”
  1. 00/18 Multi-pack index (MIDX)Derrick Stolee, Jan 7, 2018
  2. 01/18 docs: Multi-Pack Index (MIDX) Design NotesDerrick Stolee, Jan 7, 2018
  3. Jonathan TanJan 8, 2018
  4. Derrick StoleeJan 8, 2018
  5. Jonathan TanJan 8, 2018
  6. 02/18 midx: specify midx file formatDerrick Stolee, Jan 7, 2018
  7. 06/18 midx: add t5318-midx.sh test scriptDerrick Stolee, Jan 7, 2018
  8. 09/18 midx: find details of nth object in midxDerrick Stolee, Jan 7, 2018
  9. 11/18 midx: teach git-midx to clear midx filesDerrick Stolee, Jan 7, 2018
  10. 12/18 midx: teach git-midx to delete expired filesDerrick Stolee, Jan 7, 2018
  11. 13/18 t5318-midx.h: confirm git actions are stableDerrick Stolee, Jan 7, 2018
  12. 15/18 midx: use midx for approximate object countDerrick Stolee, Jan 7, 2018
  13. 18/18 packfile: use midx for object loadsDerrick Stolee, Jan 7, 2018
  14. 17/18 sha1_name: use midx for abbreviationsDerrick Stolee, Jan 7, 2018
  15. 08/18 midx: teach git-midx to read midx file detailsDerrick Stolee, Jan 7, 2018
  16. 14/18 midx: load midx files when loading packsDerrick Stolee, Jan 7, 2018
  17. 10/18 midx: use existing midx when writingDerrick Stolee, Jan 7, 2018
  18. 16/18 midx: nth_midxed_object_oid() and bsearch_midx()Derrick Stolee, Jan 7, 2018
  19. 03/18 midx: create core.midx config settingDerrick Stolee, Jan 7, 2018
  20. 07/18 midx: teach midx --write to update midx-headDerrick Stolee, Jan 7, 2018
  21. 04/18 midx: write multi-pack indexes for an object listDerrick Stolee, Jan 7, 2018
  22. 05/18 midx: create midx builtin with --write modeDerrick Stolee, Jan 7, 2018
  23. Ævar Arnfjörð BjarmasonJan 7, 2018
  24. Derrick StoleeJan 8, 2018
  25. Jeff KingJan 8, 2018
  26. Jeff KingJan 8, 2018
  27. Ævar Arnfjörð BjarmasonJan 8, 2018
  28. Johannes SchindelinJan 8, 2018
  29. Jeff KingJan 9, 2018
  30. Johannes SchindelinJan 9, 2018
  31. Stefan BellerJan 9, 2018
  32. Junio C HamanoJan 9, 2018
  33. Stefan BellerJan 9, 2018
  34. Junio C HamanoJan 9, 2018
  35. Johannes SchindelinJan 10, 2018
  36. Jeff KingJan 10, 2018
  37. Derrick StoleeJan 8, 2018
  38. Jeff KingJan 9, 2018
  39. Ævar Arnfjörð BjarmasonJan 8, 2018
  40. Ævar Arnfjörð BjarmasonJun 6, 2018
  41. 0/2 unconditional O(1) SHA-1 abbreviationÆvar Arnfjörð Bjarmason, Jun 6, 2018
  42. 1/2 config.c: use braces on multiple conditional armsÆvar Arnfjörð Bjarmason, Jun 6, 2018
  43. 2/2 sha1-name: add core.validateAbbrev & relative core.abbrevÆvar Arnfjörð Bjarmason, Jun 6, 2018
  44. Christian CouderJun 6, 2018
  45. Derrick StoleeJun 6, 2018
  46. Martin FickJan 10, 2018
  47. Derrick StoleeJan 10, 2018
  48. Martin FickJan 10, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.