git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 00/18] Multi-pack index (MIDX)

From
Derrick Stolee <stolee@gmail.com>
Date
Jan 10, 2018, 19:39 UTC
Message-ID
<7b3df316-8a35-0dcf-da17-64202f600821@gmail.com>
In-Reply-To
<2648064.NJaMxHkljU@mfick-lnx>
On 1/10/2018 1:25 PM, Martin Fick wrote:
Show 17 quoted lines
> On Sunday, January 07, 2018 01:14:41 PM Derrick Stolee
> wrote:
>> This RFC includes a new way to index the objects in
>> multiple packs using one file, called the multi-pack
>> index (MIDX).
> ...
>> The main goals of this RFC are:
>>
>> * Determine interest in this feature.
>>
>> * Find other use cases for the MIDX feature.
> My interest in this feature would be to speed up fetches
> when there is more than one large pack-file with many of the
> same objects that are in other pack-files.   What does your
> MIDX design do when it encounters multiple copies of the
> same object in different pack files?  Does it index them all,
> or does it keep a single copy?

The MIDX currently keeps only one reference to each object. Duplicates are dropped during writing. (See the care taken in commit 04/18 to avoid duplicates.) Since midx_sha1_compare() does not use anything other than the OID to order the objects, there is no decision being made about which pack is "better". The MIDX writes the first copy it finds and discards the others.

It would not be difficult to include a check in midx_sha1_compare() to favor one packfile over another based on some measurement (size? mtime?). Since this would be a heuristic at best, I left it out of the current patch.

> In our Gerrit instance (Gerrit uses jgit), we have multiple
> copies of the linux kernel repos linked together via the
> alternatives file mechanism.

GVFS also uses alternates for sharing packfiles across multiple copies of the repo. The MIDX is designed to cover all packfiles in the same directory, but is not designed to cover packfiles in multiple alternates; currently, each alternate would need its own MIDX file. Does that cause issues with your setup?

Show 27 quoted lines
>    These repos have many different
> references (mostly Gerrit change references), but they share
> most of the common objects from the mainline.  I have found
> that during a large fetch such as a clone, jgit spends a
> significant amount of extra time by having the extra large
> pack-files from the other repos visible to it, usually around
> an extra minute per instance of these (without them, the
> clone takes around 7mins).  This adds up easily with a few
> repos extra repos, it can almost double the time.
>
> My investigations have shown that this is due to jgit
> searching each of these pack files to decide which version of
> each object to send.  I don't fully understand its selection
> criteria, however if I shortcut it to just pick the first
> copy of an object that it finds, I regain my lost time.  I
> don't know if git suffers from a similar problem?  If git
> doesn't suffer from this then it likely just uses the first
> copy of an object it finds (which may not be the best object
> to send?)
>
> It would be nice if this use case could be improved with
> MIDX.  To do so, it seems that it would either require that
> MIDX either only put "the best" version of an object (i.e.
> pre-select which one to use), or include the extra
> information to help make the selection process of which copy
> to use (perhaps based on the operation being performed)
> fast.

I'm not sure if there is sufficient value in storing multiple references to the same object stored in multiple packfiles. There could be value in carefully deciding which copy is "best" during the MIDX write, but during read is not a good time to make such a decision. It also increases the size of the file to store multiple copies.

Show 6 quoted lines
> This also leads me to ask, what other additional information
> (bitmaps?) for other operations, besides object location,
> might suddenly be valuable in an index that potentially
> points to multiple copies of objects?  Would such
> information be appropriate in MIDX, or would it be better in
> another index?

For applications to bitmaps, it is probably best that we only include one copy of each object. Otherwise, we need to include extra bits in the bitmaps for those copies (when asking "is this object in the bitmap?").

Thanks for the context with Gerrit's duplicate object problem. I'll try to incorporate it in to the design document (commit 01/18) for the v1 patch.

Thanks, -Stolee

Previous: Martin FickNext: Martin Fick
Message 47 of 48 in “Multi-pack index (MIDX)”
  1. 00/18 Multi-pack index (MIDX)Derrick Stolee, Jan 7, 2018
  2. 01/18 docs: Multi-Pack Index (MIDX) Design NotesDerrick Stolee, Jan 7, 2018
  3. Jonathan TanJan 8, 2018
  4. Derrick StoleeJan 8, 2018
  5. Jonathan TanJan 8, 2018
  6. 02/18 midx: specify midx file formatDerrick Stolee, Jan 7, 2018
  7. 06/18 midx: add t5318-midx.sh test scriptDerrick Stolee, Jan 7, 2018
  8. 09/18 midx: find details of nth object in midxDerrick Stolee, Jan 7, 2018
  9. 11/18 midx: teach git-midx to clear midx filesDerrick Stolee, Jan 7, 2018
  10. 12/18 midx: teach git-midx to delete expired filesDerrick Stolee, Jan 7, 2018
  11. 13/18 t5318-midx.h: confirm git actions are stableDerrick Stolee, Jan 7, 2018
  12. 15/18 midx: use midx for approximate object countDerrick Stolee, Jan 7, 2018
  13. 18/18 packfile: use midx for object loadsDerrick Stolee, Jan 7, 2018
  14. 17/18 sha1_name: use midx for abbreviationsDerrick Stolee, Jan 7, 2018
  15. 08/18 midx: teach git-midx to read midx file detailsDerrick Stolee, Jan 7, 2018
  16. 14/18 midx: load midx files when loading packsDerrick Stolee, Jan 7, 2018
  17. 10/18 midx: use existing midx when writingDerrick Stolee, Jan 7, 2018
  18. 16/18 midx: nth_midxed_object_oid() and bsearch_midx()Derrick Stolee, Jan 7, 2018
  19. 03/18 midx: create core.midx config settingDerrick Stolee, Jan 7, 2018
  20. 07/18 midx: teach midx --write to update midx-headDerrick Stolee, Jan 7, 2018
  21. 04/18 midx: write multi-pack indexes for an object listDerrick Stolee, Jan 7, 2018
  22. 05/18 midx: create midx builtin with --write modeDerrick Stolee, Jan 7, 2018
  23. Ævar Arnfjörð BjarmasonJan 7, 2018
  24. Derrick StoleeJan 8, 2018
  25. Jeff KingJan 8, 2018
  26. Jeff KingJan 8, 2018
  27. Ævar Arnfjörð BjarmasonJan 8, 2018
  28. Johannes SchindelinJan 8, 2018
  29. Jeff KingJan 9, 2018
  30. Johannes SchindelinJan 9, 2018
  31. Stefan BellerJan 9, 2018
  32. Junio C HamanoJan 9, 2018
  33. Stefan BellerJan 9, 2018
  34. Junio C HamanoJan 9, 2018
  35. Johannes SchindelinJan 10, 2018
  36. Jeff KingJan 10, 2018
  37. Derrick StoleeJan 8, 2018
  38. Jeff KingJan 9, 2018
  39. Ævar Arnfjörð BjarmasonJan 8, 2018
  40. Ævar Arnfjörð BjarmasonJun 6, 2018
  41. 0/2 unconditional O(1) SHA-1 abbreviationÆvar Arnfjörð Bjarmason, Jun 6, 2018
  42. 1/2 config.c: use braces on multiple conditional armsÆvar Arnfjörð Bjarmason, Jun 6, 2018
  43. 2/2 sha1-name: add core.validateAbbrev & relative core.abbrevÆvar Arnfjörð Bjarmason, Jun 6, 2018
  44. Christian CouderJun 6, 2018
  45. Derrick StoleeJun 6, 2018
  46. Martin FickJan 10, 2018
  47. Derrick StoleeJan 10, 2018
  48. Martin FickJan 10, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.