{"thread":{"id":"61603","subject":"[PATCH 01/19] Documentation: describe incremental MIDX format","startedAt":"2024-06-06T23:04:25Z","lastAt":"2024-08-12T14:27:14Z","messageCount":102,"participants":["Taylor Blau","Junio C Hamano","Elijah Newren","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":19},"messages":[{"id":"496573","messageId":"cover.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":null,"subject":"[PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:22Z","receivedAt":"2024-06-06T23:04:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements incremental MIDXs, which allow for storing\na MIDX across multiple layers, each with their own distinct set of\npacks.\n\nMOTIVATION\n==========\n\nDoing so allows large repositories to make use of the MIDX feature\nwithout having to rewrite the entire MIDX every time they want to update\nthe set of packs contained in the MIDX. For extremely large\nrepositories, doing so is often infeasible.\n\nOVERVIEW\n========\n\nThis series implements the first component of incremental MIDXs, meaning\nby the end of it you can run:\n\n $ git multi-pack-index write --incremental\n\na couple of times, and produce a directory structure like:\n\n $ .git/objects/pack/multi-pack-index.d\n .git/objects/pack/multi-pack-index.d\n ├── multi-pack-index-chain\n ├── multi-pack-index-baa53bc5092bed50378fe9232ae7878828df2890.midx\n └── multi-pack-index-f60023a8a104be94eab96dd7c42a6a5db67c82ba.midx\n\nwhere each *.midx file behaves the same way as existing non-incremental\nMIDX implementation behaves today, but in a way that stitches together\nmultiple MIDX \"layers\" without having to rewrite the whole MIDX anytime\nyou want to make a modification to it.\n\nThis is \"part one\" of a multi-part series. The overview of how all of\nthese series fit together is as follows:\n\n - \"Part zero\": preparatory work like 'tb/midx-write-cleanup' and my\n series to clean up temporary file handling [1, 2].\n\n - \"Part one\": this series, which enables reading and writing\n incremental MIDXs, but does not have support for more advanced\n features like bitmaps support or rewriting parts of the MIDX chain.\n\n - \"Part two\": the next series, which builds on support for multi-pack\n reachability bitmaps in an incremental MIDX world, meaning that each\n `*.midx` layer can have its own `*.bitmap`, and the bitmaps at each\n layer can be used together.\n\n - \"Part three\": which supports more advanced management of the MIDX\n chain, like compressing intermediate layers to avoid the chain\n growing too long.\n\nParts zero, one, and two all exist, and the first two have been shared\nwith the list. Part two exists in ttaylorr/git [3], but is excluded from\nthis series to keep the length manageable. I avoided sending this series\nuntil I was confident that bitmaps worked on top of incremental MIDXs to\navoid designing ourselves into a corner.\n\nPart three doesn't exist yet, but is straightforward to do on top. None\nof the design decisions made in this series inhibit my goals for part\nthree.\n\n[1]: https://lore.kernel.org/git/cover.1717023301.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover.1717712358.git.me@ttaylorr.com/\n[3]: https://github.com/ttaylorr/git/compare/tb/incremental-midx...ttaylorr:git:tb/incremental-midx-bitmaps\n\nTaylor Blau (19):\n Documentation: describe incremental MIDX format\n midx: add new fields for incremental MIDX chains\n midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs\n midx: teach `prepare_midx_pack()` about incremental MIDXs\n midx: teach `nth_midxed_object_oid()` about incremental MIDXs\n midx: teach `nth_bitmapped_pack()` about incremental MIDXs\n midx: introduce `bsearch_one_midx()`\n midx: teach `bsearch_midx()` about incremental MIDXs\n midx: teach `nth_midxed_offset()` about incremental MIDXs\n midx: teach `fill_midx_entry()` about incremental MIDXs\n midx: remove unused `midx_locate_pack()`\n midx: teach `midx_contains_pack()` about incremental MIDXs\n midx: teach `midx_preferred_pack()` about incremental MIDXs\n midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs\n midx: support reading incremental MIDX chains\n midx: implement verification support for incremental MIDXs\n t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n t/t5313-pack-bounds-checks.sh: prepare for sub-directories\n midx: implement support for writing incremental MIDX chains\n\n Documentation/git-multi-pack-index.txt | 11 +-\n Documentation/technical/multi-pack-index.txt | 100 +++++\n builtin/multi-pack-index.c | 2 +\n builtin/repack.c | 8 +-\n ci/run-build-and-tests.sh | 2 +-\n midx-write.c | 293 +++++++++++--\n midx.c | 410 ++++++++++++++++---\n midx.h | 26 +-\n object-name.c | 99 ++---\n packfile.c | 21 +-\n packfile.h | 4 +\n t/README | 6 +-\n t/helper/test-read-midx.c | 24 +-\n t/lib-bitmap.sh | 6 +-\n t/lib-midx.sh | 28 ++\n t/t0410-partial-clone.sh | 2 -\n t/t5310-pack-bitmaps.sh | 4 -\n t/t5313-pack-bounds-checks.sh | 8 +-\n t/t5319-multi-pack-index.sh | 30 +-\n t/t5326-multi-pack-bitmaps.sh | 4 +-\n t/t5327-multi-pack-bitmaps-rev.sh | 6 +-\n t/t5332-multi-pack-reuse.sh | 2 +\n t/t5334-incremental-multi-pack-index.sh | 46 +++\n t/t7700-repack.sh | 48 +--\n 24 files changed, 935 insertions(+), 255 deletions(-)\n create mode 100755 t/t5334-incremental-multi-pack-index.sh\n\n\nbase-commit: 680474691b4639280a73baa0bb8792634f99f611\n-- \n2.45.2.437.gecb9450a0e\n"},{"id":"496572","messageId":"d3aaf18b0a8a17c16d111d93042af52268800989.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 01/19] Documentation: describe incremental MIDX format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:25Z","receivedAt":"2024-06-06T23:04:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement incremental multi-pack indexes (MIDXs) over the\nnext several commits by first describing the relevant prerequisites\n(like a new chunk in the MIDX format, the directory structure for\nincremental MIDXs, etc.)\n\nThe format is described in detail in the patch contents below, but the\nhigh-level description is as follows.\n\nIncremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and\neach `*.midx` within that directory has a single \"parent\" MIDX, which is\nthe MIDX layer immediately before it in the MIDX chain. The chain order\nresides in a file 'multi-pack-index-chain' in the same directory.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/multi-pack-index.txt | 100 +++++++++++++++++++\n 1 file changed, 100 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex f2221d2b44..d05e3d6dd9 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -61,6 +61,106 @@ Design Details\n - The MIDX file format uses a chunk-based approach (similar to the\n commit-graph file) that allows optional data to be added.\n \n+Incremental multi-pack indexes\n+------------------------------\n+\n+As repositories grow in size, it becomes more expensive to write a\n+multi-pack index (MIDX) that includes all packfiles. To accommodate\n+this, the \"incremental multi-pack indexes\" feature allows for combining\n+a \"chain\" of multi-pack indexes.\n+\n+Each individual component of the chain need only contain a small number\n+of packfiles. Appending to the chain does not invalidate earlier parts\n+of the chain, so repositories can control how much time is spent\n+updating the MIDX chain by determining the number of packs in each layer\n+of the MIDX chain.\n+\n+=== Design state\n+\n+At present, the incremental multi-pack indexes feature is missing two\n+important components:\n+\n+ - The ability to rewrite earlier portions of the MIDX chain (i.e., to\n+ \"compact\" some collection of adjacent MIDX layers into a single\n+ MIDX). At present the only supported way of shrinking a MIDX chain\n+ is to rewrite the entire chain from scratch without the `--split`\n+ flag.\n++\n+There are no fundamental limitations that stand in the way of being able\n+to implement this feature. It is omitted from the initial implementation\n+in order to reduce the complexity, but will be added later.\n+\n+ - Support for reachability bitmaps. The classic single MIDX\n+ implementation does support reachability bitmaps (see the section\n+ titled \"multi-pack-index reverse indexes\" in\n+ linkgit:gitformat-pack[5] for more details).\n++\n+As above, there are no fundamental limitations that stand in the way of\n+extending the incremental MIDX format to support reachability bitmaps.\n+The design below specifically takes this into account, and support for\n+reachability bitmaps will be added in a future patch series. It is\n+omitted from this series for the same reason as above.\n++\n+In brief, to support reachability bitmaps with the incremental MIDX\n+feature, the concept of the pseudo-pack order is extended across each\n+layer of the incremental MIDX chain to form a concatenated pseudo-pack\n+order. This concatenation takes place in the same order as the chain\n+itself (in other words, the concatenated pseudo-pack order for a chain\n+`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by\n+the pseudo-pack order for `$H2`, followed by the pseudo-pack order for\n+`$H3`).\n++\n+The layout will then be extended so that each layer of the incremental\n+MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap\n+are offset by the number of objects in the previous layers of the chain.\n+\n+=== File layout\n+\n+Instead of storing a single `multi-pack-index` file (with an optional\n+`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental\n+MIDXs are stored in the following layout:\n+\n+----\n+$GIT_DIR/objects/pack/multi-pack-index.d/\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx\n+----\n+\n+The `multi-pack-index-chain` file contains a list of the incremental\n+MIDX files in the chain, in order. The above example shows a chain whose\n+`multi-pack-index-chain` file would contain the following lines:\n+\n+----\n+$H1\n+$H2\n+$H3\n+----\n+\n+The `multi-pack-index-$H1.midx` file contains the first layer of the\n+multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains\n+the second layer of the chain, and so on.\n+\n+=== Object positions for incremental MIDXs\n+\n+In the original multi-pack-index design, we refer to objects via their\n+lexicographic position (by object IDs) within the repository's singular\n+multi-pack-index. In the incremental multi-pack-index design, we refer\n+to objects via their index into a concatenated lexicographic ordering\n+among each component in the MIDX chain.\n+\n+If `objects_nr()` is a function that returns the number of objects in a\n+given MIDX layer, then the index of an object at lexicographic position\n+`i` within, say, $H3 is defined as:\n+\n+----\n+objects_nr($H2) + objects_nr($H1) + i\n+----\n+\n+(in the C implementation, this is often computed as `i +\n+m->num_objects_in_base`).\n+\n Future Work\n -----------\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496574","messageId":"b810521bdba2d6990e3213c626ca00bfdcaaa4fe.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 02/19] midx: add new fields for incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:29Z","receivedAt":"2024-06-06T23:04:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The incremental MIDX chain feature is designed around the idea of\nindexing into a concatenated lexicographic ordering of object IDs\npresent in the MIDX.\n\nWhen given an object position, the MIDX machinery needs to be able to\nlocate both (a) which MIDX layer contains the given object, and (b) at\nwhat position *within that MIDX layer* that object appears.\n\nTo do this, three new fields are added to the `struct multi_pack_index`:\n\n - struct multi_pack_index *base_midx;\n - uint32_t num_objects_in_base;\n - uint32_t num_packs_in_base;\n\nThese three fields store the pieces of information suggested by their\nrespective field names. In turn, the `num_objects_in_base` and\n`num_packs_in_base` fields are used to crawl backwards along the\n`base_midx` pointer to locate the appropriate position for a given\nobject within the MIDX that contains it.\n\nThe following commits will update various parts of the MIDX machinery\n(as well as their callers from outside of midx.c and midx-write.c) to be\naware and make use of these fields when performing object lookups.\n\nSigned-off-by: Taylor Blau \n---\n midx.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/midx.h b/midx.h\nindex 8554f2d616..020e49f77c 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -63,6 +63,10 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_revindex;\n \tsize_t chunk_revindex_len;\n \n+\tstruct multi_pack_index *base_midx;\n+\tuint32_t num_objects_in_base;\n+\tuint32_t num_packs_in_base;\n+\n \tconst char **pack_names;\n \tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496575","messageId":"9f2cec7aa27b806bedf62826be2a58639717b289.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:32Z","receivedAt":"2024-06-06T23:04:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `nth_midxed_pack_int_id()` takes in a object position in\nMIDX lexicographic order and returns an identifier of the pack from\nwhich that object was selected in the MIDX.\n\nCurrently, the given object position is an index into the lexicographic\norder of objects in a single MIDX. Change this position to instead refer\ninto the concatenated lexicographic order of all MIDXs in a MIDX chain.\n\nThis has two visible effects within the implementation of\n`prepare_midx_pack()`:\n\n - First, the given position is now an index into the concatenated\n lexicographic order of all MIDXs in the order in which they appear\n in the MIDX chain.\n\n - Second the pack ID returned from this function is now also in the\n concatenated order of packs among all layers of the MIDX chain in\n the same order that they appear in the MIDX chain.\n\nTo do this, introduce the first of two general purpose helpers, this one\nbeing `midx_for_object()`. `midx_for_object()` takes a double pointer to\na `struct multi_pack_index` as well as an object `pos` in terms of the\nentire MIDX chain[^1].\n\nThe function chases down the '->base_midx' field until it finds the MIDX\nlayer within the chain that contains the given object. It then:\n\n - modifies the double pointer to point to the containing MIDX, instead\n of the tip of the chain, and\n\n - returns the MIDX-local position[^2] at which the given object can be\n found.\n\nUse this function within `nth_midxed_pack_int_id()` so that the `pos` it\nexpects is now relative to the entire MIDX chain, and that it returns\nthe appropriate pack position for that object.\n\n[^1]: As a reminder, this means that the object is identified among the\n objects contained in all layers of the incremental MIDX chain, not any\n particular layer. For example, consider MIDX chain with two individual\n MIDXs, one with 4 objects and another with 3 objects. If the MIDX with\n 4 objects appears earlier in the chain, then asking for pack \"6\" would\n return the second object in the MIDX with 3 objects.\n\n[^2]: Building on the previous example, asking for object 6 in a MIDX\n chain with (4, 3) objects, respectively, this would set the double\n pointer to point at the MIDX containing three objects, and would\n return an index to the second object within that MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bc4797196f..d5828b48fd 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -240,6 +240,23 @@ void close_midx(struct multi_pack_index *m)\n \tfree(m);\n }\n \n+static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n+{\n+\tstruct multi_pack_index *m = *_m;\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n+\n+\tif (pos >= m->num_objects + m->num_objects_in_base)\n+\t\tdie(_(\"invalid MIDX object position, MIDX is likely corrupt\"));\n+\n+\t*_m = m;\n+\n+\treturn pos - m->num_objects_in_base;\n+}\n+\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n@@ -331,8 +348,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n-\treturn get_be32(m->chunk_object_offsets +\n-\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n+\tpos = midx_for_object(&m, pos);\n+\n+\treturn m->num_packs_in_base + get_be32(m->chunk_object_offsets +\n+\t\t\t\t\t (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n }\n \n int fill_midx_entry(struct repository *r,\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496576","messageId":"97661bb0de966354cc19d37c70d18f324d82f693.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:35Z","receivedAt":"2024-06-06T23:04:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `prepare_midx_pack()` is part of the midx.h API and\nloads the pack identified by the MIDX-local 'pack_int_id'. This patch\nprepares that function to be aware of an incremental MIDX world.\n\nTo do this, introduce the second of the two general purpose helpers\nmentioned in the previous commit. This commit introduces\n`midx_for_pack()`, which is the pack-specific analog of\n`midx_for_object()`, and works in the same fashion.\n\nLike `midx_for_object()`, this function chases down the '->base_midx'\nfield until it finds the MIDX layer within the chain that contains the\ngiven pack.\n\nUse this function within `prepare_midx_pack()` so that the `pack_int_id`\nit expects is now relative to the entire MIDX chain, and that it\nprepares the given pack in the appropriate MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 33 +++++++++++++++++++++++++--------\n 1 file changed, 25 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d5828b48fd..7fa3a1a7f8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -257,20 +257,37 @@ static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n \treturn pos - m->num_objects_in_base;\n }\n \n-int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n+static uint32_t midx_for_pack(struct multi_pack_index **_m,\n+\t\t\t uint32_t pack_int_id)\n {\n-\tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct packed_git *p;\n+\tstruct multi_pack_index *m = *_m;\n+\twhile (m && pack_int_id < m->num_packs_in_base)\n+\t\tm = m->base_midx;\n \n-\tif (pack_int_id >= m->num_packs)\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for pack ID: %\"PRIu32, pack_int_id);\n+\n+\tif (pack_int_id >= m->num_packs + m->num_packs_in_base)\n \t\tdie(_(\"bad pack-int-id: %u (%u total packs)\"),\n-\t\t pack_int_id, m->num_packs);\n+\t\t pack_int_id, m->num_packs + m->num_packs_in_base);\n \n-\tif (m->packs[pack_int_id])\n+\t*_m = m;\n+\n+\treturn pack_int_id - m->num_packs_in_base;\n+}\n+\n+int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\n+\tif (m->packs[local_pack_int_id])\n \t\treturn 0;\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n-\t\t m->pack_names[pack_int_id]);\n+\t\t m->pack_names[local_pack_int_id]);\n \n \tp = add_packed_git(pack_name.buf, pack_name.len, m->local);\n \tstrbuf_release(&pack_name);\n@@ -279,7 +296,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n \t\treturn 1;\n \n \tp->multi_pack_index = 1;\n-\tm->packs[pack_int_id] = p;\n+\tm->packs[local_pack_int_id] = p;\n \tinstall_packed_git(r, p);\n \tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496577","messageId":"a10772585d5fa9dcfd6f5bf5ec6fa81154181144.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:38Z","receivedAt":"2024-06-06T23:04:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `nth_midxed_object_oid()` returns the object ID for a given\nobject position in the MIDX lexicographic order.\n\nTeach this function to instead operate over the concatenated\nlexicographic order defined in an earlier step so that it is able to be\nused with incremental MIDXs.\n\nTo do this, we need to both (a) adjust the bounds check for the given\n'n', as well as record the MIDX-local position after chasing the\n`->base_midx` pointer to find the MIDX which contains that object.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7fa3a1a7f8..cfab7f8113 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -335,9 +335,11 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n)\n {\n-\tif (n >= m->num_objects)\n+\tif (n >= m->num_objects + m->num_objects_in_base)\n \t\treturn NULL;\n \n+\tn = midx_for_object(&m, n);\n+\n \toidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n));\n \treturn oid;\n }\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496578","messageId":"4e960edf8a1018b8425d4d6ae5bb3553bfd38023.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:41Z","receivedAt":"2024-06-06T23:04:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach the function\n`nth_bitmapped_pack()` about incremental MIDXs by translating the given\n`pack_int_id` from the concatenated lexical order to a MIDX-local\nlexical position.\n\nWhen accessing the containing MIDX's array of packs, use the local pack\nID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset\nwhen accessing the data within that chunk.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex cfab7f8113..cdc754af97 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -308,17 +308,19 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n {\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\n \tif (!m->chunk_bitmapped_packs)\n \t\treturn error(_(\"MIDX does not contain the BTMP chunk\"));\n \n \tif (prepare_midx_pack(r, m, pack_int_id))\n \t\treturn error(_(\"could not load bitmapped pack %\"PRIu32), pack_int_id);\n \n-\tbp->p = m->packs[pack_int_id];\n+\tbp->p = m->packs[local_pack_int_id];\n \tbp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs +\n-\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id);\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id);\n \tbp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs +\n-\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id +\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id +\n \t\t\t\t sizeof(uint32_t));\n \tbp->pack_int_id = pack_int_id;\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496579","messageId":"ac10770602e0fc386aa597cd314a718b83d78c52.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 07/19] midx: introduce `bsearch_one_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:45Z","receivedAt":"2024-06-06T23:04:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `bsearch_midx()` function will be extended in a following commit to\nsearch for the location of a given object ID across all MIDXs in a chain\n(or the single non-chain MIDX if no chain is available).\n\nWhile most callers will naturally want to use the updated\n`bsearch_midx()` function, there are a handful of special cases that\nwill want finer control and will only want to search through a single\nMIDX.\n\nFor instance, the object abbreviation code, which cares about object IDs\nnear to where we'd expect to find a match in a MIDX. In that case, we\nwant to look at the nearby matches in each layer of the MIDX chain, not\njust a single one).\n\nSplit the more fine-grained control out into a separate function called\n`bsearch_one_midx()` which searches only a single MIDX.\n\nAt present both `bsearch_midx()` and `bsearch_one_midx()` have identical\nbehavior, but the following commit will rewrite the former to be aware\nof incremental MIDXs for the remaining non-special case callers.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 17 +++++++--\n midx.h | 5 ++-\n object-name.c | 99 +++++++++++++++++++++++++++------------------------\n 3 files changed, 71 insertions(+), 50 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex cdc754af97..1b4a9d5d00 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -327,10 +327,21 @@ int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \treturn 0;\n }\n \n-int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n+int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result)\n {\n-\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n-\t\t\t the_hash_algo->rawsz, result);\n+\tint ret = bsearch_hash(oid->hash, m->chunk_oid_fanout,\n+\t\t\t m->chunk_oid_lookup, the_hash_algo->rawsz,\n+\t\t\t result);\n+\tif (result)\n+\t\t*result += m->num_objects_in_base;\n+\treturn ret;\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result)\n+{\n+\t\treturn bsearch_one_midx(oid, m, result);\n }\n \n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/midx.h b/midx.h\nindex 020e49f77c..46c53d69ff 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -90,7 +90,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n-int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result);\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/object-name.c b/object-name.c\nindex 523af6f64f..3307d5200c 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -132,28 +132,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char\n static void unique_in_midx(struct multi_pack_index *m,\n \t\t\t struct disambiguate_state *ds)\n {\n-\tuint32_t num, i, first = 0;\n-\tconst struct object_id *current = NULL;\n-\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n-\t\tds->repo->hash_algo->hexsz : ds->len;\n-\tnum = m->num_objects;\n+\tfor (; m; m = m->base_midx) {\n+\t\tuint32_t num, i, first = 0;\n+\t\tconst struct object_id *current = NULL;\n+\t\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n+\t\t\tds->repo->hash_algo->hexsz : ds->len;\n \n-\tif (!num)\n-\t\treturn;\n+\t\tnum = m->num_objects + m->num_objects_in_base;\n \n-\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\t\tif (!num)\n+\t\t\tcontinue;\n \n-\t/*\n-\t * At this point, \"first\" is the location of the lowest object\n-\t * with an object name that could match \"bin_pfx\". See if we have\n-\t * 0, 1 or more objects that actually match(es).\n-\t */\n-\tfor (i = first; i < num && !ds->ambiguous; i++) {\n-\t\tstruct object_id oid;\n-\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n-\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n-\t\t\tbreak;\n-\t\tupdate_candidates(ds, current);\n+\t\tbsearch_one_midx(&ds->bin_pfx, m, &first);\n+\n+\t\t/*\n+\t\t * At this point, \"first\" is the location of the lowest\n+\t\t * object with an object name that could match\n+\t\t * \"bin_pfx\". See if we have 0, 1 or more objects that\n+\t\t * actually match(es).\n+\t\t */\n+\t\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\t\tstruct object_id oid;\n+\t\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n+\t\t\t\tbreak;\n+\t\t\tupdate_candidates(ds, current);\n+\t\t}\n \t}\n }\n \n@@ -706,37 +710,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED,\n static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n-\tint match = 0;\n-\tuint32_t num, first = 0;\n-\tstruct object_id oid;\n-\tconst struct object_id *mad_oid;\n+\tfor (; m; m = m->base_midx) {\n+\t\tint match = 0;\n+\t\tuint32_t num, first = 0;\n+\t\tstruct object_id oid;\n+\t\tconst struct object_id *mad_oid;\n \n-\tif (!m->num_objects)\n-\t\treturn;\n+\t\tif (!m->num_objects)\n+\t\t\tcontinue;\n \n-\tnum = m->num_objects;\n-\tmad_oid = mad->oid;\n-\tmatch = bsearch_midx(mad_oid, m, &first);\n+\t\tnum = m->num_objects + m->num_objects_in_base;\n+\t\tmad_oid = mad->oid;\n+\t\tmatch = bsearch_one_midx(mad_oid, m, &first);\n \n-\t/*\n-\t * first is now the position in the packfile where we would insert\n-\t * mad->hash if it does not exist (or the position of mad->hash if\n-\t * it does exist). Hence, we consider a maximum of two objects\n-\t * nearby for the abbreviation length.\n-\t */\n-\tmad->init_len = 0;\n-\tif (!match) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first))\n-\t\t\textend_abbrev_len(&oid, mad);\n-\t} else if (first < num - 1) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n-\t\t\textend_abbrev_len(&oid, mad);\n+\t\t/*\n+\t\t * first is now the position in the packfile where we\n+\t\t * would insert mad->hash if it does not exist (or the\n+\t\t * position of mad->hash if it does exist). Hence, we\n+\t\t * consider a maximum of two objects nearby for the\n+\t\t * abbreviation length.\n+\t\t */\n+\t\tmad->init_len = 0;\n+\t\tif (!match) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t} else if (first < num - 1) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t}\n+\t\tif (first > 0) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t}\n+\t\tmad->init_len = mad->cur_len;\n \t}\n-\tif (first > 0) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n-\t\t\textend_abbrev_len(&oid, mad);\n-\t}\n-\tmad->init_len = mad->cur_len;\n }\n \n static void find_abbrev_len_for_pack(struct packed_git *p,\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496580","messageId":"f3c37c83696deb03ad06c59df393e9b66a72ddab.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 08/19] midx: teach `bsearch_midx()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:48Z","receivedAt":"2024-06-06T23:04:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the special cases callers of `bsearch_midx()` have been dealt\nwith, teach `bsearch_midx()` to handle incremental MIDX chains.\n\nThe incremental MIDX-aware version of `bsearch_midx()` works by\nrepeatedly searching for a given OID in each layer along the\n`->base_midx` pointer, stopping either when an exact match is found, or\nthe end of the chain is reached.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 1b4a9d5d00..7c4f58f7f1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -341,7 +341,10 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result)\n {\n-\t\treturn bsearch_one_midx(oid, m, result);\n+\tfor (; m; m = m->base_midx)\n+\t\tif (bsearch_one_midx(oid, m, result))\n+\t\t\treturn 1;\n+\treturn 0;\n }\n \n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496581","messageId":"c28ebf22c0819dbf6b65b28654362642880b6bea.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:51Z","receivedAt":"2024-06-06T23:04:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach the function\n`nth_midxed_offset()` about incremental MIDXs.\n\nThe given object `pos` is used to find the containing MIDX, and\ntranslated back into a MIDX-local position by assigning the return value\nof `midx_for_object()` to it.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 7c4f58f7f1..d351dbb7e0 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -365,6 +365,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n \n+\tpos = midx_for_object(&m, pos);\n+\n \toffset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;\n \toffset32 = get_be32(offset_data + sizeof(uint32_t));\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496582","messageId":"984ca9dc2dbc1777586424243b8a6c1aeb565eec.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:54Z","receivedAt":"2024-06-06T23:04:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, teach the `fill_midx_entry()`\nfunction to work in a incremental MIDX-aware fashion.\n\nThis function, unlike others which accept an index into either the\nlexical order of objects or packs, takes in an object_id, and attempts\nto fill a caller-provided 'struct pack_entry' with the remaining pieces\nof information about that object from the MIDX.\n\nThe function uses `bsearch_midx()` which fills out the frame-local 'pos'\nvariable, recording the given object_id's lexical position within the\nMIDX chain, if found (if no matching object ID was found, we'll return\nimmediately without filling out the `pack_entry` structure).\n\nOnce given that position, we jump back through the `->base_midx` pointer\nto ensure that our `m` points at the MIDX layer which contains the given\nobject_id (and not an ancestor or descendant of it in the chain). Note\nthat we can drop the bounds check \"if (pos >= m->num_objects)\" because\n`midx_for_object()` performs this check for us.\n\nAfter that point, we only need to make two special considerations within\nthis function:\n\n - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()`\n is a position in the concatenated lexical order of packs, so we must\n ensure that we subtract `m->num_packs_in_base` before accessing the\n MIDX-local `packs` array.\n\n - Second, we must avoid translating the `pos` back to a MIDX-local\n index, since we use it as an argument to `nth_midxed_offset()` which\n expects a position relative to the concatenated lexical order of\n objects.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d351dbb7e0..c3802354e3 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -403,14 +403,12 @@ int fill_midx_entry(struct repository *r,\n \tif (!bsearch_midx(oid, m, &pos))\n \t\treturn 0;\n \n-\tif (pos >= m->num_objects)\n-\t\treturn 0;\n-\n+\tmidx_for_object(&m, pos);\n \tpack_int_id = nth_midxed_pack_int_id(m, pos);\n \n \tif (prepare_midx_pack(r, m, pack_int_id))\n \t\treturn 0;\n-\tp = m->packs[pack_int_id];\n+\tp = m->packs[pack_int_id - m->num_packs_in_base];\n \n \t/*\n \t* We are about to tell the caller where they can locate the\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496583","messageId":"02b1d39fd2aa0da5c2c2591aae1268498e70aa05.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 11/19] midx: remove unused `midx_locate_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:04:57Z","receivedAt":"2024-06-06T23:05:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14)\nintroduced `midx_locate_pack()`, which was described at the time as a\ncomplement to the function `midx_contains_pack()` which allowed\ncallers to determine where in the MIDX lexical order a pack appeared, as\nopposed to whether or not it was simply contained.\n\n307d75bbe6 suggests that future patches would be added which would\nintroduce callers for this new function, but none ever were, meaning the\nfunction has gone unused since its introduction.\n\nClean this up by in effect reverting 307d75bbe6, which removes the\nunused functions and inlines its definition back into\n`midx_contains_pack()`.\n\n(Looking back through the list archives when 307d75bbe6 was written,\nthis was in preparation for this[1] patch from back when we had the\nconcept of \"disjoint\" packs while developing multi-pack verbatim reuse.\nThat concept was abandoned before the series was merged, but I never\ndropped what would become 307d75bbe6 from the series, leading to the\nstate prior to this commit).\n\n[1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 13 ++-----------\n midx.h | 2 --\n 2 files changed, 2 insertions(+), 13 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex c3802354e3..186d8344dc 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -462,8 +462,7 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n-\t\t uint32_t *pos)\n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -474,11 +473,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \n \t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n-\t\tif (!cmp) {\n-\t\t\tif (pos)\n-\t\t\t\t*pos = mid;\n+\t\tif (!cmp)\n \t\t\treturn 1;\n-\t\t}\n \t\tif (cmp > 0) {\n \t\t\tfirst = mid + 1;\n \t\t\tcontinue;\n@@ -489,11 +485,6 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \treturn 0;\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n-{\n-\treturn midx_locate_pack(m, idx_or_pack_name, NULL);\n-}\n-\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\ndiff --git a/midx.h b/midx.h\nindex 46c53d69ff..86af7dfc5e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -102,8 +102,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n-int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n-\t\t uint32_t *pos);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496584","messageId":"2288683674ba6858d1ea217dc7f84bff269024b1.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:01Z","receivedAt":"2024-06-06T23:05:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle\nhas been cleaned up, teach the former about how to operate in an\nincremental MIDX-aware world in a similar fashion as in previous\ncommits.\n\nInstead of using either of the two `midx_for_object()` or\n`midx_for_pack()` helpers, this function is split into two: one that\ndetermines whether a pack is contained in a single MIDX, and another\nwhich calls the former in a loop over all MIDXs.\n\nThis approach does not require that we change any of the implementation\nin what is now `midx_contains_pack_1()` as it still operates over a\nsingle MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 186d8344dc..564e922533 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -462,7 +462,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+static int midx_contains_pack_1(struct multi_pack_index *m,\n+\t\t\t\tconst char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -485,6 +486,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \treturn 0;\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+{\n+\tfor (; m; m = m->base_midx)\n+\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496585","messageId":"53b71c6514dac9abe7472d95842b7be5675b8a4e.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:04Z","receivedAt":"2024-06-06T23:05:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `midx_preferred_pack()` is used to determine the identity\nof the preferred pack, which is the identity of a unique pack within\nthe MIDX which is used as a tie-breaker when selecting from which pack\nto represent an object that appears in multiple packs within the MIDX.\n\nHistorically we have said that the MIDX's preferred pack has the unique\nproperty that all objects from that pack are represented in the MIDX.\nBut that isn't quite true: a more precise statement would be that all\nobjects from that pack *which appear in the MIDX* are selected from that\npack.\n\nThis helps us extend the concept of preferred packs across a MIDX chain,\nwhere some object(s) in the preferred pack may appear in other packs\nin an earlier MIDX layer, in which case those object(s) will not appear\nin a subsequent MIDX layer from either the preferred pack or any other\npack.\n\nExtend the concept of preferred packs by using the pack which represents\nthe object at the first position in MIDX pseudo-pack order belonging to\nthe current MIDX layer (i.e., at position 'm->num_objects_in_base').\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 564e922533..cb7b623b5d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -497,13 +497,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\n+\t\tuint32_t midx_pos;\n \t\tif (load_midx_revindex(m) < 0) {\n \t\t\tm->preferred_pack_idx = -2;\n \t\t\treturn -1;\n \t\t}\n \n-\t\tm->preferred_pack_idx =\n-\t\t\tnth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\t\tmidx_pos = pack_pos_to_midx(m, m->num_objects_in_base);\n+\n+\t\tm->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos);\n+\n \t} else if (m->preferred_pack_idx == -2)\n \t\treturn -1; /* no revindex */\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496586","messageId":"e0e30adc6a0c625b372a4fea314b948e1e175eae.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:07Z","receivedAt":"2024-06-06T23:05:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `midx_fanout_add_midx_fanout()` is used to help construct\nthe fanout table when generating a MIDX by reusing data from an existing\nMIDX.\n\nPrepare this function to work with incremental MIDXs by making a few\nchanges:\n\n - The bounds checks need to be adjusted to start object lookups taking\n into account the number of objects in the previous MIDX layer (i.e.,\n by starting the lookups at position `m->num_objects_in_base` instead\n of position 0).\n\n - Likewise, the bounds checks need to end at `m->num_objects_in_base`\n objects after `m->num_objects`.\n\n - Finally, `midx_fanout_add_midx_fanout()` needs to recur on earlier\n MIDX layers when dealing with an incremental MIDX chain by calling\n itself when given a MIDX with a non-NULL `base_midx`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 55a6b63bac..b148ee443a 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -180,7 +180,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n-\tif (pos >= m->num_objects)\n+\tif (pos >= m->num_objects + m->num_objects_in_base)\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n@@ -231,12 +231,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n \t\t\t\t\tuint32_t cur_fanout,\n \t\t\t\t\tint preferred_pack)\n {\n-\tuint32_t start = 0, end;\n+\tuint32_t start = m->num_objects_in_base, end;\n \tuint32_t cur_object;\n \n+\tif (m->base_midx)\n+\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n+\t\t\t\t\t preferred_pack);\n+\n \tif (cur_fanout)\n-\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n-\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\t\tstart += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\tend = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);\n \n \tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\tif ((preferred_pack > -1) &&\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496587","messageId":"28579fa29266b65c7c6b915678dff35f50cc051d.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 15/19] midx: support reading incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:10Z","receivedAt":"2024-06-06T23:05:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the MIDX machinery's internals have been taught to understand\nincremental MIDXs over the previous handful of commits, the MIDX\nmachinery itself can begin reading incremental MIDXs.\n\n(Note that while the on-disk format for incremental MIDXs has been\ndefined, the writing end has not been implemented. This will take place\nin the commit after next.)\n\nThe core of this change involves following the order specified in the\nMIDX chain and opening up MIDXs in the chain one-by-one, adding them to\nthe previous layer's `->base_midx` pointer at each step.\n\nIn order to implement this, the `load_multi_pack_index()` function is\ntaught to call a new `load_multi_pack_index_chain()` function if loading\na non-incremental MIDX failed via `load_multi_pack_index_one()`.\n\nWhen loading a MIDX chain, `load_midx_chain_fd_st()` reads each line in\nthe file one-by-one and dispatches calls to\n`load_multi_pack_index_one()` to read each layer of the MIDX chain. When\na layer was successfully read, it is added to the MIDX chain by calling\n`add_midx_to_chain()` which validates the contents of the `BASE` chunk,\nperforms some bounds checks on the number of combined packs and objects,\nand attaches the new MIDX by assigning its `base_midx` pointer to the\nexisting part of the chain.\n\nAs a supplement to this, introduce a new mode in the test-read-midx\ntest-tool which allows us to read the information for a specific MIDX in\nthe chain by specifying its trailing checksum via the command-line\narguments like so:\n\n $ test-tool read-midx .git/objects [checksum]\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 184 +++++++++++++++++++++++++++++++++++---\n midx.h | 7 ++\n packfile.c | 5 +-\n t/helper/test-read-midx.c | 24 +++--\n 4 files changed, 201 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex cb7b623b5d..ac44fcefc2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -89,7 +89,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n \n-struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n+static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir,\n+\t\t\t\t\t\t\t const char *midx_name,\n+\t\t\t\t\t\t\t int local)\n {\n \tstruct multi_pack_index *m = NULL;\n \tint fd;\n@@ -97,31 +99,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tsize_t midx_size;\n \tvoid *midx_map = NULL;\n \tuint32_t hash_version;\n-\tstruct strbuf midx_name = STRBUF_INIT;\n \tuint32_t i;\n \tconst char *cur_pack_name;\n \tstruct chunkfile *cf = NULL;\n \n-\tget_midx_filename(&midx_name, object_dir);\n-\n-\tfd = git_open(midx_name.buf);\n+\tfd = git_open(midx_name);\n \n \tif (fd < 0)\n \t\tgoto cleanup_fail;\n \tif (fstat(fd, &st)) {\n-\t\terror_errno(_(\"failed to read %s\"), midx_name.buf);\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n \n \tmidx_size = xsize_t(st.st_size);\n \n \tif (midx_size < MIDX_MIN_SIZE) {\n-\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name.buf);\n+\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n \n-\tstrbuf_release(&midx_name);\n-\n \tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \n@@ -211,7 +208,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n cleanup_fail:\n \tfree(m);\n-\tstrbuf_release(&midx_name);\n \tfree_chunkfile(cf);\n \tif (midx_map)\n \t\tmunmap(midx_map, midx_size);\n@@ -220,6 +216,173 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \treturn NULL;\n }\n \n+void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir)\n+{\n+\tstrbuf_addf(buf, \"%s/pack/multi-pack-index.d\", object_dir);\n+}\n+\n+void get_midx_chain_filename(struct strbuf *buf, const char *object_dir)\n+{\n+\tget_midx_chain_dirname(buf, object_dir);\n+\tstrbuf_addstr(buf, \"/multi-pack-index-chain\");\n+}\n+\n+void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n+\t\t\t\t const unsigned char *hash, const char *ext)\n+{\n+\tget_midx_chain_dirname(buf, object_dir);\n+\tstrbuf_addf(buf, \"/multi-pack-index-%s.%s\", hash_to_hex(hash), ext);\n+}\n+\n+static int open_multi_pack_index_chain(const char *chain_file,\n+\t\t\t\t int *fd, struct stat *st)\n+{\n+\t*fd = git_open(chain_file);\n+\tif (*fd < 0)\n+\t\treturn 0;\n+\tif (fstat(*fd, st)) {\n+\t\tclose(*fd);\n+\t\treturn 0;\n+\t}\n+\tif (st->st_size < the_hash_algo->hexsz) {\n+\t\tclose(*fd);\n+\t\tif (!st->st_size) {\n+\t\t\t/* treat empty files the same as missing */\n+\t\t\terrno = ENOENT;\n+\t\t} else {\n+\t\t\twarning(_(\"multi-pack-index chain file too small\"));\n+\t\t\terrno = EINVAL;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+static int add_midx_to_chain(struct multi_pack_index *midx,\n+\t\t\t struct multi_pack_index *midx_chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tif (midx_chain) {\n+\t\tif (unsigned_add_overflows(midx_chain->num_packs,\n+\t\t\t\t\t midx_chain->num_packs_in_base)) {\n+\t\t\twarning(_(\"pack count in base MIDX too high: %\"PRIuMAX),\n+\t\t\t\t(uintmax_t)midx_chain->num_packs_in_base);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (unsigned_add_overflows(midx_chain->num_objects,\n+\t\t\t\t\t midx_chain->num_objects_in_base)) {\n+\t\t\twarning(_(\"object count in base MIDX too high: %\"PRIuMAX),\n+\t\t\t\t(uintmax_t)midx_chain->num_objects_in_base);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tmidx->num_packs_in_base = midx_chain->num_packs +\n+\t\t\tmidx_chain->num_packs_in_base;\n+\t\tmidx->num_objects_in_base = midx_chain->num_objects +\n+\t\t\tmidx_chain->num_objects_in_base;\n+\t}\n+\n+\tmidx->base_midx = midx_chain;\n+\tmidx->has_chain = 1;\n+\n+\treturn 1;\n+}\n+\n+static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir,\n+\t\t\t\t\t\t int local,\n+\t\t\t\t\t\t int fd, struct stat *st,\n+\t\t\t\t\t\t int *incomplete_chain)\n+{\n+\tstruct multi_pack_index *midx_chain = NULL;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id *layers = NULL;\n+\tint valid = 1;\n+\tuint32_t i, count;\n+\tFILE *fp = xfdopen(fd, \"r\");\n+\n+\tcount = st->st_size / (the_hash_algo->hexsz + 1);\n+\tCALLOC_ARRAY(layers, count);\n+\n+\tfor (i = 0; i < count; i++) {\n+\t\tstruct multi_pack_index *m;\n+\n+\t\tif (strbuf_getline_lf(&buf, fp) == EOF)\n+\t\t\tbreak;\n+\n+\t\tif (get_oid_hex(buf.buf, &layers[i])) {\n+\t\t\twarning(_(\"invalid multi-pack-index chain: line '%s' \"\n+\t\t\t\t \"not a hash\"),\n+\t\t\t\tbuf.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tvalid = 0;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tget_split_midx_filename_ext(&buf, object_dir, layers[i].hash,\n+\t\t\t\t\t MIDX_EXT_MIDX);\n+\t\tm = load_multi_pack_index_one(object_dir, buf.buf, local);\n+\n+\t\tif (m) {\n+\t\t\tif (add_midx_to_chain(m, midx_chain, layers, i)) {\n+\t\t\t\tmidx_chain = m;\n+\t\t\t\tvalid = 1;\n+\t\t\t} else {\n+\t\t\t\tclose_midx(m);\n+\t\t\t}\n+\t\t}\n+\t\tif (!valid) {\n+\t\t\twarning(_(\"unable to find all multi-pack index files\"));\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree(layers);\n+\tfclose(fp);\n+\tstrbuf_release(&buf);\n+\n+\t*incomplete_chain = !valid;\n+\treturn midx_chain;\n+}\n+\n+static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir,\n+\t\t\t\t\t\t\t int local)\n+{\n+\tstruct strbuf chain_file = STRBUF_INIT;\n+\tstruct stat st;\n+\tint fd;\n+\tstruct multi_pack_index *m = NULL;\n+\n+\tget_midx_chain_filename(&chain_file, object_dir);\n+\tif (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) {\n+\t\tint incomplete;\n+\t\t/* ownership of fd is taken over by load function */\n+\t\tm = load_midx_chain_fd_st(object_dir, local, fd, &st,\n+\t\t\t\t\t &incomplete);\n+\t}\n+\n+\tstrbuf_release(&chain_file);\n+\treturn m;\n+}\n+\n+struct multi_pack_index *load_multi_pack_index(const char *object_dir,\n+\t\t\t\t\t int local)\n+{\n+\tstruct strbuf midx_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m;\n+\n+\tget_midx_filename(&midx_name, object_dir);\n+\n+\tm = load_multi_pack_index_one(object_dir, midx_name.buf, local);\n+\tif (!m)\n+\t\tm = load_multi_pack_index_chain(object_dir, local);\n+\n+\tstrbuf_release(&midx_name);\n+\n+\treturn m;\n+}\n+\n void close_midx(struct multi_pack_index *m)\n {\n \tuint32_t i;\n@@ -228,6 +391,7 @@ void close_midx(struct multi_pack_index *m)\n \t\treturn;\n \n \tclose_midx(m->next);\n+\tclose_midx(m->base_midx);\n \n \tmunmap((unsigned char *)m->data, m->data_len);\n \ndiff --git a/midx.h b/midx.h\nindex 86af7dfc5e..94de16a8c4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -24,6 +24,7 @@ struct bitmapped_pack;\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n+#define MIDX_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n@@ -50,6 +51,7 @@ struct multi_pack_index {\n \tint preferred_pack_idx;\n \n \tint local;\n+\tint has_chain;\n \n \tconst unsigned char *chunk_pack_names;\n \tsize_t chunk_pack_names_len;\n@@ -80,11 +82,16 @@ struct multi_pack_index {\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n+#define MIDX_EXT_MIDX \"midx\"\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\n void get_midx_filename_ext(struct strbuf *out, const char *object_dir,\n \t\t\t const unsigned char *hash, const char *ext);\n+void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir);\n+void get_midx_chain_filename(struct strbuf *buf, const char *object_dir);\n+void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n+\t\t\t\t const unsigned char *hash, const char *ext);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\ndiff --git a/packfile.c b/packfile.c\nindex d4df7fdeea..85f0345435 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -878,7 +878,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!report_garbage)\n \t\treturn;\n \n-\tif (!strcmp(file_name, \"multi-pack-index\"))\n+\tif (!strcmp(file_name, \"multi-pack-index\") ||\n+\t !strcmp(file_name, \"multi-pack-index.d\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n \t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n@@ -1062,7 +1063,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \tprepare_packed_git(r);\n \tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n \t\tuint32_t i;\n-\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 4acae41bb9..f9148328e3 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -7,8 +7,10 @@\n #include \"packfile.h\"\n #include \"setup.h\"\n #include \"gettext.h\"\n+#include \"pack-revindex.h\"\n \n-static int read_midx_file(const char *object_dir, int show_objects)\n+static int read_midx_file(const char *object_dir, const char *checksum,\n+\t\t\t int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -19,6 +21,13 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \tif (!m)\n \t\treturn 1;\n \n+\tif (checksum) {\n+\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n+\t\t\tm = m->base_midx;\n+\t\tif (!m)\n+\t\t\treturn 1;\n+\t}\n+\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t m->signature,\n \t m->version,\n@@ -52,7 +61,8 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \t\tstruct pack_entry e;\n \n \t\tfor (i = 0; i < m->num_objects; i++) {\n-\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tnth_midxed_object_oid(&oid, m,\n+\t\t\t\t\t i + m->num_objects_in_base);\n \t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n \n \t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n@@ -109,7 +119,7 @@ static int read_midx_bitmapped_packs(const char *object_dir)\n \tif (!midx)\n \t\treturn 1;\n \n-\tfor (i = 0; i < midx->num_packs; i++) {\n+\tfor (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) {\n \t\tif (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)\n \t\t\treturn 1;\n \n@@ -125,16 +135,16 @@ static int read_midx_bitmapped_packs(const char *object_dir)\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n+\tif (!(argc == 2 || argc == 3 || argc == 4))\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n-\t\treturn read_midx_file(argv[2], 1);\n+\t\treturn read_midx_file(argv[2], argv[3], 1);\n \telse if (!strcmp(argv[1], \"--checksum\"))\n \t\treturn read_midx_checksum(argv[2]);\n \telse if (!strcmp(argv[1], \"--preferred-pack\"))\n \t\treturn read_midx_preferred_pack(argv[2]);\n \telse if (!strcmp(argv[1], \"--bitmap\"))\n \t\treturn read_midx_bitmapped_packs(argv[2]);\n-\treturn read_midx_file(argv[1], 0);\n+\treturn read_midx_file(argv[1], argv[2], 0);\n }\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496588","messageId":"f729ed76121f8f38f71d5811cdeca0045e60ce8b.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 16/19] midx: implement verification support for incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:13Z","receivedAt":"2024-06-06T23:05:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the verification implementation used by `git multi-pack-index\nverify` to perform verification for incremental MIDX chains by\nindependently validating each layer within the chain.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 47 ++++++++++++++++++++++++++++++-----------------\n midx.h | 2 ++\n 2 files changed, 32 insertions(+), 17 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex ac44fcefc2..ae3e30a062 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -467,6 +467,13 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \treturn 0;\n }\n \n+struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n+\t\t\t\t uint32_t pack_int_id)\n+{\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\treturn m->packs[local_pack_int_id];\n+}\n+\n #define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n \n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n@@ -814,6 +821,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \tuint32_t i;\n \tstruct progress *progress = NULL;\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\tstruct multi_pack_index *curr;\n \tverify_midx_error = 0;\n \n \tif (!m) {\n@@ -836,8 +844,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Looking for referenced packfiles\"),\n-\t\t\t\t\t m->num_packs);\n-\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\t\t\t m->num_packs + m->num_packs_in_base);\n+\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\tif (prepare_midx_pack(r, m, i))\n \t\t\tmidx_report(\"failed to load pack in position %d\", i);\n \n@@ -857,17 +865,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Verifying OID order in multi-pack-index\"),\n \t\t\t\t\t\t m->num_objects - 1);\n-\tfor (i = 0; i < m->num_objects - 1; i++) {\n-\t\tstruct object_id oid1, oid2;\n \n-\t\tnth_midxed_object_oid(&oid1, m, i);\n-\t\tnth_midxed_object_oid(&oid2, m, i + 1);\n+\tfor (curr = m; curr; curr = curr->base_midx) {\n+\t\tfor (i = 0; i < m->num_objects - 1; i++) {\n+\t\t\tstruct object_id oid1, oid2;\n \n-\t\tif (oidcmp(&oid1, &oid2) >= 0)\n-\t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n-\t\t\t\t i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n+\t\t\tnth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i);\n+\t\t\tnth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1);\n \n-\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\t\tif (oidcmp(&oid1, &oid2) >= 0)\n+\t\t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n+\t\t\t\t\t i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n+\n+\t\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \n@@ -877,8 +888,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t * each of the objects and only require 1 packfile to be open at a\n \t * time.\n \t */\n-\tALLOC_ARRAY(pairs, m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n+\tALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base);\n+\tfor (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {\n \t\tpairs[i].pos = i;\n \t\tpairs[i].pack_int_id = nth_midxed_pack_int_id(m, i);\n \t}\n@@ -892,16 +903,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Verifying object offsets\"), m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n+\tfor (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {\n \t\tstruct object_id oid;\n \t\tstruct pack_entry e;\n \t\toff_t m_offset, p_offset;\n \n \t\tif (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id &&\n-\t\t m->packs[pairs[i-1].pack_int_id])\n-\t\t{\n-\t\t\tclose_pack_fd(m->packs[pairs[i-1].pack_int_id]);\n-\t\t\tclose_pack_index(m->packs[pairs[i-1].pack_int_id]);\n+\t\t m->packs[pairs[i-1].pack_int_id]) {\n+\t\t\tuint32_t pack_int_id = pairs[i-1].pack_int_id;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tclose_pack_fd(p);\n+\t\t\tclose_pack_index(p);\n \t\t}\n \n \t\tnth_midxed_object_oid(&oid, m, pairs[i].pos);\ndiff --git a/midx.h b/midx.h\nindex 94de16a8c4..9d30935589 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -95,6 +95,8 @@ void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n+struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n+\t\t\t\t uint32_t pack_int_id);\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496589","messageId":"1609b8611f073e2954ec5fca875388cbeaecccdb.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:16Z","receivedAt":"2024-06-06T23:05:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Two years ago, commit ff1e653c8e2 (midx: respect\n'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new\nenvironment variable which caused the test suite to write MIDX bitmaps\nafter any 'git repack' invocation.\n\nAt the time, this was done to help flush out any bugs with MIDX bitmaps\nthat weren't explicitly covered in the t5326-multi-pack-bitmap.sh\nscript.\n\nTwo years later, that flag has served us well and is no longer providing\nmeaningful coverage, as the script in t5326 has matured substantially\nand covers many more interesting cases than it did back when ff1e653c8e2\nwas originally written.\n\nRemove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable\nas it is no longer serving a useful purpose. More importantly, removing\nthis variable clears the way for us to introduce a new one to help\nsimilarly flush out bugs related to incremental MIDX chains.\n\nBecause these incremental MIDX chains are (for now) incompatible with\nMIDX bitmaps, we cannot have both.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 12 ++----------\n ci/run-build-and-tests.sh | 1 -\n midx.h | 2 --\n t/README | 4 ----\n t/t0410-partial-clone.sh | 2 --\n t/t5310-pack-bitmaps.sh | 4 ----\n t/t5319-multi-pack-index.sh | 3 +--\n t/t5326-multi-pack-bitmaps.sh | 3 +--\n t/t5327-multi-pack-bitmaps-rev.sh | 5 ++---\n t/t7700-repack.sh | 21 +++++++--------------\n 10 files changed, 13 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 58ad82dd97..e2fec16389 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1217,10 +1217,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!write_midx &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n-\t} else if (write_bitmaps &&\n-\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n-\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n-\t\twrite_bitmaps = 0;\n \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0 && !write_midx;\n@@ -1518,12 +1514,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (run_update_server_info)\n \t\tupdate_server_info(0);\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n-\t\tunsigned flags = 0;\n-\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n-\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n-\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n-\t}\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, 0);\n \n cleanup:\n \tstring_list_clear(&names, 1);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 98dda42045..e6fd68630c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -25,7 +25,6 @@ linux-TEST-vars)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n-\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\ndiff --git a/midx.h b/midx.h\nindex 9d30935589..3714cad2cc 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,8 +29,6 @@ struct bitmapped_pack;\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n-#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n-\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex d9e0e07506..e8a11926e4 100644\n--- a/t/README\n+++ b/t/README\n@@ -469,10 +469,6 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n-'--bitmap' option on all invocations of 'git multi-pack-index write',\n-and ignores pack-objects' '--write-bitmap-index'.\n-\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 7797391c03..f6c58d80dd 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -4,8 +4,6 @@ test_description='partial clone'\n \n . ./test-lib.sh\n \n-# missing promisor objects cause repacks which write bitmaps to fail\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n # When enabled, some commands will write commit-graphs. This causes fsck\n # to fail when delete_object() is called because fsck will attempt to\n # verify the out-of-sync commit graph.\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex d7fd71360e..a6de7c5764 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n-# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n-# their place.\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n-\n # Likewise, allow individual tests to control whether or not they use\n # the boundary-based traversal.\n sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 10d2a6bf92..6e9ee23398 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -600,8 +600,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex cc7220b6c0..dff3b26849 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -4,10 +4,9 @@ test_description='exercise basic multi-pack bitmap functionality'\n . ./test-lib.sh\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n \n-# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# We'll be writing our own MIDX, so avoid getting confused by the\n # automatic ones.\n GIT_TEST_MULTI_PACK_INDEX=0\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n # This test exercise multi-pack bitmap functionality where the object order is\n # stored and read from a special chunk within the MIDX, so use the default\ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex e65e311cd7..23db949c20 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -5,10 +5,9 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)'\n . ./test-lib.sh\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n \n-# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n-# automatic ones.\n+# We'll be writing our own MIDX, so avoid getting confused by the automatic\n+# ones.\n GIT_TEST_MULTI_PACK_INDEX=0\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n # Unlike t5326, this test exercise multi-pack bitmap functionality where the\n # object order is stored in a separate .rev file.\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 127efe99f8..8f34f05087 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -70,14 +70,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n+\tgit repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -284,8 +283,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o\n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n \trm -f bare.git/objects/pack/*.bitmap &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad &&\n+\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -296,8 +294,7 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -308,8 +305,7 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad 2>stderr &&\n+\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -320,8 +316,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad 2>stderr &&\n+\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -342,9 +337,7 @@ test_expect_success 'repacking with a filter works' '\n '\n \n test_expect_success '--filter fails with --write-bitmap-index' '\n-\ttest_must_fail \\\n-\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+\ttest_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n '\n \n test_expect_success 'repacking with two filters works' '\n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496590","messageId":"76154308d1d2ea2439b25f4efc359f0820afa655.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 18/19] t/t5313-pack-bounds-checks.sh: prepare for sub-directories","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:19Z","receivedAt":"2024-06-06T23:05:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for sub-directories to appear in $GIT_DIR/objects/pack by\nadjusting the copy, remove, and chmod invocations to perform their\nbehavior recursively.\n\nThis prepares us for the new $GIT_DIR/objects/pack/multi-pack-index.d\ndirectory which will be added in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n t/t5313-pack-bounds-checks.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh\nindex ceaa6700a2..86fc73f9fb 100755\n--- a/t/t5313-pack-bounds-checks.sh\n+++ b/t/t5313-pack-bounds-checks.sh\n@@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true\n \n clear_base () {\n \ttest_when_finished 'restore_base' &&\n-\trm -f $base\n+\trm -r -f $base\n }\n \n restore_base () {\n-\tcp base-backup/* .git/objects/pack/\n+\tcp -r base-backup/* .git/objects/pack/\n }\n \n do_pack () {\n@@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' '\n \tgit commit -m base &&\n \tgit repack -ad &&\n \tbase=$(echo .git/objects/pack/*) &&\n-\tchmod +w $base &&\n+\tchmod -R +w $base &&\n \tmkdir base-backup &&\n-\tcp $base base-backup/ &&\n+\tcp -r $base base-backup/ &&\n \tobject=$(git rev-parse HEAD:file)\n '\n \n-- \n2.45.2.437.gecb9450a0e\n\n"},{"id":"496591","messageId":"97b3ea84b92c543c7bd15a2f054d0d3af1b34c67.1717715060.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH 19/19] midx: implement support for writing incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-06T23:05:22Z","receivedAt":"2024-06-06T23:05:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the rest of the MIDX subsystem and relevant callers have been\nupdated to learn about how to read and process incremental MIDX chains,\nlet's finally update the implementation in `write_midx_internal()` to be\nable to write incremental MIDX chains.\n\nThis new feature is available behind the `--incremental` option for the\n`multi-pack-index` builtin, like so:\n\n $ git multi-pack-index write --incremental\n\nThe implementation for doing so is relatively straightforward, and boils\ndown to a handful of different kinds of changes implemented in this\npatch:\n\n - The `compute_sorted_entries()` function is taught to reject objects\n which appear in any existing MIDX layer.\n\n - Functions like `write_midx_revindex()` are adjusted to write\n pack_order values which are offset by the number of objects in the\n base MIDX layer.\n\n - The end of `write_midx_internal()` is adjusted to move\n non-incremental MIDX files when necessary (i.e. when creating an\n incremental chain with an existing non-incremental MIDX in the\n repository).\n\nThere are a handful of other changes that are introduced, like new\nfunctions to clear incremental MIDX files that are unrelated to the\ncurrent chain (using the same \"keep_hash\" mechanism as in the\nnon-incremental case).\n\nThe tests explicitly exercising the new incremental MIDX feature are\nrelatively limited for two reasons:\n\n 1. Most of the \"interesting\" behavior is already thoroughly covered in\n t5319-multi-pack-index.sh, which handles the core logic of reading\n objects through a MIDX.\n\n The new tests in t5334-incremental-multi-pack-index.sh are mostly\n focused on creating and destroying incremental MIDXs, as well as\n stitching their results together across layers.\n\n 2. A new GIT_TEST environment variable is added called\n \"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\", which modifies the\n entire test suite to write incremental MIDXs after repacking when\n combined with the \"GIT_TEST_MULTI_PACK_INDEX\" variable.\n\n This exercises the long tail of other interesting behavior that is\n defined implicitly throughout the rest of the CI suite. It is\n likewise added to the linux-TEST-vars job.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 11 +-\n builtin/multi-pack-index.c | 2 +\n builtin/repack.c | 8 +-\n ci/run-build-and-tests.sh | 1 +\n midx-write.c | 281 ++++++++++++++++++++----\n midx.c | 62 +++++-\n midx.h | 4 +\n packfile.c | 16 +-\n packfile.h | 4 +\n t/README | 4 +\n t/lib-bitmap.sh | 6 +-\n t/lib-midx.sh | 28 +++\n t/t5319-multi-pack-index.sh | 27 +--\n t/t5326-multi-pack-bitmaps.sh | 1 +\n t/t5327-multi-pack-bitmaps-rev.sh | 1 +\n t/t5332-multi-pack-reuse.sh | 2 +\n t/t5334-incremental-multi-pack-index.sh | 46 ++++\n t/t7700-repack.sh | 27 +--\n 18 files changed, 436 insertions(+), 95 deletions(-)\n create mode 100755 t/t5334-incremental-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 3696506eb3..631d5c7d15 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -64,6 +64,12 @@ The file given at `` is expected to be readable, and can contain\n duplicates. (If a given OID is given more than once, it is marked as\n preferred if at least one instance of it begins with the special `+`\n marker).\n+\n+\t--incremental::\n+\t\tWrite an incremental MIDX file containing only objects\n+\t\tand packs not present in an existing MIDX layer.\n+\t\tMigrates non-incremental MIDXs to incremental ones when\n+\t\tnecessary. Incompatible with `--bitmap`.\n --\n \n verify::\n@@ -74,6 +80,8 @@ expire::\n \thave no objects referenced by the MIDX (with the exception of\n \t`.keep` packs and cruft packs). Rewrite the MIDX file afterward\n \tto remove all references to these pack-files.\n++\n+NOTE: this mode is incompatible with incremental MIDX files.\n \n repack::\n \tCreate a new pack-file containing objects in small pack-files\n@@ -95,7 +103,8 @@ repack::\n +\n If `repack.packKeptObjects` is `false`, then any pack-files with an\n associated `.keep` file will not be selected for the batch to repack.\n-\n++\n+NOTE: this mode is incompatible with incremental MIDX files.\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 8360932d2e..92b86153ba 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"progress\", &opts.flags,\n \t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n+\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n+\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex e2fec16389..e1fab4d809 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1514,8 +1514,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (run_update_server_info)\n \t\tupdate_server_info(0);\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n+\t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n+\t}\n \n cleanup:\n \tstring_list_clear(&names, 1);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex e6fd68630c..2e28d02b20 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -25,6 +25,7 @@ linux-TEST-vars)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\ndiff --git a/midx-write.c b/midx-write.c\nindex b148ee443a..241557d03e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -15,6 +15,8 @@\n #include \"refs.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"path.h\"\n+#include \"pack-revindex.h\"\n \n #define PACK_EXPIRED UINT_MAX\n #define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n@@ -23,7 +25,11 @@\n \n extern int midx_checksum_valid(struct multi_pack_index *m);\n extern void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash);\n+\t\t\t\t const char *keep_hash);\n+extern void clear_incremental_midx_files_ext(const char *object_dir,\n+\t\t\t\t\t const char *ext,\n+\t\t\t\t\t const char **keep_hashes,\n+\t\t\t\t\t uint32_t hashes_nr);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -97,6 +103,9 @@ struct write_midx_context {\n \n \tint preferred_pack_idx;\n \n+\tint incremental;\n+\tuint32_t num_multi_pack_indexes_before;\n+\n \tstruct string_list *to_include;\n };\n \n@@ -322,7 +331,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tif (ctx->m)\n+\t\tif (ctx->m && !ctx->incremental)\n \t\t\tmidx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,\n \t\t\t\t\t\t ctx->preferred_pack_idx);\n \n@@ -348,6 +357,9 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n \t\t\t\t\t\t&fanout.entries[cur_object].oid))\n \t\t\t\tcontinue;\n+\t\t\tif (ctx->incremental && ctx->m &&\n+\t\t\t midx_has_oid(ctx->m, &fanout.entries[cur_object].oid))\n+\t\t\t\tcontinue;\n \n \t\t\tALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1),\n \t\t\t\t alloc_objects);\n@@ -531,10 +543,15 @@ static int write_midx_revindex(struct hashfile *f,\n \t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n+\tuint32_t i, nr_base;\n+\n+\tif (ctx->m && ctx->incremental)\n+\t\tnr_base = ctx->m->num_objects + ctx->m->num_objects_in_base;\n+\telse\n+\t\tnr_base = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\t\thashwrite_be32(f, ctx->pack_order[i] + nr_base);\n \n \treturn 0;\n }\n@@ -563,12 +580,17 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n {\n \tstruct midx_pack_order_data *data;\n-\tuint32_t *pack_order;\n+\tuint32_t *pack_order, base_objects = 0;\n \tuint32_t i;\n \n \ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n \n+\tif (ctx->incremental && ctx->m)\n+\t\tbase_objects = ctx->m->num_objects + ctx->m->num_objects_in_base;\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tALLOC_ARRAY(data, ctx->entries_nr);\n+\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n@@ -580,12 +602,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \n \tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n \n-\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n \t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = i;\n+\t\t\tpack->bitmap_pos = i + base_objects;\n \t\tpack->bitmap_nr++;\n \t\tpack_order[i] = data[i].nr;\n \t}\n@@ -633,7 +654,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \tprepare_packing_data(the_repository, pdata);\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tuint32_t pos = ctx->pack_order[i];\n+\t\tstruct pack_midx_entry *from = &ctx->entries[pos];\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n \t\toe_set_in_pack(pdata, to,\n@@ -881,40 +903,133 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n static int fill_packs_from_midx(struct write_midx_context *ctx,\n \t\t\t\tconst char *preferred_pack_name, uint32_t flags)\n {\n-\tuint32_t i;\n+\tstruct multi_pack_index *m;\n \n-\tfor (i = 0; i < ctx->m->num_packs; i++) {\n-\t\tif (!should_include_pack(ctx, ctx->m->pack_names[i], 0))\n-\t\t\tcontinue;\n+\tfor (m = ctx->m; m; m = m->base_midx) {\n+\t\tuint32_t i;\n \n-\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n-\n-\t\tif (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) {\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n \t\t\t/*\n \t\t\t * If generating a reverse index, need to have\n \t\t\t * packed_git's loaded to compare their\n \t\t\t * mtimes and object count.\n \t\t\t *\n-\t\t\t *\n \t\t\t * If a preferred pack is specified, need to\n \t\t\t * have packed_git's loaded to ensure the chosen\n \t\t\t * preferred pack has a non-zero object count.\n \t\t\t */\n-\t\t\tif (prepare_midx_pack(the_repository, ctx->m, i))\n-\t\t\t\treturn error(_(\"could not load pack\"));\n+\t\t\tif (!should_include_pack(ctx, m->pack_names[i], 0))\n+\t\t\t\tcontinue;\n \n-\t\t\tif (open_pack_index(ctx->m->packs[i]))\n-\t\t\t\tdie(_(\"could not open index for %s\"),\n-\t\t\t\t ctx->m->packs[i]->pack_name);\n+\t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX ||\n+\t\t\t preferred_pack_name) {\n+\t\t\t\tif (prepare_midx_pack(the_repository, m,\n+\t\t\t\t\t\t m->num_packs_in_base + i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\treturn 1;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t m->packs[i]->pack_name);\n+\t\t\t}\n+\n+\t\t\tfill_pack_info(&ctx->info[ctx->nr++], m->packs[i],\n+\t\t\t\t m->pack_names[i],\n+\t\t\t\t m->num_packs_in_base + i);\n \t\t}\n-\n-\t\tfill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i],\n-\t\t\t ctx->m->pack_names[i], i);\n \t}\n-\n \treturn 0;\n }\n \n+static struct {\n+\tconst char *non_split;\n+\tconst char *split;\n+} midx_exts[] = {\n+\t{NULL, MIDX_EXT_MIDX},\n+\t{MIDX_EXT_BITMAP, MIDX_EXT_BITMAP},\n+\t{MIDX_EXT_REV, MIDX_EXT_REV},\n+};\n+\n+static int link_midx_to_chain(struct multi_pack_index *m)\n+{\n+\tstruct strbuf from = STRBUF_INIT;\n+\tstruct strbuf to = STRBUF_INIT;\n+\tint ret = 0;\n+\tsize_t i;\n+\n+\tif (!m || m->has_chain) {\n+\t\t/*\n+\t\t * Either no MIDX previously existed, or it was already\n+\t\t * part of a MIDX chain. In both cases, we have nothing\n+\t\t * to link, so return early.\n+\t\t */\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n+\t\tconst unsigned char *hash = get_midx_checksum(m);\n+\n+\t\tget_midx_filename_ext(&from, m->object_dir, hash,\n+\t\t\t\t midx_exts[i].non_split);\n+\t\tget_split_midx_filename_ext(&to, m->object_dir, hash,\n+\t\t\t\t\t midx_exts[i].split);\n+\n+\t\tif (link(from.buf, to.buf) < 0 && errno != ENOENT) {\n+\t\t\tret = error_errno(_(\"unable to link '%s' to '%s'\"),\n+\t\t\t\t\t from.buf, to.buf);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tstrbuf_reset(&from);\n+\t\tstrbuf_reset(&to);\n+\t}\n+\n+done:\n+\tstrbuf_release(&from);\n+\tstrbuf_release(&to);\n+\treturn ret;\n+}\n+\n+static void clear_midx_files(const char *object_dir,\n+\t\t\t const char **hashes,\n+\t\t\t uint32_t hashes_nr,\n+\t\t\t unsigned incremental)\n+{\n+\t/*\n+\t * if incremental:\n+\t * - remove all non-incremental MIDX files\n+\t * - remove any incremental MIDX files not in the current one\n+\t *\n+\t * if non-incremental:\n+\t * - remove all incremental MIDX files\n+\t * - remove any non-incremental MIDX files not matching the current\n+\t * hash\n+\t */\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n+\tuint32_t i, j;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tclear_incremental_midx_files_ext(object_dir, exts[i],\n+\t\t\t\t\t\t hashes, hashes_nr);\n+\t\tfor (j = 0; j < hashes_nr; j++)\n+\t\t\tclear_midx_files_ext(object_dir, exts[i], hashes[j]);\n+\t}\n+\n+\tif (incremental)\n+\t\tget_midx_filename(&buf, object_dir);\n+\telse\n+\t\tget_midx_chain_filename(&buf, object_dir);\n+\n+\tif (unlink(buf.buf) && errno != ENOENT)\n+\t\tdie_errno(_(\"failed to clear multi-pack-index at %s\"), buf.buf);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t struct string_list *packs_to_drop,\n@@ -927,16 +1042,27 @@ static int write_midx_internal(const char *object_dir,\n \tuint32_t i, start_pack;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n+\tstruct tempfile *incr;\n \tstruct write_midx_context ctx = { 0 };\n \tint bitmapped_packs_concat_len = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tconst char **keep_hashes = NULL;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n \n-\tget_midx_filename(&midx_name, object_dir);\n+\tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n+\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n+\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n+\n+\tif (ctx.incremental)\n+\t\tstrbuf_addf(&midx_name,\n+\t\t\t \"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX\",\n+\t\t\t object_dir);\n+\telse\n+\t\tget_midx_filename(&midx_name, object_dir);\n \tif (safe_create_leading_directories(midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n@@ -948,14 +1074,19 @@ static int write_midx_internal(const char *object_dir,\n \t}\n \n \tctx.nr = 0;\n-\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n \tctx.to_include = packs_to_include;\n \tALLOC_ARRAY(ctx.info, ctx.alloc);\n \n-\tif (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n-\t\t\t\t\t flags) < 0) {\n-\t\tresult = 1;\n+\tif (ctx.incremental) {\n+\t\tstruct multi_pack_index *m = ctx.m;\n+\t\twhile (m) {\n+\t\t\tctx.num_multi_pack_indexes_before++;\n+\t\t\tm = m->base_midx;\n+\t\t}\n+\t} else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n+\t\t\t\t\t\t flags) < 0) {\n \t\tgoto cleanup;\n \t}\n \n@@ -970,7 +1101,8 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n+\t !ctx.incremental &&\n \t !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n@@ -986,12 +1118,14 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t * corresponding bitmap (or one wasn't requested).\n \t\t\t */\n \t\t\tif (!want_bitmap)\n-\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n-\t\t\t\t\t\t NULL);\n+\t\t\t\tclear_midx_files_ext(object_dir, \"bitmap\", NULL);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n+\tif (ctx.incremental && !ctx.nr)\n+\t\tgoto cleanup; /* nothing to do */\n+\n \tif (preferred_pack_name) {\n \t\tctx.preferred_pack_idx = -1;\n \n@@ -1137,8 +1271,30 @@ static int write_midx_internal(const char *object_dir,\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n \n-\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\tif (ctx.incremental) {\n+\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\n+\t\tget_midx_chain_filename(&lock_name, object_dir);\n+\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\t\tstrbuf_release(&lock_name);\n+\n+\t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n+\t\tif (!incr) {\n+\t\t\terror(_(\"unable to create temporary MIDX layer\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (adjust_shared_perm(get_tempfile_path(incr))) {\n+\t\t\terror(_(\"unable to adjust shared permissions for '%s'\"),\n+\t\t\t get_tempfile_path(incr));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr));\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n+\t\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\t}\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -1231,14 +1387,55 @@ static int write_midx_internal(const char *object_dir,\n \t * have been freed in the previous if block.\n \t */\n \n+\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n+\n+\tif (ctx.incremental) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tstruct strbuf final_midx_name = STRBUF_INIT;\n+\t\tstruct multi_pack_index *m = ctx.m;\n+\n+\t\tif (!chainf) {\n+\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (link_midx_to_chain(ctx.m) < 0)\n+\t\t\treturn -1;\n+\n+\t\tget_split_midx_filename_ext(&final_midx_name, object_dir,\n+\t\t\t\t\t midx_hash, MIDX_EXT_MIDX);\n+\n+\t\tif (rename_tempfile(&incr, final_midx_name.buf) < 0) {\n+\t\t\terror_errno(_(\"unable to rename new multi-pack-index layer\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\txstrdup(hash_to_hex(midx_hash));\n+\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n+\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\n+\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m)));\n+\t\t\tm = m->base_midx;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t} else {\n+\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\txstrdup(hash_to_hex(midx_hash));\n+\t}\n+\n \tif (ctx.m)\n \t\tclose_object_store(the_repository->objects);\n \n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+\tclear_midx_files(object_dir, keep_hashes,\n+\t\t\t ctx.num_multi_pack_indexes_before + 1,\n+\t\t\t ctx.incremental);\n \n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n@@ -1253,6 +1450,11 @@ static int write_midx_internal(const char *object_dir,\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n+\tif (keep_hashes) {\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)\n+\t\t\tfree((char *)keep_hashes[i]);\n+\t\tfree(keep_hashes);\n+\t}\n \tstrbuf_release(&midx_name);\n \n \ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n@@ -1289,6 +1491,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tif (!m)\n \t\treturn 0;\n \n+\tif (m->base_midx)\n+\t\tdie(_(\"cannot expire packs from an incremental multi-pack-index\"));\n+\n \tCALLOC_ARRAY(count, m->num_packs);\n \n \tif (flags & MIDX_PROGRESS)\n@@ -1463,6 +1668,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \n \tif (!m)\n \t\treturn 0;\n+\tif (m->base_midx)\n+\t\tdie(_(\"cannot repack an incremental multi-pack-index\"));\n \n \tCALLOC_ARRAY(include_pack, m->num_packs);\n \ndiff --git a/midx.c b/midx.c\nindex ae3e30a062..5aa7e2a6e6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -14,7 +14,10 @@\n \n int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t unsigned char *keep_hash);\n+\t\t\t const char *keep_hash);\n+void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t char **keep_hashes,\n+\t\t\t\t uint32_t hashes_nr);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -518,6 +521,11 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \treturn 0;\n }\n \n+int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid)\n+{\n+\treturn bsearch_midx(oid, m, NULL);\n+}\n+\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n)\n@@ -719,7 +727,8 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar *keep;\n+\tchar **keep;\n+\tuint32_t keep_nr;\n \tconst char *ext;\n };\n \n@@ -727,32 +736,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n+\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tif (data->keep && !strcmp(data->keep, file_name))\n-\t\treturn;\n-\n+\tfor (i = 0; i < data->keep_nr; i++) {\n+\t\tif (!strcmp(data->keep[i], file_name))\n+\t\t\treturn;\n+\t}\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n \n void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t unsigned char *keep_hash)\n+\t\t\t const char *keep_hash)\n {\n \tstruct clear_midx_data data;\n \tmemset(&data, 0, sizeof(struct clear_midx_data));\n \n-\tif (keep_hash)\n-\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n-\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tif (keep_hash) {\n+\t\tALLOC_ARRAY(data.keep, 1);\n+\n+\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n+\t\tdata.keep_nr = 1;\n+\t}\n \tdata.ext = ext;\n \n \tfor_each_file_in_pack_dir(object_dir,\n \t\t\t\t clear_midx_file_ext,\n \t\t\t\t &data);\n \n+\tif (keep_hash)\n+\t\tfree(data.keep[0]);\n+\tfree(data.keep);\n+}\n+\n+void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t char **keep_hashes,\n+\t\t\t\t uint32_t hashes_nr)\n+{\n+\tstruct clear_midx_data data;\n+\tuint32_t i;\n+\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tALLOC_ARRAY(data.keep, hashes_nr);\n+\tfor (i = 0; i < hashes_nr; i++)\n+\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n+\t\t\t\t ext);\n+\tdata.keep_nr = hashes_nr;\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_subdir(object_dir, \"multi-pack-index.d\",\n+\t\t\t\t clear_midx_file_ext, &data);\n+\n+\tfor (i = 0; i < hashes_nr; i++)\n+\t\tfree(data.keep[i]);\n \tfree(data.keep);\n }\n \n@@ -770,8 +810,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx.buf))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx.buf);\n \n-\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n-\tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL);\n \n \tstrbuf_release(&midx);\n }\ndiff --git a/midx.h b/midx.h\nindex 3714cad2cc..42d4f8d149 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,6 +29,8 @@ struct bitmapped_pack;\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\n@@ -77,6 +79,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n+#define MIDX_WRITE_INCREMENTAL (1 << 5)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n@@ -101,6 +104,7 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result);\n+int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/packfile.c b/packfile.c\nindex 85f0345435..2c335f4c4d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -813,9 +813,10 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-void for_each_file_in_pack_dir(const char *objdir,\n-\t\t\t each_file_in_pack_dir_fn fn,\n-\t\t\t void *data)\n+void for_each_file_in_pack_subdir(const char *objdir,\n+\t\t\t\t const char *subdir,\n+\t\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n@@ -824,6 +825,8 @@ void for_each_file_in_pack_dir(const char *objdir,\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n+\tif (subdir)\n+\t\tstrbuf_addf(&path, \"/%s\", subdir);\n \tdir = opendir(path.buf);\n \tif (!dir) {\n \t\tif (errno != ENOENT)\n@@ -845,6 +848,13 @@ void for_each_file_in_pack_dir(const char *objdir,\n \tstrbuf_release(&path);\n }\n \n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n+{\n+\tfor_each_file_in_pack_subdir(objdir, NULL, fn, data);\n+}\n+\n struct prepare_pack_data {\n \tstruct repository *r;\n \tstruct string_list *garbage;\ndiff --git a/packfile.h b/packfile.h\nindex 28c8fd3e39..07ba2c0be0 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n \t\t\t\t const char *file_name, void *data);\n+void for_each_file_in_pack_subdir(const char *objdir,\n+\t\t\t\t const char *subdir,\n+\t\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t\t void *data);\n void for_each_file_in_pack_dir(const char *objdir,\n \t\t\t each_file_in_pack_dir_fn fn,\n \t\t\t void *data);\ndiff --git a/t/README b/t/README\nindex e8a11926e4..e93a29de1b 100644\n--- a/t/README\n+++ b/t/README\n@@ -469,6 +469,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=, when true, sets\n+the '--incremental' option on all invocations of 'git multi-pack-index\n+write'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex f595937094..62aa6744a6 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,6 +1,8 @@\n # Helpers for scripts testing bitmap functionality; see t5310 for\n # example usage.\n \n+. \"$TEST_DIRECTORY\"/lib-midx.sh\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n@@ -264,10 +266,6 @@ have_delta () {\n \ttest_cmp expect actual\n }\n \n-midx_checksum () {\n-\ttest-tool read-midx --checksum \"$1\"\n-}\n-\n # midx_pack_source \n midx_pack_source () {\n \ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\ndiff --git a/t/lib-midx.sh b/t/lib-midx.sh\nindex 1261994744..e38c609604 100644\n--- a/t/lib-midx.sh\n+++ b/t/lib-midx.sh\n@@ -6,3 +6,31 @@ test_midx_consistent () {\n \ttest_cmp expect actual &&\n \tgit multi-pack-index --object-dir=$1 verify\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"$1\"\n+}\n+\n+midx_git_two_modes () {\n+\tgit -c core.multiPackIndex=false $1 >expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\tif [ \"$2\" = \"sorted\" ]\n+\tthen\n+\t\tsort expect.sorted &&\n+\t\tmv expect.sorted expect &&\n+\t\tsort actual.sorted &&\n+\t\tmv actual.sorted actual\n+\tfi &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx () {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\" &&\n+\t\tmidx_git_two_modes \"count-objects --verbose\" &&\n+\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check\" &&\n+\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check --unordered\" sorted\n+\t'\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 6e9ee23398..4b0b5a5c9f 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,8 +3,11 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-chunk.sh\n+. \"$TEST_DIRECTORY\"/lib-midx.sh\n \n GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n objdir=.git/objects\n \n HASH_LEN=$(test_oid rawsz)\n@@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' '\n \tmidx_read_expect 1 18 4 $objdir\n '\n \n-midx_git_two_modes () {\n-\tgit -c core.multiPackIndex=false $1 >expect &&\n-\tgit -c core.multiPackIndex=true $1 >actual &&\n-\tif [ \"$2\" = \"sorted\" ]\n-\tthen\n-\t\tsort expect.sorted &&\n-\t\tmv expect.sorted expect &&\n-\t\tsort actual.sorted &&\n-\t\tmv actual.sorted actual\n-\tfi &&\n-\ttest_cmp expect actual\n-}\n-\n-compare_results_with_midx () {\n-\tMSG=$1\n-\ttest_expect_success \"check normal git operations: $MSG\" '\n-\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\" &&\n-\t\tmidx_git_two_modes \"count-objects --verbose\" &&\n-\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check\" &&\n-\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check --unordered\" sorted\n-\t'\n-}\n-\n test_expect_success 'write midx with one v2 pack' '\n \tgit pack-objects --index-version=2,0x40 $objdir/pack/test &2 &&\n \tincrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack err &&\n+\t\tgit repack -Adl --write-bitmap-index 2>err &&\n \t\tcat >expect <<-EOF &&\n \t\twarning: disabling bitmap writing, as some objects are not being packed\n \t\tEOF\n@@ -533,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' '\n test_expect_success '--write-midx unchanged' '\n \t(\n \t\tcd midx &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack &&\n+\t\tgit repack &&\n \t\ttest_path_is_missing $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\t\tgit repack --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -550,7 +553,7 @@ test_expect_success '--write-midx with a new pack' '\n \t\tcd midx &&\n \t\ttest_commit loose &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\t\tgit repack --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -561,7 +564,7 @@ test_expect_success '--write-midx with a new pack' '\n test_expect_success '--write-midx with -b' '\n \t(\n \t\tcd midx &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&\n+\t\tgit repack -mb &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-*.bitmap &&\n@@ -574,7 +577,7 @@ test_expect_success '--write-midx with -d' '\n \t\tcd midx &&\n \t\ttest_commit repack &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&\n+\t\tgit repack -Ad --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -587,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t\tcd midx &&\n \n \t\ttest_commit repack-2 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\t\tgit repack -Adb --write-midx &&\n \n \t\tchecksum=$(midx_checksum $objdir) &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$checksum.bitmap &&\n \n \t\ttest_commit repack-3 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\t\tgit repack -Adb --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-$checksum.bitmap &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n \t\ttest_commit repack-4 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n+\t\tgit repack -Adb &&\n \n \t\tfind $objdir/pack -type f -name \"multi-pack-index*\" >files &&\n \t\ttest_must_be_empty files\n@@ -622,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '\n \t\tgit log --format=\"create refs/tags/%s/%s %H\" HEAD >refs &&\n \t\tgit update-ref --stdin This series implements incremental MIDXs, which allow for storing\n> a MIDX across multiple layers, each with their own distinct set of\n> packs.\n\nI forgot to mention, this series is based off a merge with current\nmaster and 'tb/midx-write-cleanup'.\n\nThe latter topic is marked to merge into 'master', but hasn't been\npushed out yet, hence the dependency on a merge with that and 'master'\ninstead of just 'master'.\n\nThanks,\nTaylor\n"},{"id":"496662","messageId":"xmqqwmn0d4dc.fsf@gitster.g","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"Re: [PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-07T17:55:43Z","receivedAt":"2024-06-07T17:55:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Part three doesn't exist yet, but is straightforward to do on top. None\n> of the design decisions made in this series inhibit my goals for part\n> three.\n\nNice to always see the bigger picture to come to understand where\nthe current series fits, but the above is a bit peculiar thing to\nsay. Of course there should be no design decision the currently\nposted series makes that would block your future work---otherwise\nyou would not be posting it. The real question is rather the future\nand yet to be written work is still feasible after the design\ndecisions the current series made are found to be broken and need to\nbe revised (if it happens---but we do not know until we see reviews).\n\nThanks.\n"},{"id":"496663","messageId":"xmqqr0d8d2mu.fsf@gitster.g","threadId":"61603","inReplyTo":"ZmJBAjraBh6O/G1d@nand.local","subject":"Re: [PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-07T18:33:13Z","receivedAt":"2024-06-07T18:33:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> I forgot to mention, this series is based off a merge with current\n> master and 'tb/midx-write-cleanup'.\n\nI think I saw \"am -3\" fall back to three-way at around [17/19] for\nt0410 while applying on that base, but it wasn't anything \"am -3\"\ncouldn't handle.\n\nQueued.\n\nThanks.\n"},{"id":"496669","messageId":"ZmNtnap8NWmHM6am@nand.local","threadId":"61603","inReplyTo":"xmqqr0d8d2mu.fsf@gitster.g","subject":"Re: [PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-07T20:29:17Z","receivedAt":"2024-06-07T20:29:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 07, 2024 at 11:33:13AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > I forgot to mention, this series is based off a merge with current\n> > master and 'tb/midx-write-cleanup'.\n>\n> I think I saw \"am -3\" fall back to three-way at around [17/19] for\n> t0410 while applying on that base, but it wasn't anything \"am -3\"\n> couldn't handle.\n>\n> Queued.\n\nGreat, thanks. Sorry again for forgetting to mention it sooner.\n\nThanks,\nTaylor\n"},{"id":"496670","messageId":"ZmNuEinqUV7DJQKQ@nand.local","threadId":"61603","inReplyTo":"xmqqwmn0d4dc.fsf@gitster.g","subject":"Re: [PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-06-07T20:31:14Z","receivedAt":"2024-06-07T20:31:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 07, 2024 at 10:55:43AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Part three doesn't exist yet, but is straightforward to do on top. None\n> > of the design decisions made in this series inhibit my goals for part\n> > three.\n>\n> Nice to always see the bigger picture to come to understand where\n> the current series fits, but the above is a bit peculiar thing to\n> say. Of course there should be no design decision the currently\n> posted series makes that would block your future work---otherwise\n> you would not be posting it.i\n\nYeah. What I was trying to say was that part two actually exists, and\nworks in practice rather than just thinking that it would work without\nhaving actually demonstrated anything ;-).\n\n> The real question is rather the future and yet to be written work is\n> still feasible after the design decisions the current series made are\n> found to be broken and need to be revised (if it happens---but we do\n> not know until we see reviews).\n\nIndeed. I'll make sure that before I push out a new round that the\nrebased part two still works as I expect it to.\n\nCertainly all of this could be avoided by combining the two together,\nbut I think the result is just too large to review.\n\nThanks,\nTaylor\n"},{"id":"497665","messageId":"xmqq4j9g7gmx.fsf@gitster.g","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"Re: [PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-25T23:21:42Z","receivedAt":"2024-06-25T23:21:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> This series implements incremental MIDXs, which allow for storing\n> a MIDX across multiple layers, each with their own distinct set of\n> packs.\n\nSo, ... it is unfortunate that this hasn't seen any responses (not\neven a question, let alone a proper review) and almost 3 weeks have\npassed.\n\nAny takers?\n\nThanks.\n"},{"id":"497669","messageId":"CABPp-BGNXO3wZAYGz1SVCSb_yHzU3E1fynD5yDDyKxo1_55r8A@mail.gmail.com","threadId":"61603","inReplyTo":"xmqq4j9g7gmx.fsf@gitster.g","subject":"Re: [PATCH 00/19] midx: incremental multi-pack indexes, part one","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2024-06-26T00:44:59Z","receivedAt":"2024-06-26T00:45:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jun 25, 2024 at 4:21 PM Junio C Hamano wrote:\n>\n> Taylor Blau writes:\n>\n> > This series implements incremental MIDXs, which allow for storing\n> > a MIDX across multiple layers, each with their own distinct set of\n> > packs.\n>\n> So, ... it is unfortunate that this hasn't seen any responses (not\n> even a question, let alone a proper review) and almost 3 weeks have\n> passed.\n>\n> Any takers?\n>\n> Thanks.\n\nI've got it on my list, and I'll try to look at it soon. It'll take a\nbit longer since I'm not familiar with the area.\n"},{"id":"498861","messageId":"cover.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH v2 00/19] midx: incremental multi-pack indexes, part one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:11:54Z","receivedAt":"2024-07-17T21:11:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements incremental MIDXs, which allow for storing\na MIDX across multiple layers, each with their own distinct set of\npacks.\n\nThis round is mostly unchanged from the previous since there has not yet\nbeen substantial review. But it does rebase to current 'master' (which\nis 04f5a52757 (Post 2.46-rc0 batch #2, 2024-07-16), at the time of\nwriting).\n\nImportantly, this rebase moves this topic to be based on an ancestor of\n0c5a62f14b (midx-write.c: do not read existing MIDX with\n`packs_to_include`, 2024-06-11), which resulted in a non-trivial\nconflict prior to this rebase.\n\nThe rest of the topic is unchanged. I don't expect that we'll see much\nreview here for the next couple of weeks while we are in the -rc phase,\nbut I figured it would be useful to have it on the list for folks that\nare interested in taking a look.\n\nThanks in advance for any review! :-)\n\nTaylor Blau (19):\n Documentation: describe incremental MIDX format\n midx: add new fields for incremental MIDX chains\n midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs\n midx: teach `prepare_midx_pack()` about incremental MIDXs\n midx: teach `nth_midxed_object_oid()` about incremental MIDXs\n midx: teach `nth_bitmapped_pack()` about incremental MIDXs\n midx: introduce `bsearch_one_midx()`\n midx: teach `bsearch_midx()` about incremental MIDXs\n midx: teach `nth_midxed_offset()` about incremental MIDXs\n midx: teach `fill_midx_entry()` about incremental MIDXs\n midx: remove unused `midx_locate_pack()`\n midx: teach `midx_contains_pack()` about incremental MIDXs\n midx: teach `midx_preferred_pack()` about incremental MIDXs\n midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs\n midx: support reading incremental MIDX chains\n midx: implement verification support for incremental MIDXs\n t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n t/t5313-pack-bounds-checks.sh: prepare for sub-directories\n midx: implement support for writing incremental MIDX chains\n\n Documentation/git-multi-pack-index.txt | 11 +-\n Documentation/technical/multi-pack-index.txt | 100 +++++\n builtin/multi-pack-index.c | 2 +\n builtin/repack.c | 8 +-\n ci/run-build-and-tests.sh | 2 +-\n midx-write.c | 326 ++++++++++++---\n midx.c | 410 ++++++++++++++++---\n midx.h | 26 +-\n object-name.c | 99 ++---\n packfile.c | 21 +-\n packfile.h | 4 +\n t/README | 6 +-\n t/helper/test-read-midx.c | 24 +-\n t/lib-bitmap.sh | 6 +-\n t/lib-midx.sh | 28 ++\n t/t0410-partial-clone.sh | 2 -\n t/t5310-pack-bitmaps.sh | 4 -\n t/t5313-pack-bounds-checks.sh | 8 +-\n t/t5319-multi-pack-index.sh | 30 +-\n t/t5326-multi-pack-bitmaps.sh | 4 +-\n t/t5327-multi-pack-bitmaps-rev.sh | 6 +-\n t/t5332-multi-pack-reuse.sh | 2 +\n t/t5334-incremental-multi-pack-index.sh | 46 +++\n t/t7700-repack.sh | 48 +--\n 24 files changed, 959 insertions(+), 264 deletions(-)\n create mode 100755 t/t5334-incremental-multi-pack-index.sh\n\nRange-diff against v1:\n 1: e5ce916f67 = 1: 014588b3ec Documentation: describe incremental MIDX format\n 2: 6569289ca7 = 2: 337ebc6de7 midx: add new fields for incremental MIDX chains\n 3: d2e845a9d4 = 3: f449a72877 midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs\n 4: 2100c6ddfa = 4: f88569c819 midx: teach `prepare_midx_pack()` about incremental MIDXs\n 5: 454c3d2fe7 ! 5: ec57ff4349 midx: teach `nth_midxed_object_oid()` about incremental MIDXs\n @@ midx.c: struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \n +\tn = midx_for_object(&m, n);\n +\n - \toidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n));\n + \toidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n),\n + \t\tthe_repository->hash_algo);\n \treturn oid;\n - }\n 6: 7d945c41bc = 6: 650b8c8c21 midx: teach `nth_bitmapped_pack()` about incremental MIDXs\n 7: 4d4d924aa2 = 7: bfd1dadbf1 midx: introduce `bsearch_one_midx()`\n 8: 86d88bc6a3 = 8: 38bd45bd24 midx: teach `bsearch_midx()` about incremental MIDXs\n 9: eb9ed10ca3 = 9: 342ed56033 midx: teach `nth_midxed_offset()` about incremental MIDXs\n10: 36cfdd9b95 = 10: 2b335c45ae midx: teach `fill_midx_entry()` about incremental MIDXs\n11: 1ae5fd7e89 = 11: 22de5898f3 midx: remove unused `midx_locate_pack()`\n12: e3319967b9 = 12: fb60f2b022 midx: teach `midx_contains_pack()` about incremental MIDXs\n13: 3b8dffa051 = 13: 38b642d404 midx: teach `midx_preferred_pack()` about incremental MIDXs\n14: 35fbe05a4a = 14: 594386da10 midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs\n15: a5eedb15fa = 15: dad130799c midx: support reading incremental MIDX chains\n16: 186b15e6bd ! 16: ad976ef413 midx: implement verification support for incremental MIDXs\n @@ midx.c: int verify_midx_file(struct repository *r, const char *object_dir, unsig\n -\t\t{\n -\t\t\tclose_pack_fd(m->packs[pairs[i-1].pack_int_id]);\n -\t\t\tclose_pack_index(m->packs[pairs[i-1].pack_int_id]);\n -+\t\t m->packs[pairs[i-1].pack_int_id]) {\n ++\t\t nth_midxed_pack(m, pairs[i-1].pack_int_id)) {\n +\t\t\tuint32_t pack_int_id = pairs[i-1].pack_int_id;\n +\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n +\n17: 94362c057a ! 17: 23912425bf t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n @@ t/README: GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n \n ## t/t0410-partial-clone.sh ##\n @@ t/t0410-partial-clone.sh: test_description='partial clone'\n - \n . ./test-lib.sh\n + . \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n -# missing promisor objects cause repacks which write bitmaps to fail\n -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n18: 4442e7ca52 = 18: 814da1916d t/t5313-pack-bounds-checks.sh: prepare for sub-directories\n19: 0cbe34b0bd ! 19: e2b5961b45 midx: implement support for writing incremental MIDX chains\n @@ midx-write.c\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n +@@ midx-write.c: struct write_midx_context {\n + \tsize_t nr;\n + \tsize_t alloc;\n + \tstruct multi_pack_index *m;\n ++\tstruct multi_pack_index *base_midx;\n + \tstruct progress *progress;\n + \tunsigned pack_paths_checked;\n + \n @@ midx-write.c: struct write_midx_context {\n \n \tint preferred_pack_idx;\n @@ midx-write.c: struct write_midx_context {\n \tstruct string_list *to_include;\n };\n \n +@@ midx-write.c: static int should_include_pack(const struct write_midx_context *ctx,\n + \t */\n + \tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n + \t\treturn 0;\n ++\telse if (ctx->base_midx && midx_contains_pack(ctx->base_midx,\n ++\t\t\t\t\t\t file_name))\n ++\t\treturn 0;\n + \telse if (ctx->to_include &&\n + \t\t !string_list_has_string(ctx->to_include, file_name))\n + \t\treturn 0;\n @@ midx-write.c: static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n @@ midx-write.c: static void compute_sorted_entries(struct write_midx_context *ctx,\n \t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n \t\t\t\t\t\t&fanout.entries[cur_object].oid))\n \t\t\t\tcontinue;\n -+\t\t\tif (ctx->incremental && ctx->m &&\n -+\t\t\t midx_has_oid(ctx->m, &fanout.entries[cur_object].oid))\n ++\t\t\tif (ctx->incremental && ctx->base_midx &&\n ++\t\t\t midx_has_oid(ctx->base_midx,\n ++\t\t\t\t\t &fanout.entries[cur_object].oid))\n +\t\t\t\tcontinue;\n \n \t\t\tALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1),\n @@ midx-write.c: static int write_midx_revindex(struct hashfile *f,\n -\tuint32_t i;\n +\tuint32_t i, nr_base;\n +\n -+\tif (ctx->m && ctx->incremental)\n -+\t\tnr_base = ctx->m->num_objects + ctx->m->num_objects_in_base;\n ++\tif (ctx->incremental && ctx->base_midx)\n ++\t\tnr_base = ctx->base_midx->num_objects +\n ++\t\t\tctx->base_midx->num_objects_in_base;\n +\telse\n +\t\tnr_base = 0;\n \n @@ midx-write.c: static int midx_pack_order_cmp(const void *va, const void *vb)\n \n \ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n \n -+\tif (ctx->incremental && ctx->m)\n -+\t\tbase_objects = ctx->m->num_objects + ctx->m->num_objects_in_base;\n ++\tif (ctx->incremental && ctx->base_midx)\n ++\t\tbase_objects = ctx->base_midx->num_objects +\n ++\t\t\tctx->base_midx->num_objects_in_base;\n +\n +\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tALLOC_ARRAY(data, ctx->entries_nr);\n @@ midx-write.c: static struct multi_pack_index *lookup_multi_pack_index(struct rep\n +\tstruct multi_pack_index *m;\n \n -\tfor (i = 0; i < ctx->m->num_packs; i++) {\n --\t\tif (!should_include_pack(ctx, ctx->m->pack_names[i], 0))\n --\t\t\tcontinue;\n +-\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n +\tfor (m = ctx->m; m; m = m->base_midx) {\n +\t\tuint32_t i;\n - \n --\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n --\n --\t\tif (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) {\n ++\n +\t\tfor (i = 0; i < m->num_packs; i++) {\n ++\t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n + \n +-\t\tif (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) {\n \t\t\t/*\n \t\t\t * If generating a reverse index, need to have\n \t\t\t * packed_git's loaded to compare their\n @@ midx-write.c: static struct multi_pack_index *lookup_multi_pack_index(struct rep\n \t\t\t */\n -\t\t\tif (prepare_midx_pack(the_repository, ctx->m, i))\n -\t\t\t\treturn error(_(\"could not load pack\"));\n -+\t\t\tif (!should_include_pack(ctx, m->pack_names[i], 0))\n -+\t\t\t\tcontinue;\n - \n --\t\t\tif (open_pack_index(ctx->m->packs[i]))\n --\t\t\t\tdie(_(\"could not open index for %s\"),\n --\t\t\t\t ctx->m->packs[i]->pack_name);\n -+\t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n -+\n +\t\t\tif (flags & MIDX_WRITE_REV_INDEX ||\n +\t\t\t preferred_pack_name) {\n +\t\t\t\tif (prepare_midx_pack(the_repository, m,\n @@ midx-write.c: static struct multi_pack_index *lookup_multi_pack_index(struct rep\n +\t\t\t\t\terror(_(\"could not load pack\"));\n +\t\t\t\t\treturn 1;\n +\t\t\t\t}\n -+\n + \n +-\t\t\tif (open_pack_index(ctx->m->packs[i]))\n +-\t\t\t\tdie(_(\"could not open index for %s\"),\n +-\t\t\t\t ctx->m->packs[i]->pack_name);\n +\t\t\t\tif (open_pack_index(m->packs[i]))\n +\t\t\t\t\tdie(_(\"could not open index for %s\"),\n +\t\t\t\t\t m->packs[i]->pack_name);\n @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n \tif (safe_create_leading_directories(midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n -@@ midx-write.c: static int write_midx_internal(const char *object_dir,\n + \n +-\tif (!packs_to_include) {\n +-\t\t/*\n +-\t\t * Only reference an existing MIDX when not filtering which\n +-\t\t * packs to include, since all packs and objects are copied\n +-\t\t * blindly from an existing MIDX if one is present.\n +-\t\t */\n +-\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n +-\t}\n ++\tif (!packs_to_include || ctx.incremental) {\n ++\t\tstruct multi_pack_index *m = lookup_multi_pack_index(the_repository,\n ++\t\t\t\t\t\t\t\t object_dir);\n ++\t\tif (m && !midx_checksum_valid(m)) {\n ++\t\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n ++\t\t\tm = NULL;\n ++\t\t}\n + \n +-\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n +-\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n +-\t\tctx.m = NULL;\n ++\t\tif (m) {\n ++\t\t\t/*\n ++\t\t\t * Only reference an existing MIDX when not filtering\n ++\t\t\t * which packs to include, since all packs and objects\n ++\t\t\t * are copied blindly from an existing MIDX if one is\n ++\t\t\t * present.\n ++\t\t\t */\n ++\t\t\tif (ctx.incremental)\n ++\t\t\t\tctx.base_midx = m;\n ++\t\t\telse if (!packs_to_include)\n ++\t\t\t\tctx.m = m;\n ++\t\t}\n \t}\n \n \tctx.nr = 0;\n -\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n +\tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n - \tctx.to_include = packs_to_include;\n \tALLOC_ARRAY(ctx.info, ctx.alloc);\n \n -\tif (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n -\t\t\t\t\t flags) < 0) {\n -\t\tresult = 1;\n +\tif (ctx.incremental) {\n -+\t\tstruct multi_pack_index *m = ctx.m;\n ++\t\tstruct multi_pack_index *m = ctx.base_midx;\n +\t\twhile (m) {\n +\t\t\tctx.num_multi_pack_indexes_before++;\n +\t\t\tm = m->base_midx;\n @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n \t * have been freed in the previous if block.\n \t */\n \n +-\tif (ctx.m)\n +\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n +\n +\tif (ctx.incremental) {\n +\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n +\t\tstruct strbuf final_midx_name = STRBUF_INIT;\n -+\t\tstruct multi_pack_index *m = ctx.m;\n ++\t\tstruct multi_pack_index *m = ctx.base_midx;\n +\n +\t\tif (!chainf) {\n +\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n +\t\t\treturn -1;\n +\t\t}\n +\n -+\t\tif (link_midx_to_chain(ctx.m) < 0)\n ++\t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n +\t\t\treturn -1;\n +\n +\t\tget_split_midx_filename_ext(&final_midx_name, object_dir,\n @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n +\t\t\txstrdup(hash_to_hex(midx_hash));\n +\t}\n +\n - \tif (ctx.m)\n ++\tif (ctx.m || ctx.base_midx)\n \t\tclose_object_store(the_repository->objects);\n \n \tif (commit_lock_file(&lk) < 0)\n\nbase-commit: 04f5a52757cd92347271e24f7cbdfe15dafce3b7\n-- \n2.46.0.rc0.94.g9b2aff57b3\n"},{"id":"498862","messageId":"014588b3ecf2e32a6ab1af8e77234dfb2f9ea75a.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 01/19] Documentation: describe incremental MIDX format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:11:58Z","receivedAt":"2024-07-17T21:12:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement incremental multi-pack indexes (MIDXs) over the\nnext several commits by first describing the relevant prerequisites\n(like a new chunk in the MIDX format, the directory structure for\nincremental MIDXs, etc.)\n\nThe format is described in detail in the patch contents below, but the\nhigh-level description is as follows.\n\nIncremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and\neach `*.midx` within that directory has a single \"parent\" MIDX, which is\nthe MIDX layer immediately before it in the MIDX chain. The chain order\nresides in a file 'multi-pack-index-chain' in the same directory.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/multi-pack-index.txt | 100 +++++++++++++++++++\n 1 file changed, 100 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex f2221d2b44..d05e3d6dd9 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -61,6 +61,106 @@ Design Details\n - The MIDX file format uses a chunk-based approach (similar to the\n commit-graph file) that allows optional data to be added.\n \n+Incremental multi-pack indexes\n+------------------------------\n+\n+As repositories grow in size, it becomes more expensive to write a\n+multi-pack index (MIDX) that includes all packfiles. To accommodate\n+this, the \"incremental multi-pack indexes\" feature allows for combining\n+a \"chain\" of multi-pack indexes.\n+\n+Each individual component of the chain need only contain a small number\n+of packfiles. Appending to the chain does not invalidate earlier parts\n+of the chain, so repositories can control how much time is spent\n+updating the MIDX chain by determining the number of packs in each layer\n+of the MIDX chain.\n+\n+=== Design state\n+\n+At present, the incremental multi-pack indexes feature is missing two\n+important components:\n+\n+ - The ability to rewrite earlier portions of the MIDX chain (i.e., to\n+ \"compact\" some collection of adjacent MIDX layers into a single\n+ MIDX). At present the only supported way of shrinking a MIDX chain\n+ is to rewrite the entire chain from scratch without the `--split`\n+ flag.\n++\n+There are no fundamental limitations that stand in the way of being able\n+to implement this feature. It is omitted from the initial implementation\n+in order to reduce the complexity, but will be added later.\n+\n+ - Support for reachability bitmaps. The classic single MIDX\n+ implementation does support reachability bitmaps (see the section\n+ titled \"multi-pack-index reverse indexes\" in\n+ linkgit:gitformat-pack[5] for more details).\n++\n+As above, there are no fundamental limitations that stand in the way of\n+extending the incremental MIDX format to support reachability bitmaps.\n+The design below specifically takes this into account, and support for\n+reachability bitmaps will be added in a future patch series. It is\n+omitted from this series for the same reason as above.\n++\n+In brief, to support reachability bitmaps with the incremental MIDX\n+feature, the concept of the pseudo-pack order is extended across each\n+layer of the incremental MIDX chain to form a concatenated pseudo-pack\n+order. This concatenation takes place in the same order as the chain\n+itself (in other words, the concatenated pseudo-pack order for a chain\n+`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by\n+the pseudo-pack order for `$H2`, followed by the pseudo-pack order for\n+`$H3`).\n++\n+The layout will then be extended so that each layer of the incremental\n+MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap\n+are offset by the number of objects in the previous layers of the chain.\n+\n+=== File layout\n+\n+Instead of storing a single `multi-pack-index` file (with an optional\n+`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental\n+MIDXs are stored in the following layout:\n+\n+----\n+$GIT_DIR/objects/pack/multi-pack-index.d/\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx\n+----\n+\n+The `multi-pack-index-chain` file contains a list of the incremental\n+MIDX files in the chain, in order. The above example shows a chain whose\n+`multi-pack-index-chain` file would contain the following lines:\n+\n+----\n+$H1\n+$H2\n+$H3\n+----\n+\n+The `multi-pack-index-$H1.midx` file contains the first layer of the\n+multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains\n+the second layer of the chain, and so on.\n+\n+=== Object positions for incremental MIDXs\n+\n+In the original multi-pack-index design, we refer to objects via their\n+lexicographic position (by object IDs) within the repository's singular\n+multi-pack-index. In the incremental multi-pack-index design, we refer\n+to objects via their index into a concatenated lexicographic ordering\n+among each component in the MIDX chain.\n+\n+If `objects_nr()` is a function that returns the number of objects in a\n+given MIDX layer, then the index of an object at lexicographic position\n+`i` within, say, $H3 is defined as:\n+\n+----\n+objects_nr($H2) + objects_nr($H1) + i\n+----\n+\n+(in the C implementation, this is often computed as `i +\n+m->num_objects_in_base`).\n+\n Future Work\n -----------\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498863","messageId":"337ebc6de7bdf6ff3b4b09c2bea3df2802174e8b.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 02/19] midx: add new fields for incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:01Z","receivedAt":"2024-07-17T21:12:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The incremental MIDX chain feature is designed around the idea of\nindexing into a concatenated lexicographic ordering of object IDs\npresent in the MIDX.\n\nWhen given an object position, the MIDX machinery needs to be able to\nlocate both (a) which MIDX layer contains the given object, and (b) at\nwhat position *within that MIDX layer* that object appears.\n\nTo do this, three new fields are added to the `struct multi_pack_index`:\n\n - struct multi_pack_index *base_midx;\n - uint32_t num_objects_in_base;\n - uint32_t num_packs_in_base;\n\nThese three fields store the pieces of information suggested by their\nrespective field names. In turn, the `num_objects_in_base` and\n`num_packs_in_base` fields are used to crawl backwards along the\n`base_midx` pointer to locate the appropriate position for a given\nobject within the MIDX that contains it.\n\nThe following commits will update various parts of the MIDX machinery\n(as well as their callers from outside of midx.c and midx-write.c) to be\naware and make use of these fields when performing object lookups.\n\nSigned-off-by: Taylor Blau \n---\n midx.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/midx.h b/midx.h\nindex 8554f2d616..020e49f77c 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -63,6 +63,10 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_revindex;\n \tsize_t chunk_revindex_len;\n \n+\tstruct multi_pack_index *base_midx;\n+\tuint32_t num_objects_in_base;\n+\tuint32_t num_packs_in_base;\n+\n \tconst char **pack_names;\n \tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498864","messageId":"f449a72877de97d89e31a56724b4f65be2f33f20.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:04Z","receivedAt":"2024-07-17T21:12:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `nth_midxed_pack_int_id()` takes in a object position in\nMIDX lexicographic order and returns an identifier of the pack from\nwhich that object was selected in the MIDX.\n\nCurrently, the given object position is an index into the lexicographic\norder of objects in a single MIDX. Change this position to instead refer\ninto the concatenated lexicographic order of all MIDXs in a MIDX chain.\n\nThis has two visible effects within the implementation of\n`prepare_midx_pack()`:\n\n - First, the given position is now an index into the concatenated\n lexicographic order of all MIDXs in the order in which they appear\n in the MIDX chain.\n\n - Second the pack ID returned from this function is now also in the\n concatenated order of packs among all layers of the MIDX chain in\n the same order that they appear in the MIDX chain.\n\nTo do this, introduce the first of two general purpose helpers, this one\nbeing `midx_for_object()`. `midx_for_object()` takes a double pointer to\na `struct multi_pack_index` as well as an object `pos` in terms of the\nentire MIDX chain[^1].\n\nThe function chases down the '->base_midx' field until it finds the MIDX\nlayer within the chain that contains the given object. It then:\n\n - modifies the double pointer to point to the containing MIDX, instead\n of the tip of the chain, and\n\n - returns the MIDX-local position[^2] at which the given object can be\n found.\n\nUse this function within `nth_midxed_pack_int_id()` so that the `pos` it\nexpects is now relative to the entire MIDX chain, and that it returns\nthe appropriate pack position for that object.\n\n[^1]: As a reminder, this means that the object is identified among the\n objects contained in all layers of the incremental MIDX chain, not any\n particular layer. For example, consider MIDX chain with two individual\n MIDXs, one with 4 objects and another with 3 objects. If the MIDX with\n 4 objects appears earlier in the chain, then asking for pack \"6\" would\n return the second object in the MIDX with 3 objects.\n\n[^2]: Building on the previous example, asking for object 6 in a MIDX\n chain with (4, 3) objects, respectively, this would set the double\n pointer to point at the MIDX containing three objects, and would\n return an index to the second object within that MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3992b05465..39d358da20 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -242,6 +242,23 @@ void close_midx(struct multi_pack_index *m)\n \tfree(m);\n }\n \n+static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n+{\n+\tstruct multi_pack_index *m = *_m;\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n+\n+\tif (pos >= m->num_objects + m->num_objects_in_base)\n+\t\tdie(_(\"invalid MIDX object position, MIDX is likely corrupt\"));\n+\n+\t*_m = m;\n+\n+\treturn pos - m->num_objects_in_base;\n+}\n+\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n@@ -334,8 +351,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n-\treturn get_be32(m->chunk_object_offsets +\n-\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n+\tpos = midx_for_object(&m, pos);\n+\n+\treturn m->num_packs_in_base + get_be32(m->chunk_object_offsets +\n+\t\t\t\t\t (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n }\n \n int fill_midx_entry(struct repository *r,\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498865","messageId":"f88569c819292a824c78cdffd4e1fbc329f07f8e.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:07Z","receivedAt":"2024-07-17T21:12:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `prepare_midx_pack()` is part of the midx.h API and\nloads the pack identified by the MIDX-local 'pack_int_id'. This patch\nprepares that function to be aware of an incremental MIDX world.\n\nTo do this, introduce the second of the two general purpose helpers\nmentioned in the previous commit. This commit introduces\n`midx_for_pack()`, which is the pack-specific analog of\n`midx_for_object()`, and works in the same fashion.\n\nLike `midx_for_object()`, this function chases down the '->base_midx'\nfield until it finds the MIDX layer within the chain that contains the\ngiven pack.\n\nUse this function within `prepare_midx_pack()` so that the `pack_int_id`\nit expects is now relative to the entire MIDX chain, and that it\nprepares the given pack in the appropriate MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 33 +++++++++++++++++++++++++--------\n 1 file changed, 25 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 39d358da20..da5e0bb940 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -259,20 +259,37 @@ static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n \treturn pos - m->num_objects_in_base;\n }\n \n-int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n+static uint32_t midx_for_pack(struct multi_pack_index **_m,\n+\t\t\t uint32_t pack_int_id)\n {\n-\tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct packed_git *p;\n+\tstruct multi_pack_index *m = *_m;\n+\twhile (m && pack_int_id < m->num_packs_in_base)\n+\t\tm = m->base_midx;\n \n-\tif (pack_int_id >= m->num_packs)\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for pack ID: %\"PRIu32, pack_int_id);\n+\n+\tif (pack_int_id >= m->num_packs + m->num_packs_in_base)\n \t\tdie(_(\"bad pack-int-id: %u (%u total packs)\"),\n-\t\t pack_int_id, m->num_packs);\n+\t\t pack_int_id, m->num_packs + m->num_packs_in_base);\n \n-\tif (m->packs[pack_int_id])\n+\t*_m = m;\n+\n+\treturn pack_int_id - m->num_packs_in_base;\n+}\n+\n+int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\n+\tif (m->packs[local_pack_int_id])\n \t\treturn 0;\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n-\t\t m->pack_names[pack_int_id]);\n+\t\t m->pack_names[local_pack_int_id]);\n \n \tp = add_packed_git(pack_name.buf, pack_name.len, m->local);\n \tstrbuf_release(&pack_name);\n@@ -281,7 +298,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n \t\treturn 1;\n \n \tp->multi_pack_index = 1;\n-\tm->packs[pack_int_id] = p;\n+\tm->packs[local_pack_int_id] = p;\n \tinstall_packed_git(r, p);\n \tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498866","messageId":"ec57ff434900f2b95e31fbdf854b5ebbf46b5c78.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:10Z","receivedAt":"2024-07-17T21:12:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `nth_midxed_object_oid()` returns the object ID for a given\nobject position in the MIDX lexicographic order.\n\nTeach this function to instead operate over the concatenated\nlexicographic order defined in an earlier step so that it is able to be\nused with incremental MIDXs.\n\nTo do this, we need to both (a) adjust the bounds check for the given\n'n', as well as record the MIDX-local position after chasing the\n`->base_midx` pointer to find the MIDX which contains that object.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex da5e0bb940..d470a88755 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -337,9 +337,11 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n)\n {\n-\tif (n >= m->num_objects)\n+\tif (n >= m->num_objects + m->num_objects_in_base)\n \t\treturn NULL;\n \n+\tn = midx_for_object(&m, n);\n+\n \toidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n),\n \t\tthe_repository->hash_algo);\n \treturn oid;\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498867","messageId":"650b8c8c21b7e8a6e4f65d9b47185a875f0022bb.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:13Z","receivedAt":"2024-07-17T21:12:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach the function\n`nth_bitmapped_pack()` about incremental MIDXs by translating the given\n`pack_int_id` from the concatenated lexical order to a MIDX-local\nlexical position.\n\nWhen accessing the containing MIDX's array of packs, use the local pack\nID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset\nwhen accessing the data within that chunk.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d470a88755..b6c3cd3e59 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -310,17 +310,19 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n {\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\n \tif (!m->chunk_bitmapped_packs)\n \t\treturn error(_(\"MIDX does not contain the BTMP chunk\"));\n \n \tif (prepare_midx_pack(r, m, pack_int_id))\n \t\treturn error(_(\"could not load bitmapped pack %\"PRIu32), pack_int_id);\n \n-\tbp->p = m->packs[pack_int_id];\n+\tbp->p = m->packs[local_pack_int_id];\n \tbp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs +\n-\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id);\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id);\n \tbp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs +\n-\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id +\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id +\n \t\t\t\t sizeof(uint32_t));\n \tbp->pack_int_id = pack_int_id;\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498868","messageId":"bfd1dadbf15cf735392ca15b52834f104cbd6538.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 07/19] midx: introduce `bsearch_one_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:16Z","receivedAt":"2024-07-17T21:12:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `bsearch_midx()` function will be extended in a following commit to\nsearch for the location of a given object ID across all MIDXs in a chain\n(or the single non-chain MIDX if no chain is available).\n\nWhile most callers will naturally want to use the updated\n`bsearch_midx()` function, there are a handful of special cases that\nwill want finer control and will only want to search through a single\nMIDX.\n\nFor instance, the object abbreviation code, which cares about object IDs\nnear to where we'd expect to find a match in a MIDX. In that case, we\nwant to look at the nearby matches in each layer of the MIDX chain, not\njust a single one).\n\nSplit the more fine-grained control out into a separate function called\n`bsearch_one_midx()` which searches only a single MIDX.\n\nAt present both `bsearch_midx()` and `bsearch_one_midx()` have identical\nbehavior, but the following commit will rewrite the former to be aware\nof incremental MIDXs for the remaining non-special case callers.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 17 +++++++--\n midx.h | 5 ++-\n object-name.c | 99 +++++++++++++++++++++++++++------------------------\n 3 files changed, 71 insertions(+), 50 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex b6c3cd3e59..bb3fa43492 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -329,10 +329,21 @@ int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \treturn 0;\n }\n \n-int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n+int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result)\n {\n-\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n-\t\t\t the_hash_algo->rawsz, result);\n+\tint ret = bsearch_hash(oid->hash, m->chunk_oid_fanout,\n+\t\t\t m->chunk_oid_lookup, the_hash_algo->rawsz,\n+\t\t\t result);\n+\tif (result)\n+\t\t*result += m->num_objects_in_base;\n+\treturn ret;\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result)\n+{\n+\t\treturn bsearch_one_midx(oid, m, result);\n }\n \n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/midx.h b/midx.h\nindex 020e49f77c..46c53d69ff 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -90,7 +90,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n-int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result);\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/object-name.c b/object-name.c\nindex 527b853ac4..e23b3e695a 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -134,28 +134,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char\n static void unique_in_midx(struct multi_pack_index *m,\n \t\t\t struct disambiguate_state *ds)\n {\n-\tuint32_t num, i, first = 0;\n-\tconst struct object_id *current = NULL;\n-\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n-\t\tds->repo->hash_algo->hexsz : ds->len;\n-\tnum = m->num_objects;\n+\tfor (; m; m = m->base_midx) {\n+\t\tuint32_t num, i, first = 0;\n+\t\tconst struct object_id *current = NULL;\n+\t\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n+\t\t\tds->repo->hash_algo->hexsz : ds->len;\n \n-\tif (!num)\n-\t\treturn;\n+\t\tnum = m->num_objects + m->num_objects_in_base;\n \n-\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\t\tif (!num)\n+\t\t\tcontinue;\n \n-\t/*\n-\t * At this point, \"first\" is the location of the lowest object\n-\t * with an object name that could match \"bin_pfx\". See if we have\n-\t * 0, 1 or more objects that actually match(es).\n-\t */\n-\tfor (i = first; i < num && !ds->ambiguous; i++) {\n-\t\tstruct object_id oid;\n-\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n-\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n-\t\t\tbreak;\n-\t\tupdate_candidates(ds, current);\n+\t\tbsearch_one_midx(&ds->bin_pfx, m, &first);\n+\n+\t\t/*\n+\t\t * At this point, \"first\" is the location of the lowest\n+\t\t * object with an object name that could match\n+\t\t * \"bin_pfx\". See if we have 0, 1 or more objects that\n+\t\t * actually match(es).\n+\t\t */\n+\t\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\t\tstruct object_id oid;\n+\t\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n+\t\t\t\tbreak;\n+\t\t\tupdate_candidates(ds, current);\n+\t\t}\n \t}\n }\n \n@@ -708,37 +712,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED,\n static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n-\tint match = 0;\n-\tuint32_t num, first = 0;\n-\tstruct object_id oid;\n-\tconst struct object_id *mad_oid;\n+\tfor (; m; m = m->base_midx) {\n+\t\tint match = 0;\n+\t\tuint32_t num, first = 0;\n+\t\tstruct object_id oid;\n+\t\tconst struct object_id *mad_oid;\n \n-\tif (!m->num_objects)\n-\t\treturn;\n+\t\tif (!m->num_objects)\n+\t\t\tcontinue;\n \n-\tnum = m->num_objects;\n-\tmad_oid = mad->oid;\n-\tmatch = bsearch_midx(mad_oid, m, &first);\n+\t\tnum = m->num_objects + m->num_objects_in_base;\n+\t\tmad_oid = mad->oid;\n+\t\tmatch = bsearch_one_midx(mad_oid, m, &first);\n \n-\t/*\n-\t * first is now the position in the packfile where we would insert\n-\t * mad->hash if it does not exist (or the position of mad->hash if\n-\t * it does exist). Hence, we consider a maximum of two objects\n-\t * nearby for the abbreviation length.\n-\t */\n-\tmad->init_len = 0;\n-\tif (!match) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first))\n-\t\t\textend_abbrev_len(&oid, mad);\n-\t} else if (first < num - 1) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n-\t\t\textend_abbrev_len(&oid, mad);\n+\t\t/*\n+\t\t * first is now the position in the packfile where we\n+\t\t * would insert mad->hash if it does not exist (or the\n+\t\t * position of mad->hash if it does exist). Hence, we\n+\t\t * consider a maximum of two objects nearby for the\n+\t\t * abbreviation length.\n+\t\t */\n+\t\tmad->init_len = 0;\n+\t\tif (!match) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t} else if (first < num - 1) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t}\n+\t\tif (first > 0) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t}\n+\t\tmad->init_len = mad->cur_len;\n \t}\n-\tif (first > 0) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n-\t\t\textend_abbrev_len(&oid, mad);\n-\t}\n-\tmad->init_len = mad->cur_len;\n }\n \n static void find_abbrev_len_for_pack(struct packed_git *p,\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498869","messageId":"38bd45bd24436d3404b089dff5c3d85f4026da97.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 08/19] midx: teach `bsearch_midx()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:19Z","receivedAt":"2024-07-17T21:12:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the special cases callers of `bsearch_midx()` have been dealt\nwith, teach `bsearch_midx()` to handle incremental MIDX chains.\n\nThe incremental MIDX-aware version of `bsearch_midx()` works by\nrepeatedly searching for a given OID in each layer along the\n`->base_midx` pointer, stopping either when an exact match is found, or\nthe end of the chain is reached.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex bb3fa43492..cd6e4afde4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -343,7 +343,10 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result)\n {\n-\t\treturn bsearch_one_midx(oid, m, result);\n+\tfor (; m; m = m->base_midx)\n+\t\tif (bsearch_one_midx(oid, m, result))\n+\t\t\treturn 1;\n+\treturn 0;\n }\n \n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498870","messageId":"342ed560331d95fb0a1d86743dbb999ec7327313.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:22Z","receivedAt":"2024-07-17T21:12:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach the function\n`nth_midxed_offset()` about incremental MIDXs.\n\nThe given object `pos` is used to find the containing MIDX, and\ntranslated back into a MIDX-local position by assigning the return value\nof `midx_for_object()` to it.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex cd6e4afde4..f87df6bede 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -368,6 +368,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n \n+\tpos = midx_for_object(&m, pos);\n+\n \toffset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;\n \toffset32 = get_be32(offset_data + sizeof(uint32_t));\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498871","messageId":"2b335c45ae7832b886ef9adccc2530e4ca53d267.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:25Z","receivedAt":"2024-07-17T21:12:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, teach the `fill_midx_entry()`\nfunction to work in a incremental MIDX-aware fashion.\n\nThis function, unlike others which accept an index into either the\nlexical order of objects or packs, takes in an object_id, and attempts\nto fill a caller-provided 'struct pack_entry' with the remaining pieces\nof information about that object from the MIDX.\n\nThe function uses `bsearch_midx()` which fills out the frame-local 'pos'\nvariable, recording the given object_id's lexical position within the\nMIDX chain, if found (if no matching object ID was found, we'll return\nimmediately without filling out the `pack_entry` structure).\n\nOnce given that position, we jump back through the `->base_midx` pointer\nto ensure that our `m` points at the MIDX layer which contains the given\nobject_id (and not an ancestor or descendant of it in the chain). Note\nthat we can drop the bounds check \"if (pos >= m->num_objects)\" because\n`midx_for_object()` performs this check for us.\n\nAfter that point, we only need to make two special considerations within\nthis function:\n\n - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()`\n is a position in the concatenated lexical order of packs, so we must\n ensure that we subtract `m->num_packs_in_base` before accessing the\n MIDX-local `packs` array.\n\n - Second, we must avoid translating the `pos` back to a MIDX-local\n index, since we use it as an argument to `nth_midxed_offset()` which\n expects a position relative to the concatenated lexical order of\n objects.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f87df6bede..dea78474a2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -406,14 +406,12 @@ int fill_midx_entry(struct repository *r,\n \tif (!bsearch_midx(oid, m, &pos))\n \t\treturn 0;\n \n-\tif (pos >= m->num_objects)\n-\t\treturn 0;\n-\n+\tmidx_for_object(&m, pos);\n \tpack_int_id = nth_midxed_pack_int_id(m, pos);\n \n \tif (prepare_midx_pack(r, m, pack_int_id))\n \t\treturn 0;\n-\tp = m->packs[pack_int_id];\n+\tp = m->packs[pack_int_id - m->num_packs_in_base];\n \n \t/*\n \t* We are about to tell the caller where they can locate the\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498872","messageId":"22de5898f3fec0cfff4289660b5f52aa31533fe1.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 11/19] midx: remove unused `midx_locate_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:29Z","receivedAt":"2024-07-17T21:12:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14)\nintroduced `midx_locate_pack()`, which was described at the time as a\ncomplement to the function `midx_contains_pack()` which allowed\ncallers to determine where in the MIDX lexical order a pack appeared, as\nopposed to whether or not it was simply contained.\n\n307d75bbe6 suggests that future patches would be added which would\nintroduce callers for this new function, but none ever were, meaning the\nfunction has gone unused since its introduction.\n\nClean this up by in effect reverting 307d75bbe6, which removes the\nunused functions and inlines its definition back into\n`midx_contains_pack()`.\n\n(Looking back through the list archives when 307d75bbe6 was written,\nthis was in preparation for this[1] patch from back when we had the\nconcept of \"disjoint\" packs while developing multi-pack verbatim reuse.\nThat concept was abandoned before the series was merged, but I never\ndropped what would become 307d75bbe6 from the series, leading to the\nstate prior to this commit).\n\n[1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 13 ++-----------\n midx.h | 2 --\n 2 files changed, 2 insertions(+), 13 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dea78474a2..59097808a9 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -465,8 +465,7 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n-\t\t uint32_t *pos)\n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -477,11 +476,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \n \t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n-\t\tif (!cmp) {\n-\t\t\tif (pos)\n-\t\t\t\t*pos = mid;\n+\t\tif (!cmp)\n \t\t\treturn 1;\n-\t\t}\n \t\tif (cmp > 0) {\n \t\t\tfirst = mid + 1;\n \t\t\tcontinue;\n@@ -492,11 +488,6 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \treturn 0;\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n-{\n-\treturn midx_locate_pack(m, idx_or_pack_name, NULL);\n-}\n-\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\ndiff --git a/midx.h b/midx.h\nindex 46c53d69ff..86af7dfc5e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -102,8 +102,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n-int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n-\t\t uint32_t *pos);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498873","messageId":"fb60f2b022d1dc8c0e6df53253dc924f8d00a382.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:32Z","receivedAt":"2024-07-17T21:12:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle\nhas been cleaned up, teach the former about how to operate in an\nincremental MIDX-aware world in a similar fashion as in previous\ncommits.\n\nInstead of using either of the two `midx_for_object()` or\n`midx_for_pack()` helpers, this function is split into two: one that\ndetermines whether a pack is contained in a single MIDX, and another\nwhich calls the former in a loop over all MIDXs.\n\nThis approach does not require that we change any of the implementation\nin what is now `midx_contains_pack_1()` as it still operates over a\nsingle MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 59097808a9..0fa8febb9d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -465,7 +465,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+static int midx_contains_pack_1(struct multi_pack_index *m,\n+\t\t\t\tconst char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -488,6 +489,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \treturn 0;\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+{\n+\tfor (; m; m = m->base_midx)\n+\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498874","messageId":"38b642d40423f2e14dd47d676de54f264b9d6f22.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:35Z","receivedAt":"2024-07-17T21:12:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `midx_preferred_pack()` is used to determine the identity\nof the preferred pack, which is the identity of a unique pack within\nthe MIDX which is used as a tie-breaker when selecting from which pack\nto represent an object that appears in multiple packs within the MIDX.\n\nHistorically we have said that the MIDX's preferred pack has the unique\nproperty that all objects from that pack are represented in the MIDX.\nBut that isn't quite true: a more precise statement would be that all\nobjects from that pack *which appear in the MIDX* are selected from that\npack.\n\nThis helps us extend the concept of preferred packs across a MIDX chain,\nwhere some object(s) in the preferred pack may appear in other packs\nin an earlier MIDX layer, in which case those object(s) will not appear\nin a subsequent MIDX layer from either the preferred pack or any other\npack.\n\nExtend the concept of preferred packs by using the pack which represents\nthe object at the first position in MIDX pseudo-pack order belonging to\nthe current MIDX layer (i.e., at position 'm->num_objects_in_base').\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 0fa8febb9d..d2dbea41e4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -500,13 +500,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\n+\t\tuint32_t midx_pos;\n \t\tif (load_midx_revindex(m) < 0) {\n \t\t\tm->preferred_pack_idx = -2;\n \t\t\treturn -1;\n \t\t}\n \n-\t\tm->preferred_pack_idx =\n-\t\t\tnth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\t\tmidx_pos = pack_pos_to_midx(m, m->num_objects_in_base);\n+\n+\t\tm->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos);\n+\n \t} else if (m->preferred_pack_idx == -2)\n \t\treturn -1; /* no revindex */\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498875","messageId":"594386da10bc3f3d6364916201438bf453b70098.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:38Z","receivedAt":"2024-07-17T21:12:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `midx_fanout_add_midx_fanout()` is used to help construct\nthe fanout table when generating a MIDX by reusing data from an existing\nMIDX.\n\nPrepare this function to work with incremental MIDXs by making a few\nchanges:\n\n - The bounds checks need to be adjusted to start object lookups taking\n into account the number of objects in the previous MIDX layer (i.e.,\n by starting the lookups at position `m->num_objects_in_base` instead\n of position 0).\n\n - Likewise, the bounds checks need to end at `m->num_objects_in_base`\n objects after `m->num_objects`.\n\n - Finally, `midx_fanout_add_midx_fanout()` needs to recur on earlier\n MIDX layers when dealing with an incremental MIDX chain by calling\n itself when given a MIDX with a non-NULL `base_midx`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 478b42e720..d5275d719b 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -196,7 +196,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n-\tif (pos >= m->num_objects)\n+\tif (pos >= m->num_objects + m->num_objects_in_base)\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n@@ -247,12 +247,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n \t\t\t\t\tuint32_t cur_fanout,\n \t\t\t\t\tint preferred_pack)\n {\n-\tuint32_t start = 0, end;\n+\tuint32_t start = m->num_objects_in_base, end;\n \tuint32_t cur_object;\n \n+\tif (m->base_midx)\n+\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n+\t\t\t\t\t preferred_pack);\n+\n \tif (cur_fanout)\n-\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n-\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\t\tstart += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\tend = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);\n \n \tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\tif ((preferred_pack > -1) &&\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498876","messageId":"dad130799c0d41d14375539bb12978e37849a721.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 15/19] midx: support reading incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:41Z","receivedAt":"2024-07-17T21:12:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the MIDX machinery's internals have been taught to understand\nincremental MIDXs over the previous handful of commits, the MIDX\nmachinery itself can begin reading incremental MIDXs.\n\n(Note that while the on-disk format for incremental MIDXs has been\ndefined, the writing end has not been implemented. This will take place\nin the commit after next.)\n\nThe core of this change involves following the order specified in the\nMIDX chain and opening up MIDXs in the chain one-by-one, adding them to\nthe previous layer's `->base_midx` pointer at each step.\n\nIn order to implement this, the `load_multi_pack_index()` function is\ntaught to call a new `load_multi_pack_index_chain()` function if loading\na non-incremental MIDX failed via `load_multi_pack_index_one()`.\n\nWhen loading a MIDX chain, `load_midx_chain_fd_st()` reads each line in\nthe file one-by-one and dispatches calls to\n`load_multi_pack_index_one()` to read each layer of the MIDX chain. When\na layer was successfully read, it is added to the MIDX chain by calling\n`add_midx_to_chain()` which validates the contents of the `BASE` chunk,\nperforms some bounds checks on the number of combined packs and objects,\nand attaches the new MIDX by assigning its `base_midx` pointer to the\nexisting part of the chain.\n\nAs a supplement to this, introduce a new mode in the test-read-midx\ntest-tool which allows us to read the information for a specific MIDX in\nthe chain by specifying its trailing checksum via the command-line\narguments like so:\n\n $ test-tool read-midx .git/objects [checksum]\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 184 +++++++++++++++++++++++++++++++++++---\n midx.h | 7 ++\n packfile.c | 5 +-\n t/helper/test-read-midx.c | 24 +++--\n 4 files changed, 201 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d2dbea41e4..0bfd17c021 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -91,7 +91,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n \n-struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n+static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir,\n+\t\t\t\t\t\t\t const char *midx_name,\n+\t\t\t\t\t\t\t int local)\n {\n \tstruct multi_pack_index *m = NULL;\n \tint fd;\n@@ -99,31 +101,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tsize_t midx_size;\n \tvoid *midx_map = NULL;\n \tuint32_t hash_version;\n-\tstruct strbuf midx_name = STRBUF_INIT;\n \tuint32_t i;\n \tconst char *cur_pack_name;\n \tstruct chunkfile *cf = NULL;\n \n-\tget_midx_filename(&midx_name, object_dir);\n-\n-\tfd = git_open(midx_name.buf);\n+\tfd = git_open(midx_name);\n \n \tif (fd < 0)\n \t\tgoto cleanup_fail;\n \tif (fstat(fd, &st)) {\n-\t\terror_errno(_(\"failed to read %s\"), midx_name.buf);\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n \n \tmidx_size = xsize_t(st.st_size);\n \n \tif (midx_size < MIDX_MIN_SIZE) {\n-\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name.buf);\n+\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n \n-\tstrbuf_release(&midx_name);\n-\n \tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \n@@ -213,7 +210,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n cleanup_fail:\n \tfree(m);\n-\tstrbuf_release(&midx_name);\n \tfree_chunkfile(cf);\n \tif (midx_map)\n \t\tmunmap(midx_map, midx_size);\n@@ -222,6 +218,173 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \treturn NULL;\n }\n \n+void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir)\n+{\n+\tstrbuf_addf(buf, \"%s/pack/multi-pack-index.d\", object_dir);\n+}\n+\n+void get_midx_chain_filename(struct strbuf *buf, const char *object_dir)\n+{\n+\tget_midx_chain_dirname(buf, object_dir);\n+\tstrbuf_addstr(buf, \"/multi-pack-index-chain\");\n+}\n+\n+void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n+\t\t\t\t const unsigned char *hash, const char *ext)\n+{\n+\tget_midx_chain_dirname(buf, object_dir);\n+\tstrbuf_addf(buf, \"/multi-pack-index-%s.%s\", hash_to_hex(hash), ext);\n+}\n+\n+static int open_multi_pack_index_chain(const char *chain_file,\n+\t\t\t\t int *fd, struct stat *st)\n+{\n+\t*fd = git_open(chain_file);\n+\tif (*fd < 0)\n+\t\treturn 0;\n+\tif (fstat(*fd, st)) {\n+\t\tclose(*fd);\n+\t\treturn 0;\n+\t}\n+\tif (st->st_size < the_hash_algo->hexsz) {\n+\t\tclose(*fd);\n+\t\tif (!st->st_size) {\n+\t\t\t/* treat empty files the same as missing */\n+\t\t\terrno = ENOENT;\n+\t\t} else {\n+\t\t\twarning(_(\"multi-pack-index chain file too small\"));\n+\t\t\terrno = EINVAL;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+static int add_midx_to_chain(struct multi_pack_index *midx,\n+\t\t\t struct multi_pack_index *midx_chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tif (midx_chain) {\n+\t\tif (unsigned_add_overflows(midx_chain->num_packs,\n+\t\t\t\t\t midx_chain->num_packs_in_base)) {\n+\t\t\twarning(_(\"pack count in base MIDX too high: %\"PRIuMAX),\n+\t\t\t\t(uintmax_t)midx_chain->num_packs_in_base);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (unsigned_add_overflows(midx_chain->num_objects,\n+\t\t\t\t\t midx_chain->num_objects_in_base)) {\n+\t\t\twarning(_(\"object count in base MIDX too high: %\"PRIuMAX),\n+\t\t\t\t(uintmax_t)midx_chain->num_objects_in_base);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tmidx->num_packs_in_base = midx_chain->num_packs +\n+\t\t\tmidx_chain->num_packs_in_base;\n+\t\tmidx->num_objects_in_base = midx_chain->num_objects +\n+\t\t\tmidx_chain->num_objects_in_base;\n+\t}\n+\n+\tmidx->base_midx = midx_chain;\n+\tmidx->has_chain = 1;\n+\n+\treturn 1;\n+}\n+\n+static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir,\n+\t\t\t\t\t\t int local,\n+\t\t\t\t\t\t int fd, struct stat *st,\n+\t\t\t\t\t\t int *incomplete_chain)\n+{\n+\tstruct multi_pack_index *midx_chain = NULL;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id *layers = NULL;\n+\tint valid = 1;\n+\tuint32_t i, count;\n+\tFILE *fp = xfdopen(fd, \"r\");\n+\n+\tcount = st->st_size / (the_hash_algo->hexsz + 1);\n+\tCALLOC_ARRAY(layers, count);\n+\n+\tfor (i = 0; i < count; i++) {\n+\t\tstruct multi_pack_index *m;\n+\n+\t\tif (strbuf_getline_lf(&buf, fp) == EOF)\n+\t\t\tbreak;\n+\n+\t\tif (get_oid_hex(buf.buf, &layers[i])) {\n+\t\t\twarning(_(\"invalid multi-pack-index chain: line '%s' \"\n+\t\t\t\t \"not a hash\"),\n+\t\t\t\tbuf.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tvalid = 0;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tget_split_midx_filename_ext(&buf, object_dir, layers[i].hash,\n+\t\t\t\t\t MIDX_EXT_MIDX);\n+\t\tm = load_multi_pack_index_one(object_dir, buf.buf, local);\n+\n+\t\tif (m) {\n+\t\t\tif (add_midx_to_chain(m, midx_chain, layers, i)) {\n+\t\t\t\tmidx_chain = m;\n+\t\t\t\tvalid = 1;\n+\t\t\t} else {\n+\t\t\t\tclose_midx(m);\n+\t\t\t}\n+\t\t}\n+\t\tif (!valid) {\n+\t\t\twarning(_(\"unable to find all multi-pack index files\"));\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree(layers);\n+\tfclose(fp);\n+\tstrbuf_release(&buf);\n+\n+\t*incomplete_chain = !valid;\n+\treturn midx_chain;\n+}\n+\n+static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir,\n+\t\t\t\t\t\t\t int local)\n+{\n+\tstruct strbuf chain_file = STRBUF_INIT;\n+\tstruct stat st;\n+\tint fd;\n+\tstruct multi_pack_index *m = NULL;\n+\n+\tget_midx_chain_filename(&chain_file, object_dir);\n+\tif (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) {\n+\t\tint incomplete;\n+\t\t/* ownership of fd is taken over by load function */\n+\t\tm = load_midx_chain_fd_st(object_dir, local, fd, &st,\n+\t\t\t\t\t &incomplete);\n+\t}\n+\n+\tstrbuf_release(&chain_file);\n+\treturn m;\n+}\n+\n+struct multi_pack_index *load_multi_pack_index(const char *object_dir,\n+\t\t\t\t\t int local)\n+{\n+\tstruct strbuf midx_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m;\n+\n+\tget_midx_filename(&midx_name, object_dir);\n+\n+\tm = load_multi_pack_index_one(object_dir, midx_name.buf, local);\n+\tif (!m)\n+\t\tm = load_multi_pack_index_chain(object_dir, local);\n+\n+\tstrbuf_release(&midx_name);\n+\n+\treturn m;\n+}\n+\n void close_midx(struct multi_pack_index *m)\n {\n \tuint32_t i;\n@@ -230,6 +393,7 @@ void close_midx(struct multi_pack_index *m)\n \t\treturn;\n \n \tclose_midx(m->next);\n+\tclose_midx(m->base_midx);\n \n \tmunmap((unsigned char *)m->data, m->data_len);\n \ndiff --git a/midx.h b/midx.h\nindex 86af7dfc5e..94de16a8c4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -24,6 +24,7 @@ struct bitmapped_pack;\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n+#define MIDX_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n@@ -50,6 +51,7 @@ struct multi_pack_index {\n \tint preferred_pack_idx;\n \n \tint local;\n+\tint has_chain;\n \n \tconst unsigned char *chunk_pack_names;\n \tsize_t chunk_pack_names_len;\n@@ -80,11 +82,16 @@ struct multi_pack_index {\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n+#define MIDX_EXT_MIDX \"midx\"\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\n void get_midx_filename_ext(struct strbuf *out, const char *object_dir,\n \t\t\t const unsigned char *hash, const char *ext);\n+void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir);\n+void get_midx_chain_filename(struct strbuf *buf, const char *object_dir);\n+void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n+\t\t\t\t const unsigned char *hash, const char *ext);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\ndiff --git a/packfile.c b/packfile.c\nindex 813584646f..1eb18e3041 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -880,7 +880,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!report_garbage)\n \t\treturn;\n \n-\tif (!strcmp(file_name, \"multi-pack-index\"))\n+\tif (!strcmp(file_name, \"multi-pack-index\") ||\n+\t !strcmp(file_name, \"multi-pack-index.d\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n \t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n@@ -1064,7 +1065,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \tprepare_packed_git(r);\n \tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n \t\tuint32_t i;\n-\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 83effc2b5f..69757e94fc 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -9,8 +9,10 @@\n #include \"packfile.h\"\n #include \"setup.h\"\n #include \"gettext.h\"\n+#include \"pack-revindex.h\"\n \n-static int read_midx_file(const char *object_dir, int show_objects)\n+static int read_midx_file(const char *object_dir, const char *checksum,\n+\t\t\t int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -21,6 +23,13 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \tif (!m)\n \t\treturn 1;\n \n+\tif (checksum) {\n+\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n+\t\t\tm = m->base_midx;\n+\t\tif (!m)\n+\t\t\treturn 1;\n+\t}\n+\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t m->signature,\n \t m->version,\n@@ -54,7 +63,8 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \t\tstruct pack_entry e;\n \n \t\tfor (i = 0; i < m->num_objects; i++) {\n-\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tnth_midxed_object_oid(&oid, m,\n+\t\t\t\t\t i + m->num_objects_in_base);\n \t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n \n \t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n@@ -111,7 +121,7 @@ static int read_midx_bitmapped_packs(const char *object_dir)\n \tif (!midx)\n \t\treturn 1;\n \n-\tfor (i = 0; i < midx->num_packs; i++) {\n+\tfor (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) {\n \t\tif (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)\n \t\t\treturn 1;\n \n@@ -127,16 +137,16 @@ static int read_midx_bitmapped_packs(const char *object_dir)\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n+\tif (!(argc == 2 || argc == 3 || argc == 4))\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n-\t\treturn read_midx_file(argv[2], 1);\n+\t\treturn read_midx_file(argv[2], argv[3], 1);\n \telse if (!strcmp(argv[1], \"--checksum\"))\n \t\treturn read_midx_checksum(argv[2]);\n \telse if (!strcmp(argv[1], \"--preferred-pack\"))\n \t\treturn read_midx_preferred_pack(argv[2]);\n \telse if (!strcmp(argv[1], \"--bitmap\"))\n \t\treturn read_midx_bitmapped_packs(argv[2]);\n-\treturn read_midx_file(argv[1], 0);\n+\treturn read_midx_file(argv[1], argv[2], 0);\n }\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498877","messageId":"ad976ef4137d31929a0feef205396d7642de7da0.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 16/19] midx: implement verification support for incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:44Z","receivedAt":"2024-07-17T21:12:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the verification implementation used by `git multi-pack-index\nverify` to perform verification for incremental MIDX chains by\nindependently validating each layer within the chain.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 47 ++++++++++++++++++++++++++++++-----------------\n midx.h | 2 ++\n 2 files changed, 32 insertions(+), 17 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 0bfd17c021..21a9dbe23a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -469,6 +469,13 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \treturn 0;\n }\n \n+struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n+\t\t\t\t uint32_t pack_int_id)\n+{\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\treturn m->packs[local_pack_int_id];\n+}\n+\n #define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n \n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n@@ -817,6 +824,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \tuint32_t i;\n \tstruct progress *progress = NULL;\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\tstruct multi_pack_index *curr;\n \tverify_midx_error = 0;\n \n \tif (!m) {\n@@ -839,8 +847,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Looking for referenced packfiles\"),\n-\t\t\t\t\t m->num_packs);\n-\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\t\t\t m->num_packs + m->num_packs_in_base);\n+\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\tif (prepare_midx_pack(r, m, i))\n \t\t\tmidx_report(\"failed to load pack in position %d\", i);\n \n@@ -860,17 +868,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Verifying OID order in multi-pack-index\"),\n \t\t\t\t\t\t m->num_objects - 1);\n-\tfor (i = 0; i < m->num_objects - 1; i++) {\n-\t\tstruct object_id oid1, oid2;\n \n-\t\tnth_midxed_object_oid(&oid1, m, i);\n-\t\tnth_midxed_object_oid(&oid2, m, i + 1);\n+\tfor (curr = m; curr; curr = curr->base_midx) {\n+\t\tfor (i = 0; i < m->num_objects - 1; i++) {\n+\t\t\tstruct object_id oid1, oid2;\n \n-\t\tif (oidcmp(&oid1, &oid2) >= 0)\n-\t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n-\t\t\t\t i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n+\t\t\tnth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i);\n+\t\t\tnth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1);\n \n-\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\t\tif (oidcmp(&oid1, &oid2) >= 0)\n+\t\t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n+\t\t\t\t\t i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n+\n+\t\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \n@@ -880,8 +891,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t * each of the objects and only require 1 packfile to be open at a\n \t * time.\n \t */\n-\tALLOC_ARRAY(pairs, m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n+\tALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base);\n+\tfor (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {\n \t\tpairs[i].pos = i;\n \t\tpairs[i].pack_int_id = nth_midxed_pack_int_id(m, i);\n \t}\n@@ -895,16 +906,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Verifying object offsets\"), m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n+\tfor (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {\n \t\tstruct object_id oid;\n \t\tstruct pack_entry e;\n \t\toff_t m_offset, p_offset;\n \n \t\tif (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id &&\n-\t\t m->packs[pairs[i-1].pack_int_id])\n-\t\t{\n-\t\t\tclose_pack_fd(m->packs[pairs[i-1].pack_int_id]);\n-\t\t\tclose_pack_index(m->packs[pairs[i-1].pack_int_id]);\n+\t\t nth_midxed_pack(m, pairs[i-1].pack_int_id)) {\n+\t\t\tuint32_t pack_int_id = pairs[i-1].pack_int_id;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tclose_pack_fd(p);\n+\t\t\tclose_pack_index(p);\n \t\t}\n \n \t\tnth_midxed_object_oid(&oid, m, pairs[i].pos);\ndiff --git a/midx.h b/midx.h\nindex 94de16a8c4..9d30935589 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -95,6 +95,8 @@ void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n+struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n+\t\t\t\t uint32_t pack_int_id);\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498878","messageId":"23912425bf7c0106ed388f7712e7bb822572fe5d.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:47Z","receivedAt":"2024-07-17T21:12:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Two years ago, commit ff1e653c8e2 (midx: respect\n'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new\nenvironment variable which caused the test suite to write MIDX bitmaps\nafter any 'git repack' invocation.\n\nAt the time, this was done to help flush out any bugs with MIDX bitmaps\nthat weren't explicitly covered in the t5326-multi-pack-bitmap.sh\nscript.\n\nTwo years later, that flag has served us well and is no longer providing\nmeaningful coverage, as the script in t5326 has matured substantially\nand covers many more interesting cases than it did back when ff1e653c8e2\nwas originally written.\n\nRemove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable\nas it is no longer serving a useful purpose. More importantly, removing\nthis variable clears the way for us to introduce a new one to help\nsimilarly flush out bugs related to incremental MIDX chains.\n\nBecause these incremental MIDX chains are (for now) incompatible with\nMIDX bitmaps, we cannot have both.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 12 ++----------\n ci/run-build-and-tests.sh | 1 -\n midx.h | 2 --\n t/README | 4 ----\n t/t0410-partial-clone.sh | 2 --\n t/t5310-pack-bitmaps.sh | 4 ----\n t/t5319-multi-pack-index.sh | 3 +--\n t/t5326-multi-pack-bitmaps.sh | 3 +--\n t/t5327-multi-pack-bitmaps-rev.sh | 5 ++---\n t/t7700-repack.sh | 21 +++++++--------------\n 10 files changed, 13 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f0317fa94a..8499bf0e12 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1217,10 +1217,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!write_midx &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n-\t} else if (write_bitmaps &&\n-\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n-\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n-\t\twrite_bitmaps = 0;\n \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0 && !write_midx;\n@@ -1518,12 +1514,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (run_update_server_info)\n \t\tupdate_server_info(0);\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n-\t\tunsigned flags = 0;\n-\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n-\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n-\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n-\t}\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, 0);\n \n cleanup:\n \tstring_list_clear(&names, 1);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 98dda42045..e6fd68630c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -25,7 +25,6 @@ linux-TEST-vars)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n-\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\ndiff --git a/midx.h b/midx.h\nindex 9d30935589..3714cad2cc 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,8 +29,6 @@ struct bitmapped_pack;\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n-#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n-\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex d9e0e07506..e8a11926e4 100644\n--- a/t/README\n+++ b/t/README\n@@ -469,10 +469,6 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n-'--bitmap' option on all invocations of 'git multi-pack-index write',\n-and ignores pack-objects' '--write-bitmap-index'.\n-\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 2c30c86e7b..34bdb3ab1f 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -5,8 +5,6 @@ test_description='partial clone'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n-# missing promisor objects cause repacks which write bitmaps to fail\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n # When enabled, some commands will write commit-graphs. This causes fsck\n # to fail when delete_object() is called because fsck will attempt to\n # verify the out-of-sync commit graph.\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex d7fd71360e..a6de7c5764 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n-# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n-# their place.\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n-\n # Likewise, allow individual tests to control whether or not they use\n # the boundary-based traversal.\n sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 10d2a6bf92..6e9ee23398 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -600,8 +600,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 916da389b6..1cb3e3ff08 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -4,10 +4,9 @@ test_description='exercise basic multi-pack bitmap functionality'\n . ./test-lib.sh\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n \n-# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# We'll be writing our own MIDX, so avoid getting confused by the\n # automatic ones.\n GIT_TEST_MULTI_PACK_INDEX=0\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n # This test exercise multi-pack bitmap functionality where the object order is\n # stored and read from a special chunk within the MIDX, so use the default\ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex e65e311cd7..23db949c20 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -5,10 +5,9 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)'\n . ./test-lib.sh\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n \n-# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n-# automatic ones.\n+# We'll be writing our own MIDX, so avoid getting confused by the automatic\n+# ones.\n GIT_TEST_MULTI_PACK_INDEX=0\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n # Unlike t5326, this test exercise multi-pack bitmap functionality where the\n # object order is stored in a separate .rev file.\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 127efe99f8..8f34f05087 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -70,14 +70,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n+\tgit repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -284,8 +283,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o\n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n \trm -f bare.git/objects/pack/*.bitmap &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad &&\n+\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -296,8 +294,7 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -308,8 +305,7 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad 2>stderr &&\n+\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -320,8 +316,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad 2>stderr &&\n+\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -342,9 +337,7 @@ test_expect_success 'repacking with a filter works' '\n '\n \n test_expect_success '--filter fails with --write-bitmap-index' '\n-\ttest_must_fail \\\n-\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+\ttest_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n '\n \n test_expect_success 'repacking with two filters works' '\n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498879","messageId":"814da1916d36173ffb18db64bcce8ea489b7da10.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 18/19] t/t5313-pack-bounds-checks.sh: prepare for sub-directories","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:50Z","receivedAt":"2024-07-17T21:12:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for sub-directories to appear in $GIT_DIR/objects/pack by\nadjusting the copy, remove, and chmod invocations to perform their\nbehavior recursively.\n\nThis prepares us for the new $GIT_DIR/objects/pack/multi-pack-index.d\ndirectory which will be added in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n t/t5313-pack-bounds-checks.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh\nindex ceaa6700a2..86fc73f9fb 100755\n--- a/t/t5313-pack-bounds-checks.sh\n+++ b/t/t5313-pack-bounds-checks.sh\n@@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true\n \n clear_base () {\n \ttest_when_finished 'restore_base' &&\n-\trm -f $base\n+\trm -r -f $base\n }\n \n restore_base () {\n-\tcp base-backup/* .git/objects/pack/\n+\tcp -r base-backup/* .git/objects/pack/\n }\n \n do_pack () {\n@@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' '\n \tgit commit -m base &&\n \tgit repack -ad &&\n \tbase=$(echo .git/objects/pack/*) &&\n-\tchmod +w $base &&\n+\tchmod -R +w $base &&\n \tmkdir base-backup &&\n-\tcp $base base-backup/ &&\n+\tcp -r $base base-backup/ &&\n \tobject=$(git rev-parse HEAD:file)\n '\n \n-- \n2.46.0.rc0.94.g9b2aff57b3\n\n"},{"id":"498880","messageId":"e2b5961b4556122e594b657efe2f1d3337368cdd.1721250704.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"[PATCH v2 19/19] midx: implement support for writing incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-17T21:12:53Z","receivedAt":"2024-07-17T21:12:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the rest of the MIDX subsystem and relevant callers have been\nupdated to learn about how to read and process incremental MIDX chains,\nlet's finally update the implementation in `write_midx_internal()` to be\nable to write incremental MIDX chains.\n\nThis new feature is available behind the `--incremental` option for the\n`multi-pack-index` builtin, like so:\n\n $ git multi-pack-index write --incremental\n\nThe implementation for doing so is relatively straightforward, and boils\ndown to a handful of different kinds of changes implemented in this\npatch:\n\n - The `compute_sorted_entries()` function is taught to reject objects\n which appear in any existing MIDX layer.\n\n - Functions like `write_midx_revindex()` are adjusted to write\n pack_order values which are offset by the number of objects in the\n base MIDX layer.\n\n - The end of `write_midx_internal()` is adjusted to move\n non-incremental MIDX files when necessary (i.e. when creating an\n incremental chain with an existing non-incremental MIDX in the\n repository).\n\nThere are a handful of other changes that are introduced, like new\nfunctions to clear incremental MIDX files that are unrelated to the\ncurrent chain (using the same \"keep_hash\" mechanism as in the\nnon-incremental case).\n\nThe tests explicitly exercising the new incremental MIDX feature are\nrelatively limited for two reasons:\n\n 1. Most of the \"interesting\" behavior is already thoroughly covered in\n t5319-multi-pack-index.sh, which handles the core logic of reading\n objects through a MIDX.\n\n The new tests in t5334-incremental-multi-pack-index.sh are mostly\n focused on creating and destroying incremental MIDXs, as well as\n stitching their results together across layers.\n\n 2. A new GIT_TEST environment variable is added called\n \"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\", which modifies the\n entire test suite to write incremental MIDXs after repacking when\n combined with the \"GIT_TEST_MULTI_PACK_INDEX\" variable.\n\n This exercises the long tail of other interesting behavior that is\n defined implicitly throughout the rest of the CI suite. It is\n likewise added to the linux-TEST-vars job.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 11 +-\n builtin/multi-pack-index.c | 2 +\n builtin/repack.c | 8 +-\n ci/run-build-and-tests.sh | 1 +\n midx-write.c | 314 ++++++++++++++++++++----\n midx.c | 62 ++++-\n midx.h | 4 +\n packfile.c | 16 +-\n packfile.h | 4 +\n t/README | 4 +\n t/lib-bitmap.sh | 6 +-\n t/lib-midx.sh | 28 +++\n t/t5319-multi-pack-index.sh | 27 +-\n t/t5326-multi-pack-bitmaps.sh | 1 +\n t/t5327-multi-pack-bitmaps-rev.sh | 1 +\n t/t5332-multi-pack-reuse.sh | 2 +\n t/t5334-incremental-multi-pack-index.sh | 46 ++++\n t/t7700-repack.sh | 27 +-\n 18 files changed, 460 insertions(+), 104 deletions(-)\n create mode 100755 t/t5334-incremental-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 3696506eb3..631d5c7d15 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -64,6 +64,12 @@ The file given at `` is expected to be readable, and can contain\n duplicates. (If a given OID is given more than once, it is marked as\n preferred if at least one instance of it begins with the special `+`\n marker).\n+\n+\t--incremental::\n+\t\tWrite an incremental MIDX file containing only objects\n+\t\tand packs not present in an existing MIDX layer.\n+\t\tMigrates non-incremental MIDXs to incremental ones when\n+\t\tnecessary. Incompatible with `--bitmap`.\n --\n \n verify::\n@@ -74,6 +80,8 @@ expire::\n \thave no objects referenced by the MIDX (with the exception of\n \t`.keep` packs and cruft packs). Rewrite the MIDX file afterward\n \tto remove all references to these pack-files.\n++\n+NOTE: this mode is incompatible with incremental MIDX files.\n \n repack::\n \tCreate a new pack-file containing objects in small pack-files\n@@ -95,7 +103,8 @@ repack::\n +\n If `repack.packKeptObjects` is `false`, then any pack-files with an\n associated `.keep` file will not be selected for the batch to repack.\n-\n++\n+NOTE: this mode is incompatible with incremental MIDX files.\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 9cf1a32d65..8805cbbeb3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"progress\", &opts.flags,\n \t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n+\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n+\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 8499bf0e12..7608430a37 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1514,8 +1514,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (run_update_server_info)\n \t\tupdate_server_info(0);\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n+\t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n+\t}\n \n cleanup:\n \tstring_list_clear(&names, 1);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex e6fd68630c..2e28d02b20 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -25,6 +25,7 @@ linux-TEST-vars)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\ndiff --git a/midx-write.c b/midx-write.c\nindex d5275d719b..a94cb28bfd 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -17,6 +17,8 @@\n #include \"refs.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"path.h\"\n+#include \"pack-revindex.h\"\n \n #define PACK_EXPIRED UINT_MAX\n #define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n@@ -25,7 +27,11 @@\n \n extern int midx_checksum_valid(struct multi_pack_index *m);\n extern void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash);\n+\t\t\t\t const char *keep_hash);\n+extern void clear_incremental_midx_files_ext(const char *object_dir,\n+\t\t\t\t\t const char *ext,\n+\t\t\t\t\t const char **keep_hashes,\n+\t\t\t\t\t uint32_t hashes_nr);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -86,6 +92,7 @@ struct write_midx_context {\n \tsize_t nr;\n \tsize_t alloc;\n \tstruct multi_pack_index *m;\n+\tstruct multi_pack_index *base_midx;\n \tstruct progress *progress;\n \tunsigned pack_paths_checked;\n \n@@ -99,6 +106,9 @@ struct write_midx_context {\n \n \tint preferred_pack_idx;\n \n+\tint incremental;\n+\tuint32_t num_multi_pack_indexes_before;\n+\n \tstruct string_list *to_include;\n };\n \n@@ -122,6 +132,9 @@ static int should_include_pack(const struct write_midx_context *ctx,\n \t */\n \tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\treturn 0;\n+\telse if (ctx->base_midx && midx_contains_pack(ctx->base_midx,\n+\t\t\t\t\t\t file_name))\n+\t\treturn 0;\n \telse if (ctx->to_include &&\n \t\t !string_list_has_string(ctx->to_include, file_name))\n \t\treturn 0;\n@@ -338,7 +351,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tif (ctx->m)\n+\t\tif (ctx->m && !ctx->incremental)\n \t\t\tmidx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,\n \t\t\t\t\t\t ctx->preferred_pack_idx);\n \n@@ -364,6 +377,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n \t\t\t\t\t\t&fanout.entries[cur_object].oid))\n \t\t\t\tcontinue;\n+\t\t\tif (ctx->incremental && ctx->base_midx &&\n+\t\t\t midx_has_oid(ctx->base_midx,\n+\t\t\t\t\t &fanout.entries[cur_object].oid))\n+\t\t\t\tcontinue;\n \n \t\t\tALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1),\n \t\t\t\t alloc_objects);\n@@ -547,10 +564,16 @@ static int write_midx_revindex(struct hashfile *f,\n \t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n+\tuint32_t i, nr_base;\n+\n+\tif (ctx->incremental && ctx->base_midx)\n+\t\tnr_base = ctx->base_midx->num_objects +\n+\t\t\tctx->base_midx->num_objects_in_base;\n+\telse\n+\t\tnr_base = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\t\thashwrite_be32(f, ctx->pack_order[i] + nr_base);\n \n \treturn 0;\n }\n@@ -579,12 +602,18 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n {\n \tstruct midx_pack_order_data *data;\n-\tuint32_t *pack_order;\n+\tuint32_t *pack_order, base_objects = 0;\n \tuint32_t i;\n \n \ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n \n+\tif (ctx->incremental && ctx->base_midx)\n+\t\tbase_objects = ctx->base_midx->num_objects +\n+\t\t\tctx->base_midx->num_objects_in_base;\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tALLOC_ARRAY(data, ctx->entries_nr);\n+\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n@@ -596,12 +625,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \n \tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n \n-\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n \t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = i;\n+\t\t\tpack->bitmap_pos = i + base_objects;\n \t\tpack->bitmap_nr++;\n \t\tpack_order[i] = data[i].nr;\n \t}\n@@ -649,7 +677,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \tprepare_packing_data(the_repository, pdata);\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tuint32_t pos = ctx->pack_order[i];\n+\t\tstruct pack_midx_entry *from = &ctx->entries[pos];\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n \t\toe_set_in_pack(pdata, to,\n@@ -897,37 +926,130 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n static int fill_packs_from_midx(struct write_midx_context *ctx,\n \t\t\t\tconst char *preferred_pack_name, uint32_t flags)\n {\n-\tuint32_t i;\n+\tstruct multi_pack_index *m;\n \n-\tfor (i = 0; i < ctx->m->num_packs; i++) {\n-\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n+\tfor (m = ctx->m; m; m = m->base_midx) {\n+\t\tuint32_t i;\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tif (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) {\n \t\t\t/*\n \t\t\t * If generating a reverse index, need to have\n \t\t\t * packed_git's loaded to compare their\n \t\t\t * mtimes and object count.\n \t\t\t *\n-\t\t\t *\n \t\t\t * If a preferred pack is specified, need to\n \t\t\t * have packed_git's loaded to ensure the chosen\n \t\t\t * preferred pack has a non-zero object count.\n \t\t\t */\n-\t\t\tif (prepare_midx_pack(the_repository, ctx->m, i))\n-\t\t\t\treturn error(_(\"could not load pack\"));\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX ||\n+\t\t\t preferred_pack_name) {\n+\t\t\t\tif (prepare_midx_pack(the_repository, m,\n+\t\t\t\t\t\t m->num_packs_in_base + i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\treturn 1;\n+\t\t\t\t}\n \n-\t\t\tif (open_pack_index(ctx->m->packs[i]))\n-\t\t\t\tdie(_(\"could not open index for %s\"),\n-\t\t\t\t ctx->m->packs[i]->pack_name);\n+\t\t\t\tif (open_pack_index(m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t m->packs[i]->pack_name);\n+\t\t\t}\n+\n+\t\t\tfill_pack_info(&ctx->info[ctx->nr++], m->packs[i],\n+\t\t\t\t m->pack_names[i],\n+\t\t\t\t m->num_packs_in_base + i);\n \t\t}\n-\n-\t\tfill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i],\n-\t\t\t ctx->m->pack_names[i], i);\n \t}\n-\n \treturn 0;\n }\n \n+static struct {\n+\tconst char *non_split;\n+\tconst char *split;\n+} midx_exts[] = {\n+\t{NULL, MIDX_EXT_MIDX},\n+\t{MIDX_EXT_BITMAP, MIDX_EXT_BITMAP},\n+\t{MIDX_EXT_REV, MIDX_EXT_REV},\n+};\n+\n+static int link_midx_to_chain(struct multi_pack_index *m)\n+{\n+\tstruct strbuf from = STRBUF_INIT;\n+\tstruct strbuf to = STRBUF_INIT;\n+\tint ret = 0;\n+\tsize_t i;\n+\n+\tif (!m || m->has_chain) {\n+\t\t/*\n+\t\t * Either no MIDX previously existed, or it was already\n+\t\t * part of a MIDX chain. In both cases, we have nothing\n+\t\t * to link, so return early.\n+\t\t */\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n+\t\tconst unsigned char *hash = get_midx_checksum(m);\n+\n+\t\tget_midx_filename_ext(&from, m->object_dir, hash,\n+\t\t\t\t midx_exts[i].non_split);\n+\t\tget_split_midx_filename_ext(&to, m->object_dir, hash,\n+\t\t\t\t\t midx_exts[i].split);\n+\n+\t\tif (link(from.buf, to.buf) < 0 && errno != ENOENT) {\n+\t\t\tret = error_errno(_(\"unable to link '%s' to '%s'\"),\n+\t\t\t\t\t from.buf, to.buf);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tstrbuf_reset(&from);\n+\t\tstrbuf_reset(&to);\n+\t}\n+\n+done:\n+\tstrbuf_release(&from);\n+\tstrbuf_release(&to);\n+\treturn ret;\n+}\n+\n+static void clear_midx_files(const char *object_dir,\n+\t\t\t const char **hashes,\n+\t\t\t uint32_t hashes_nr,\n+\t\t\t unsigned incremental)\n+{\n+\t/*\n+\t * if incremental:\n+\t * - remove all non-incremental MIDX files\n+\t * - remove any incremental MIDX files not in the current one\n+\t *\n+\t * if non-incremental:\n+\t * - remove all incremental MIDX files\n+\t * - remove any non-incremental MIDX files not matching the current\n+\t * hash\n+\t */\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n+\tuint32_t i, j;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tclear_incremental_midx_files_ext(object_dir, exts[i],\n+\t\t\t\t\t\t hashes, hashes_nr);\n+\t\tfor (j = 0; j < hashes_nr; j++)\n+\t\t\tclear_midx_files_ext(object_dir, exts[i], hashes[j]);\n+\t}\n+\n+\tif (incremental)\n+\t\tget_midx_filename(&buf, object_dir);\n+\telse\n+\t\tget_midx_chain_filename(&buf, object_dir);\n+\n+\tif (unlink(buf.buf) && errno != ENOENT)\n+\t\tdie_errno(_(\"failed to clear multi-pack-index at %s\"), buf.buf);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t struct string_list *packs_to_drop,\n@@ -940,42 +1062,66 @@ static int write_midx_internal(const char *object_dir,\n \tuint32_t i, start_pack;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n+\tstruct tempfile *incr;\n \tstruct write_midx_context ctx = { 0 };\n \tint bitmapped_packs_concat_len = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tconst char **keep_hashes = NULL;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n \n-\tget_midx_filename(&midx_name, object_dir);\n+\tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n+\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n+\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n+\n+\tif (ctx.incremental)\n+\t\tstrbuf_addf(&midx_name,\n+\t\t\t \"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX\",\n+\t\t\t object_dir);\n+\telse\n+\t\tget_midx_filename(&midx_name, object_dir);\n \tif (safe_create_leading_directories(midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n \n-\tif (!packs_to_include) {\n-\t\t/*\n-\t\t * Only reference an existing MIDX when not filtering which\n-\t\t * packs to include, since all packs and objects are copied\n-\t\t * blindly from an existing MIDX if one is present.\n-\t\t */\n-\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n-\t}\n+\tif (!packs_to_include || ctx.incremental) {\n+\t\tstruct multi_pack_index *m = lookup_multi_pack_index(the_repository,\n+\t\t\t\t\t\t\t\t object_dir);\n+\t\tif (m && !midx_checksum_valid(m)) {\n+\t\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n+\t\t\tm = NULL;\n+\t\t}\n \n-\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n-\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n-\t\tctx.m = NULL;\n+\t\tif (m) {\n+\t\t\t/*\n+\t\t\t * Only reference an existing MIDX when not filtering\n+\t\t\t * which packs to include, since all packs and objects\n+\t\t\t * are copied blindly from an existing MIDX if one is\n+\t\t\t * present.\n+\t\t\t */\n+\t\t\tif (ctx.incremental)\n+\t\t\t\tctx.base_midx = m;\n+\t\t\telse if (!packs_to_include)\n+\t\t\t\tctx.m = m;\n+\t\t}\n \t}\n \n \tctx.nr = 0;\n-\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n \tALLOC_ARRAY(ctx.info, ctx.alloc);\n \n-\tif (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n-\t\t\t\t\t flags) < 0) {\n-\t\tresult = 1;\n+\tif (ctx.incremental) {\n+\t\tstruct multi_pack_index *m = ctx.base_midx;\n+\t\twhile (m) {\n+\t\t\tctx.num_multi_pack_indexes_before++;\n+\t\t\tm = m->base_midx;\n+\t\t}\n+\t} else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n+\t\t\t\t\t\t flags) < 0) {\n \t\tgoto cleanup;\n \t}\n \n@@ -992,7 +1138,8 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n+\t !ctx.incremental &&\n \t !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n@@ -1008,12 +1155,14 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t * corresponding bitmap (or one wasn't requested).\n \t\t\t */\n \t\t\tif (!want_bitmap)\n-\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n-\t\t\t\t\t\t NULL);\n+\t\t\t\tclear_midx_files_ext(object_dir, \"bitmap\", NULL);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n+\tif (ctx.incremental && !ctx.nr)\n+\t\tgoto cleanup; /* nothing to do */\n+\n \tif (preferred_pack_name) {\n \t\tctx.preferred_pack_idx = -1;\n \n@@ -1159,8 +1308,30 @@ static int write_midx_internal(const char *object_dir,\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n \n-\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\tif (ctx.incremental) {\n+\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\n+\t\tget_midx_chain_filename(&lock_name, object_dir);\n+\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\t\tstrbuf_release(&lock_name);\n+\n+\t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n+\t\tif (!incr) {\n+\t\t\terror(_(\"unable to create temporary MIDX layer\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (adjust_shared_perm(get_tempfile_path(incr))) {\n+\t\t\terror(_(\"unable to adjust shared permissions for '%s'\"),\n+\t\t\t get_tempfile_path(incr));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr));\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n+\t\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\t}\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -1253,14 +1424,55 @@ static int write_midx_internal(const char *object_dir,\n \t * have been freed in the previous if block.\n \t */\n \n-\tif (ctx.m)\n+\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n+\n+\tif (ctx.incremental) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tstruct strbuf final_midx_name = STRBUF_INIT;\n+\t\tstruct multi_pack_index *m = ctx.base_midx;\n+\n+\t\tif (!chainf) {\n+\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n+\t\t\treturn -1;\n+\n+\t\tget_split_midx_filename_ext(&final_midx_name, object_dir,\n+\t\t\t\t\t midx_hash, MIDX_EXT_MIDX);\n+\n+\t\tif (rename_tempfile(&incr, final_midx_name.buf) < 0) {\n+\t\t\terror_errno(_(\"unable to rename new multi-pack-index layer\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\txstrdup(hash_to_hex(midx_hash));\n+\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n+\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\n+\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m)));\n+\t\t\tm = m->base_midx;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t} else {\n+\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\txstrdup(hash_to_hex(midx_hash));\n+\t}\n+\n+\tif (ctx.m || ctx.base_midx)\n \t\tclose_object_store(the_repository->objects);\n \n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+\tclear_midx_files(object_dir, keep_hashes,\n+\t\t\t ctx.num_multi_pack_indexes_before + 1,\n+\t\t\t ctx.incremental);\n \n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n@@ -1275,6 +1487,11 @@ static int write_midx_internal(const char *object_dir,\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n+\tif (keep_hashes) {\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)\n+\t\t\tfree((char *)keep_hashes[i]);\n+\t\tfree(keep_hashes);\n+\t}\n \tstrbuf_release(&midx_name);\n \n \ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n@@ -1311,6 +1528,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tif (!m)\n \t\treturn 0;\n \n+\tif (m->base_midx)\n+\t\tdie(_(\"cannot expire packs from an incremental multi-pack-index\"));\n+\n \tCALLOC_ARRAY(count, m->num_packs);\n \n \tif (flags & MIDX_PROGRESS)\n@@ -1485,6 +1705,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \n \tif (!m)\n \t\treturn 0;\n+\tif (m->base_midx)\n+\t\tdie(_(\"cannot repack an incremental multi-pack-index\"));\n \n \tCALLOC_ARRAY(include_pack, m->num_packs);\n \ndiff --git a/midx.c b/midx.c\nindex 21a9dbe23a..6372138ba8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -16,7 +16,10 @@\n \n int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t unsigned char *keep_hash);\n+\t\t\t const char *keep_hash);\n+void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t char **keep_hashes,\n+\t\t\t\t uint32_t hashes_nr);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -520,6 +523,11 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \treturn 0;\n }\n \n+int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid)\n+{\n+\treturn bsearch_midx(oid, m, NULL);\n+}\n+\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n)\n@@ -722,7 +730,8 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar *keep;\n+\tchar **keep;\n+\tuint32_t keep_nr;\n \tconst char *ext;\n };\n \n@@ -730,32 +739,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n+\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tif (data->keep && !strcmp(data->keep, file_name))\n-\t\treturn;\n-\n+\tfor (i = 0; i < data->keep_nr; i++) {\n+\t\tif (!strcmp(data->keep[i], file_name))\n+\t\t\treturn;\n+\t}\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n \n void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t unsigned char *keep_hash)\n+\t\t\t const char *keep_hash)\n {\n \tstruct clear_midx_data data;\n \tmemset(&data, 0, sizeof(struct clear_midx_data));\n \n-\tif (keep_hash)\n-\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n-\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tif (keep_hash) {\n+\t\tALLOC_ARRAY(data.keep, 1);\n+\n+\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n+\t\tdata.keep_nr = 1;\n+\t}\n \tdata.ext = ext;\n \n \tfor_each_file_in_pack_dir(object_dir,\n \t\t\t\t clear_midx_file_ext,\n \t\t\t\t &data);\n \n+\tif (keep_hash)\n+\t\tfree(data.keep[0]);\n+\tfree(data.keep);\n+}\n+\n+void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t char **keep_hashes,\n+\t\t\t\t uint32_t hashes_nr)\n+{\n+\tstruct clear_midx_data data;\n+\tuint32_t i;\n+\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tALLOC_ARRAY(data.keep, hashes_nr);\n+\tfor (i = 0; i < hashes_nr; i++)\n+\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n+\t\t\t\t ext);\n+\tdata.keep_nr = hashes_nr;\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_subdir(object_dir, \"multi-pack-index.d\",\n+\t\t\t\t clear_midx_file_ext, &data);\n+\n+\tfor (i = 0; i < hashes_nr; i++)\n+\t\tfree(data.keep[i]);\n \tfree(data.keep);\n }\n \n@@ -773,8 +813,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx.buf))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx.buf);\n \n-\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n-\tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL);\n \n \tstrbuf_release(&midx);\n }\ndiff --git a/midx.h b/midx.h\nindex 3714cad2cc..42d4f8d149 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,6 +29,8 @@ struct bitmapped_pack;\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\n@@ -77,6 +79,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n+#define MIDX_WRITE_INCREMENTAL (1 << 5)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n@@ -101,6 +104,7 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result);\n+int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/packfile.c b/packfile.c\nindex 1eb18e3041..cf12a539ea 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -815,9 +815,10 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-void for_each_file_in_pack_dir(const char *objdir,\n-\t\t\t each_file_in_pack_dir_fn fn,\n-\t\t\t void *data)\n+void for_each_file_in_pack_subdir(const char *objdir,\n+\t\t\t\t const char *subdir,\n+\t\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n@@ -826,6 +827,8 @@ void for_each_file_in_pack_dir(const char *objdir,\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n+\tif (subdir)\n+\t\tstrbuf_addf(&path, \"/%s\", subdir);\n \tdir = opendir(path.buf);\n \tif (!dir) {\n \t\tif (errno != ENOENT)\n@@ -847,6 +850,13 @@ void for_each_file_in_pack_dir(const char *objdir,\n \tstrbuf_release(&path);\n }\n \n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n+{\n+\tfor_each_file_in_pack_subdir(objdir, NULL, fn, data);\n+}\n+\n struct prepare_pack_data {\n \tstruct repository *r;\n \tstruct string_list *garbage;\ndiff --git a/packfile.h b/packfile.h\nindex eb18ec15db..0f78658229 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n \t\t\t\t const char *file_name, void *data);\n+void for_each_file_in_pack_subdir(const char *objdir,\n+\t\t\t\t const char *subdir,\n+\t\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t\t void *data);\n void for_each_file_in_pack_dir(const char *objdir,\n \t\t\t each_file_in_pack_dir_fn fn,\n \t\t\t void *data);\ndiff --git a/t/README b/t/README\nindex e8a11926e4..e93a29de1b 100644\n--- a/t/README\n+++ b/t/README\n@@ -469,6 +469,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=, when true, sets\n+the '--incremental' option on all invocations of 'git multi-pack-index\n+write'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex f595937094..62aa6744a6 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,6 +1,8 @@\n # Helpers for scripts testing bitmap functionality; see t5310 for\n # example usage.\n \n+. \"$TEST_DIRECTORY\"/lib-midx.sh\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n@@ -264,10 +266,6 @@ have_delta () {\n \ttest_cmp expect actual\n }\n \n-midx_checksum () {\n-\ttest-tool read-midx --checksum \"$1\"\n-}\n-\n # midx_pack_source \n midx_pack_source () {\n \ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\ndiff --git a/t/lib-midx.sh b/t/lib-midx.sh\nindex 1261994744..e38c609604 100644\n--- a/t/lib-midx.sh\n+++ b/t/lib-midx.sh\n@@ -6,3 +6,31 @@ test_midx_consistent () {\n \ttest_cmp expect actual &&\n \tgit multi-pack-index --object-dir=$1 verify\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"$1\"\n+}\n+\n+midx_git_two_modes () {\n+\tgit -c core.multiPackIndex=false $1 >expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\tif [ \"$2\" = \"sorted\" ]\n+\tthen\n+\t\tsort expect.sorted &&\n+\t\tmv expect.sorted expect &&\n+\t\tsort actual.sorted &&\n+\t\tmv actual.sorted actual\n+\tfi &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx () {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\" &&\n+\t\tmidx_git_two_modes \"count-objects --verbose\" &&\n+\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check\" &&\n+\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check --unordered\" sorted\n+\t'\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 6e9ee23398..4b0b5a5c9f 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,8 +3,11 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-chunk.sh\n+. \"$TEST_DIRECTORY\"/lib-midx.sh\n \n GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n objdir=.git/objects\n \n HASH_LEN=$(test_oid rawsz)\n@@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' '\n \tmidx_read_expect 1 18 4 $objdir\n '\n \n-midx_git_two_modes () {\n-\tgit -c core.multiPackIndex=false $1 >expect &&\n-\tgit -c core.multiPackIndex=true $1 >actual &&\n-\tif [ \"$2\" = \"sorted\" ]\n-\tthen\n-\t\tsort expect.sorted &&\n-\t\tmv expect.sorted expect &&\n-\t\tsort actual.sorted &&\n-\t\tmv actual.sorted actual\n-\tfi &&\n-\ttest_cmp expect actual\n-}\n-\n-compare_results_with_midx () {\n-\tMSG=$1\n-\ttest_expect_success \"check normal git operations: $MSG\" '\n-\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\" &&\n-\t\tmidx_git_two_modes \"count-objects --verbose\" &&\n-\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check\" &&\n-\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check --unordered\" sorted\n-\t'\n-}\n-\n test_expect_success 'write midx with one v2 pack' '\n \tgit pack-objects --index-version=2,0x40 $objdir/pack/test &2 &&\n \tincrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack err &&\n+\t\tgit repack -Adl --write-bitmap-index 2>err &&\n \t\tcat >expect <<-EOF &&\n \t\twarning: disabling bitmap writing, as some objects are not being packed\n \t\tEOF\n@@ -533,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' '\n test_expect_success '--write-midx unchanged' '\n \t(\n \t\tcd midx &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack &&\n+\t\tgit repack &&\n \t\ttest_path_is_missing $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\t\tgit repack --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -550,7 +553,7 @@ test_expect_success '--write-midx with a new pack' '\n \t\tcd midx &&\n \t\ttest_commit loose &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\t\tgit repack --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -561,7 +564,7 @@ test_expect_success '--write-midx with a new pack' '\n test_expect_success '--write-midx with -b' '\n \t(\n \t\tcd midx &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&\n+\t\tgit repack -mb &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-*.bitmap &&\n@@ -574,7 +577,7 @@ test_expect_success '--write-midx with -d' '\n \t\tcd midx &&\n \t\ttest_commit repack &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&\n+\t\tgit repack -Ad --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -587,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t\tcd midx &&\n \n \t\ttest_commit repack-2 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\t\tgit repack -Adb --write-midx &&\n \n \t\tchecksum=$(midx_checksum $objdir) &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$checksum.bitmap &&\n \n \t\ttest_commit repack-3 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\t\tgit repack -Adb --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-$checksum.bitmap &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n \t\ttest_commit repack-4 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n+\t\tgit repack -Adb &&\n \n \t\tfind $objdir/pack -type f -name \"multi-pack-index*\" >files &&\n \t\ttest_must_be_empty files\n@@ -622,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '\n \t\tgit log --format=\"create refs/tags/%s/%s %H\" HEAD >refs &&\n \t\tgit update-ref --stdin +Each individual component of the chain need only contain a small number\n> +of packfiles. Appending to the chain does not invalidate earlier parts\n> +of the chain, so repositories can control how much time is spent\n> +updating the MIDX chain by determining the number of packs in each layer\n> +of the MIDX chain.\n> +\n> +=== Design state\n> +\n> +At present, the incremental multi-pack indexes feature is missing two\n> +important components:\n> +\n> + - The ability to rewrite earlier portions of the MIDX chain (i.e., to\n> + \"compact\" some collection of adjacent MIDX layers into a single\n> + MIDX). At present the only supported way of shrinking a MIDX chain\n> + is to rewrite the entire chain from scratch without the `--split`\n> + flag.\n> ++\n> +There are no fundamental limitations that stand in the way of being able\n> +to implement this feature. It is omitted from the initial implementation\n> +in order to reduce the complexity, but will be added later.\n> +\n> + - Support for reachability bitmaps. The classic single MIDX\n> + implementation does support reachability bitmaps (see the section\n> + titled \"multi-pack-index reverse indexes\" in\n> + linkgit:gitformat-pack[5] for more details).\n> ++\n> +As above, there are no fundamental limitations that stand in the way of\n> +extending the incremental MIDX format to support reachability bitmaps.\n> +The design below specifically takes this into account, and support for\n> +reachability bitmaps will be added in a future patch series. It is\n> +omitted from this series for the same reason as above.\n\nIt is nice that you added a bit of a roadmap here about what is\nimplemented and what is not, and that the design takes into account\nfuture directions (especially incremental bitmap generation).\n\nIt does feel a little funny to say \"this series\" in text that will go\ninto the repository (i.e., somebody reading the checked out file will\nsay \"huh? which series?\"). I'm not sure how to word it better, except to\nmaybe just say \"in the future\" and \"it is omitted for now\" (and\nobviously it's a pretty minor point).\n\n> +In brief, to support reachability bitmaps with the incremental MIDX\n> +feature, the concept of the pseudo-pack order is extended across each\n> +layer of the incremental MIDX chain to form a concatenated pseudo-pack\n> +order. This concatenation takes place in the same order as the chain\n> +itself (in other words, the concatenated pseudo-pack order for a chain\n> +`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by\n> +the pseudo-pack order for `$H2`, followed by the pseudo-pack order for\n> +`$H3`).\n\nOK, that makes sense. It's how I'd have intuitively assumed it to be,\nand most importantly, it should allow appending to the chain without\nregenerating (or even translating) earlier bitmaps.\n\n> +=== File layout\n> +\n> +Instead of storing a single `multi-pack-index` file (with an optional\n> +`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental\n> +MIDXs are stored in the following layout:\n> +\n> +----\n> +$GIT_DIR/objects/pack/multi-pack-index.d/\n> +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain\n> +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx\n> +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx\n> +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx\n> +----\n> +\n> +The `multi-pack-index-chain` file contains a list of the incremental\n> +MIDX files in the chain, in order. The above example shows a chain whose\n> +`multi-pack-index-chain` file would contain the following lines:\n> +\n> +----\n> +$H1\n> +$H2\n> +$H3\n> +----\n> +\n> +The `multi-pack-index-$H1.midx` file contains the first layer of the\n> +multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains\n> +the second layer of the chain, and so on.\n\nMakes sense. How does the chained multi-pack-index.d interact with a\nsingular multi-pack-index? Generally we should not have both at the same\ntime, but I'd imagine they both exist for a brief period when moving\nfrom one to another.\n\nI assume the rules are the same as for commit-graphs, which use the same\non-disk structure. I can't think of a reason to prefer one over the\nother but this might be a good place to document what does/should\nhappen.\n\n> +=== Object positions for incremental MIDXs\n> +\n> +In the original multi-pack-index design, we refer to objects via their\n> +lexicographic position (by object IDs) within the repository's singular\n> +multi-pack-index. In the incremental multi-pack-index design, we refer\n> +to objects via their index into a concatenated lexicographic ordering\n> +among each component in the MIDX chain.\n\nHow do duplicate objects work here? I guess there aren't any duplicates\nin the midx itself, only in the constituent packfiles. So from the\nperspective of this section, I guess it doesn't matter? And from the\nperspective of bitmaps (where the duplicate issue came up before), it is\nbusiness as usual: the midx revindex gives the bit order, and we'd\npresumably concatenate the individual revindexes in chain order.\n\n(Mostly just thinking out loud; I'm not sure there's much for you to\nanswer there).\n\n\nLooking good so far...\n\n-Peff\n"},{"id":"499778","messageId":"20240801092142.GB1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"337ebc6de7bdf6ff3b4b09c2bea3df2802174e8b.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 02/19] midx: add new fields for incremental MIDX chains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T09:21:42Z","receivedAt":"2024-08-01T09:21:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:01PM -0400, Taylor Blau wrote:\n\n> The incremental MIDX chain feature is designed around the idea of\n> indexing into a concatenated lexicographic ordering of object IDs\n> present in the MIDX.\n> \n> When given an object position, the MIDX machinery needs to be able to\n> locate both (a) which MIDX layer contains the given object, and (b) at\n> what position *within that MIDX layer* that object appears.\n> \n> To do this, three new fields are added to the `struct multi_pack_index`:\n> \n> - struct multi_pack_index *base_midx;\n> - uint32_t num_objects_in_base;\n> - uint32_t num_packs_in_base;\n> \n> These three fields store the pieces of information suggested by their\n> respective field names. In turn, the `num_objects_in_base` and\n> `num_packs_in_base` fields are used to crawl backwards along the\n> `base_midx` pointer to locate the appropriate position for a given\n> object within the MIDX that contains it.\n\nOK, so base_midx is a back-pointer. I think in theory you could compute\nnum_objects_in_base on the fly by doing that crawl yourself, but we'd\nwant to be able to do it in constant time, rather than O(# of midx)?\n\nMakes sense.\n\n-Peff\n"},{"id":"499779","messageId":"20240801093006.GC1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"f449a72877de97d89e31a56724b4f65be2f33f20.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T09:30:06Z","receivedAt":"2024-08-01T09:30:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:04PM -0400, Taylor Blau wrote:\n\n> [^1]: As a reminder, this means that the object is identified among the\n> objects contained in all layers of the incremental MIDX chain, not any\n> particular layer. For example, consider MIDX chain with two individual\n> MIDXs, one with 4 objects and another with 3 objects. If the MIDX with\n> 4 objects appears earlier in the chain, then asking for pack \"6\" would\n> return the second object in the MIDX with 3 objects.\n\nI think this is \"object 6\" in the final sentence?\n\nOtherwise, the explanation lays things out pretty well. Let's look at\nthe code.\n\n> +static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n> +{\n> +\tstruct multi_pack_index *m = *_m;\n> +\twhile (m && pos < m->num_objects_in_base)\n> +\t\tm = m->base_midx;\n\nOK, so given a global position, we walk backwards until we find the\ncorrect midx...\n\n> +\tif (!m)\n> +\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n> +\n> +\tif (pos >= m->num_objects + m->num_objects_in_base)\n> +\t\tdie(_(\"invalid MIDX object position, MIDX is likely corrupt\"));\n\n...and we double check that the given base claims to have that position.\nSeems obvious.\n\n> +\t*_m = m;\n> +\n> +\treturn pos - m->num_objects_in_base;\n\nAnd then we adjust it into a per-midx position.\n\n> @@ -334,8 +351,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n> \n> uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n> {\n> -\treturn get_be32(m->chunk_object_offsets +\n> -\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n> +\tpos = midx_for_object(&m, pos);\n> +\n> +\treturn m->num_packs_in_base + get_be32(m->chunk_object_offsets +\n> +\t\t\t\t\t (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n> }\n\nOK, so now this function translates a global position into a local one,\nand then we get the pack id for the local midx/pos, and then turn it\nback into a global pack id.\n\nThat all makes sense, but you definitely have to read carefully to make\nsure which positions/ids are global within the chain and which are local\nto a midx.\n\nI wonder if the type system can help us annotate them, but I suspect it\nbecomes awkward. Just typedef-ing them to uint32_t means the compiler\nwon't warn us when we use one in the wrong spot. Sticking them in\nstructs would solve that, but then using them is painful. Let's keep\nreading and see if it's even an issue in practice.\n\n-Peff\n"},{"id":"499787","messageId":"20240801093550.GD1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"f88569c819292a824c78cdffd4e1fbc329f07f8e.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T09:35:50Z","receivedAt":"2024-08-01T09:35:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:07PM -0400, Taylor Blau wrote:\n\n> The function `prepare_midx_pack()` is part of the midx.h API and\n> loads the pack identified by the MIDX-local 'pack_int_id'. This patch\n> prepares that function to be aware of an incremental MIDX world.\n> \n> To do this, introduce the second of the two general purpose helpers\n> mentioned in the previous commit. This commit introduces\n> `midx_for_pack()`, which is the pack-specific analog of\n> `midx_for_object()`, and works in the same fashion.\n> \n> Like `midx_for_object()`, this function chases down the '->base_midx'\n> field until it finds the MIDX layer within the chain that contains the\n> given pack.\n> \n> Use this function within `prepare_midx_pack()` so that the `pack_int_id`\n> it expects is now relative to the entire MIDX chain, and that it\n> prepares the given pack in the appropriate MIDX.\n\nOK, I'm adequately prepared for more global/local confusion. :)\n\n> -int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n> +static uint32_t midx_for_pack(struct multi_pack_index **_m,\n> +\t\t\t uint32_t pack_int_id)\n> {\n> -\tstruct strbuf pack_name = STRBUF_INIT;\n> -\tstruct packed_git *p;\n> +\tstruct multi_pack_index *m = *_m;\n> +\twhile (m && pack_int_id < m->num_packs_in_base)\n> +\t\tm = m->base_midx;\n\nOK, so we chase down the pack id as before...\n\n> +\tif (!m)\n> +\t\tBUG(\"NULL multi-pack-index for pack ID: %\"PRIu32, pack_int_id);\n> +\n> +\tif (pack_int_id >= m->num_packs + m->num_packs_in_base)\n> \t\tdie(_(\"bad pack-int-id: %u (%u total packs)\"),\n> -\t\t pack_int_id, m->num_packs);\n> +\t\t pack_int_id, m->num_packs + m->num_packs_in_base);\n\n...with the same sanity checks...\n\n> +\t*_m = m;\n> +\n> +\treturn pack_int_id - m->num_packs_in_base;\n\n...and the same global to local offset conversion. Looks good so far.\n\n> +int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n> +\t\t uint32_t pack_int_id)\n> +{\n> +\tstruct strbuf pack_name = STRBUF_INIT;\n> +\tstruct packed_git *p;\n> +\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n\nThis one uses a separate variable with the word \"local\" in it. Helpful. :)\n\n> +\tif (m->packs[local_pack_int_id])\n> \t\treturn 0;\n> \n> \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n> -\t\t m->pack_names[pack_int_id]);\n> +\t\t m->pack_names[local_pack_int_id]);\n\nOK, and then this is just existing lazy-load of the pack struct. Good.\n\nI guess if you just reused pack_int_id for the local id, the diff would\nbe much smaller (this part would remain exactly the same). I dunno which\nis better, but it was a little curious that the two patches differed in\napproach. Probably not worth caring too much about, though.\n\n-Peff\n"},{"id":"499788","messageId":"20240801093805.GE1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"ec57ff434900f2b95e31fbdf854b5ebbf46b5c78.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T09:38:05Z","receivedAt":"2024-08-01T09:38:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:10PM -0400, Taylor Blau wrote:\n\n> The function `nth_midxed_object_oid()` returns the object ID for a given\n> object position in the MIDX lexicographic order.\n> \n> Teach this function to instead operate over the concatenated\n> lexicographic order defined in an earlier step so that it is able to be\n> used with incremental MIDXs.\n> \n> To do this, we need to both (a) adjust the bounds check for the given\n> 'n', as well as record the MIDX-local position after chasing the\n> `->base_midx` pointer to find the MIDX which contains that object.\n\nYep, this makes sense. The hard thing about reviewing this, I think, is\nthat each individual step like this is going to make sense, but I'll\nhave very little clue what spots (if any) were missed.\n\nTo some degree I think the proof will be in the pudding. If you missed\nany helpers, then the end result is going to crash and burn quite badly\nwhen used with a chained midx, and we'd see it in the test suite. And\nthe nice thing is that most of this is abstracted inside these helpers,\nso we know the set of tricky places is generally limited to the helpers,\nand not arbitrary bits of midx code.\n\n-Peff\n"},{"id":"499789","messageId":"20240801093947.GF1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"650b8c8c21b7e8a6e4f65d9b47185a875f0022bb.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T09:39:47Z","receivedAt":"2024-08-01T09:39:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:13PM -0400, Taylor Blau wrote:\n\n> In a similar fashion as in previous commits, teach the function\n> `nth_bitmapped_pack()` about incremental MIDXs by translating the given\n> `pack_int_id` from the concatenated lexical order to a MIDX-local\n> lexical position.\n> \n> When accessing the containing MIDX's array of packs, use the local pack\n> ID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset\n> when accessing the data within that chunk.\n\nOK, makes sense.\n\n> int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n> \t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n> {\n> +\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n> +\n\nHeh, after the last one reused the \"n\" variable, now we are back to a\nseparate local variable. Not wrong, but curious to go back and forth.\n\n-Peff\n"},{"id":"499790","messageId":"20240801100635.GG1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"bfd1dadbf15cf735392ca15b52834f104cbd6538.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 07/19] midx: introduce `bsearch_one_midx()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:06:35Z","receivedAt":"2024-08-01T10:06:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:16PM -0400, Taylor Blau wrote:\n\n> The `bsearch_midx()` function will be extended in a following commit to\n> search for the location of a given object ID across all MIDXs in a chain\n> (or the single non-chain MIDX if no chain is available).\n> \n> While most callers will naturally want to use the updated\n> `bsearch_midx()` function, there are a handful of special cases that\n> will want finer control and will only want to search through a single\n> MIDX.\n> \n> For instance, the object abbreviation code, which cares about object IDs\n> near to where we'd expect to find a match in a MIDX. In that case, we\n> want to look at the nearby matches in each layer of the MIDX chain, not\n> just a single one).\n\nHmm. That seems like a weird thing for the object abbreviation code to\nwant, just because the layers of the midx are essentially random with\nrespect to object names. So you have to search each layer individually\nwhen looking for a contiguous segment of hash names. But maybe that's\nwhy you want the one_midx() variant. Let's see...\n\n> static void unique_in_midx(struct multi_pack_index *m,\n> \t\t\t struct disambiguate_state *ds)\n> {\n> -\tuint32_t num, i, first = 0;\n> -\tconst struct object_id *current = NULL;\n> -\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n> -\t\tds->repo->hash_algo->hexsz : ds->len;\n> -\tnum = m->num_objects;\n> +\tfor (; m; m = m->base_midx) {\n> +\t\tuint32_t num, i, first = 0;\n> +\t\tconst struct object_id *current = NULL;\n> +\t\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n> +\t\t\tds->repo->hash_algo->hexsz : ds->len;\n> \n> -\tif (!num)\n> -\t\treturn;\n> +\t\tnum = m->num_objects + m->num_objects_in_base;\n> \n> -\tbsearch_midx(&ds->bin_pfx, m, &first);\n> +\t\tif (!num)\n> +\t\t\tcontinue;\n> \n> -\t/*\n> -\t * At this point, \"first\" is the location of the lowest object\n> -\t * with an object name that could match \"bin_pfx\". See if we have\n> -\t * 0, 1 or more objects that actually match(es).\n> -\t */\n> -\tfor (i = first; i < num && !ds->ambiguous; i++) {\n> -\t\tstruct object_id oid;\n> -\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n> -\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n> -\t\t\tbreak;\n> -\t\tupdate_candidates(ds, current);\n> +\t\tbsearch_one_midx(&ds->bin_pfx, m, &first);\n> +\n> +\t\t/*\n> +\t\t * At this point, \"first\" is the location of the lowest\n> +\t\t * object with an object name that could match\n> +\t\t * \"bin_pfx\". See if we have 0, 1 or more objects that\n> +\t\t * actually match(es).\n> +\t\t */\n> +\t\tfor (i = first; i < num && !ds->ambiguous; i++) {\n> +\t\t\tstruct object_id oid;\n> +\t\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n> +\t\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n> +\t\t\t\tbreak;\n> +\t\t\tupdate_candidates(ds, current);\n> +\t\t}\n> \t}\n> }\n\nThis is much easier to read with \"-w\", of course. So yeah, the gist of\nit is that we're going to loop over items in the chain via the base_midx\npointer, and then search each individually. So that makes sense.\n\nOne thing that confused me, though, is setting \"num\". From the \"-w\"\ndiff:\n\n - num = m->num_objects;\n +\n + num = m->num_objects + m->num_objects_in_base;\n \n if (!num)\n - return;\n + continue;\n\nBefore we only had one midx, so that was our limit. But now we are\nlooking at \"num\" as a limit in the global size of the chained midx.\nWhich feels weird, since we're just considering a single layer here. We\nseem to use \"num\" in two ways:\n\n - we return if it's 0 (or now continue to the next layer). But\n wouldn't we want to do that per-layer? I don't think it will produce\n wrong answers, but we're less likely to kick in this early return\n (though it's not clear to me when it would ever kick in really; a\n zero-length midx?).\n\n - later we loop \"i\" from \"first\", using \"num\" as a boundary. But this\n \"i\" is a global position, since that's what bsearch_one_midx()\n returns, and what nth_midxed_object_oid() expects.\n\n So I think it's correct, though it feels like bsearch_one_midx()\n should still return the position within that midx (and then\n bsearch_midx() could add back m->num_objects_in_base to get a global\n position). And then I guess likewise there would need to be a\n midx-local version of nth_midxed_object_oid().\n\n I'm not sure if that would make things simpler, or just add to the\n confusion, though. It's easy to get the global/local functions mixed\n up, since of course the global ones also have to take a \"struct\n multi_pack_index\". We could make them take a\n \"multi_pack_index_chain\" instead, but now all of the other code\n which wants to treat chains and single midx files the same has to\n care about the distinction (probably by wrapping the single midx in\n a one-entry chain struct).\n\n\n> @@ -708,37 +712,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED,\n> static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n> \t\t\t\t struct min_abbrev_data *mad)\n\nAnd likewise here (again, much more readable with \"-w\"). Interestingly,\nin this one...\n\n> {\n> -\tint match = 0;\n> -\tuint32_t num, first = 0;\n> -\tstruct object_id oid;\n> -\tconst struct object_id *mad_oid;\n> +\tfor (; m; m = m->base_midx) {\n> +\t\tint match = 0;\n> +\t\tuint32_t num, first = 0;\n> +\t\tstruct object_id oid;\n> +\t\tconst struct object_id *mad_oid;\n> \n> -\tif (!m->num_objects)\n> -\t\treturn;\n> +\t\tif (!m->num_objects)\n> +\t\t\tcontinue;\n\nWe do the early return/continue directly on the layer's m->num_objects,\nwhich makes sense.\n\n> -\tnum = m->num_objects;\n> -\tmad_oid = mad->oid;\n> -\tmatch = bsearch_midx(mad_oid, m, &first);\n> +\t\tnum = m->num_objects + m->num_objects_in_base;\n> +\t\tmad_oid = mad->oid;\n> +\t\tmatch = bsearch_one_midx(mad_oid, m, &first);\n\nBut then of course we go back to the same global \"num\", as we must.\n\nSo I think it's all correct, minus the early continue on \"!num\" in the\nfirst function.\n\n-Peff\n"},{"id":"499791","messageId":"20240801100746.GH1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"38bd45bd24436d3404b089dff5c3d85f4026da97.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 08/19] midx: teach `bsearch_midx()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:07:46Z","receivedAt":"2024-08-01T10:07:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:19PM -0400, Taylor Blau wrote:\n\n> Now that the special cases callers of `bsearch_midx()` have been dealt\n> with, teach `bsearch_midx()` to handle incremental MIDX chains.\n> \n> The incremental MIDX-aware version of `bsearch_midx()` works by\n> repeatedly searching for a given OID in each layer along the\n> `->base_midx` pointer, stopping either when an exact match is found, or\n> the end of the chain is reached.\n\nOK. I think this could have just happened in the last patch, but no big\ndeal either way.\n\n-Peff\n"},{"id":"499792","messageId":"20240801100852.GI1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"342ed560331d95fb0a1d86743dbb999ec7327313.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:08:52Z","receivedAt":"2024-08-01T10:08:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:22PM -0400, Taylor Blau wrote:\n\n> In a similar fashion as in previous commits, teach the function\n> `nth_midxed_offset()` about incremental MIDXs.\n> \n> The given object `pos` is used to find the containing MIDX, and\n> translated back into a MIDX-local position by assigning the return value\n> of `midx_for_object()` to it.\n\nMakes sense.\n\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -368,6 +368,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n> \tconst unsigned char *offset_data;\n> \tuint32_t offset32;\n> \n> +\tpos = midx_for_object(&m, pos);\n> +\n\n\"pos\" reused again! :)\n\nIt certainly makes the diffs nicer to read.\n\n-Peff\n"},{"id":"499793","messageId":"20240801101215.GJ1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"2b335c45ae7832b886ef9adccc2530e4ca53d267.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:12:15Z","receivedAt":"2024-08-01T10:12:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:25PM -0400, Taylor Blau wrote:\n\n> After that point, we only need to make two special considerations within\n> this function:\n> \n> - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()`\n> is a position in the concatenated lexical order of packs, so we must\n> ensure that we subtract `m->num_packs_in_base` before accessing the\n> MIDX-local `packs` array.\n> \n> - Second, we must avoid translating the `pos` back to a MIDX-local\n> index, since we use it as an argument to `nth_midxed_offset()` which\n> expects a position relative to the concatenated lexical order of\n> objects.\n\nOK. I think this is correct, but this would be another place where we\ncould use an nth_midxed_offset_one() function if we had one.\n\nMy thinking was that we'd avoid walking back over the midx chain again.\nBut I guess we don't actually do that, because our midx_for_object()\nwill have overwritten our \"m\" variable, as well. So inside\nnth_midxed_offset_one() we'll immediately realize that the global\nposition is inside the midx we passed in. A little extra arithmetic, but\nthere's no pointer chasing.\n\n-Peff\n"},{"id":"499794","messageId":"20240801101454.GK1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"22de5898f3fec0cfff4289660b5f52aa31533fe1.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 11/19] midx: remove unused `midx_locate_pack()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:14:54Z","receivedAt":"2024-08-01T10:14:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:29PM -0400, Taylor Blau wrote:\n\n> Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14)\n> introduced `midx_locate_pack()`, which was described at the time as a\n> complement to the function `midx_contains_pack()` which allowed\n> callers to determine where in the MIDX lexical order a pack appeared, as\n> opposed to whether or not it was simply contained.\n> \n> 307d75bbe6 suggests that future patches would be added which would\n> introduce callers for this new function, but none ever were, meaning the\n> function has gone unused since its introduction.\n> \n> Clean this up by in effect reverting 307d75bbe6, which removes the\n> unused functions and inlines its definition back into\n> `midx_contains_pack()`.\n> \n> (Looking back through the list archives when 307d75bbe6 was written,\n> this was in preparation for this[1] patch from back when we had the\n> concept of \"disjoint\" packs while developing multi-pack verbatim reuse.\n> That concept was abandoned before the series was merged, but I never\n> dropped what would become 307d75bbe6 from the series, leading to the\n> state prior to this commit).\n> \n> [1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/\n\nNice description of the history. I wish all patches which said \"eh, this\nis unused, let's remove it\" went to the same trouble to make sure we\naren't missing something subtle.\n\n-Peff\n"},{"id":"499795","messageId":"20240801101751.GL1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"fb60f2b022d1dc8c0e6df53253dc924f8d00a382.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:17:51Z","receivedAt":"2024-08-01T10:17:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:32PM -0400, Taylor Blau wrote:\n\n> Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle\n> has been cleaned up, teach the former about how to operate in an\n> incremental MIDX-aware world in a similar fashion as in previous\n> commits.\n> \n> Instead of using either of the two `midx_for_object()` or\n> `midx_for_pack()` helpers, this function is split into two: one that\n> determines whether a pack is contained in a single MIDX, and another\n> which calls the former in a loop over all MIDXs.\n> \n> This approach does not require that we change any of the implementation\n> in what is now `midx_contains_pack_1()` as it still operates over a\n> single MIDX.\n\nMakes sense. There is no ordering or relationship for which packs might\nbe in which midx, so we have to just walk them linearly and check each\npart of the chain.\n\n-Peff\n"},{"id":"499796","messageId":"20240801102517.GM1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"38b642d40423f2e14dd47d676de54f264b9d6f22.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:25:17Z","receivedAt":"2024-08-01T10:25:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:35PM -0400, Taylor Blau wrote:\n\n> The function `midx_preferred_pack()` is used to determine the identity\n> of the preferred pack, which is the identity of a unique pack within\n> the MIDX which is used as a tie-breaker when selecting from which pack\n> to represent an object that appears in multiple packs within the MIDX.\n> \n> Historically we have said that the MIDX's preferred pack has the unique\n> property that all objects from that pack are represented in the MIDX.\n> But that isn't quite true: a more precise statement would be that all\n> objects from that pack *which appear in the MIDX* are selected from that\n> pack.\n> \n> This helps us extend the concept of preferred packs across a MIDX chain,\n> where some object(s) in the preferred pack may appear in other packs\n> in an earlier MIDX layer, in which case those object(s) will not appear\n> in a subsequent MIDX layer from either the preferred pack or any other\n> pack.\n\nOK, that matches my intuition for how the preferred concept should\nexist. I'm not quite clear on how that will affect the code, though.\n\n> diff --git a/midx.c b/midx.c\n> index 0fa8febb9d..d2dbea41e4 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -500,13 +500,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n> int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n> {\n> \tif (m->preferred_pack_idx == -1) {\n> +\t\tuint32_t midx_pos;\n> \t\tif (load_midx_revindex(m) < 0) {\n> \t\t\tm->preferred_pack_idx = -2;\n> \t\t\treturn -1;\n> \t\t}\n> \n> -\t\tm->preferred_pack_idx =\n> -\t\t\tnth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n> +\t\tmidx_pos = pack_pos_to_midx(m, m->num_objects_in_base);\n> +\n> +\t\tm->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos);\n> +\n\nOK, so rather than looking for the pack of object 0, we're looking for\nthe first one in _this_ layer, since the position is global within the\nmidx. That makes some sense, but is pack_pos_to_midx() ready for that?\nIt looks like it just looks at m->revindex_data. Are we going to be\ngenerating a revindex for the whole chain? I'd think that each layer\nwould have its own revindex (and any trickiness would happen at the\ngeneration stage, making sure we don't insert objects that are already\nmentioned in earlier layers).\n\n-Peff\n"},{"id":"499797","messageId":"20240801102906.GN1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"594386da10bc3f3d6364916201438bf453b70098.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:29:06Z","receivedAt":"2024-08-01T10:29:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:38PM -0400, Taylor Blau wrote:\n\n> The function `midx_fanout_add_midx_fanout()` is used to help construct\n> the fanout table when generating a MIDX by reusing data from an existing\n> MIDX.\n\nI'm not sure I understand the original function enough to know if we're\ndoing the right thing. But I notice that after your series, we can only\nget into midx_fanout_add_midx_fanout() if !ctx->incremental. So is this\ncode even used for an incremental midx?\n\nOr is it used if we are writing a non-incremental midx, but trying to\nreuse data from a chained one?\n\n-Peff\n"},{"id":"499811","messageId":"20240801104026.GO1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"dad130799c0d41d14375539bb12978e37849a721.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 15/19] midx: support reading incremental MIDX chains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:40:26Z","receivedAt":"2024-08-01T10:40:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:41PM -0400, Taylor Blau wrote:\n\n> Now that the MIDX machinery's internals have been taught to understand\n> incremental MIDXs over the previous handful of commits, the MIDX\n> machinery itself can begin reading incremental MIDXs.\n> \n> (Note that while the on-disk format for incremental MIDXs has been\n> defined, the writing end has not been implemented. This will take place\n> in the commit after next.)\n> \n> The core of this change involves following the order specified in the\n> MIDX chain and opening up MIDXs in the chain one-by-one, adding them to\n> the previous layer's `->base_midx` pointer at each step.\n\nThis makes it sound like reading a chain file of:\n\n multi-pack-index-$H1.midx\n multi-pack-index-$H2.midx\n multi-pack-index-$H3.midx\n\nwill have H1's base_midx pointing to H2. But the design document from\nthe first patch made me think it went the other way (H1 is the oldest\nmidx, then H2, then H3). For many things the ordering doesn't matter,\nbut I'd think the pseudo-pack order would go from the root of the\nbase_midx walk to the tip. That is, the base_midx pointers go in reverse\nchronological order.\n\nLooking at the code, I think it's doing what I expect. Not sure if I'm\nmis-reading what you wrote above, or if it's wrong.\n\n> [...]\n\nThe code itself all looked reasonable. There are a scary number of spots\nwhere we have to do global/local position conversion. It's hard to know\nif you got them all.\n\n-Peff\n"},{"id":"499823","messageId":"20240801104142.GP1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"ad976ef4137d31929a0feef205396d7642de7da0.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 16/19] midx: implement verification support for incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:41:42Z","receivedAt":"2024-08-01T10:41:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:44PM -0400, Taylor Blau wrote:\n\n> Teach the verification implementation used by `git multi-pack-index\n> verify` to perform verification for incremental MIDX chains by\n> independently validating each layer within the chain.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> midx.c | 47 ++++++++++++++++++++++++++++++-----------------\n> midx.h | 2 ++\n> 2 files changed, 32 insertions(+), 17 deletions(-)\n\nAnother one that benefits from \"-w\". Again, all looked good but I have\nto just assume that you touched all of the spots that needed it.\n\n-Peff\n"},{"id":"499825","messageId":"20240801104647.GQ1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"23912425bf7c0106ed388f7712e7bb822572fe5d.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T10:46:47Z","receivedAt":"2024-08-01T10:46:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:47PM -0400, Taylor Blau wrote:\n\n> Two years ago, commit ff1e653c8e2 (midx: respect\n> 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new\n> environment variable which caused the test suite to write MIDX bitmaps\n> after any 'git repack' invocation.\n> \n> At the time, this was done to help flush out any bugs with MIDX bitmaps\n> that weren't explicitly covered in the t5326-multi-pack-bitmap.sh\n> script.\n> \n> Two years later, that flag has served us well and is no longer providing\n> meaningful coverage, as the script in t5326 has matured substantially\n> and covers many more interesting cases than it did back when ff1e653c8e2\n> was originally written.\n\nI do think it could be providing some value still, just because other\nscripts may create unusual setups that will exercise the code in\ndifferent ways. That said...\n\n> Remove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable\n> as it is no longer serving a useful purpose. More importantly, removing\n> this variable clears the way for us to introduce a new one to help\n> similarly flush out bugs related to incremental MIDX chains.\n> \n> Because these incremental MIDX chains are (for now) incompatible with\n> MIDX bitmaps, we cannot have both.\n\n...if it is one or the other, I think it is better to test the new code.\n\nAnd I do think that midx bitmap code is less likely to be exercised in\ninteresting ways by random parts of the test suite (versus something\nlike GIT_TEST_DEFAULT_HASH, whose effects are pervasive). So I think\nthis is a good tradeoff.\n\n> builtin/repack.c | 12 ++----------\n> ci/run-build-and-tests.sh | 1 -\n> midx.h | 2 --\n> t/README | 4 ----\n> t/t0410-partial-clone.sh | 2 --\n> t/t5310-pack-bitmaps.sh | 4 ----\n> t/t5319-multi-pack-index.sh | 3 +--\n> t/t5326-multi-pack-bitmaps.sh | 3 +--\n> t/t5327-multi-pack-bitmaps-rev.sh | 5 ++---\n> t/t7700-repack.sh | 21 +++++++--------------\n> 10 files changed, 13 insertions(+), 44 deletions(-)\n\nPatch looks good.\n\n-Peff\n"},{"id":"499835","messageId":"20240801110722.GR1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"e2b5961b4556122e594b657efe2f1d3337368cdd.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 19/19] midx: implement support for writing incremental MIDX chains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T11:07:22Z","receivedAt":"2024-08-01T11:07:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:12:53PM -0400, Taylor Blau wrote:\n\n> The implementation for doing so is relatively straightforward, and boils\n> down to a handful of different kinds of changes implemented in this\n> patch:\n> \n> - The `compute_sorted_entries()` function is taught to reject objects\n> which appear in any existing MIDX layer.\n\nOK, I think this is one part I was looking for earlier but didn't see.\nThe implementation looks pretty easy (we can always ask about just the\nearlier layers by feeding the base_midx pointer to midx_has_oid(), etc).\n\n> - Functions like `write_midx_revindex()` are adjusted to write\n> pack_order values which are offset by the number of objects in the\n> base MIDX layer.\n> \n> - The end of `write_midx_internal()` is adjusted to move\n> non-incremental MIDX files when necessary (i.e. when creating an\n> incremental chain with an existing non-incremental MIDX in the\n> repository).\n> \n> There are a handful of other changes that are introduced, like new\n> functions to clear incremental MIDX files that are unrelated to the\n> current chain (using the same \"keep_hash\" mechanism as in the\n> non-incremental case).\n\nThat all makes sense. I wondered a bit about selection of packs, size of\nincremental, etc. We'd probably want a geometric-ish progression, just\nlike with packs, to balance cost of generation versus cost of lookups.\nBut I guess we get that for free to some degree with \"repack\n--geometric\", assuming that our incremental midx is just covering the\nnew packfiles.\n\n> The tests explicitly exercising the new incremental MIDX feature are\n> relatively limited for two reasons:\n> \n> 1. Most of the \"interesting\" behavior is already thoroughly covered in\n> t5319-multi-pack-index.sh, which handles the core logic of reading\n> objects through a MIDX.\n> \n> The new tests in t5334-incremental-multi-pack-index.sh are mostly\n> focused on creating and destroying incremental MIDXs, as well as\n> stitching their results together across layers.\n\nDo you mean here that t5319 will get coverage when\nGIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL is set? In the long run, I\nwonder if we should pull t5319's tests into lib-midx.sh and run them in\nincremental and non-incremental modes.\n\n> 2. A new GIT_TEST environment variable is added called\n> \"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\", which modifies the\n> entire test suite to write incremental MIDXs after repacking when\n> combined with the \"GIT_TEST_MULTI_PACK_INDEX\" variable.\n> \n> This exercises the long tail of other interesting behavior that is\n> defined implicitly throughout the rest of the CI suite. It is\n> likewise added to the linux-TEST-vars job.\n\nMakes sense.\n\n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index 8f34f05087..be1188e736 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -7,6 +7,9 @@ test_description='git repack works correctly'\n> . \"${TEST_DIRECTORY}/lib-midx.sh\"\n> . \"${TEST_DIRECTORY}/lib-terminal.sh\"\n> \n> +GIT_TEST_MULTI_PACK_INDEX=0\n> +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n> [...]\n> -\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adl --write-bitmap-index 2>err &&\n> +\t\tgit repack -Adl --write-bitmap-index 2>err &&\n\nThis pulls the GIT_TEST_MULTI_PACK_INDEX=0 out of the individual tests\nand sets it for the whole file. Are we losing some coverage for the\nother tests? I doubt it's that big a deal either way (and I can\ncertainly see the argument that t7700, which is concerned with the\ndetails of repacking, should be in control of the details of midx\ngeneration). But I wonder if just setting WRITE_INCREMENTAL=0 would be\nenough?\n\nThe rest of the changes all looked pretty reasonable, though again, it's\neasy to review code you wrote and say \"that looks good\" but not realize\nany gotchas that both of us missed.\n\n-Peff\n"},{"id":"499836","messageId":"20240801111410.GS1159276@coredump.intra.peff.net","threadId":"61603","inReplyTo":"cover.1721250704.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/19] midx: incremental multi-pack indexes, part one","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-01T11:14:10Z","receivedAt":"2024-08-01T11:14:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 17, 2024 at 05:11:54PM -0400, Taylor Blau wrote:\n\n> This series implements incremental MIDXs, which allow for storing\n> a MIDX across multiple layers, each with their own distinct set of\n> packs.\n> \n> This round is mostly unchanged from the previous since there has not yet\n> been substantial review. But it does rebase to current 'master' (which\n> is 04f5a52757 (Post 2.46-rc0 batch #2, 2024-07-16), at the time of\n> writing).\n> \n> Importantly, this rebase moves this topic to be based on an ancestor of\n> 0c5a62f14b (midx-write.c: do not read existing MIDX with\n> `packs_to_include`, 2024-06-11), which resulted in a non-trivial\n> conflict prior to this rebase.\n> \n> The rest of the topic is unchanged. I don't expect that we'll see much\n> review here for the next couple of weeks while we are in the -rc phase,\n> but I figured it would be useful to have it on the list for folks that\n> are interested in taking a look.\n> \n> Thanks in advance for any review! :-)\n\nI gave it a pretty thorough look. Everything looks good for the most\npart. I left a few comments, but mostly just thinking my way through\nthings.\n\nThe trickiest parts were:\n\n - the confusion between when we want local per-layer positions versus\n global positions within the whole chainfile, or whether functions\n are operating on a single layer versus the whole chain. I mused a\n bit on how we could do it differently, but ultimately I'm not sure\n there any good solutions.\n\n - the changes you did make look good, but it's hard to know if there's\n code lurking that still needs to be adjusted for chained midx's. For\n that I think I'd turn more towards testing than code review. I'm not\n sure how much interesting coverage we're getting from the GIT_TEST\n variable, just because the repositories made in most of the tests\n are so trivial.\n\n I'd love to see the results on a real workload (both a big repo, but\n also how things behave over days or weeks of repository maintenance\n done with incremental midxs). I know that can be hard to do, though.\n\n-Peff\n"},{"id":"499890","messageId":"ZqvZeWDGDAeZNZjW@nand.local","threadId":"61603","inReplyTo":"20240801091952.GA1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 01/19] Documentation: describe incremental MIDX format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T18:52:41Z","receivedAt":"2024-08-01T18:52:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 05:19:52AM -0400, Jeff King wrote:\n> > +As above, there are no fundamental limitations that stand in the way of\n> > +extending the incremental MIDX format to support reachability bitmaps.\n> > +The design below specifically takes this into account, and support for\n> > +reachability bitmaps will be added in a future patch series. It is\n> > +omitted from this series for the same reason as above.\n>\n> It is nice that you added a bit of a roadmap here about what is\n> implemented and what is not, and that the design takes into account\n> future directions (especially incremental bitmap generation).\n>\n> It does feel a little funny to say \"this series\" in text that will go\n> into the repository (i.e., somebody reading the checked out file will\n> say \"huh? which series?\"). I'm not sure how to word it better, except to\n> maybe just say \"in the future\" and \"it is omitted for now\" (and\n> obviously it's a pretty minor point).\n\ns/this series/the current implementation/ ?\n\nDefinitely an oversight on my part, thanks for catching. I'll squash it\nin.\n\n> > +The `multi-pack-index-$H1.midx` file contains the first layer of the\n> > +multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains\n> > +the second layer of the chain, and so on.\n>\n> Makes sense. How does the chained multi-pack-index.d interact with a\n> singular multi-pack-index? Generally we should not have both at the same\n> time, but I'd imagine they both exist for a brief period when moving\n> from one to another.\n>\n> I assume the rules are the same as for commit-graphs, which use the same\n> on-disk structure. I can't think of a reason to prefer one over the\n> other but this might be a good place to document what does/should\n> happen.\n\nThe commit-graph code reads the non-chained commit-graph first (see\ncommit-graph.c::read_commit_graph_one() for exact details, paraphrased\nhere):\n\n struct commit_graph *g = load_commit_graph_v1(...);\n if (!g)\n g = load_commit_graph_chain(...);\n return g;\n\nand I matched the same for the MIDX code. I think there are reasonable\narguments for preferring either one over the other, so I think the\neasiest thing to do is just throw our hands up and stick with the\nconvention ;-).\n\n> > +=== Object positions for incremental MIDXs\n> > +\n> > +In the original multi-pack-index design, we refer to objects via their\n> > +lexicographic position (by object IDs) within the repository's singular\n> > +multi-pack-index. In the incremental multi-pack-index design, we refer\n> > +to objects via their index into a concatenated lexicographic ordering\n> > +among each component in the MIDX chain.\n>\n> How do duplicate objects work here? I guess there aren't any duplicates\n> in the midx itself, only in the constituent packfiles. So from the\n> perspective of this section, I guess it doesn't matter? And from the\n> perspective of bitmaps (where the duplicate issue came up before), it is\n> business as usual: the midx revindex gives the bit order, and we'd\n> presumably concatenate the individual revindexes in chain order.\n>\n> (Mostly just thinking out loud; I'm not sure there's much for you to\n> answer there).\n\nRight. In a pre-incremental MIDX world, MIDXs contain no duplicate\nobject entries with respect to themselves. In this new world, the same\nis true, with the additional property that MIDXs also contain no\nduplicates with respect to their ancestors (when part of a MIDX chain).\n\n> Looking good so far...\n>\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"499892","messageId":"ZqvaAiCCmb7AmW35@nand.local","threadId":"61603","inReplyTo":"20240801092142.GB1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 02/19] midx: add new fields for incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T18:54:58Z","receivedAt":"2024-08-01T18:55:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 05:21:42AM -0400, Jeff King wrote:\n> On Wed, Jul 17, 2024 at 05:12:01PM -0400, Taylor Blau wrote:\n>\n> > The incremental MIDX chain feature is designed around the idea of\n> > indexing into a concatenated lexicographic ordering of object IDs\n> > present in the MIDX.\n> >\n> > When given an object position, the MIDX machinery needs to be able to\n> > locate both (a) which MIDX layer contains the given object, and (b) at\n> > what position *within that MIDX layer* that object appears.\n> >\n> > To do this, three new fields are added to the `struct multi_pack_index`:\n> >\n> > - struct multi_pack_index *base_midx;\n> > - uint32_t num_objects_in_base;\n> > - uint32_t num_packs_in_base;\n> >\n> > These three fields store the pieces of information suggested by their\n> > respective field names. In turn, the `num_objects_in_base` and\n> > `num_packs_in_base` fields are used to crawl backwards along the\n> > `base_midx` pointer to locate the appropriate position for a given\n> > object within the MIDX that contains it.\n>\n> OK, so base_midx is a back-pointer. I think in theory you could compute\n> num_objects_in_base on the fly by doing that crawl yourself, but we'd\n> want to be able to do it in constant time, rather than O(# of midx)?\n>\n> Makes sense.\n\nYep. As you have seen in the later patches in this series, we end up\nneeding to read num_objects_in_base and num_packs_in_base quite\nfrequently, so it's nice to have them precomputed.\n\nWe could compute them lazily, but they're easy to build up in\nmidx.c::add_midx_to_chain(), where we're already validating that, for\ne.g.:\n\n if (unsigned_add_overflows(midx_chain->num_packs,\n midx_chain->num_packs_in_base)) {\n /* ... */\n }\n midx->num_packs_in_base = midx_chain->num_packs +\n midx_chain->num_packs_in_base;\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"499893","messageId":"ZqvajDLI6Rc6KXMX@nand.local","threadId":"61603","inReplyTo":"20240801093006.GC1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T18:57:16Z","receivedAt":"2024-08-01T18:57:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 05:30:06AM -0400, Jeff King wrote:\n> On Wed, Jul 17, 2024 at 05:12:04PM -0400, Taylor Blau wrote:\n>\n> > [^1]: As a reminder, this means that the object is identified among the\n> > objects contained in all layers of the incremental MIDX chain, not any\n> > particular layer. For example, consider MIDX chain with two individual\n> > MIDXs, one with 4 objects and another with 3 objects. If the MIDX with\n> > 4 objects appears earlier in the chain, then asking for pack \"6\" would\n> > return the second object in the MIDX with 3 objects.\n>\n> I think this is \"object 6\" in the final sentence?\n\nOops, yes. Thanks for spotting, this was as easy as s/pack/object on the\nsecond to last line in the paragraph quoted above.\n\n> OK, so now this function translates a global position into a local one,\n> and then we get the pack id for the local midx/pos, and then turn it\n> back into a global pack id.\n>\n> That all makes sense, but you definitely have to read carefully to make\n> sure which positions/ids are global within the chain and which are local\n> to a midx.\n\nExactly.\n\nThanks,\nTaylor\n"},{"id":"499894","messageId":"ZqvbVQKW7lhtt0A+@nand.local","threadId":"61603","inReplyTo":"20240801093550.GD1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T19:00:37Z","receivedAt":"2024-08-01T19:00:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 05:35:50AM -0400, Jeff King wrote:\n> OK, I'm adequately prepared for more global/local confusion. :)\n\nGood, since there is definitely more where that came from ;-).\n\n> I guess if you just reused pack_int_id for the local id, the diff would\n> be much smaller (this part would remain exactly the same). I dunno which\n> is better, but it was a little curious that the two patches differed in\n> approach. Probably not worth caring too much about, though.\n\nI meandered a lot about different approaches before I arrived at what\nbecame midx_for_pack() and midx_for_object(). So I think declaring a new\nlocal_pack_int_id was a relic from when perhaps the function returned\nvoid and expected to have a uint32_t* to write the translated pack ID\nto.\n\nLater translations make use of the:\n\n pack_int_id = midx_for_pack(&m, pack_int_id);\n\npattern when they do not care about the global pack ID, and this one\nshould as well. I'll update the patch to do that.\n\nThanks,\nTaylor\n"},{"id":"499895","messageId":"ZqvcAQABDIthFUPH@nand.local","threadId":"61603","inReplyTo":"20240801093805.GE1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T19:03:29Z","receivedAt":"2024-08-01T19:03:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 05:38:05AM -0400, Jeff King wrote:\n> To some degree I think the proof will be in the pudding. If you missed\n> any helpers, then the end result is going to crash and burn quite badly\n> when used with a chained midx, and we'd see it in the test suite. And\n> the nice thing is that most of this is abstracted inside these helpers,\n> so we know the set of tricky places is generally limited to the helpers,\n> and not arbitrary bits of midx code.\n\nI think back in the old days, I might have considered experimenting with\nGitHub's fork of Git to build up this feature before sharing it with the\nlist.\n\nBut experience has taught me that it's far better to share early and\noften. Doing so gets you the benefit of having many eyes on the feature,\nnot just from individuals at GitHub.\n\nSelfishly, it also reduces the pain of having to change some on-disk\nformat that has already been widely rolled out within GitHub's\ninfrastructure and similar, but the former motivation is much more\ncompelling.\n\nIn terms of \"the pudding\" here, I think that marking this feature as\nexperimental / incomplete is a good way for us to push this forward and\nbuild up some real-world experience with brave users via a tagged\nversion of Git. Then we can refine it until we are confident it has\ngraduated the \"experimental\" phase.\n\nThanks,\nTaylor\n"},{"id":"499896","messageId":"ZqvdDkMZA8Yxcm/q@nand.local","threadId":"61603","inReplyTo":"20240801093947.GF1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T19:07:58Z","receivedAt":"2024-08-01T19:08:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 05:39:47AM -0400, Jeff King wrote:\n> > int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n> > \t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n> > {\n> > +\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n> > +\n>\n> Heh, after the last one reused the \"n\" variable, now we are back to a\n> separate local variable. Not wrong, but curious to go back and forth.\n\nThis one we care about having both, for a couple of reasons:\nprepare_midx_pack() still expects us to have the global pack_int_id, and\njust as well for bp->pack_int_id.\n\nWe could write this as:\n\n pack_int_id = midx_for_pack(&m, pack_int_id);\n if (prepare_midx_pack(r, m, pack_int_id + m->num_packs_in_base))\n return -1;\n\nBut I found it easier to have a separate local_-prefixed variable for\nwhen referring to the MIDX-local pack identifier.\n\nI'll add a short note in the commit message explaining why we took this\napproach in this commit.\n\nThanks,\nTaylor\n"},{"id":"499898","messageId":"Zqvn+qGz5hireM5X@nand.local","threadId":"61603","inReplyTo":"20240801100635.GG1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 07/19] midx: introduce `bsearch_one_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T19:54:34Z","receivedAt":"2024-08-01T19:54:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:06:35AM -0400, Jeff King wrote:\n> One thing that confused me, though, is setting \"num\". From the \"-w\"\n> diff:\n>\n> - num = m->num_objects;\n> +\n> + num = m->num_objects + m->num_objects_in_base;\n>\n> if (!num)\n> - return;\n> + continue;\n>\n> Before we only had one midx, so that was our limit. But now we are\n> looking at \"num\" as a limit in the global size of the chained midx.\n> Which feels weird, since we're just considering a single layer here. We\n> seem to use \"num\" in two ways:\n>\n> - we return if it's 0 (or now continue to the next layer). But\n> wouldn't we want to do that per-layer? I don't think it will produce\n> wrong answers, but we're less likely to kick in this early return\n> (though it's not clear to me when it would ever kick in really; a\n> zero-length midx?).\n\nThis is definitely a bug. We should certainly do something like:\n\n for (; m; m = m->base_midx) {\n uint32_t num;\n if (!m->num_objects)\n continue;\n\n num = m->num_objects + m->num_objects_in_base;\n /* ... */\n }\n\nI'll go ahead and fix this one up locally, which is easy enough to do.\n\n> So I think it's correct, though it feels like bsearch_one_midx()\n> should still return the position within that midx (and then\n> bsearch_midx() could add back m->num_objects_in_base to get a global\n> position). And then I guess likewise there would need to be a\n> midx-local version of nth_midxed_object_oid().\n\nLike many of the other changes in this series, it's really a matter of\nwhere you put the complexity: either it's in the callers or in the\nfunction itself.\n\nI think here I prefer having bsearch_one_midx() return the global\nposition, since it is directly usable in other top-level functions\nwithin the MIDX API, like being able to pass it directly to\nnth_midxed_object_oid() and etc. below.\n\nThanks,\nTaylor\n"},{"id":"499899","messageId":"ZqvpfRJC7+rCMwWV@nand.local","threadId":"61603","inReplyTo":"20240801101215.GJ1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:01:01Z","receivedAt":"2024-08-01T20:01:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:12:15AM -0400, Jeff King wrote:\n> On Wed, Jul 17, 2024 at 05:12:25PM -0400, Taylor Blau wrote:\n>\n> > After that point, we only need to make two special considerations within\n> > this function:\n> >\n> > - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()`\n> > is a position in the concatenated lexical order of packs, so we must\n> > ensure that we subtract `m->num_packs_in_base` before accessing the\n> > MIDX-local `packs` array.\n> >\n> > - Second, we must avoid translating the `pos` back to a MIDX-local\n> > index, since we use it as an argument to `nth_midxed_offset()` which\n> > expects a position relative to the concatenated lexical order of\n> > objects.\n>\n> OK. I think this is correct, but this would be another place where we\n> could use an nth_midxed_offset_one() function if we had one.\n\nYeah.\n\n> My thinking was that we'd avoid walking back over the midx chain again.\n> But I guess we don't actually do that, because our midx_for_object()\n> will have overwritten our \"m\" variable, as well. So inside\n> nth_midxed_offset_one() we'll immediately realize that the global\n> position is inside the midx we passed in. A little extra arithmetic, but\n> there's no pointer chasing.\n\nYup, exactly.\n\nThanks,\nTaylor\n"},{"id":"499900","messageId":"ZqvplfibWoQFhgEi@nand.local","threadId":"61603","inReplyTo":"20240801101454.GK1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 11/19] midx: remove unused `midx_locate_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:01:25Z","receivedAt":"2024-08-01T20:01:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:14:54AM -0400, Jeff King wrote:\n> Nice description of the history. I wish all patches which said \"eh, this\n> is unused, let's remove it\" went to the same trouble to make sure we\n> aren't missing something subtle.\n\n:-), thanks.\n\n> -Peff\nThanks,\nTaylor\n"},{"id":"499901","messageId":"ZqvqkHph6YYyEHy2@nand.local","threadId":"61603","inReplyTo":"20240801102517.GM1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:05:36Z","receivedAt":"2024-08-01T20:05:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:25:17AM -0400, Jeff King wrote:\n> > diff --git a/midx.c b/midx.c\n> > index 0fa8febb9d..d2dbea41e4 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -500,13 +500,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n> > int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n> > {\n> > \tif (m->preferred_pack_idx == -1) {\n> > +\t\tuint32_t midx_pos;\n> > \t\tif (load_midx_revindex(m) < 0) {\n> > \t\t\tm->preferred_pack_idx = -2;\n> > \t\t\treturn -1;\n> > \t\t}\n> >\n> > -\t\tm->preferred_pack_idx =\n> > -\t\t\tnth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n> > +\t\tmidx_pos = pack_pos_to_midx(m, m->num_objects_in_base);\n> > +\n> > +\t\tm->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos);\n> > +\n>\n> OK, so rather than looking for the pack of object 0, we're looking for\n> the first one in _this_ layer, since the position is global within the\n> midx. That makes some sense, but is pack_pos_to_midx() ready for that?\n> It looks like it just looks at m->revindex_data. Are we going to be\n> generating a revindex for the whole chain? I'd think that each layer\n> would have its own revindex (and any trickiness would happen at the\n> generation stage, making sure we don't insert objects that are already\n> mentioned in earlier layers).\n\npack_pos_to_midx() is kind of ready, and kind of not.\n\nThe way that the pseudo-pack order is constructed within the\nmidx-write.c code, we will write reverse indexes (within each MIDX layer\nitself as a separate chunk) that contain data for each object within\nthat layer in the expected reverse index format.\n\nBut we don't bother writing any reverse indexes for MIDXs which are\nincremental at this point in the multi-series plan, since we just bail\nif the BITMAP flag is set (saying that it is unsupported at this point).\n\nArguably we could have just left this hunk / patch out of the series as\na whole. It's this kind of stuff that's really at the boundary between\nadjacent \"phases\" that I think is awkward no matter which way you slice\nit.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"499902","messageId":"ZqvrZCGYI0J226kn@nand.local","threadId":"61603","inReplyTo":"20240801102906.GN1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:09:08Z","receivedAt":"2024-08-01T20:09:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:29:06AM -0400, Jeff King wrote:\n> On Wed, Jul 17, 2024 at 05:12:38PM -0400, Taylor Blau wrote:\n>\n> > The function `midx_fanout_add_midx_fanout()` is used to help construct\n> > the fanout table when generating a MIDX by reusing data from an existing\n> > MIDX.\n>\n> I'm not sure I understand the original function enough to know if we're\n> doing the right thing. But I notice that after your series, we can only\n> get into midx_fanout_add_midx_fanout() if !ctx->incremental. So is this\n> code even used for an incremental midx?\n\nOriginally it was, but after 0c5a62f14b (midx-write.c: do not read\nexisting MIDX with `packs_to_include`, 2024-06-11) we no longer use this\nfunction in that path. But...\n\n> Or is it used if we are writing a non-incremental midx, but trying to\n> reuse data from a chained one?\n\n...we would use it in this one, so I think the patch stands. I added a\nnote at the end of the commit message to make sure that we don't forget\nwhich paths do and don't reach this function.\n\nThanks,\nTaylor\n"},{"id":"499903","messageId":"ZqvxrH9x6hMN6hjf@nand.local","threadId":"61603","inReplyTo":"20240801104026.GO1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 15/19] midx: support reading incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:35:56Z","receivedAt":"2024-08-01T20:35:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:40:26AM -0400, Jeff King wrote:\n> On Wed, Jul 17, 2024 at 05:12:41PM -0400, Taylor Blau wrote:\n>\n> > Now that the MIDX machinery's internals have been taught to understand\n> > incremental MIDXs over the previous handful of commits, the MIDX\n> > machinery itself can begin reading incremental MIDXs.\n> >\n> > (Note that while the on-disk format for incremental MIDXs has been\n> > defined, the writing end has not been implemented. This will take place\n> > in the commit after next.)\n> >\n> > The core of this change involves following the order specified in the\n> > MIDX chain and opening up MIDXs in the chain one-by-one, adding them to\n> > the previous layer's `->base_midx` pointer at each step.\n>\n> This makes it sound like reading a chain file of:\n>\n> multi-pack-index-$H1.midx\n> multi-pack-index-$H2.midx\n> multi-pack-index-$H3.midx\n>\n> will have H1's base_midx pointing to H2. But the design document from\n> the first patch made me think it went the other way (H1 is the oldest\n> midx, then H2, then H3). For many things the ordering doesn't matter,\n> but I'd think the pseudo-pack order would go from the root of the\n> base_midx walk to the tip. That is, the base_midx pointers go in reverse\n> chronological order.\n>\n> Looking at the code, I think it's doing what I expect. Not sure if I'm\n> mis-reading what you wrote above, or if it's wrong.\n\nThe patch message is just plain wrong here. I switched the sentence\nbeginning with \"The core of this change involves [...]\" to add \"in\nreverse\" to clarify what's going on here.\n\n> > [...]\n>\n> The code itself all looked reasonable. There are a scary number of spots\n> where we have to do global/local position conversion. It's hard to know\n> if you got them all.\n\nAgreed. If you have ideas to make it less scary, do let me know ;-).\n\nThanks,\nTaylor\n"},{"id":"499904","messageId":"Zqvx32vv8u4WDTKg@nand.local","threadId":"61603","inReplyTo":"20240801104647.GQ1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:36:47Z","receivedAt":"2024-08-01T20:36:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 06:46:47AM -0400, Jeff King wrote:\n> ...if it is one or the other, I think it is better to test the new code.\n>\n> And I do think that midx bitmap code is less likely to be exercised in\n> interesting ways by random parts of the test suite (versus something\n> like GIT_TEST_DEFAULT_HASH, whose effects are pervasive). So I think\n> this is a good tradeoff.\n\nThanks, I appreciate your careful reasoning here.\n\n> Patch looks good.\n\nThanks also for the review :-).\n\nThanks,\nTaylor\n"},{"id":"499905","messageId":"ZqvybqlvW/wcKV5g@nand.local","threadId":"61603","inReplyTo":"20240801110722.GR1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 19/19] midx: implement support for writing incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:39:10Z","receivedAt":"2024-08-01T20:39:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 07:07:22AM -0400, Jeff King wrote:\n> > The tests explicitly exercising the new incremental MIDX feature are\n> > relatively limited for two reasons:\n> >\n> > 1. Most of the \"interesting\" behavior is already thoroughly covered in\n> > t5319-multi-pack-index.sh, which handles the core logic of reading\n> > objects through a MIDX.\n> >\n> > The new tests in t5334-incremental-multi-pack-index.sh are mostly\n> > focused on creating and destroying incremental MIDXs, as well as\n> > stitching their results together across layers.\n>\n> Do you mean here that t5319 will get coverage when\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL is set? In the long run, I\n> wonder if we should pull t5319's tests into lib-midx.sh and run them in\n> incremental and non-incremental modes.\n\nI don't know. Part of me thinks that that would be a good idea, but part\nof me also thinks that it would be (a) painful (since many of those\ntests assume that exactly one MIDX exists in the repository\nbefore/during/after each test), and (b) not particularly useful (since\nmuch of the interesting behavior occurs when multiple MIDXs contain\npacks with overlapping objects, see (a)).\n\nThanks,\nTaylor\n"},{"id":"499906","messageId":"ZqvzCUHgDKj6TNQK@nand.local","threadId":"61603","inReplyTo":"20240801111410.GS1159276@coredump.intra.peff.net","subject":"Re: [PATCH v2 00/19] midx: incremental multi-pack indexes, part one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-01T20:41:45Z","receivedAt":"2024-08-01T20:41:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 01, 2024 at 07:14:10AM -0400, Jeff King wrote:\n> On Wed, Jul 17, 2024 at 05:11:54PM -0400, Taylor Blau wrote:\n>\n> > This series implements incremental MIDXs, which allow for storing\n> > a MIDX across multiple layers, each with their own distinct set of\n> > packs.\n> >\n> > This round is mostly unchanged from the previous since there has not yet\n> > been substantial review. But it does rebase to current 'master' (which\n> > is 04f5a52757 (Post 2.46-rc0 batch #2, 2024-07-16), at the time of\n> > writing).\n> >\n> > Importantly, this rebase moves this topic to be based on an ancestor of\n> > 0c5a62f14b (midx-write.c: do not read existing MIDX with\n> > `packs_to_include`, 2024-06-11), which resulted in a non-trivial\n> > conflict prior to this rebase.\n> >\n> > The rest of the topic is unchanged. I don't expect that we'll see much\n> > review here for the next couple of weeks while we are in the -rc phase,\n> > but I figured it would be useful to have it on the list for folks that\n> > are interested in taking a look.\n> >\n> > Thanks in advance for any review! :-)\n>\n> I gave it a pretty thorough look. Everything looks good for the most\n> part. I left a few comments, but mostly just thinking my way through\n> things.\n\nThanks very much.\n\nI squashed all of the feedback that I got from your review into a local\ncopy, which I'll submit as \"v3\" (probably next week, as I am gone for a\nlong weekend starting ~tomorrow and would like to leave others some time\nto review as well).\n\n> - the changes you did make look good, but it's hard to know if there's\n> code lurking that still needs to be adjusted for chained midx's. For\n> that I think I'd turn more towards testing than code review. I'm not\n> sure how much interesting coverage we're getting from the GIT_TEST\n> variable, just because the repositories made in most of the tests\n> are so trivial.\n>\n> I'd love to see the results on a real workload (both a big repo, but\n> also how things behave over days or weeks of repository maintenance\n> done with incremental midxs). I know that can be hard to do, though.\n\nYeah, I agree that this is the biggest gap in this series and the\noverall plan right now. I have some more detailed comments in [1] that I\nthink are useful to the overall approach.\n\nIt basically boils down to declaring the feature \"experimental\" and\nletting users that are comfortable testing on the bleeding edge help us\niron out any bugs (along with rolling it out at GitHub once all the dust\nhas settled on this and subsequent parts).\n\nThanks again for a detailed and helpful review :-).\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/ZqvcAQABDIthFUPH@nand.local/\n"},{"id":"500210","messageId":"cover.1722958595.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1717715060.git.me@ttaylorr.com","subject":"[PATCH v3 00/19] midx: incremental multi-pack indexes, part one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:36:36Z","receivedAt":"2024-08-06T15:36:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements incremental MIDXs, which allow for storing\na MIDX across multiple layers, each with their own distinct set of\npacks.\n\nThis round is also similar to the previous one, but is rebased on\ncurrent 'master' (406f326d27 (The second batch, 2024-08-01)) and has\nbeen updated in response to review from Peff on the previous round.\n\nAs usual, a range-diff is below, but the main changes since last time\nare as follows:\n\n - Documentation improvements to clarify what happens when both an\n incremental- and non-incremental MIDX are both present in a\n repository.\n\n - Commit message typofix on 3/19 to fix an error in one of the\n technical examples.\n\n - Dropped a custom 'local_pack_int_id' in 4/19 to make the remaining\n diff easier to read.\n\n - Minor bugfix in 7/19 where we incorrectly terminated the object\n abbreviation disambiguation step for incremental MIDXs.\n\n - Various additional bits of information in the commit message to\n explain anything that was subtle.\n\nThanks in advance for any review! :-)\n\nTaylor Blau (19):\n Documentation: describe incremental MIDX format\n midx: add new fields for incremental MIDX chains\n midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs\n midx: teach `prepare_midx_pack()` about incremental MIDXs\n midx: teach `nth_midxed_object_oid()` about incremental MIDXs\n midx: teach `nth_bitmapped_pack()` about incremental MIDXs\n midx: introduce `bsearch_one_midx()`\n midx: teach `bsearch_midx()` about incremental MIDXs\n midx: teach `nth_midxed_offset()` about incremental MIDXs\n midx: teach `fill_midx_entry()` about incremental MIDXs\n midx: remove unused `midx_locate_pack()`\n midx: teach `midx_contains_pack()` about incremental MIDXs\n midx: teach `midx_preferred_pack()` about incremental MIDXs\n midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs\n midx: support reading incremental MIDX chains\n midx: implement verification support for incremental MIDXs\n t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n t/t5313-pack-bounds-checks.sh: prepare for sub-directories\n midx: implement support for writing incremental MIDX chains\n\n Documentation/git-multi-pack-index.txt | 11 +-\n Documentation/technical/multi-pack-index.txt | 103 +++++\n builtin/multi-pack-index.c | 2 +\n builtin/repack.c | 8 +-\n ci/run-build-and-tests.sh | 2 +-\n midx-write.c | 326 ++++++++++++---\n midx.c | 405 ++++++++++++++++---\n midx.h | 26 +-\n object-name.c | 99 ++---\n packfile.c | 21 +-\n packfile.h | 4 +\n t/README | 6 +-\n t/helper/test-read-midx.c | 24 +-\n t/lib-bitmap.sh | 6 +-\n t/lib-midx.sh | 28 ++\n t/t0410-partial-clone.sh | 2 -\n t/t5310-pack-bitmaps.sh | 4 -\n t/t5313-pack-bounds-checks.sh | 8 +-\n t/t5319-multi-pack-index.sh | 30 +-\n t/t5326-multi-pack-bitmaps.sh | 4 +-\n t/t5327-multi-pack-bitmaps-rev.sh | 6 +-\n t/t5332-multi-pack-reuse.sh | 2 +\n t/t5334-incremental-multi-pack-index.sh | 46 +++\n t/t7700-repack.sh | 48 +--\n 24 files changed, 960 insertions(+), 261 deletions(-)\n create mode 100755 t/t5334-incremental-multi-pack-index.sh\n\nRange-diff against v2:\n 1: 014588b3ec ! 1: 90b21b11ed Documentation: describe incremental MIDX format\n @@ Documentation/technical/multi-pack-index.txt: Design Details\n +extending the incremental MIDX format to support reachability bitmaps.\n +The design below specifically takes this into account, and support for\n +reachability bitmaps will be added in a future patch series. It is\n -+omitted from this series for the same reason as above.\n ++omitted from the current implementation for the same reason as above.\n ++\n +In brief, to support reachability bitmaps with the incremental MIDX\n +feature, the concept of the pseudo-pack order is extended across each\n @@ Documentation/technical/multi-pack-index.txt: Design Details\n +multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains\n +the second layer of the chain, and so on.\n +\n ++When both an incremental- and non-incremental MIDX are present, the\n ++non-incremental MIDX is always read first.\n ++\n +=== Object positions for incremental MIDXs\n +\n +In the original multi-pack-index design, we refer to objects via their\n 2: 337ebc6de7 = 2: 0d3b19c59f midx: add new fields for incremental MIDX chains\n 3: f449a72877 ! 3: 5cd742b677 midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs\n @@ Commit message\n objects contained in all layers of the incremental MIDX chain, not any\n particular layer. For example, consider MIDX chain with two individual\n MIDXs, one with 4 objects and another with 3 objects. If the MIDX with\n - 4 objects appears earlier in the chain, then asking for pack \"6\" would\n + 4 objects appears earlier in the chain, then asking for object 6 would\n return the second object in the MIDX with 3 objects.\n \n [^2]: Building on the previous example, asking for object 6 in a MIDX\n 4: f88569c819 ! 4: 372104c73d midx: teach `prepare_midx_pack()` about incremental MIDXs\n @@ midx.c: static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t p\n \t\tdie(_(\"bad pack-int-id: %u (%u total packs)\"),\n -\t\t pack_int_id, m->num_packs);\n +\t\t pack_int_id, m->num_packs + m->num_packs_in_base);\n - \n --\tif (m->packs[pack_int_id])\n ++\n +\t*_m = m;\n +\n +\treturn pack_int_id - m->num_packs_in_base;\n @@ midx.c: static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t p\n +{\n +\tstruct strbuf pack_name = STRBUF_INIT;\n +\tstruct packed_git *p;\n -+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n +\n -+\tif (m->packs[local_pack_int_id])\n ++\tpack_int_id = midx_for_pack(&m, pack_int_id);\n + \n + \tif (m->packs[pack_int_id])\n \t\treturn 0;\n - \n - \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n --\t\t m->pack_names[pack_int_id]);\n -+\t\t m->pack_names[local_pack_int_id]);\n - \n - \tp = add_packed_git(pack_name.buf, pack_name.len, m->local);\n - \tstrbuf_release(&pack_name);\n -@@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n - \t\treturn 1;\n - \n - \tp->multi_pack_index = 1;\n --\tm->packs[pack_int_id] = p;\n -+\tm->packs[local_pack_int_id] = p;\n - \tinstall_packed_git(r, p);\n - \tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n - \n 5: ec57ff4349 = 5: e68a3ceff9 midx: teach `nth_midxed_object_oid()` about incremental MIDXs\n 6: 650b8c8c21 ! 6: ff2d7bc5ca midx: teach `nth_bitmapped_pack()` about incremental MIDXs\n @@ Commit message\n ID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset\n when accessing the data within that chunk.\n \n + (Note that the both the call to prepare_midx_pack() and the assignment\n + of bp->pack_int_id both care about the global pack_int_id, so avoid\n + shadowing the given 'pack_int_id' parameter).\n +\n Signed-off-by: Taylor Blau \n \n ## midx.c ##\n 7: bfd1dadbf1 ! 7: 32c3fceada midx: introduce `bsearch_one_midx()`\n @@ object-name.c: static int match_hash(unsigned len, const unsigned char *a, const\n \n -\tif (!num)\n -\t\treturn;\n -+\t\tnum = m->num_objects + m->num_objects_in_base;\n ++\t\tif (!m->num_objects)\n ++\t\t\tcontinue;\n \n -\tbsearch_midx(&ds->bin_pfx, m, &first);\n -+\t\tif (!num)\n -+\t\t\tcontinue;\n ++\t\tnum = m->num_objects + m->num_objects_in_base;\n \n -\t/*\n -\t * At this point, \"first\" is the location of the lowest object\n 8: 38bd45bd24 = 8: 16db6c98ce midx: teach `bsearch_midx()` about incremental MIDXs\n 9: 342ed56033 = 9: 761c7c59ba midx: teach `nth_midxed_offset()` about incremental MIDXs\n10: 2b335c45ae = 10: 8366456d29 midx: teach `fill_midx_entry()` about incremental MIDXs\n11: 22de5898f3 = 11: 909d927c47 midx: remove unused `midx_locate_pack()`\n12: fb60f2b022 = 12: 71127601b5 midx: teach `midx_contains_pack()` about incremental MIDXs\n13: 38b642d404 = 13: 2f98ebb141 midx: teach `midx_preferred_pack()` about incremental MIDXs\n14: 594386da10 ! 14: 550ae2dc93 midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs\n @@ Commit message\n MIDX layers when dealing with an incremental MIDX chain by calling\n itself when given a MIDX with a non-NULL `base_midx`.\n \n + Note that after 0c5a62f14b (midx-write.c: do not read existing MIDX with\n + `packs_to_include`, 2024-06-11), we do not use this function with an\n + existing MIDX (incremental or not) when generating a MIDX with\n + --stdin-packs, and likewise for incremental MIDXs.\n +\n + But it is still used when adding the fanout table from an incremental\n + MIDX when generating a non-incremental MIDX (without --stdin-packs, of\n + course).\n +\n Signed-off-by: Taylor Blau \n \n ## midx-write.c ##\n15: dad130799c ! 15: 9ae1bc415e midx: support reading incremental MIDX chains\n @@ Commit message\n in the commit after next.)\n \n The core of this change involves following the order specified in the\n - MIDX chain and opening up MIDXs in the chain one-by-one, adding them to\n - the previous layer's `->base_midx` pointer at each step.\n + MIDX chain in reverse and opening up MIDXs in the chain one-by-one,\n + adding them to the previous layer's `->base_midx` pointer at each step.\n \n In order to implement this, the `load_multi_pack_index()` function is\n taught to call a new `load_multi_pack_index_chain()` function if loading\n16: ad976ef413 = 16: 3d4181df51 midx: implement verification support for incremental MIDXs\n17: 23912425bf = 17: 3b268f91bf t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n18: 814da1916d = 18: 09d74f8942 t/t5313-pack-bounds-checks.sh: prepare for sub-directories\n19: e2b5961b45 = 19: 5d467d38a8 midx: implement support for writing incremental MIDX chains\n\nbase-commit: 406f326d271e0bacecdb00425422c5fa3f314930\n-- \n2.46.0.46.g406f326d27.dirty\n"},{"id":"500211","messageId":"90b21b11ed7ba11e58a3bfda2c70672f0d7fa16d.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 01/19] Documentation: describe incremental MIDX format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:36:41Z","receivedAt":"2024-08-06T15:36:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement incremental multi-pack indexes (MIDXs) over the\nnext several commits by first describing the relevant prerequisites\n(like a new chunk in the MIDX format, the directory structure for\nincremental MIDXs, etc.)\n\nThe format is described in detail in the patch contents below, but the\nhigh-level description is as follows.\n\nIncremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and\neach `*.midx` within that directory has a single \"parent\" MIDX, which is\nthe MIDX layer immediately before it in the MIDX chain. The chain order\nresides in a file 'multi-pack-index-chain' in the same directory.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/multi-pack-index.txt | 103 +++++++++++++++++++\n 1 file changed, 103 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex f2221d2b44..cc063b30be 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -61,6 +61,109 @@ Design Details\n - The MIDX file format uses a chunk-based approach (similar to the\n commit-graph file) that allows optional data to be added.\n \n+Incremental multi-pack indexes\n+------------------------------\n+\n+As repositories grow in size, it becomes more expensive to write a\n+multi-pack index (MIDX) that includes all packfiles. To accommodate\n+this, the \"incremental multi-pack indexes\" feature allows for combining\n+a \"chain\" of multi-pack indexes.\n+\n+Each individual component of the chain need only contain a small number\n+of packfiles. Appending to the chain does not invalidate earlier parts\n+of the chain, so repositories can control how much time is spent\n+updating the MIDX chain by determining the number of packs in each layer\n+of the MIDX chain.\n+\n+=== Design state\n+\n+At present, the incremental multi-pack indexes feature is missing two\n+important components:\n+\n+ - The ability to rewrite earlier portions of the MIDX chain (i.e., to\n+ \"compact\" some collection of adjacent MIDX layers into a single\n+ MIDX). At present the only supported way of shrinking a MIDX chain\n+ is to rewrite the entire chain from scratch without the `--split`\n+ flag.\n++\n+There are no fundamental limitations that stand in the way of being able\n+to implement this feature. It is omitted from the initial implementation\n+in order to reduce the complexity, but will be added later.\n+\n+ - Support for reachability bitmaps. The classic single MIDX\n+ implementation does support reachability bitmaps (see the section\n+ titled \"multi-pack-index reverse indexes\" in\n+ linkgit:gitformat-pack[5] for more details).\n++\n+As above, there are no fundamental limitations that stand in the way of\n+extending the incremental MIDX format to support reachability bitmaps.\n+The design below specifically takes this into account, and support for\n+reachability bitmaps will be added in a future patch series. It is\n+omitted from the current implementation for the same reason as above.\n++\n+In brief, to support reachability bitmaps with the incremental MIDX\n+feature, the concept of the pseudo-pack order is extended across each\n+layer of the incremental MIDX chain to form a concatenated pseudo-pack\n+order. This concatenation takes place in the same order as the chain\n+itself (in other words, the concatenated pseudo-pack order for a chain\n+`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by\n+the pseudo-pack order for `$H2`, followed by the pseudo-pack order for\n+`$H3`).\n++\n+The layout will then be extended so that each layer of the incremental\n+MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap\n+are offset by the number of objects in the previous layers of the chain.\n+\n+=== File layout\n+\n+Instead of storing a single `multi-pack-index` file (with an optional\n+`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental\n+MIDXs are stored in the following layout:\n+\n+----\n+$GIT_DIR/objects/pack/multi-pack-index.d/\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx\n+$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx\n+----\n+\n+The `multi-pack-index-chain` file contains a list of the incremental\n+MIDX files in the chain, in order. The above example shows a chain whose\n+`multi-pack-index-chain` file would contain the following lines:\n+\n+----\n+$H1\n+$H2\n+$H3\n+----\n+\n+The `multi-pack-index-$H1.midx` file contains the first layer of the\n+multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains\n+the second layer of the chain, and so on.\n+\n+When both an incremental- and non-incremental MIDX are present, the\n+non-incremental MIDX is always read first.\n+\n+=== Object positions for incremental MIDXs\n+\n+In the original multi-pack-index design, we refer to objects via their\n+lexicographic position (by object IDs) within the repository's singular\n+multi-pack-index. In the incremental multi-pack-index design, we refer\n+to objects via their index into a concatenated lexicographic ordering\n+among each component in the MIDX chain.\n+\n+If `objects_nr()` is a function that returns the number of objects in a\n+given MIDX layer, then the index of an object at lexicographic position\n+`i` within, say, $H3 is defined as:\n+\n+----\n+objects_nr($H2) + objects_nr($H1) + i\n+----\n+\n+(in the C implementation, this is often computed as `i +\n+m->num_objects_in_base`).\n+\n Future Work\n -----------\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500212","messageId":"0d3b19c59ff31eb17d283c61f615957e89137fb1.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 02/19] midx: add new fields for incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:36:44Z","receivedAt":"2024-08-06T15:36:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The incremental MIDX chain feature is designed around the idea of\nindexing into a concatenated lexicographic ordering of object IDs\npresent in the MIDX.\n\nWhen given an object position, the MIDX machinery needs to be able to\nlocate both (a) which MIDX layer contains the given object, and (b) at\nwhat position *within that MIDX layer* that object appears.\n\nTo do this, three new fields are added to the `struct multi_pack_index`:\n\n - struct multi_pack_index *base_midx;\n - uint32_t num_objects_in_base;\n - uint32_t num_packs_in_base;\n\nThese three fields store the pieces of information suggested by their\nrespective field names. In turn, the `num_objects_in_base` and\n`num_packs_in_base` fields are used to crawl backwards along the\n`base_midx` pointer to locate the appropriate position for a given\nobject within the MIDX that contains it.\n\nThe following commits will update various parts of the MIDX machinery\n(as well as their callers from outside of midx.c and midx-write.c) to be\naware and make use of these fields when performing object lookups.\n\nSigned-off-by: Taylor Blau \n---\n midx.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/midx.h b/midx.h\nindex 8554f2d616..020e49f77c 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -63,6 +63,10 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_revindex;\n \tsize_t chunk_revindex_len;\n \n+\tstruct multi_pack_index *base_midx;\n+\tuint32_t num_objects_in_base;\n+\tuint32_t num_packs_in_base;\n+\n \tconst char **pack_names;\n \tstruct packed_git **packs;\n \tchar object_dir[FLEX_ARRAY];\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500213","messageId":"5cd742b67761913bd9693102a8b823f353b60f7b.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:18Z","receivedAt":"2024-08-06T15:37:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `nth_midxed_pack_int_id()` takes in a object position in\nMIDX lexicographic order and returns an identifier of the pack from\nwhich that object was selected in the MIDX.\n\nCurrently, the given object position is an index into the lexicographic\norder of objects in a single MIDX. Change this position to instead refer\ninto the concatenated lexicographic order of all MIDXs in a MIDX chain.\n\nThis has two visible effects within the implementation of\n`prepare_midx_pack()`:\n\n - First, the given position is now an index into the concatenated\n lexicographic order of all MIDXs in the order in which they appear\n in the MIDX chain.\n\n - Second the pack ID returned from this function is now also in the\n concatenated order of packs among all layers of the MIDX chain in\n the same order that they appear in the MIDX chain.\n\nTo do this, introduce the first of two general purpose helpers, this one\nbeing `midx_for_object()`. `midx_for_object()` takes a double pointer to\na `struct multi_pack_index` as well as an object `pos` in terms of the\nentire MIDX chain[^1].\n\nThe function chases down the '->base_midx' field until it finds the MIDX\nlayer within the chain that contains the given object. It then:\n\n - modifies the double pointer to point to the containing MIDX, instead\n of the tip of the chain, and\n\n - returns the MIDX-local position[^2] at which the given object can be\n found.\n\nUse this function within `nth_midxed_pack_int_id()` so that the `pos` it\nexpects is now relative to the entire MIDX chain, and that it returns\nthe appropriate pack position for that object.\n\n[^1]: As a reminder, this means that the object is identified among the\n objects contained in all layers of the incremental MIDX chain, not any\n particular layer. For example, consider MIDX chain with two individual\n MIDXs, one with 4 objects and another with 3 objects. If the MIDX with\n 4 objects appears earlier in the chain, then asking for object 6 would\n return the second object in the MIDX with 3 objects.\n\n[^2]: Building on the previous example, asking for object 6 in a MIDX\n chain with (4, 3) objects, respectively, this would set the double\n pointer to point at the MIDX containing three objects, and would\n return an index to the second object within that MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3992b05465..39d358da20 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -242,6 +242,23 @@ void close_midx(struct multi_pack_index *m)\n \tfree(m);\n }\n \n+static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n+{\n+\tstruct multi_pack_index *m = *_m;\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n+\n+\tif (pos >= m->num_objects + m->num_objects_in_base)\n+\t\tdie(_(\"invalid MIDX object position, MIDX is likely corrupt\"));\n+\n+\t*_m = m;\n+\n+\treturn pos - m->num_objects_in_base;\n+}\n+\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n@@ -334,8 +351,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n-\treturn get_be32(m->chunk_object_offsets +\n-\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n+\tpos = midx_for_object(&m, pos);\n+\n+\treturn m->num_packs_in_base + get_be32(m->chunk_object_offsets +\n+\t\t\t\t\t (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n }\n \n int fill_midx_entry(struct repository *r,\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500214","messageId":"372104c73ded1dd7035b16f5aa48827560c4e064.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:21Z","receivedAt":"2024-08-06T15:37:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `prepare_midx_pack()` is part of the midx.h API and\nloads the pack identified by the MIDX-local 'pack_int_id'. This patch\nprepares that function to be aware of an incremental MIDX world.\n\nTo do this, introduce the second of the two general purpose helpers\nmentioned in the previous commit. This commit introduces\n`midx_for_pack()`, which is the pack-specific analog of\n`midx_for_object()`, and works in the same fashion.\n\nLike `midx_for_object()`, this function chases down the '->base_midx'\nfield until it finds the MIDX layer within the chain that contains the\ngiven pack.\n\nUse this function within `prepare_midx_pack()` so that the `pack_int_id`\nit expects is now relative to the entire MIDX chain, and that it\nprepares the given pack in the appropriate MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 28 +++++++++++++++++++++++-----\n 1 file changed, 23 insertions(+), 5 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 39d358da20..07b3981a7a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -259,14 +259,32 @@ static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)\n \treturn pos - m->num_objects_in_base;\n }\n \n-int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n+static uint32_t midx_for_pack(struct multi_pack_index **_m,\n+\t\t\t uint32_t pack_int_id)\n {\n-\tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct packed_git *p;\n+\tstruct multi_pack_index *m = *_m;\n+\twhile (m && pack_int_id < m->num_packs_in_base)\n+\t\tm = m->base_midx;\n \n-\tif (pack_int_id >= m->num_packs)\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for pack ID: %\"PRIu32, pack_int_id);\n+\n+\tif (pack_int_id >= m->num_packs + m->num_packs_in_base)\n \t\tdie(_(\"bad pack-int-id: %u (%u total packs)\"),\n-\t\t pack_int_id, m->num_packs);\n+\t\t pack_int_id, m->num_packs + m->num_packs_in_base);\n+\n+\t*_m = m;\n+\n+\treturn pack_int_id - m->num_packs_in_base;\n+}\n+\n+int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t uint32_t pack_int_id)\n+{\n+\tstruct strbuf pack_name = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\tpack_int_id = midx_for_pack(&m, pack_int_id);\n \n \tif (m->packs[pack_int_id])\n \t\treturn 0;\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500215","messageId":"e68a3ceff9a3dc3d9c2504ed13217cac150c3640.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:24Z","receivedAt":"2024-08-06T15:37:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `nth_midxed_object_oid()` returns the object ID for a given\nobject position in the MIDX lexicographic order.\n\nTeach this function to instead operate over the concatenated\nlexicographic order defined in an earlier step so that it is able to be\nused with incremental MIDXs.\n\nTo do this, we need to both (a) adjust the bounds check for the given\n'n', as well as record the MIDX-local position after chasing the\n`->base_midx` pointer to find the MIDX which contains that object.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 07b3981a7a..64a051cca1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -338,9 +338,11 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n)\n {\n-\tif (n >= m->num_objects)\n+\tif (n >= m->num_objects + m->num_objects_in_base)\n \t\treturn NULL;\n \n+\tn = midx_for_object(&m, n);\n+\n \toidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n),\n \t\tthe_repository->hash_algo);\n \treturn oid;\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500216","messageId":"ff2d7bc5ca079dcf9b571822227a578ac0072952.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:27Z","receivedAt":"2024-08-06T15:37:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach the function\n`nth_bitmapped_pack()` about incremental MIDXs by translating the given\n`pack_int_id` from the concatenated lexical order to a MIDX-local\nlexical position.\n\nWhen accessing the containing MIDX's array of packs, use the local pack\nID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset\nwhen accessing the data within that chunk.\n\n(Note that the both the call to prepare_midx_pack() and the assignment\nof bp->pack_int_id both care about the global pack_int_id, so avoid\nshadowing the given 'pack_int_id' parameter).\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 64a051cca1..25350152f1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -311,17 +311,19 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n {\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\n \tif (!m->chunk_bitmapped_packs)\n \t\treturn error(_(\"MIDX does not contain the BTMP chunk\"));\n \n \tif (prepare_midx_pack(r, m, pack_int_id))\n \t\treturn error(_(\"could not load bitmapped pack %\"PRIu32), pack_int_id);\n \n-\tbp->p = m->packs[pack_int_id];\n+\tbp->p = m->packs[local_pack_int_id];\n \tbp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs +\n-\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id);\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id);\n \tbp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs +\n-\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id +\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id +\n \t\t\t\t sizeof(uint32_t));\n \tbp->pack_int_id = pack_int_id;\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500217","messageId":"32c3fceada73e8d53f4239eafc1f440c072920c8.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 07/19] midx: introduce `bsearch_one_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:30Z","receivedAt":"2024-08-06T15:37:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `bsearch_midx()` function will be extended in a following commit to\nsearch for the location of a given object ID across all MIDXs in a chain\n(or the single non-chain MIDX if no chain is available).\n\nWhile most callers will naturally want to use the updated\n`bsearch_midx()` function, there are a handful of special cases that\nwill want finer control and will only want to search through a single\nMIDX.\n\nFor instance, the object abbreviation code, which cares about object IDs\nnear to where we'd expect to find a match in a MIDX. In that case, we\nwant to look at the nearby matches in each layer of the MIDX chain, not\njust a single one).\n\nSplit the more fine-grained control out into a separate function called\n`bsearch_one_midx()` which searches only a single MIDX.\n\nAt present both `bsearch_midx()` and `bsearch_one_midx()` have identical\nbehavior, but the following commit will rewrite the former to be aware\nof incremental MIDXs for the remaining non-special case callers.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 17 +++++++--\n midx.h | 5 ++-\n object-name.c | 99 +++++++++++++++++++++++++++------------------------\n 3 files changed, 71 insertions(+), 50 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 25350152f1..bd6e3f26c9 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -330,10 +330,21 @@ int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \treturn 0;\n }\n \n-int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n+int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result)\n {\n-\treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n-\t\t\t the_hash_algo->rawsz, result);\n+\tint ret = bsearch_hash(oid->hash, m->chunk_oid_fanout,\n+\t\t\t m->chunk_oid_lookup, the_hash_algo->rawsz,\n+\t\t\t result);\n+\tif (result)\n+\t\t*result += m->num_objects_in_base;\n+\treturn ret;\n+}\n+\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result)\n+{\n+\t\treturn bsearch_one_midx(oid, m, result);\n }\n \n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/midx.h b/midx.h\nindex 020e49f77c..46c53d69ff 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -90,7 +90,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n-int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result);\n+int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n+\t\t uint32_t *result);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/object-name.c b/object-name.c\nindex 527b853ac4..739d46f9cf 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -134,28 +134,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char\n static void unique_in_midx(struct multi_pack_index *m,\n \t\t\t struct disambiguate_state *ds)\n {\n-\tuint32_t num, i, first = 0;\n-\tconst struct object_id *current = NULL;\n-\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n-\t\tds->repo->hash_algo->hexsz : ds->len;\n-\tnum = m->num_objects;\n+\tfor (; m; m = m->base_midx) {\n+\t\tuint32_t num, i, first = 0;\n+\t\tconst struct object_id *current = NULL;\n+\t\tint len = ds->len > ds->repo->hash_algo->hexsz ?\n+\t\t\tds->repo->hash_algo->hexsz : ds->len;\n \n-\tif (!num)\n-\t\treturn;\n+\t\tif (!m->num_objects)\n+\t\t\tcontinue;\n \n-\tbsearch_midx(&ds->bin_pfx, m, &first);\n+\t\tnum = m->num_objects + m->num_objects_in_base;\n \n-\t/*\n-\t * At this point, \"first\" is the location of the lowest object\n-\t * with an object name that could match \"bin_pfx\". See if we have\n-\t * 0, 1 or more objects that actually match(es).\n-\t */\n-\tfor (i = first; i < num && !ds->ambiguous; i++) {\n-\t\tstruct object_id oid;\n-\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n-\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n-\t\t\tbreak;\n-\t\tupdate_candidates(ds, current);\n+\t\tbsearch_one_midx(&ds->bin_pfx, m, &first);\n+\n+\t\t/*\n+\t\t * At this point, \"first\" is the location of the lowest\n+\t\t * object with an object name that could match\n+\t\t * \"bin_pfx\". See if we have 0, 1 or more objects that\n+\t\t * actually match(es).\n+\t\t */\n+\t\tfor (i = first; i < num && !ds->ambiguous; i++) {\n+\t\t\tstruct object_id oid;\n+\t\t\tcurrent = nth_midxed_object_oid(&oid, m, i);\n+\t\t\tif (!match_hash(len, ds->bin_pfx.hash, current->hash))\n+\t\t\t\tbreak;\n+\t\t\tupdate_candidates(ds, current);\n+\t\t}\n \t}\n }\n \n@@ -708,37 +712,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED,\n static void find_abbrev_len_for_midx(struct multi_pack_index *m,\n \t\t\t\t struct min_abbrev_data *mad)\n {\n-\tint match = 0;\n-\tuint32_t num, first = 0;\n-\tstruct object_id oid;\n-\tconst struct object_id *mad_oid;\n+\tfor (; m; m = m->base_midx) {\n+\t\tint match = 0;\n+\t\tuint32_t num, first = 0;\n+\t\tstruct object_id oid;\n+\t\tconst struct object_id *mad_oid;\n \n-\tif (!m->num_objects)\n-\t\treturn;\n+\t\tif (!m->num_objects)\n+\t\t\tcontinue;\n \n-\tnum = m->num_objects;\n-\tmad_oid = mad->oid;\n-\tmatch = bsearch_midx(mad_oid, m, &first);\n+\t\tnum = m->num_objects + m->num_objects_in_base;\n+\t\tmad_oid = mad->oid;\n+\t\tmatch = bsearch_one_midx(mad_oid, m, &first);\n \n-\t/*\n-\t * first is now the position in the packfile where we would insert\n-\t * mad->hash if it does not exist (or the position of mad->hash if\n-\t * it does exist). Hence, we consider a maximum of two objects\n-\t * nearby for the abbreviation length.\n-\t */\n-\tmad->init_len = 0;\n-\tif (!match) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first))\n-\t\t\textend_abbrev_len(&oid, mad);\n-\t} else if (first < num - 1) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n-\t\t\textend_abbrev_len(&oid, mad);\n+\t\t/*\n+\t\t * first is now the position in the packfile where we\n+\t\t * would insert mad->hash if it does not exist (or the\n+\t\t * position of mad->hash if it does exist). Hence, we\n+\t\t * consider a maximum of two objects nearby for the\n+\t\t * abbreviation length.\n+\t\t */\n+\t\tmad->init_len = 0;\n+\t\tif (!match) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t} else if (first < num - 1) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first + 1))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t}\n+\t\tif (first > 0) {\n+\t\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n+\t\t\t\textend_abbrev_len(&oid, mad);\n+\t\t}\n+\t\tmad->init_len = mad->cur_len;\n \t}\n-\tif (first > 0) {\n-\t\tif (nth_midxed_object_oid(&oid, m, first - 1))\n-\t\t\textend_abbrev_len(&oid, mad);\n-\t}\n-\tmad->init_len = mad->cur_len;\n }\n \n static void find_abbrev_len_for_pack(struct packed_git *p,\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500218","messageId":"16db6c98cecb103c9c4aae263dc8b9071be2caea.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 08/19] midx: teach `bsearch_midx()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:33Z","receivedAt":"2024-08-06T15:37:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the special cases callers of `bsearch_midx()` have been dealt\nwith, teach `bsearch_midx()` to handle incremental MIDX chains.\n\nThe incremental MIDX-aware version of `bsearch_midx()` works by\nrepeatedly searching for a given OID in each layer along the\n`->base_midx` pointer, stopping either when an exact match is found, or\nthe end of the chain is reached.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex bd6e3f26c9..83857cbd1e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -344,7 +344,10 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result)\n {\n-\t\treturn bsearch_one_midx(oid, m, result);\n+\tfor (; m; m = m->base_midx)\n+\t\tif (bsearch_one_midx(oid, m, result))\n+\t\t\treturn 1;\n+\treturn 0;\n }\n \n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500219","messageId":"761c7c59ba1a245732d86a4e1d6c24baf742a1cb.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:36Z","receivedAt":"2024-08-06T15:37:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach the function\n`nth_midxed_offset()` about incremental MIDXs.\n\nThe given object `pos` is used to find the containing MIDX, and\ntranslated back into a MIDX-local position by assigning the return value\nof `midx_for_object()` to it.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 83857cbd1e..346e58dec7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -369,6 +369,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n \n+\tpos = midx_for_object(&m, pos);\n+\n \toffset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;\n \toffset32 = get_be32(offset_data + sizeof(uint32_t));\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500220","messageId":"8366456d29b870533b5a788a372375b11103b8c7.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:40Z","receivedAt":"2024-08-06T15:37:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, teach the `fill_midx_entry()`\nfunction to work in a incremental MIDX-aware fashion.\n\nThis function, unlike others which accept an index into either the\nlexical order of objects or packs, takes in an object_id, and attempts\nto fill a caller-provided 'struct pack_entry' with the remaining pieces\nof information about that object from the MIDX.\n\nThe function uses `bsearch_midx()` which fills out the frame-local 'pos'\nvariable, recording the given object_id's lexical position within the\nMIDX chain, if found (if no matching object ID was found, we'll return\nimmediately without filling out the `pack_entry` structure).\n\nOnce given that position, we jump back through the `->base_midx` pointer\nto ensure that our `m` points at the MIDX layer which contains the given\nobject_id (and not an ancestor or descendant of it in the chain). Note\nthat we can drop the bounds check \"if (pos >= m->num_objects)\" because\n`midx_for_object()` performs this check for us.\n\nAfter that point, we only need to make two special considerations within\nthis function:\n\n - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()`\n is a position in the concatenated lexical order of packs, so we must\n ensure that we subtract `m->num_packs_in_base` before accessing the\n MIDX-local `packs` array.\n\n - Second, we must avoid translating the `pos` back to a MIDX-local\n index, since we use it as an argument to `nth_midxed_offset()` which\n expects a position relative to the concatenated lexical order of\n objects.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 346e58dec7..5e4e6f9b65 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -407,14 +407,12 @@ int fill_midx_entry(struct repository *r,\n \tif (!bsearch_midx(oid, m, &pos))\n \t\treturn 0;\n \n-\tif (pos >= m->num_objects)\n-\t\treturn 0;\n-\n+\tmidx_for_object(&m, pos);\n \tpack_int_id = nth_midxed_pack_int_id(m, pos);\n \n \tif (prepare_midx_pack(r, m, pack_int_id))\n \t\treturn 0;\n-\tp = m->packs[pack_int_id];\n+\tp = m->packs[pack_int_id - m->num_packs_in_base];\n \n \t/*\n \t* We are about to tell the caller where they can locate the\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500221","messageId":"909d927c470f24c723ba067fce2016ab80b01885.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 11/19] midx: remove unused `midx_locate_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:42Z","receivedAt":"2024-08-06T15:37:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14)\nintroduced `midx_locate_pack()`, which was described at the time as a\ncomplement to the function `midx_contains_pack()` which allowed\ncallers to determine where in the MIDX lexical order a pack appeared, as\nopposed to whether or not it was simply contained.\n\n307d75bbe6 suggests that future patches would be added which would\nintroduce callers for this new function, but none ever were, meaning the\nfunction has gone unused since its introduction.\n\nClean this up by in effect reverting 307d75bbe6, which removes the\nunused functions and inlines its definition back into\n`midx_contains_pack()`.\n\n(Looking back through the list archives when 307d75bbe6 was written,\nthis was in preparation for this[1] patch from back when we had the\nconcept of \"disjoint\" packs while developing multi-pack verbatim reuse.\nThat concept was abandoned before the series was merged, but I never\ndropped what would become 307d75bbe6 from the series, leading to the\nstate prior to this commit).\n\n[1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 13 ++-----------\n midx.h | 2 --\n 2 files changed, 2 insertions(+), 13 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 5e4e6f9b65..50f131e59a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -466,8 +466,7 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n-\t\t uint32_t *pos)\n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -478,11 +477,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \n \t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n-\t\tif (!cmp) {\n-\t\t\tif (pos)\n-\t\t\t\t*pos = mid;\n+\t\tif (!cmp)\n \t\t\treturn 1;\n-\t\t}\n \t\tif (cmp > 0) {\n \t\t\tfirst = mid + 1;\n \t\t\tcontinue;\n@@ -493,11 +489,6 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \treturn 0;\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n-{\n-\treturn midx_locate_pack(m, idx_or_pack_name, NULL);\n-}\n-\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\ndiff --git a/midx.h b/midx.h\nindex 46c53d69ff..86af7dfc5e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -102,8 +102,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n-int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n-\t\t uint32_t *pos);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500222","messageId":"71127601b5d412eff69d6451e328e4b1c74b4caa.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:46Z","receivedAt":"2024-08-06T15:37:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle\nhas been cleaned up, teach the former about how to operate in an\nincremental MIDX-aware world in a similar fashion as in previous\ncommits.\n\nInstead of using either of the two `midx_for_object()` or\n`midx_for_pack()` helpers, this function is split into two: one that\ndetermines whether a pack is contained in a single MIDX, and another\nwhich calls the former in a loop over all MIDXs.\n\nThis approach does not require that we change any of the implementation\nin what is now `midx_contains_pack_1()` as it still operates over a\nsingle MIDX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 50f131e59a..454c27b673 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -466,7 +466,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+static int midx_contains_pack_1(struct multi_pack_index *m,\n+\t\t\t\tconst char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -489,6 +490,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \treturn 0;\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+{\n+\tfor (; m; m = m->base_midx)\n+\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500223","messageId":"2f98ebb141e7d9231d9f7688b3ebab849e8fa689.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:49Z","receivedAt":"2024-08-06T15:37:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `midx_preferred_pack()` is used to determine the identity\nof the preferred pack, which is the identity of a unique pack within\nthe MIDX which is used as a tie-breaker when selecting from which pack\nto represent an object that appears in multiple packs within the MIDX.\n\nHistorically we have said that the MIDX's preferred pack has the unique\nproperty that all objects from that pack are represented in the MIDX.\nBut that isn't quite true: a more precise statement would be that all\nobjects from that pack *which appear in the MIDX* are selected from that\npack.\n\nThis helps us extend the concept of preferred packs across a MIDX chain,\nwhere some object(s) in the preferred pack may appear in other packs\nin an earlier MIDX layer, in which case those object(s) will not appear\nin a subsequent MIDX layer from either the preferred pack or any other\npack.\n\nExtend the concept of preferred packs by using the pack which represents\nthe object at the first position in MIDX pseudo-pack order belonging to\nthe current MIDX layer (i.e., at position 'm->num_objects_in_base').\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 454c27b673..918349576f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -501,13 +501,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n {\n \tif (m->preferred_pack_idx == -1) {\n+\t\tuint32_t midx_pos;\n \t\tif (load_midx_revindex(m) < 0) {\n \t\t\tm->preferred_pack_idx = -2;\n \t\t\treturn -1;\n \t\t}\n \n-\t\tm->preferred_pack_idx =\n-\t\t\tnth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\t\tmidx_pos = pack_pos_to_midx(m, m->num_objects_in_base);\n+\n+\t\tm->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos);\n+\n \t} else if (m->preferred_pack_idx == -2)\n \t\treturn -1; /* no revindex */\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500224","messageId":"550ae2dc933c6d0c45fead9737e405c4025af9ad.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:52Z","receivedAt":"2024-08-06T15:37:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `midx_fanout_add_midx_fanout()` is used to help construct\nthe fanout table when generating a MIDX by reusing data from an existing\nMIDX.\n\nPrepare this function to work with incremental MIDXs by making a few\nchanges:\n\n - The bounds checks need to be adjusted to start object lookups taking\n into account the number of objects in the previous MIDX layer (i.e.,\n by starting the lookups at position `m->num_objects_in_base` instead\n of position 0).\n\n - Likewise, the bounds checks need to end at `m->num_objects_in_base`\n objects after `m->num_objects`.\n\n - Finally, `midx_fanout_add_midx_fanout()` needs to recur on earlier\n MIDX layers when dealing with an incremental MIDX chain by calling\n itself when given a MIDX with a non-NULL `base_midx`.\n\nNote that after 0c5a62f14b (midx-write.c: do not read existing MIDX with\n`packs_to_include`, 2024-06-11), we do not use this function with an\nexisting MIDX (incremental or not) when generating a MIDX with\n--stdin-packs, and likewise for incremental MIDXs.\n\nBut it is still used when adding the fanout table from an incremental\nMIDX when generating a non-incremental MIDX (without --stdin-packs, of\ncourse).\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex a77ee73c68..0accbdbb04 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -196,7 +196,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n-\tif (pos >= m->num_objects)\n+\tif (pos >= m->num_objects + m->num_objects_in_base)\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n@@ -247,12 +247,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n \t\t\t\t\tuint32_t cur_fanout,\n \t\t\t\t\tint preferred_pack)\n {\n-\tuint32_t start = 0, end;\n+\tuint32_t start = m->num_objects_in_base, end;\n \tuint32_t cur_object;\n \n+\tif (m->base_midx)\n+\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n+\t\t\t\t\t preferred_pack);\n+\n \tif (cur_fanout)\n-\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n-\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\t\tstart += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\tend = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);\n \n \tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\tif ((preferred_pack > -1) &&\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500225","messageId":"9ae1bc415e9973a403a9bb3adb8a97df3bdfad2a.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 15/19] midx: support reading incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:55Z","receivedAt":"2024-08-06T15:37:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the MIDX machinery's internals have been taught to understand\nincremental MIDXs over the previous handful of commits, the MIDX\nmachinery itself can begin reading incremental MIDXs.\n\n(Note that while the on-disk format for incremental MIDXs has been\ndefined, the writing end has not been implemented. This will take place\nin the commit after next.)\n\nThe core of this change involves following the order specified in the\nMIDX chain in reverse and opening up MIDXs in the chain one-by-one,\nadding them to the previous layer's `->base_midx` pointer at each step.\n\nIn order to implement this, the `load_multi_pack_index()` function is\ntaught to call a new `load_multi_pack_index_chain()` function if loading\na non-incremental MIDX failed via `load_multi_pack_index_one()`.\n\nWhen loading a MIDX chain, `load_midx_chain_fd_st()` reads each line in\nthe file one-by-one and dispatches calls to\n`load_multi_pack_index_one()` to read each layer of the MIDX chain. When\na layer was successfully read, it is added to the MIDX chain by calling\n`add_midx_to_chain()` which validates the contents of the `BASE` chunk,\nperforms some bounds checks on the number of combined packs and objects,\nand attaches the new MIDX by assigning its `base_midx` pointer to the\nexisting part of the chain.\n\nAs a supplement to this, introduce a new mode in the test-read-midx\ntest-tool which allows us to read the information for a specific MIDX in\nthe chain by specifying its trailing checksum via the command-line\narguments like so:\n\n $ test-tool read-midx .git/objects [checksum]\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 184 +++++++++++++++++++++++++++++++++++---\n midx.h | 7 ++\n packfile.c | 5 +-\n t/helper/test-read-midx.c | 24 +++--\n 4 files changed, 201 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 918349576f..54c06cbb86 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -91,7 +91,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n \n-struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n+static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir,\n+\t\t\t\t\t\t\t const char *midx_name,\n+\t\t\t\t\t\t\t int local)\n {\n \tstruct multi_pack_index *m = NULL;\n \tint fd;\n@@ -99,31 +101,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tsize_t midx_size;\n \tvoid *midx_map = NULL;\n \tuint32_t hash_version;\n-\tstruct strbuf midx_name = STRBUF_INIT;\n \tuint32_t i;\n \tconst char *cur_pack_name;\n \tstruct chunkfile *cf = NULL;\n \n-\tget_midx_filename(&midx_name, object_dir);\n-\n-\tfd = git_open(midx_name.buf);\n+\tfd = git_open(midx_name);\n \n \tif (fd < 0)\n \t\tgoto cleanup_fail;\n \tif (fstat(fd, &st)) {\n-\t\terror_errno(_(\"failed to read %s\"), midx_name.buf);\n+\t\terror_errno(_(\"failed to read %s\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n \n \tmidx_size = xsize_t(st.st_size);\n \n \tif (midx_size < MIDX_MIN_SIZE) {\n-\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name.buf);\n+\t\terror(_(\"multi-pack-index file %s is too small\"), midx_name);\n \t\tgoto cleanup_fail;\n \t}\n \n-\tstrbuf_release(&midx_name);\n-\n \tmidx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \n@@ -213,7 +210,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n cleanup_fail:\n \tfree(m);\n-\tstrbuf_release(&midx_name);\n \tfree_chunkfile(cf);\n \tif (midx_map)\n \t\tmunmap(midx_map, midx_size);\n@@ -222,6 +218,173 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \treturn NULL;\n }\n \n+void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir)\n+{\n+\tstrbuf_addf(buf, \"%s/pack/multi-pack-index.d\", object_dir);\n+}\n+\n+void get_midx_chain_filename(struct strbuf *buf, const char *object_dir)\n+{\n+\tget_midx_chain_dirname(buf, object_dir);\n+\tstrbuf_addstr(buf, \"/multi-pack-index-chain\");\n+}\n+\n+void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n+\t\t\t\t const unsigned char *hash, const char *ext)\n+{\n+\tget_midx_chain_dirname(buf, object_dir);\n+\tstrbuf_addf(buf, \"/multi-pack-index-%s.%s\", hash_to_hex(hash), ext);\n+}\n+\n+static int open_multi_pack_index_chain(const char *chain_file,\n+\t\t\t\t int *fd, struct stat *st)\n+{\n+\t*fd = git_open(chain_file);\n+\tif (*fd < 0)\n+\t\treturn 0;\n+\tif (fstat(*fd, st)) {\n+\t\tclose(*fd);\n+\t\treturn 0;\n+\t}\n+\tif (st->st_size < the_hash_algo->hexsz) {\n+\t\tclose(*fd);\n+\t\tif (!st->st_size) {\n+\t\t\t/* treat empty files the same as missing */\n+\t\t\terrno = ENOENT;\n+\t\t} else {\n+\t\t\twarning(_(\"multi-pack-index chain file too small\"));\n+\t\t\terrno = EINVAL;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+static int add_midx_to_chain(struct multi_pack_index *midx,\n+\t\t\t struct multi_pack_index *midx_chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tif (midx_chain) {\n+\t\tif (unsigned_add_overflows(midx_chain->num_packs,\n+\t\t\t\t\t midx_chain->num_packs_in_base)) {\n+\t\t\twarning(_(\"pack count in base MIDX too high: %\"PRIuMAX),\n+\t\t\t\t(uintmax_t)midx_chain->num_packs_in_base);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (unsigned_add_overflows(midx_chain->num_objects,\n+\t\t\t\t\t midx_chain->num_objects_in_base)) {\n+\t\t\twarning(_(\"object count in base MIDX too high: %\"PRIuMAX),\n+\t\t\t\t(uintmax_t)midx_chain->num_objects_in_base);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tmidx->num_packs_in_base = midx_chain->num_packs +\n+\t\t\tmidx_chain->num_packs_in_base;\n+\t\tmidx->num_objects_in_base = midx_chain->num_objects +\n+\t\t\tmidx_chain->num_objects_in_base;\n+\t}\n+\n+\tmidx->base_midx = midx_chain;\n+\tmidx->has_chain = 1;\n+\n+\treturn 1;\n+}\n+\n+static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir,\n+\t\t\t\t\t\t int local,\n+\t\t\t\t\t\t int fd, struct stat *st,\n+\t\t\t\t\t\t int *incomplete_chain)\n+{\n+\tstruct multi_pack_index *midx_chain = NULL;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id *layers = NULL;\n+\tint valid = 1;\n+\tuint32_t i, count;\n+\tFILE *fp = xfdopen(fd, \"r\");\n+\n+\tcount = st->st_size / (the_hash_algo->hexsz + 1);\n+\tCALLOC_ARRAY(layers, count);\n+\n+\tfor (i = 0; i < count; i++) {\n+\t\tstruct multi_pack_index *m;\n+\n+\t\tif (strbuf_getline_lf(&buf, fp) == EOF)\n+\t\t\tbreak;\n+\n+\t\tif (get_oid_hex(buf.buf, &layers[i])) {\n+\t\t\twarning(_(\"invalid multi-pack-index chain: line '%s' \"\n+\t\t\t\t \"not a hash\"),\n+\t\t\t\tbuf.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tvalid = 0;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tget_split_midx_filename_ext(&buf, object_dir, layers[i].hash,\n+\t\t\t\t\t MIDX_EXT_MIDX);\n+\t\tm = load_multi_pack_index_one(object_dir, buf.buf, local);\n+\n+\t\tif (m) {\n+\t\t\tif (add_midx_to_chain(m, midx_chain, layers, i)) {\n+\t\t\t\tmidx_chain = m;\n+\t\t\t\tvalid = 1;\n+\t\t\t} else {\n+\t\t\t\tclose_midx(m);\n+\t\t\t}\n+\t\t}\n+\t\tif (!valid) {\n+\t\t\twarning(_(\"unable to find all multi-pack index files\"));\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree(layers);\n+\tfclose(fp);\n+\tstrbuf_release(&buf);\n+\n+\t*incomplete_chain = !valid;\n+\treturn midx_chain;\n+}\n+\n+static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir,\n+\t\t\t\t\t\t\t int local)\n+{\n+\tstruct strbuf chain_file = STRBUF_INIT;\n+\tstruct stat st;\n+\tint fd;\n+\tstruct multi_pack_index *m = NULL;\n+\n+\tget_midx_chain_filename(&chain_file, object_dir);\n+\tif (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) {\n+\t\tint incomplete;\n+\t\t/* ownership of fd is taken over by load function */\n+\t\tm = load_midx_chain_fd_st(object_dir, local, fd, &st,\n+\t\t\t\t\t &incomplete);\n+\t}\n+\n+\tstrbuf_release(&chain_file);\n+\treturn m;\n+}\n+\n+struct multi_pack_index *load_multi_pack_index(const char *object_dir,\n+\t\t\t\t\t int local)\n+{\n+\tstruct strbuf midx_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m;\n+\n+\tget_midx_filename(&midx_name, object_dir);\n+\n+\tm = load_multi_pack_index_one(object_dir, midx_name.buf, local);\n+\tif (!m)\n+\t\tm = load_multi_pack_index_chain(object_dir, local);\n+\n+\tstrbuf_release(&midx_name);\n+\n+\treturn m;\n+}\n+\n void close_midx(struct multi_pack_index *m)\n {\n \tuint32_t i;\n@@ -230,6 +393,7 @@ void close_midx(struct multi_pack_index *m)\n \t\treturn;\n \n \tclose_midx(m->next);\n+\tclose_midx(m->base_midx);\n \n \tmunmap((unsigned char *)m->data, m->data_len);\n \ndiff --git a/midx.h b/midx.h\nindex 86af7dfc5e..94de16a8c4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -24,6 +24,7 @@ struct bitmapped_pack;\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n #define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n+#define MIDX_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n@@ -50,6 +51,7 @@ struct multi_pack_index {\n \tint preferred_pack_idx;\n \n \tint local;\n+\tint has_chain;\n \n \tconst unsigned char *chunk_pack_names;\n \tsize_t chunk_pack_names_len;\n@@ -80,11 +82,16 @@ struct multi_pack_index {\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n+#define MIDX_EXT_MIDX \"midx\"\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\n void get_midx_filename_ext(struct strbuf *out, const char *object_dir,\n \t\t\t const unsigned char *hash, const char *ext);\n+void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir);\n+void get_midx_chain_filename(struct strbuf *buf, const char *object_dir);\n+void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n+\t\t\t\t const unsigned char *hash, const char *ext);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\ndiff --git a/packfile.c b/packfile.c\nindex 813584646f..1eb18e3041 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -880,7 +880,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!report_garbage)\n \t\treturn;\n \n-\tif (!strcmp(file_name, \"multi-pack-index\"))\n+\tif (!strcmp(file_name, \"multi-pack-index\") ||\n+\t !strcmp(file_name, \"multi-pack-index.d\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n \t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n@@ -1064,7 +1065,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \tprepare_packed_git(r);\n \tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n \t\tuint32_t i;\n-\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 83effc2b5f..69757e94fc 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -9,8 +9,10 @@\n #include \"packfile.h\"\n #include \"setup.h\"\n #include \"gettext.h\"\n+#include \"pack-revindex.h\"\n \n-static int read_midx_file(const char *object_dir, int show_objects)\n+static int read_midx_file(const char *object_dir, const char *checksum,\n+\t\t\t int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -21,6 +23,13 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \tif (!m)\n \t\treturn 1;\n \n+\tif (checksum) {\n+\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n+\t\t\tm = m->base_midx;\n+\t\tif (!m)\n+\t\t\treturn 1;\n+\t}\n+\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t m->signature,\n \t m->version,\n@@ -54,7 +63,8 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \t\tstruct pack_entry e;\n \n \t\tfor (i = 0; i < m->num_objects; i++) {\n-\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tnth_midxed_object_oid(&oid, m,\n+\t\t\t\t\t i + m->num_objects_in_base);\n \t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n \n \t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n@@ -111,7 +121,7 @@ static int read_midx_bitmapped_packs(const char *object_dir)\n \tif (!midx)\n \t\treturn 1;\n \n-\tfor (i = 0; i < midx->num_packs; i++) {\n+\tfor (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) {\n \t\tif (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)\n \t\t\treturn 1;\n \n@@ -127,16 +137,16 @@ static int read_midx_bitmapped_packs(const char *object_dir)\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n+\tif (!(argc == 2 || argc == 3 || argc == 4))\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n-\t\treturn read_midx_file(argv[2], 1);\n+\t\treturn read_midx_file(argv[2], argv[3], 1);\n \telse if (!strcmp(argv[1], \"--checksum\"))\n \t\treturn read_midx_checksum(argv[2]);\n \telse if (!strcmp(argv[1], \"--preferred-pack\"))\n \t\treturn read_midx_preferred_pack(argv[2]);\n \telse if (!strcmp(argv[1], \"--bitmap\"))\n \t\treturn read_midx_bitmapped_packs(argv[2]);\n-\treturn read_midx_file(argv[1], 0);\n+\treturn read_midx_file(argv[1], argv[2], 0);\n }\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500226","messageId":"3d4181df518ece2268534182a11f66b2d84302e9.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 16/19] midx: implement verification support for incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:37:58Z","receivedAt":"2024-08-06T15:38:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the verification implementation used by `git multi-pack-index\nverify` to perform verification for incremental MIDX chains by\nindependently validating each layer within the chain.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 47 ++++++++++++++++++++++++++++++-----------------\n midx.h | 2 ++\n 2 files changed, 32 insertions(+), 17 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 54c06cbb86..a53d65702d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -470,6 +470,13 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \treturn 0;\n }\n \n+struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n+\t\t\t\t uint32_t pack_int_id)\n+{\n+\tuint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);\n+\treturn m->packs[local_pack_int_id];\n+}\n+\n #define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n \n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n@@ -818,6 +825,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \tuint32_t i;\n \tstruct progress *progress = NULL;\n \tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\tstruct multi_pack_index *curr;\n \tverify_midx_error = 0;\n \n \tif (!m) {\n@@ -840,8 +848,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Looking for referenced packfiles\"),\n-\t\t\t\t\t m->num_packs);\n-\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\t\t\t m->num_packs + m->num_packs_in_base);\n+\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\tif (prepare_midx_pack(r, m, i))\n \t\t\tmidx_report(\"failed to load pack in position %d\", i);\n \n@@ -861,17 +869,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Verifying OID order in multi-pack-index\"),\n \t\t\t\t\t\t m->num_objects - 1);\n-\tfor (i = 0; i < m->num_objects - 1; i++) {\n-\t\tstruct object_id oid1, oid2;\n \n-\t\tnth_midxed_object_oid(&oid1, m, i);\n-\t\tnth_midxed_object_oid(&oid2, m, i + 1);\n+\tfor (curr = m; curr; curr = curr->base_midx) {\n+\t\tfor (i = 0; i < m->num_objects - 1; i++) {\n+\t\t\tstruct object_id oid1, oid2;\n \n-\t\tif (oidcmp(&oid1, &oid2) >= 0)\n-\t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n-\t\t\t\t i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n+\t\t\tnth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i);\n+\t\t\tnth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1);\n \n-\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\t\tif (oidcmp(&oid1, &oid2) >= 0)\n+\t\t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n+\t\t\t\t\t i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n+\n+\t\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \n@@ -881,8 +892,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t * each of the objects and only require 1 packfile to be open at a\n \t * time.\n \t */\n-\tALLOC_ARRAY(pairs, m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n+\tALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base);\n+\tfor (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {\n \t\tpairs[i].pos = i;\n \t\tpairs[i].pack_int_id = nth_midxed_pack_int_id(m, i);\n \t}\n@@ -896,16 +907,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Verifying object offsets\"), m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n+\tfor (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {\n \t\tstruct object_id oid;\n \t\tstruct pack_entry e;\n \t\toff_t m_offset, p_offset;\n \n \t\tif (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id &&\n-\t\t m->packs[pairs[i-1].pack_int_id])\n-\t\t{\n-\t\t\tclose_pack_fd(m->packs[pairs[i-1].pack_int_id]);\n-\t\t\tclose_pack_index(m->packs[pairs[i-1].pack_int_id]);\n+\t\t nth_midxed_pack(m, pairs[i-1].pack_int_id)) {\n+\t\t\tuint32_t pack_int_id = pairs[i-1].pack_int_id;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tclose_pack_fd(p);\n+\t\t\tclose_pack_index(p);\n \t\t}\n \n \t\tnth_midxed_object_oid(&oid, m, pairs[i].pos);\ndiff --git a/midx.h b/midx.h\nindex 94de16a8c4..9d30935589 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -95,6 +95,8 @@ void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n+struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n+\t\t\t\t uint32_t pack_int_id);\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500227","messageId":"3b268f91bf3ec74cee5fc3ceaab19d700e30233e.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:38:01Z","receivedAt":"2024-08-06T15:38:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Two years ago, commit ff1e653c8e2 (midx: respect\n'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new\nenvironment variable which caused the test suite to write MIDX bitmaps\nafter any 'git repack' invocation.\n\nAt the time, this was done to help flush out any bugs with MIDX bitmaps\nthat weren't explicitly covered in the t5326-multi-pack-bitmap.sh\nscript.\n\nTwo years later, that flag has served us well and is no longer providing\nmeaningful coverage, as the script in t5326 has matured substantially\nand covers many more interesting cases than it did back when ff1e653c8e2\nwas originally written.\n\nRemove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable\nas it is no longer serving a useful purpose. More importantly, removing\nthis variable clears the way for us to introduce a new one to help\nsimilarly flush out bugs related to incremental MIDX chains.\n\nBecause these incremental MIDX chains are (for now) incompatible with\nMIDX bitmaps, we cannot have both.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 12 ++----------\n ci/run-build-and-tests.sh | 1 -\n midx.h | 2 --\n t/README | 4 ----\n t/t0410-partial-clone.sh | 2 --\n t/t5310-pack-bitmaps.sh | 4 ----\n t/t5319-multi-pack-index.sh | 3 +--\n t/t5326-multi-pack-bitmaps.sh | 3 +--\n t/t5327-multi-pack-bitmaps-rev.sh | 5 ++---\n t/t7700-repack.sh | 21 +++++++--------------\n 10 files changed, 13 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f0317fa94a..8499bf0e12 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1217,10 +1217,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!write_midx &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n-\t} else if (write_bitmaps &&\n-\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n-\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n-\t\twrite_bitmaps = 0;\n \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0 && !write_midx;\n@@ -1518,12 +1514,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (run_update_server_info)\n \t\tupdate_server_info(0);\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n-\t\tunsigned flags = 0;\n-\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n-\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n-\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n-\t}\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, 0);\n \n cleanup:\n \tstring_list_clear(&names, 1);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 98dda42045..e6fd68630c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -25,7 +25,6 @@ linux-TEST-vars)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n-\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\ndiff --git a/midx.h b/midx.h\nindex 9d30935589..3714cad2cc 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,8 +29,6 @@ struct bitmapped_pack;\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n-#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n-\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex 724ee58195..3cee0c05e2 100644\n--- a/t/README\n+++ b/t/README\n@@ -445,10 +445,6 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n-'--bitmap' option on all invocations of 'git multi-pack-index write',\n-and ignores pack-objects' '--write-bitmap-index'.\n-\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 2c30c86e7b..34bdb3ab1f 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -5,8 +5,6 @@ test_description='partial clone'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n-# missing promisor objects cause repacks which write bitmaps to fail\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n # When enabled, some commands will write commit-graphs. This causes fsck\n # to fail when delete_object() is called because fsck will attempt to\n # verify the out-of-sync commit graph.\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex d7fd71360e..a6de7c5764 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n-# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n-# their place.\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n-\n # Likewise, allow individual tests to control whether or not they use\n # the boundary-based traversal.\n sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ace5ac3b61..8c54fc0655 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -600,8 +600,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 916da389b6..1cb3e3ff08 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -4,10 +4,9 @@ test_description='exercise basic multi-pack bitmap functionality'\n . ./test-lib.sh\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n \n-# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# We'll be writing our own MIDX, so avoid getting confused by the\n # automatic ones.\n GIT_TEST_MULTI_PACK_INDEX=0\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n # This test exercise multi-pack bitmap functionality where the object order is\n # stored and read from a special chunk within the MIDX, so use the default\ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex e65e311cd7..23db949c20 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -5,10 +5,9 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)'\n . ./test-lib.sh\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n \n-# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n-# automatic ones.\n+# We'll be writing our own MIDX, so avoid getting confused by the automatic\n+# ones.\n GIT_TEST_MULTI_PACK_INDEX=0\n-GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n # Unlike t5326, this test exercise multi-pack bitmap functionality where the\n # object order is stored in a separate .rev file.\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 127efe99f8..8f34f05087 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -70,14 +70,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n+\tgit repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -284,8 +283,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o\n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n \trm -f bare.git/objects/pack/*.bitmap &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad &&\n+\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -296,8 +294,7 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -308,8 +305,7 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad 2>stderr &&\n+\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -320,8 +316,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -ad 2>stderr &&\n+\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -342,9 +337,7 @@ test_expect_success 'repacking with a filter works' '\n '\n \n test_expect_success '--filter fails with --write-bitmap-index' '\n-\ttest_must_fail \\\n-\t\tenv GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n-\t\tgit -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n+\ttest_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none\n '\n \n test_expect_success 'repacking with two filters works' '\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500228","messageId":"09d74f89423e1a3b5486aa9691ffdb8dea2600f6.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 18/19] t/t5313-pack-bounds-checks.sh: prepare for sub-directories","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:38:04Z","receivedAt":"2024-08-06T15:38:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for sub-directories to appear in $GIT_DIR/objects/pack by\nadjusting the copy, remove, and chmod invocations to perform their\nbehavior recursively.\n\nThis prepares us for the new $GIT_DIR/objects/pack/multi-pack-index.d\ndirectory which will be added in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n t/t5313-pack-bounds-checks.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh\nindex ceaa6700a2..86fc73f9fb 100755\n--- a/t/t5313-pack-bounds-checks.sh\n+++ b/t/t5313-pack-bounds-checks.sh\n@@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true\n \n clear_base () {\n \ttest_when_finished 'restore_base' &&\n-\trm -f $base\n+\trm -r -f $base\n }\n \n restore_base () {\n-\tcp base-backup/* .git/objects/pack/\n+\tcp -r base-backup/* .git/objects/pack/\n }\n \n do_pack () {\n@@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' '\n \tgit commit -m base &&\n \tgit repack -ad &&\n \tbase=$(echo .git/objects/pack/*) &&\n-\tchmod +w $base &&\n+\tchmod -R +w $base &&\n \tmkdir base-backup &&\n-\tcp $base base-backup/ &&\n+\tcp -r $base base-backup/ &&\n \tobject=$(git rev-parse HEAD:file)\n '\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500229","messageId":"5d467d38a8d454228032610b3506040b63eb6369.1722958596.git.me@ttaylorr.com","threadId":"61603","inReplyTo":"cover.1722958595.git.me@ttaylorr.com","subject":"[PATCH v3 19/19] midx: implement support for writing incremental MIDX chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-06T15:38:07Z","receivedAt":"2024-08-06T15:38:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the rest of the MIDX subsystem and relevant callers have been\nupdated to learn about how to read and process incremental MIDX chains,\nlet's finally update the implementation in `write_midx_internal()` to be\nable to write incremental MIDX chains.\n\nThis new feature is available behind the `--incremental` option for the\n`multi-pack-index` builtin, like so:\n\n $ git multi-pack-index write --incremental\n\nThe implementation for doing so is relatively straightforward, and boils\ndown to a handful of different kinds of changes implemented in this\npatch:\n\n - The `compute_sorted_entries()` function is taught to reject objects\n which appear in any existing MIDX layer.\n\n - Functions like `write_midx_revindex()` are adjusted to write\n pack_order values which are offset by the number of objects in the\n base MIDX layer.\n\n - The end of `write_midx_internal()` is adjusted to move\n non-incremental MIDX files when necessary (i.e. when creating an\n incremental chain with an existing non-incremental MIDX in the\n repository).\n\nThere are a handful of other changes that are introduced, like new\nfunctions to clear incremental MIDX files that are unrelated to the\ncurrent chain (using the same \"keep_hash\" mechanism as in the\nnon-incremental case).\n\nThe tests explicitly exercising the new incremental MIDX feature are\nrelatively limited for two reasons:\n\n 1. Most of the \"interesting\" behavior is already thoroughly covered in\n t5319-multi-pack-index.sh, which handles the core logic of reading\n objects through a MIDX.\n\n The new tests in t5334-incremental-multi-pack-index.sh are mostly\n focused on creating and destroying incremental MIDXs, as well as\n stitching their results together across layers.\n\n 2. A new GIT_TEST environment variable is added called\n \"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\", which modifies the\n entire test suite to write incremental MIDXs after repacking when\n combined with the \"GIT_TEST_MULTI_PACK_INDEX\" variable.\n\n This exercises the long tail of other interesting behavior that is\n defined implicitly throughout the rest of the CI suite. It is\n likewise added to the linux-TEST-vars job.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 11 +-\n builtin/multi-pack-index.c | 2 +\n builtin/repack.c | 8 +-\n ci/run-build-and-tests.sh | 1 +\n midx-write.c | 314 ++++++++++++++++++++----\n midx.c | 62 ++++-\n midx.h | 4 +\n packfile.c | 16 +-\n packfile.h | 4 +\n t/README | 4 +\n t/lib-bitmap.sh | 6 +-\n t/lib-midx.sh | 28 +++\n t/t5319-multi-pack-index.sh | 27 +-\n t/t5326-multi-pack-bitmaps.sh | 1 +\n t/t5327-multi-pack-bitmaps-rev.sh | 1 +\n t/t5332-multi-pack-reuse.sh | 2 +\n t/t5334-incremental-multi-pack-index.sh | 46 ++++\n t/t7700-repack.sh | 27 +-\n 18 files changed, 460 insertions(+), 104 deletions(-)\n create mode 100755 t/t5334-incremental-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 3696506eb3..631d5c7d15 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -64,6 +64,12 @@ The file given at `` is expected to be readable, and can contain\n duplicates. (If a given OID is given more than once, it is marked as\n preferred if at least one instance of it begins with the special `+`\n marker).\n+\n+\t--incremental::\n+\t\tWrite an incremental MIDX file containing only objects\n+\t\tand packs not present in an existing MIDX layer.\n+\t\tMigrates non-incremental MIDXs to incremental ones when\n+\t\tnecessary. Incompatible with `--bitmap`.\n --\n \n verify::\n@@ -74,6 +80,8 @@ expire::\n \thave no objects referenced by the MIDX (with the exception of\n \t`.keep` packs and cruft packs). Rewrite the MIDX file afterward\n \tto remove all references to these pack-files.\n++\n+NOTE: this mode is incompatible with incremental MIDX files.\n \n repack::\n \tCreate a new pack-file containing objects in small pack-files\n@@ -95,7 +103,8 @@ repack::\n +\n If `repack.packKeptObjects` is `false`, then any pack-files with an\n associated `.keep` file will not be selected for the batch to repack.\n-\n++\n+NOTE: this mode is incompatible with incremental MIDX files.\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 9cf1a32d65..8805cbbeb3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"progress\", &opts.flags,\n \t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n+\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n+\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 8499bf0e12..7608430a37 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1514,8 +1514,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (run_update_server_info)\n \t\tupdate_server_info(0);\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n+\t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n+\t}\n \n cleanup:\n \tstring_list_clear(&names, 1);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex e6fd68630c..2e28d02b20 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -25,6 +25,7 @@ linux-TEST-vars)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\ndiff --git a/midx-write.c b/midx-write.c\nindex 0accbdbb04..e74b3d82fa 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -17,6 +17,8 @@\n #include \"refs.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"path.h\"\n+#include \"pack-revindex.h\"\n \n #define PACK_EXPIRED UINT_MAX\n #define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n@@ -25,7 +27,11 @@\n \n extern int midx_checksum_valid(struct multi_pack_index *m);\n extern void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash);\n+\t\t\t\t const char *keep_hash);\n+extern void clear_incremental_midx_files_ext(const char *object_dir,\n+\t\t\t\t\t const char *ext,\n+\t\t\t\t\t const char **keep_hashes,\n+\t\t\t\t\t uint32_t hashes_nr);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -86,6 +92,7 @@ struct write_midx_context {\n \tsize_t nr;\n \tsize_t alloc;\n \tstruct multi_pack_index *m;\n+\tstruct multi_pack_index *base_midx;\n \tstruct progress *progress;\n \tunsigned pack_paths_checked;\n \n@@ -99,6 +106,9 @@ struct write_midx_context {\n \n \tint preferred_pack_idx;\n \n+\tint incremental;\n+\tuint32_t num_multi_pack_indexes_before;\n+\n \tstruct string_list *to_include;\n };\n \n@@ -122,6 +132,9 @@ static int should_include_pack(const struct write_midx_context *ctx,\n \t */\n \tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\treturn 0;\n+\telse if (ctx->base_midx && midx_contains_pack(ctx->base_midx,\n+\t\t\t\t\t\t file_name))\n+\t\treturn 0;\n \telse if (ctx->to_include &&\n \t\t !string_list_has_string(ctx->to_include, file_name))\n \t\treturn 0;\n@@ -338,7 +351,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tif (ctx->m)\n+\t\tif (ctx->m && !ctx->incremental)\n \t\t\tmidx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,\n \t\t\t\t\t\t ctx->preferred_pack_idx);\n \n@@ -364,6 +377,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n \t\t\t\t\t\t&fanout.entries[cur_object].oid))\n \t\t\t\tcontinue;\n+\t\t\tif (ctx->incremental && ctx->base_midx &&\n+\t\t\t midx_has_oid(ctx->base_midx,\n+\t\t\t\t\t &fanout.entries[cur_object].oid))\n+\t\t\t\tcontinue;\n \n \t\t\tALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1),\n \t\t\t\t alloc_objects);\n@@ -547,10 +564,16 @@ static int write_midx_revindex(struct hashfile *f,\n \t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n+\tuint32_t i, nr_base;\n+\n+\tif (ctx->incremental && ctx->base_midx)\n+\t\tnr_base = ctx->base_midx->num_objects +\n+\t\t\tctx->base_midx->num_objects_in_base;\n+\telse\n+\t\tnr_base = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\t\thashwrite_be32(f, ctx->pack_order[i] + nr_base);\n \n \treturn 0;\n }\n@@ -579,12 +602,18 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n {\n \tstruct midx_pack_order_data *data;\n-\tuint32_t *pack_order;\n+\tuint32_t *pack_order, base_objects = 0;\n \tuint32_t i;\n \n \ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n \n+\tif (ctx->incremental && ctx->base_midx)\n+\t\tbase_objects = ctx->base_midx->num_objects +\n+\t\t\tctx->base_midx->num_objects_in_base;\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tALLOC_ARRAY(data, ctx->entries_nr);\n+\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n@@ -596,12 +625,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \n \tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n \n-\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n \t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = i;\n+\t\t\tpack->bitmap_pos = i + base_objects;\n \t\tpack->bitmap_nr++;\n \t\tpack_order[i] = data[i].nr;\n \t}\n@@ -649,7 +677,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \tprepare_packing_data(the_repository, pdata);\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tuint32_t pos = ctx->pack_order[i];\n+\t\tstruct pack_midx_entry *from = &ctx->entries[pos];\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n \t\toe_set_in_pack(pdata, to,\n@@ -897,37 +926,130 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n static int fill_packs_from_midx(struct write_midx_context *ctx,\n \t\t\t\tconst char *preferred_pack_name, uint32_t flags)\n {\n-\tuint32_t i;\n+\tstruct multi_pack_index *m;\n \n-\tfor (i = 0; i < ctx->m->num_packs; i++) {\n-\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n+\tfor (m = ctx->m; m; m = m->base_midx) {\n+\t\tuint32_t i;\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tif (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) {\n \t\t\t/*\n \t\t\t * If generating a reverse index, need to have\n \t\t\t * packed_git's loaded to compare their\n \t\t\t * mtimes and object count.\n \t\t\t *\n-\t\t\t *\n \t\t\t * If a preferred pack is specified, need to\n \t\t\t * have packed_git's loaded to ensure the chosen\n \t\t\t * preferred pack has a non-zero object count.\n \t\t\t */\n-\t\t\tif (prepare_midx_pack(the_repository, ctx->m, i))\n-\t\t\t\treturn error(_(\"could not load pack\"));\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX ||\n+\t\t\t preferred_pack_name) {\n+\t\t\t\tif (prepare_midx_pack(the_repository, m,\n+\t\t\t\t\t\t m->num_packs_in_base + i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\treturn 1;\n+\t\t\t\t}\n \n-\t\t\tif (open_pack_index(ctx->m->packs[i]))\n-\t\t\t\tdie(_(\"could not open index for %s\"),\n-\t\t\t\t ctx->m->packs[i]->pack_name);\n+\t\t\t\tif (open_pack_index(m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t m->packs[i]->pack_name);\n+\t\t\t}\n+\n+\t\t\tfill_pack_info(&ctx->info[ctx->nr++], m->packs[i],\n+\t\t\t\t m->pack_names[i],\n+\t\t\t\t m->num_packs_in_base + i);\n \t\t}\n-\n-\t\tfill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i],\n-\t\t\t ctx->m->pack_names[i], i);\n \t}\n-\n \treturn 0;\n }\n \n+static struct {\n+\tconst char *non_split;\n+\tconst char *split;\n+} midx_exts[] = {\n+\t{NULL, MIDX_EXT_MIDX},\n+\t{MIDX_EXT_BITMAP, MIDX_EXT_BITMAP},\n+\t{MIDX_EXT_REV, MIDX_EXT_REV},\n+};\n+\n+static int link_midx_to_chain(struct multi_pack_index *m)\n+{\n+\tstruct strbuf from = STRBUF_INIT;\n+\tstruct strbuf to = STRBUF_INIT;\n+\tint ret = 0;\n+\tsize_t i;\n+\n+\tif (!m || m->has_chain) {\n+\t\t/*\n+\t\t * Either no MIDX previously existed, or it was already\n+\t\t * part of a MIDX chain. In both cases, we have nothing\n+\t\t * to link, so return early.\n+\t\t */\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n+\t\tconst unsigned char *hash = get_midx_checksum(m);\n+\n+\t\tget_midx_filename_ext(&from, m->object_dir, hash,\n+\t\t\t\t midx_exts[i].non_split);\n+\t\tget_split_midx_filename_ext(&to, m->object_dir, hash,\n+\t\t\t\t\t midx_exts[i].split);\n+\n+\t\tif (link(from.buf, to.buf) < 0 && errno != ENOENT) {\n+\t\t\tret = error_errno(_(\"unable to link '%s' to '%s'\"),\n+\t\t\t\t\t from.buf, to.buf);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tstrbuf_reset(&from);\n+\t\tstrbuf_reset(&to);\n+\t}\n+\n+done:\n+\tstrbuf_release(&from);\n+\tstrbuf_release(&to);\n+\treturn ret;\n+}\n+\n+static void clear_midx_files(const char *object_dir,\n+\t\t\t const char **hashes,\n+\t\t\t uint32_t hashes_nr,\n+\t\t\t unsigned incremental)\n+{\n+\t/*\n+\t * if incremental:\n+\t * - remove all non-incremental MIDX files\n+\t * - remove any incremental MIDX files not in the current one\n+\t *\n+\t * if non-incremental:\n+\t * - remove all incremental MIDX files\n+\t * - remove any non-incremental MIDX files not matching the current\n+\t * hash\n+\t */\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n+\tuint32_t i, j;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tclear_incremental_midx_files_ext(object_dir, exts[i],\n+\t\t\t\t\t\t hashes, hashes_nr);\n+\t\tfor (j = 0; j < hashes_nr; j++)\n+\t\t\tclear_midx_files_ext(object_dir, exts[i], hashes[j]);\n+\t}\n+\n+\tif (incremental)\n+\t\tget_midx_filename(&buf, object_dir);\n+\telse\n+\t\tget_midx_chain_filename(&buf, object_dir);\n+\n+\tif (unlink(buf.buf) && errno != ENOENT)\n+\t\tdie_errno(_(\"failed to clear multi-pack-index at %s\"), buf.buf);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t struct string_list *packs_to_drop,\n@@ -940,42 +1062,66 @@ static int write_midx_internal(const char *object_dir,\n \tuint32_t i, start_pack;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n+\tstruct tempfile *incr;\n \tstruct write_midx_context ctx = { 0 };\n \tint bitmapped_packs_concat_len = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tconst char **keep_hashes = NULL;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n \n-\tget_midx_filename(&midx_name, object_dir);\n+\tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n+\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n+\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n+\n+\tif (ctx.incremental)\n+\t\tstrbuf_addf(&midx_name,\n+\t\t\t \"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX\",\n+\t\t\t object_dir);\n+\telse\n+\t\tget_midx_filename(&midx_name, object_dir);\n \tif (safe_create_leading_directories(midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n \n-\tif (!packs_to_include) {\n-\t\t/*\n-\t\t * Only reference an existing MIDX when not filtering which\n-\t\t * packs to include, since all packs and objects are copied\n-\t\t * blindly from an existing MIDX if one is present.\n-\t\t */\n-\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n-\t}\n+\tif (!packs_to_include || ctx.incremental) {\n+\t\tstruct multi_pack_index *m = lookup_multi_pack_index(the_repository,\n+\t\t\t\t\t\t\t\t object_dir);\n+\t\tif (m && !midx_checksum_valid(m)) {\n+\t\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n+\t\t\tm = NULL;\n+\t\t}\n \n-\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n-\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n-\t\tctx.m = NULL;\n+\t\tif (m) {\n+\t\t\t/*\n+\t\t\t * Only reference an existing MIDX when not filtering\n+\t\t\t * which packs to include, since all packs and objects\n+\t\t\t * are copied blindly from an existing MIDX if one is\n+\t\t\t * present.\n+\t\t\t */\n+\t\t\tif (ctx.incremental)\n+\t\t\t\tctx.base_midx = m;\n+\t\t\telse if (!packs_to_include)\n+\t\t\t\tctx.m = m;\n+\t\t}\n \t}\n \n \tctx.nr = 0;\n-\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n \tALLOC_ARRAY(ctx.info, ctx.alloc);\n \n-\tif (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n-\t\t\t\t\t flags) < 0) {\n-\t\tresult = 1;\n+\tif (ctx.incremental) {\n+\t\tstruct multi_pack_index *m = ctx.base_midx;\n+\t\twhile (m) {\n+\t\t\tctx.num_multi_pack_indexes_before++;\n+\t\t\tm = m->base_midx;\n+\t\t}\n+\t} else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,\n+\t\t\t\t\t\t flags) < 0) {\n \t\tgoto cleanup;\n \t}\n \n@@ -992,7 +1138,8 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n+\t !ctx.incremental &&\n \t !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n@@ -1008,12 +1155,14 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t * corresponding bitmap (or one wasn't requested).\n \t\t\t */\n \t\t\tif (!want_bitmap)\n-\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n-\t\t\t\t\t\t NULL);\n+\t\t\t\tclear_midx_files_ext(object_dir, \"bitmap\", NULL);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n+\tif (ctx.incremental && !ctx.nr)\n+\t\tgoto cleanup; /* nothing to do */\n+\n \tif (preferred_pack_name) {\n \t\tctx.preferred_pack_idx = -1;\n \n@@ -1159,8 +1308,30 @@ static int write_midx_internal(const char *object_dir,\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n \n-\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\tif (ctx.incremental) {\n+\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\n+\t\tget_midx_chain_filename(&lock_name, object_dir);\n+\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\t\tstrbuf_release(&lock_name);\n+\n+\t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n+\t\tif (!incr) {\n+\t\t\terror(_(\"unable to create temporary MIDX layer\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (adjust_shared_perm(get_tempfile_path(incr))) {\n+\t\t\terror(_(\"unable to adjust shared permissions for '%s'\"),\n+\t\t\t get_tempfile_path(incr));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr));\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n+\t\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\t}\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -1253,14 +1424,55 @@ static int write_midx_internal(const char *object_dir,\n \t * have been freed in the previous if block.\n \t */\n \n-\tif (ctx.m)\n+\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n+\n+\tif (ctx.incremental) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tstruct strbuf final_midx_name = STRBUF_INIT;\n+\t\tstruct multi_pack_index *m = ctx.base_midx;\n+\n+\t\tif (!chainf) {\n+\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n+\t\t\treturn -1;\n+\n+\t\tget_split_midx_filename_ext(&final_midx_name, object_dir,\n+\t\t\t\t\t midx_hash, MIDX_EXT_MIDX);\n+\n+\t\tif (rename_tempfile(&incr, final_midx_name.buf) < 0) {\n+\t\t\terror_errno(_(\"unable to rename new multi-pack-index layer\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\txstrdup(hash_to_hex(midx_hash));\n+\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n+\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\n+\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m)));\n+\t\t\tm = m->base_midx;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t} else {\n+\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\txstrdup(hash_to_hex(midx_hash));\n+\t}\n+\n+\tif (ctx.m || ctx.base_midx)\n \t\tclose_object_store(the_repository->objects);\n \n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+\tclear_midx_files(object_dir, keep_hashes,\n+\t\t\t ctx.num_multi_pack_indexes_before + 1,\n+\t\t\t ctx.incremental);\n \n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n@@ -1275,6 +1487,11 @@ static int write_midx_internal(const char *object_dir,\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n+\tif (keep_hashes) {\n+\t\tfor (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)\n+\t\t\tfree((char *)keep_hashes[i]);\n+\t\tfree(keep_hashes);\n+\t}\n \tstrbuf_release(&midx_name);\n \n \ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n@@ -1311,6 +1528,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tif (!m)\n \t\treturn 0;\n \n+\tif (m->base_midx)\n+\t\tdie(_(\"cannot expire packs from an incremental multi-pack-index\"));\n+\n \tCALLOC_ARRAY(count, m->num_packs);\n \n \tif (flags & MIDX_PROGRESS)\n@@ -1485,6 +1705,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \n \tif (!m)\n \t\treturn 0;\n+\tif (m->base_midx)\n+\t\tdie(_(\"cannot repack an incremental multi-pack-index\"));\n \n \tCALLOC_ARRAY(include_pack, m->num_packs);\n \ndiff --git a/midx.c b/midx.c\nindex a53d65702d..c867b2b6c2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -16,7 +16,10 @@\n \n int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t unsigned char *keep_hash);\n+\t\t\t const char *keep_hash);\n+void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t char **keep_hashes,\n+\t\t\t\t uint32_t hashes_nr);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -521,6 +524,11 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \treturn 0;\n }\n \n+int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid)\n+{\n+\treturn bsearch_midx(oid, m, NULL);\n+}\n+\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n)\n@@ -723,7 +731,8 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar *keep;\n+\tchar **keep;\n+\tuint32_t keep_nr;\n \tconst char *ext;\n };\n \n@@ -731,32 +740,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n+\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tif (data->keep && !strcmp(data->keep, file_name))\n-\t\treturn;\n-\n+\tfor (i = 0; i < data->keep_nr; i++) {\n+\t\tif (!strcmp(data->keep[i], file_name))\n+\t\t\treturn;\n+\t}\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n \n void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t unsigned char *keep_hash)\n+\t\t\t const char *keep_hash)\n {\n \tstruct clear_midx_data data;\n \tmemset(&data, 0, sizeof(struct clear_midx_data));\n \n-\tif (keep_hash)\n-\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n-\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tif (keep_hash) {\n+\t\tALLOC_ARRAY(data.keep, 1);\n+\n+\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n+\t\tdata.keep_nr = 1;\n+\t}\n \tdata.ext = ext;\n \n \tfor_each_file_in_pack_dir(object_dir,\n \t\t\t\t clear_midx_file_ext,\n \t\t\t\t &data);\n \n+\tif (keep_hash)\n+\t\tfree(data.keep[0]);\n+\tfree(data.keep);\n+}\n+\n+void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t char **keep_hashes,\n+\t\t\t\t uint32_t hashes_nr)\n+{\n+\tstruct clear_midx_data data;\n+\tuint32_t i;\n+\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tALLOC_ARRAY(data.keep, hashes_nr);\n+\tfor (i = 0; i < hashes_nr; i++)\n+\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n+\t\t\t\t ext);\n+\tdata.keep_nr = hashes_nr;\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_subdir(object_dir, \"multi-pack-index.d\",\n+\t\t\t\t clear_midx_file_ext, &data);\n+\n+\tfor (i = 0; i < hashes_nr; i++)\n+\t\tfree(data.keep[i]);\n \tfree(data.keep);\n }\n \n@@ -774,8 +814,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx.buf))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx.buf);\n \n-\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n-\tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL);\n \n \tstrbuf_release(&midx);\n }\ndiff --git a/midx.h b/midx.h\nindex 3714cad2cc..42d4f8d149 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,6 +29,8 @@ struct bitmapped_pack;\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\n@@ -77,6 +79,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n+#define MIDX_WRITE_INCREMENTAL (1 << 5)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n@@ -101,6 +104,7 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,\n \t\t uint32_t *result);\n+int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\ndiff --git a/packfile.c b/packfile.c\nindex 1eb18e3041..cf12a539ea 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -815,9 +815,10 @@ static void report_pack_garbage(struct string_list *list)\n \treport_helper(list, seen_bits, first, list->nr);\n }\n \n-void for_each_file_in_pack_dir(const char *objdir,\n-\t\t\t each_file_in_pack_dir_fn fn,\n-\t\t\t void *data)\n+void for_each_file_in_pack_subdir(const char *objdir,\n+\t\t\t\t const char *subdir,\n+\t\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t dirnamelen;\n@@ -826,6 +827,8 @@ void for_each_file_in_pack_dir(const char *objdir,\n \n \tstrbuf_addstr(&path, objdir);\n \tstrbuf_addstr(&path, \"/pack\");\n+\tif (subdir)\n+\t\tstrbuf_addf(&path, \"/%s\", subdir);\n \tdir = opendir(path.buf);\n \tif (!dir) {\n \t\tif (errno != ENOENT)\n@@ -847,6 +850,13 @@ void for_each_file_in_pack_dir(const char *objdir,\n \tstrbuf_release(&path);\n }\n \n+void for_each_file_in_pack_dir(const char *objdir,\n+\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t void *data)\n+{\n+\tfor_each_file_in_pack_subdir(objdir, NULL, fn, data);\n+}\n+\n struct prepare_pack_data {\n \tstruct repository *r;\n \tstruct string_list *garbage;\ndiff --git a/packfile.h b/packfile.h\nindex eb18ec15db..0f78658229 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,\n \t\t\t\t const char *file_name, void *data);\n+void for_each_file_in_pack_subdir(const char *objdir,\n+\t\t\t\t const char *subdir,\n+\t\t\t\t each_file_in_pack_dir_fn fn,\n+\t\t\t\t void *data);\n void for_each_file_in_pack_dir(const char *objdir,\n \t\t\t each_file_in_pack_dir_fn fn,\n \t\t\t void *data);\ndiff --git a/t/README b/t/README\nindex 3cee0c05e2..44c02d8129 100644\n--- a/t/README\n+++ b/t/README\n@@ -445,6 +445,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=, when true, sets\n+the '--incremental' option on all invocations of 'git multi-pack-index\n+write'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex f595937094..62aa6744a6 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,6 +1,8 @@\n # Helpers for scripts testing bitmap functionality; see t5310 for\n # example usage.\n \n+. \"$TEST_DIRECTORY\"/lib-midx.sh\n+\n objdir=.git/objects\n midx=$objdir/pack/multi-pack-index\n \n@@ -264,10 +266,6 @@ have_delta () {\n \ttest_cmp expect actual\n }\n \n-midx_checksum () {\n-\ttest-tool read-midx --checksum \"$1\"\n-}\n-\n # midx_pack_source \n midx_pack_source () {\n \ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\ndiff --git a/t/lib-midx.sh b/t/lib-midx.sh\nindex 1261994744..e38c609604 100644\n--- a/t/lib-midx.sh\n+++ b/t/lib-midx.sh\n@@ -6,3 +6,31 @@ test_midx_consistent () {\n \ttest_cmp expect actual &&\n \tgit multi-pack-index --object-dir=$1 verify\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"$1\"\n+}\n+\n+midx_git_two_modes () {\n+\tgit -c core.multiPackIndex=false $1 >expect &&\n+\tgit -c core.multiPackIndex=true $1 >actual &&\n+\tif [ \"$2\" = \"sorted\" ]\n+\tthen\n+\t\tsort expect.sorted &&\n+\t\tmv expect.sorted expect &&\n+\t\tsort actual.sorted &&\n+\t\tmv actual.sorted actual\n+\tfi &&\n+\ttest_cmp expect actual\n+}\n+\n+compare_results_with_midx () {\n+\tMSG=$1\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n+\t\tmidx_git_two_modes \"log --raw\" &&\n+\t\tmidx_git_two_modes \"count-objects --verbose\" &&\n+\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check\" &&\n+\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check --unordered\" sorted\n+\t'\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 8c54fc0655..ce1b58c732 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,8 +3,11 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-chunk.sh\n+. \"$TEST_DIRECTORY\"/lib-midx.sh\n \n GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n objdir=.git/objects\n \n HASH_LEN=$(test_oid rawsz)\n@@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' '\n \tmidx_read_expect 1 18 4 $objdir\n '\n \n-midx_git_two_modes () {\n-\tgit -c core.multiPackIndex=false $1 >expect &&\n-\tgit -c core.multiPackIndex=true $1 >actual &&\n-\tif [ \"$2\" = \"sorted\" ]\n-\tthen\n-\t\tsort expect.sorted &&\n-\t\tmv expect.sorted expect &&\n-\t\tsort actual.sorted &&\n-\t\tmv actual.sorted actual\n-\tfi &&\n-\ttest_cmp expect actual\n-}\n-\n-compare_results_with_midx () {\n-\tMSG=$1\n-\ttest_expect_success \"check normal git operations: $MSG\" '\n-\t\tmidx_git_two_modes \"rev-list --objects --all\" &&\n-\t\tmidx_git_two_modes \"log --raw\" &&\n-\t\tmidx_git_two_modes \"count-objects --verbose\" &&\n-\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check\" &&\n-\t\tmidx_git_two_modes \"cat-file --batch-all-objects --batch-check --unordered\" sorted\n-\t'\n-}\n-\n test_expect_success 'write midx with one v2 pack' '\n \tgit pack-objects --index-version=2,0x40 $objdir/pack/test &2 &&\n \tincrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack err &&\n+\t\tgit repack -Adl --write-bitmap-index 2>err &&\n \t\tcat >expect <<-EOF &&\n \t\twarning: disabling bitmap writing, as some objects are not being packed\n \t\tEOF\n@@ -533,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' '\n test_expect_success '--write-midx unchanged' '\n \t(\n \t\tcd midx &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack &&\n+\t\tgit repack &&\n \t\ttest_path_is_missing $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\t\tgit repack --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -550,7 +553,7 @@ test_expect_success '--write-midx with a new pack' '\n \t\tcd midx &&\n \t\ttest_commit loose &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\t\tgit repack --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -561,7 +564,7 @@ test_expect_success '--write-midx with a new pack' '\n test_expect_success '--write-midx with -b' '\n \t(\n \t\tcd midx &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&\n+\t\tgit repack -mb &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-*.bitmap &&\n@@ -574,7 +577,7 @@ test_expect_success '--write-midx with -d' '\n \t\tcd midx &&\n \t\ttest_commit repack &&\n \n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&\n+\t\tgit repack -Ad --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-*.bitmap &&\n@@ -587,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t\tcd midx &&\n \n \t\ttest_commit repack-2 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\t\tgit repack -Adb --write-midx &&\n \n \t\tchecksum=$(midx_checksum $objdir) &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$checksum.bitmap &&\n \n \t\ttest_commit repack-3 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\t\tgit repack -Adb --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-$checksum.bitmap &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n \t\ttest_commit repack-4 &&\n-\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n+\t\tgit repack -Adb &&\n \n \t\tfind $objdir/pack -type f -name \"multi-pack-index*\" >files &&\n \t\ttest_must_be_empty files\n@@ -622,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '\n \t\tgit log --format=\"create refs/tags/%s/%s %H\" HEAD >refs &&\n \t\tgit update-ref --stdin As usual, a range-diff is below, but the main changes since last time\n> are as follows:\n> \n> - Documentation improvements to clarify what happens when both an\n> incremental- and non-incremental MIDX are both present in a\n> repository.\n> \n> - Commit message typofix on 3/19 to fix an error in one of the\n> technical examples.\n> \n> - Dropped a custom 'local_pack_int_id' in 4/19 to make the remaining\n> diff easier to read.\n> \n> - Minor bugfix in 7/19 where we incorrectly terminated the object\n> abbreviation disambiguation step for incremental MIDXs.\n> \n> - Various additional bits of information in the commit message to\n> explain anything that was subtle.\n\nThis all looks good to me.\n\nI read over your responses to my previous review, but as you answered\nall of my questions I didn't respond to each individually. :)\n\nI looked over the changes in this iteration and they address all the\nsmall points I brought up. In the bigger picture, I do think there are\nprobably still issues lurking around the global/local pack and objection\nposition ids. But where you have the series now seems like a good\ncut-off point:\n\n - I suspect pack_pos_to_midx() might need to be adjusted. But it's not\n an issue that can be triggered until until we support incremental\n midx bitmaps. And that should definitely go in its own series (and\n preparing is kind of pointless because we don't know what the\n correct interface will be until mid-way through that topic).\n\n - I won't be surprised if there is some global/local bug that shakes\n out in the long run. But I don't have a clever way of preventing it\n or avoiding the need to deal with the distinction. So I think the\n best path is forward, and to let the shaking commence.\n\n The most important thing to that the new on-disk files are sane,\n since those are hard to walk back later. A simple bug in the code\n can always be fixed. Likewise, the changes here are unlikely to\n create a bug for anybody using a single unchained midx. So even if\n there is a bug, it will only be triggerable for people using the\n experimental mode.\n\n-Peff\n"}]}