{"thread":{"id":"61958","subject":"[PATCH 00/13] midx: incremental multi-pack indexes, part two","startedAt":"2024-08-15T21:01:21Z","lastAt":"2025-03-26T18:08:41Z","messageCount":135,"participants":["Taylor Blau","Junio C Hamano","Jeff King","Patrick Steinhardt","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"501045","messageId":"cover.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":null,"subject":"[PATCH 00/13] midx: incremental multi-pack indexes, part two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:14Z","receivedAt":"2024-08-15T21:01:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series is based on 'master', with an additional merge between\ntb/incremental-midx-part-1[1] and my newer series to fix a handful of\nbugs related to pseudo-merge bitmaps[2].\n\nThis is the second of three series to implement support for incremental\nmulti-pack indexes (MIDXs). This series brings support for bitmaps that\nare tied to incremental MIDXs in addition to regular MIDX bitmaps.\n\nThe details are laid out in the commits themselves, but the high-level\napproach is as follows:\n\n  - Each layer in the incremental MIDX chain has its own corresponding\n    *.bitmap file. Each bitmap contains commits / pseudo-merges which\n    are selected only from the commits in that layer. Likewise, only\n    that layer's objects are written in the type-level bitmaps.\n\n  - The reachability traversal is only conducted on the top-most bitmap\n    corresponding to the most recent layer in the incremental MIDX\n    chain. Earlier layers may be consulted to retrieve commit /\n    pseudo-merge reachability bitmaps, but only the top-most bitmap's\n    \"result\" and \"haves\" fields are used.\n\n  - In essence, the top-most bitmap is the only one that \"matters\", and\n    earlier bitmaps are merely used to look up commit and pseudo-merge\n    bitmaps from that layer.\n\n  - Whenever we need to look at the type-level bitmaps corresponding to\n    the whole incremental MIDX chain, a new \"ewah_or_iterator\" is used.\n    This works in concept like a typical ewah_iterator, except works\n    over many EWAH bitmaps in parallel, OR-ing their results together\n    before returning them to the user.\n\n    In effect, this allows us to treat the union of all type-level\n    bitmaps (each of which only stores information about the objects its\n    corresponding layer within the incremental MIDX chain) as a single\n    type-level bitmap corresponding to all of the objects across every\n    layer of the incremental MIDX chain.\n\nThe sum total of this series is that we are able to append new commits /\npseudo-merges to a repository's reachability bitmaps without having to\nrewrite existing bitmaps, making the operation much cheaper to perform\nin large repositories.\n\nThe series is laid out roughly as follows:\n\n  - The first patch describes the technical details of incremental MIDX\n    bitmaps.\n\n  - The second patch adjusts the pack-revindex internals to prepare for\n    incremental MIDX bitmaps.\n\n  - The next seven patches adjust various components of the pack-bitmap\n    internals to do the same.\n\n  - The next three patches introduce and adjust callers to use the\n    ewah_or_iterator (as above).\n\n  - The final patch implements writing incremental MIDX bitmaps, and\n    introduces tests.\n\nAfter this series, the remaining goals for this project include being\nable to compact contiguous runs of incremental MIDX layers into a single\nlayer to support growing the chain of MIDX layers without the chain\nitself becoming too long.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1722958595.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover.1723743050.git.me@ttaylorr.com/\n\nTaylor Blau (13):\n  Documentation: describe incremental MIDX bitmaps\n  pack-revindex: prepare for incremental MIDX bitmaps\n  pack-bitmap.c: open and store incremental bitmap layers\n  pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n  pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n  pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n  pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n  pack-bitmap.c: compute disk-usage with incremental MIDXs\n  pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n  ewah: implement `struct ewah_or_iterator`\n  pack-bitmap.c: keep track of each layer's type bitmaps\n  pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n  midx: implement writing incremental MIDX bitmaps\n\n Documentation/technical/multi-pack-index.txt |  64 ++++\n builtin/pack-objects.c                       |   3 +-\n ewah/ewah_bitmap.c                           |  33 ++\n ewah/ewok.h                                  |  12 +\n midx-write.c                                 |  35 +-\n pack-bitmap-write.c                          |  65 +++-\n pack-bitmap.c                                | 328 ++++++++++++++-----\n pack-bitmap.h                                |   4 +-\n pack-revindex.c                              |  32 +-\n t/t5334-incremental-multi-pack-index.sh      |  84 +++++\n 10 files changed, 548 insertions(+), 112 deletions(-)\n\n\nbase-commit: f28e4ef872c247b0b35f48b1c3d2c5f77753b908\n-- \n2.46.0.86.ge766d390f0.dirty\n"},{"id":"501046","messageId":"d1b8d11b37fa5ef6f031d3070669a213e721bab0.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:21Z","receivedAt":"2024-08-15T21:01:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement support for reachability bitmaps for the new\nincremental multi-pack index (MIDX) feature over the following commits.\n\nThis commit begins by first describing the relevant format and usage\ndetails for incremental MIDX bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/technical/multi-pack-index.txt | 64 ++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex cc063b30be..a063262c36 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -164,6 +164,70 @@ objects_nr($H2) + objects_nr($H1) + i\n (in the C implementation, this is often computed as `i +\n m->num_objects_in_base`).\n \n+=== Pseudo-pack order for incremental MIDXs\n+\n+The original implementation of multi-pack reachability bitmaps defined\n+the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n+titled \"multi-pack-index reverse indexes\") roughly as follows:\n+\n+____\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+____\n+\n+In the incremental MIDX design, we extend this definition to include\n+objects from multiple layers of the MIDX chain. The pseudo-pack order\n+for incremental MIDXs is determined by concatenating the pseudo-pack\n+ordering for each layer of the MIDX chain in order. Formally two objects\n+`o1` and `o2` are compared as follows:\n+\n+1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n+  `o1` is considered less than `o2`.\n+2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n+  MIDX layer has no base, then If one of `pack(o1)` and `pack(o2)` is\n+  preferred and the other is not, then the preferred one sorts first. If\n+  there is a base layer (i.e. the MIDX layer is not the first layer in\n+  the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n+  pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n+  appears earlier, than the opposite is true.\n+3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n+  same MIDX layer. Sort `o1` and `o2` by their offset within their\n+  containing packfile.\n+\n+=== Reachability bitmaps and incremental MIDXs\n+\n+Each layer of an incremental MIDX chain may have its objects (and the\n+objects from any previous layer in the same MIDX chain) represented in\n+its own `*.bitmap` file.\n+\n+The structure of a `*.bitmap` file belonging to an incremental MIDX\n+chain is identical to that of a non-incremental MIDX bitmap, or a\n+classic single-pack bitmap. Since objects are added to the end of the\n+incremental MIDX's pseudo-pack order (see: above), it is possible to\n+extend a bitmap when appending to the end of a MIDX chain.\n+\n+(Note: it is possible likewise to compress a contiguous sequence of MIDX\n+incremental layers, and their `*.bitmap`(s) into a single layer and\n+`*.bitmap`, but this is not yet implemented.)\n+\n+The object positions used are global within the pseudo-pack order, so\n+subsequent layers will have, for example, `m->num_objects_in_base`\n+number of `0` bits in each of their four type bitmaps. This follows from\n+the fact that we only write type bitmap entries for objects present in\n+the layer immediately corresponding to the bitmap).\n+\n+Note also that only the bitmap pertaining to the most recent layer in an\n+incremental MIDX chain is used to store reachability information about\n+the interesting and uninteresting objects in a reachability query.\n+Earlier bitmap layers are only used to look up commit and pseudo-merge\n+bitmaps from that layer, as well as the type-level bitmaps for objects\n+in that layer.\n+\n+To simplify the implementation, type-level bitmaps are iterated\n+simultaneously, and their results are OR'd together to avoid recursively\n+calling internal bitmap functions.\n+\n Future Work\n -----------\n \n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501047","messageId":"f5d0866e5cb3f7f5695062e12b10072fa05ecff3.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:25Z","receivedAt":"2024-08-15T21:01:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the reverse index machinery to handle object lookups in an\nincremental MIDX bitmap. These changes are broken out across a few\nfunctions:\n\n  - load_midx_revindex() learns to use the appropriate MIDX filename\n    depending on whether the given 'struct multi_pack_index *' is\n    incremental or not.\n\n  - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n    object position in the MIDX pseudo-pack order, and finds the\n    earliest containing MIDX (similar to midx.c::midx_for_object().\n\n  - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n    number of objects in the base (since 'vb - midx->revindx_data' is\n    relative to the containing MIDX, and pack_pos_to_midx() expects a\n    global position).\n\n    Likewise, this function adjusts its output by adding\n    m->num_objects_in_base to return a global position out through the\n    `*pos` pointer.\n\nTogether, these changes are sufficient to use the multi-pack index's\nreverse index format for incremental multi-pack reachability bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c   | 30 ++++++++++++++++++++----------\n pack-revindex.c | 32 +++++++++++++++++++++++---------\n 2 files changed, 43 insertions(+), 19 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2e657a2aa4..0a7039d955 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n }\n \n+static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n+{\n+\tif (index->midx) {\n+\t\tstruct multi_pack_index *m = index->midx;\n+\t\treturn m->num_objects + m->num_objects_in_base;\n+\t}\n+\treturn index->pack->num_objects;\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\n@@ -925,7 +934,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\t\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -993,7 +1002,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -1498,7 +1507,8 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\tif (!bitmap_get(objects,\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -1677,7 +1687,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tsize_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n+\t\tsize_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(to_filter, pos) &&\n \t\t    !bitmap_get(tips, pos))\n@@ -1703,7 +1713,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < bitmap_num_objects(bitmap_git)) {\n+\tif (pos < bitmap_non_extended_bits(bitmap_git)) {\n \t\tstruct packed_git *pack;\n \t\toff_t ofs;\n \n@@ -1726,7 +1736,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -1878,7 +1888,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n \tuint32_t objects_nr;\n \tsize_t i, pos;\n \n-\tobjects_nr = bitmap_num_objects(bitmap_git);\n+\tobjects_nr = bitmap_non_extended_bits(bitmap_git);\n \tpos = objects_nr / BITS_IN_EWORD;\n \n \tif (pos > result->word_alloc)\n@@ -2399,7 +2409,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(objects,\n-\t\t\t       st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t       st_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcount++;\n \t}\n \n@@ -2798,7 +2808,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n \t\t    \"extension\");\n \n-\tnum_objects = bitmap_num_objects(bitmap_git);\n+\tnum_objects = bitmap_non_extended_bits(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -2941,7 +2951,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \t\tstruct object *obj = eindex->objects[i];\n \n \t\tif (!bitmap_get(result,\n-\t\t\t\tst_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 22d3c23464..ce3f7ae214 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -383,8 +383,12 @@ int load_midx_revindex(struct multi_pack_index *m)\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t   \"source\", \"rev\");\n \n-\tget_midx_filename_ext(&revindex_name, m->object_dir,\n-\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n+\tif (m->has_chain)\n+\t\tget_split_midx_filename_ext(&revindex_name, m->object_dir,\n+\t\t\t\t\t    get_midx_checksum(m), MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(&revindex_name, m->object_dir,\n+\t\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n \t\t\t\t      m->num_objects,\n@@ -471,11 +475,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \n uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n {\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n \tif (!m->revindex_data)\n \t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n-\tif (m->num_objects <= pos)\n+\tif (m->num_objects + m->num_objects_in_base <= pos)\n \t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n-\treturn get_be32(m->revindex_data + pos);\n+\treturn get_be32(m->revindex_data + pos - m->num_objects_in_base);\n }\n \n struct midx_pack_key {\n@@ -491,7 +499,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n \tconst struct midx_pack_key *key = va;\n \tstruct multi_pack_index *midx = key->midx;\n \n-\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tsize_t pos = (uint32_t*)vb - (const uint32_t *)midx->revindex_data;\n+\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n \toff_t versus_offset;\n \n@@ -529,9 +538,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n {\n \tuint32_t *found;\n \n-\tif (key->pack >= m->num_packs)\n+\tif (key->pack >= m->num_packs + m->num_packs_in_base)\n \t\tBUG(\"MIDX pack lookup out of bounds (%\"PRIu32\" >= %\"PRIu32\")\",\n-\t\t    key->pack, m->num_packs);\n+\t\t    key->pack, m->num_packs + m->num_packs_in_base);\n \t/*\n \t * The preferred pack sorts first, so determine its identifier by\n \t * looking at the first object in pseudo-pack order.\n@@ -551,7 +560,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n \tif (!found)\n \t\treturn -1;\n \n-\t*pos = found - m->revindex_data;\n+\t*pos = (found - m->revindex_data) + m->num_objects_in_base;\n+\n \treturn 0;\n }\n \n@@ -559,9 +569,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n {\n \tstruct midx_pack_key key;\n \n+\twhile (m && at < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, at);\n \tif (!m->revindex_data)\n \t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n-\tif (m->num_objects <= at)\n+\tif (m->num_objects + m->num_objects_in_base <= at)\n \t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n \n \tkey.pack = nth_midxed_pack_int_id(m, at);\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501048","messageId":"43444efc21415b8817522cbd7b590e1ce38636d5.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:28Z","receivedAt":"2024-08-15T21:01:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the pack-bitmap machinery to work with incremental MIDXs by\nadding a new \"base\" field to keep track of the bitmap index associated\nwith the previous MIDX layer.\n\nThe changes in this commit are mostly boilerplate to open the correct\nbitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\nensures that the correct packs and their reverse indexes are loaded\nacross MIDX layers, etc.\n\nWhile we're at it, keep track of a base_nr field to indicate how many\nbitmap layers (including the current bitmap) exist. This will be used in\na future commit to allocate an array of 'struct ewah_bitmap' pointers to\ncollect all of the respective type bitmaps among all layers to\ninitialize a multi-EWAH iterator.\n\nSubsequent commits will teach the functions within the pack-bitmap\nmachinery how to interact with these new fields.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 63 ++++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 50 insertions(+), 13 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 0a7039d955..c27383c027 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -54,6 +54,13 @@ struct bitmap_index {\n \tstruct packed_git *pack;\n \tstruct multi_pack_index *midx;\n \n+\t/*\n+\t * If using a multi-pack index chain, 'base' points to the\n+\t * bitmap index corresponding to this bitmap's midx->base_midx.\n+\t */\n+\tstruct bitmap_index *base;\n+\tuint32_t base_nr;\n+\n \t/* mmapped buffer of the whole bitmap index */\n \tunsigned char *map;\n \tsize_t map_size; /* size of the mmaped buffer */\n@@ -377,8 +384,13 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n char *midx_bitmap_filename(struct multi_pack_index *midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tget_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx),\n-\t\t\t      MIDX_EXT_BITMAP);\n+\tif (midx->has_chain)\n+\t\tget_split_midx_filename_ext(&buf, midx->object_dir,\n+\t\t\t\t\t    get_midx_checksum(midx),\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(&buf, midx->object_dir,\n+\t\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n }\n@@ -397,10 +409,17 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n {\n \tstruct stat st;\n \tchar *bitmap_name = midx_bitmap_filename(midx);\n-\tint fd = git_open(bitmap_name);\n+\tint fd;\n \tuint32_t i, preferred_pack;\n \tstruct packed_git *preferred;\n \n+\tfd = git_open(bitmap_name);\n+\tif (fd < 0 && errno == ENOENT) {\n+\t\tFREE_AND_NULL(bitmap_name);\n+\t\tbitmap_name = midx_bitmap_filename(midx);\n+\t\tfd = git_open(bitmap_name);\n+\t}\n+\n \tif (fd < 0) {\n \t\tif (errno != ENOENT)\n \t\t\twarning_errno(\"cannot open '%s'\", bitmap_name);\n@@ -446,7 +465,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\tfor (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n \t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i)) {\n \t\t\twarning(_(\"could not open pack %s\"),\n \t\t\t\tbitmap_git->midx->pack_names[i]);\n@@ -459,13 +478,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tpreferred = bitmap_git->midx->packs[preferred_pack];\n+\tpreferred = nth_midxed_pack(bitmap_git->midx, preferred_pack);\n \tif (!is_pack_valid(preferred)) {\n \t\twarning(_(\"preferred pack (%s) is invalid\"),\n \t\t\tpreferred->pack_name);\n \t\tgoto cleanup;\n \t}\n \n+\tif (midx->base_midx) {\n+\t\tbitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n+\t\tbitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n+\t} else {\n+\t\tbitmap_git->base_nr = 1;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -516,6 +542,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tbitmap_git->map_pos = 0;\n+\tbitmap_git->base_nr = 1;\n \tclose(fd);\n \n \tif (load_bitmap_header(bitmap_git) < 0) {\n@@ -535,8 +562,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)\n {\n \tif (bitmap_is_midx(bitmap_git)) {\n-\t\tuint32_t i;\n-\t\tint ret;\n+\t\tstruct multi_pack_index *m;\n \n \t\t/*\n \t\t * The multi-pack-index's .rev file is already loaded via\n@@ -545,10 +571,15 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \t\t * But we still need to open the individual pack .rev files,\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tret = load_pack_revindex(r, bitmap_git->midx->packs[i]);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n+\t\tfor (m = bitmap_git->midx; m; m = m->base_midx) {\n+\t\t\tuint32_t i;\n+\t\t\tint ret;\n+\n+\t\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\tret = load_pack_revindex(r, m->packs[i]);\n+\t\t\t\tif (ret)\n+\t\t\t\t\treturn ret;\n+\t\t\t}\n \t\t}\n \t\treturn 0;\n \t}\n@@ -574,6 +605,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n+\tif (bitmap_git->base) {\n+\t\tif (!bitmap_is_midx(bitmap_git))\n+\t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n+\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\t\tgoto failed;\n+\t}\n+\n \treturn 0;\n \n failed:\n@@ -658,10 +696,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n \n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n {\n-\tstruct repository *r = the_repository;\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n+\tif (!open_midx_bitmap_1(bitmap_git, midx))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501049","messageId":"44871306487e5c4c93f619b3c81b651cab3099d6.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:31Z","receivedAt":"2024-08-15T21:01:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\nEWAH-compressed bitmap corresponding to some given commit object.\n\nTeach this function about incremental MIDX bitmaps by teaching it to\nrecur on earlier bitmap layers when it fails to find a given commit in\nthe current layer.\n\nThe changes to do so are as follows:\n\n  - Avoid initializing hash_pos at its declaration, since\n    bitmap_for_commit() is now a recursive function and may receive a\n    NULL bitmap_index pointer as its first argument.\n\n  - In cases where we would previously return NULL (to indicate that a\n    lookup failed and the given bitmap_index does not contain an entry\n    corresponding to the given commit), recursively call the function on\n    the previous bitmap layer.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c27383c027..88623d9e06 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -946,18 +946,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t      struct commit *commit)\n {\n-\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n-\t\t\t\t\t   commit->object.oid);\n+\tkhiter_t hash_pos;\n+\tif (!bitmap_git)\n+\t\treturn NULL;\n+\n+\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501050","messageId":"b7eae5dc616f8aa93a304b6fdbfe20b8fa576d48.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 05/13] pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:34Z","receivedAt":"2024-08-15T21:01:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Signed-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 88623d9e06..f91ab1b572 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1631,7 +1631,7 @@ static void show_objects_for_type(\n \t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n \n \t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n-\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\t} else {\n \t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n \t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501051","messageId":"01b8bd22cd9a6486cb79500558bd1eb2da832856.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:37Z","receivedAt":"2024-08-15T21:01:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Signed-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f91ab1b572..2b3c53d882 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2320,7 +2320,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tmulti_pack_reuse = 0;\n \n \tif (multi_pack_reuse) {\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\t\tstruct bitmapped_pack pack;\n \t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n@@ -2344,14 +2345,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tuint32_t pack_int_id;\n \n \t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n \t\t\tuint32_t preferred_pack_pos;\n \n-\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n+\t\t\twhile (m->base_midx)\n+\t\t\t\tm = m->base_midx;\n+\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n \t\t\t\treturn;\n \t\t\t}\n \n-\t\t\tpack = bitmap_git->midx->packs[preferred_pack_pos];\n+\t\t\tpack = nth_midxed_pack(m, preferred_pack_pos);\n \t\t\tpack_int_id = preferred_pack_pos;\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501052","messageId":"928a4eabc8b305f7608c082dcb61cd25f2387d3d.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:40Z","receivedAt":"2024-08-15T21:01:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Signed-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 105 ++++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 84 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2b3c53d882..5fea2714c1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -943,8 +943,9 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n \treturn NULL;\n }\n \n-struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n-\t\t\t\t      struct commit *commit)\n+static struct ewah_bitmap *find_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t  struct commit *commit,\n+\t\t\t\t\t\t  struct bitmap_index **found)\n {\n \tkhiter_t hash_pos;\n \tif (!bitmap_git)\n@@ -954,18 +955,30 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n+\t\tif (found)\n+\t\t\t*found = bitmap_git;\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n+\tif (found)\n+\t\t*found = bitmap_git;\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t      struct commit *commit)\n+{\n+\treturn find_bitmap_for_commit(bitmap_git, commit, NULL);\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n@@ -2489,6 +2502,8 @@ struct bitmap_test_data {\n \tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n+\n+\tstruct bitmap_test_data *base_tdata;\n };\n \n static void test_bitmap_type(struct bitmap_test_data *tdata,\n@@ -2497,6 +2512,11 @@ static void test_bitmap_type(struct bitmap_test_data *tdata,\n \tenum object_type bitmap_type = OBJ_NONE;\n \tint bitmaps_nr = 0;\n \n+\tif (bitmap_is_midx(tdata->bitmap_git)) {\n+\t\twhile (pos < tdata->bitmap_git->midx->num_objects_in_base)\n+\t\t\ttdata = tdata->base_tdata;\n+\t}\n+\n \tif (bitmap_get(tdata->commits, pos)) {\n \t\tbitmap_type = OBJ_COMMIT;\n \t\tbitmaps_nr++;\n@@ -2560,13 +2580,57 @@ static void test_show_commit(struct commit *commit, void *data)\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n+static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t total = 0;\n+\tdo {\n+\t\ttotal = st_add(total, bitmap_git->entry_count);\n+\t\tbitmap_git = bitmap_git->base;\n+\t} while (bitmap_git);\n+\n+\treturn total;\n+}\n+\n+static void prepare_bitmap_test_data(struct bitmap_test_data *tdata,\n+\t\t\t\t     struct bitmap_index *bitmap_git)\n+{\n+\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n+\n+\ttdata->bitmap_git = bitmap_git;\n+\ttdata->base = bitmap_new();\n+\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n+\n+\tif (bitmap_git->base) {\n+\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n+\t\tprepare_bitmap_test_data(tdata->base_tdata, bitmap_git->base);\n+\t}\n+}\n+\n+static void free_bitmap_test_data(struct bitmap_test_data *tdata)\n+{\n+\tif (!tdata)\n+\t\treturn;\n+\n+\tfree_bitmap_test_data(tdata->base_tdata);\n+\tfree(tdata->base_tdata);\n+\n+\tbitmap_free(tdata->base);\n+\tbitmap_free(tdata->commits);\n+\tbitmap_free(tdata->trees);\n+\tbitmap_free(tdata->blobs);\n+\tbitmap_free(tdata->tags);\n+}\n+\n void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n-\tstruct bitmap_index *bitmap_git;\n+\tstruct bitmap_index *bitmap_git, *found;\n \tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n@@ -2575,17 +2639,26 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n \t\tbitmap_git->version,\n \t\tbitmap_git->entry_count,\n-\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n+\t\tbitmap_total_entry_count(bitmap_git));\n \n \troot = revs->pending.objects[0].item;\n-\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n+\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n \n \tif (bm) {\n \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n-\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n+\t\t\toid_to_hex(&root->oid),\n+\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n+\n+\t\tif (bitmap_is_midx(found))\n+\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n+\t\t\t\t   hash_to_hex(get_midx_checksum(found->midx)));\n+\t\telse\n+\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n+\t\t\t\t   hash_to_hex(found->pack->hash));\n \n \t\tresult = ewah_to_bitmap(bm);\n \t}\n@@ -2602,14 +2675,8 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n-\ttdata.bitmap_git = bitmap_git;\n-\ttdata.base = bitmap_new();\n-\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n-\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n-\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n-\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n+\tprepare_bitmap_test_data(&tdata, bitmap_git);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n-\ttdata.seen = 0;\n \n \ttraverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);\n \n@@ -2621,11 +2688,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tdie(_(\"mismatch in bitmap results\"));\n \n \tbitmap_free(result);\n-\tbitmap_free(tdata.base);\n-\tbitmap_free(tdata.commits);\n-\tbitmap_free(tdata.trees);\n-\tbitmap_free(tdata.blobs);\n-\tbitmap_free(tdata.tags);\n+\tfree_bitmap_test_data(&tdata);\n \tfree_bitmap_index(bitmap_git);\n }\n \n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501053","messageId":"129f55ac285de29b9dc84aa67a58600dcf2ce41c.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 08/13] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:42Z","receivedAt":"2024-08-15T21:01:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Signed-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5fea2714c1..d3bb78237e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1774,7 +1774,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n \t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n \n-\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n@@ -3020,7 +3020,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \n \t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n-\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tstruct packed_git *pack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \n \t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n \t\t\t\t\tstruct object_id oid;\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501054","messageId":"81839fc1d1a60b467b7de61222795a3a0a7423aa.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 09/13] pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:45Z","receivedAt":"2024-08-15T21:01:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for using pseudo-merges with incremental MIDX bitmaps by\nattempting to apply pseudo-merges from each layer when encountering a\ngiven commit during a walk.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d3bb78237e..1fa101bb33 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1087,10 +1087,15 @@ static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git\n \t\t\t\t\t\t struct commit *commit,\n \t\t\t\t\t\t uint32_t commit_pos)\n {\n-\tint ret;\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tint ret = 0;\n \n-\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t     result, commit, commit_pos);\n+\twhile (curr) {\n+\t\tret += apply_pseudo_merges_for_commit(&curr->pseudo_merges,\n+\t\t\t\t\t\t      result, commit,\n+\t\t\t\t\t\t      commit_pos);\n+\t\tcurr = curr->base;\n+\t}\n \n \tif (ret)\n \t\tpseudo_merges_satisfied_nr += ret;\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501055","messageId":"63019e952abfd70865466019cbc38bfdb8841301.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:48Z","receivedAt":"2024-08-15T21:01:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"While individual bitmap layers store different commit, type-level, and\npseudo-merge bitmaps, only the top-most layer is used to compute\nreachability traversals.\n\nMany functions which implement the aforementioned traversal rely on\nenumerating the results according to the type-level bitmaps, and so\nwould benefit from a conceptual type-level bitmap that spans multiple\nlayers.\n\nImplement `struct ewah_or_iterator` which is capable of enumerating\nmultiple EWAH bitmaps at once, and OR-ing the results together. When\ninitialized with, for example, all of the commit type bitmaps from each\nlayer, callers can pretend as if they are enumerating a large type-level\nbitmap which contains the commits from *all* bitmap layers.\n\nThere are a couple of alternative approaches which were considered:\n\n  - Decompress each EWAH bitmap and OR them together, enumerating a\n    single (non-EWAH) bitmap. This would work, but has the disadvantage\n    of decompressing a potentially large bitmap, which may not be\n    necessary if the caller does not wish to read all of it.\n\n  - Recursively call bitmap internal functions, reusing the \"result\" and\n    \"haves\" bitmap from the top-most layer. This approach resembles the\n    original implementation of this feature, but is inefficient in that\n    it both (a) requires significant refactoring to implement, and (b)\n    enumerates large sections of later bitmaps which are all zeros (as\n    they pertain to objects in earlier layers).\n\n    (b) is not so bad in and of itself, but can cause significant\n    slow-downs when combined with expensive loop bodies.\n\nThis approach (enumerating an OR'd together version of all of the\ntype-level bitmaps from each layer) produces a significantly more\nstraightforward implementation with significantly less refactoring\nrequired in order to make it work.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ewah/ewah_bitmap.c | 33 +++++++++++++++++++++++++++++++++\n ewah/ewok.h        | 12 ++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex 8785cbc54a..b3a7ada071 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -372,6 +372,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n \t\tread_new_rlw(it);\n }\n \n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr)\n+{\n+\tsize_t i;\n+\n+\tmemset(it, 0, sizeof(*it));\n+\n+\tALLOC_ARRAY(it->its, nr);\n+\tfor (i = 0; i < nr; i++)\n+\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n+}\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n+{\n+\teword_t buf, out = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tfor (i = 0; i < it->nr; i++)\n+\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n+\t\t\tout |= buf;\n+\t\t\tret = 1;\n+\t\t}\n+\n+\t*next = out;\n+\treturn ret;\n+}\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it)\n+{\n+\tfree(it->its);\n+}\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 5e357e2493..4b70641045 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -148,6 +148,18 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent);\n  */\n int ewah_iterator_next(eword_t *next, struct ewah_iterator *it);\n \n+struct ewah_or_iterator {\n+\tstruct ewah_iterator *its;\n+\tsize_t nr;\n+};\n+\n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr);\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it);\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501056","messageId":"01508e4ff56531a271beca95b5a8e64bf842fa58.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:52Z","receivedAt":"2024-08-15T21:01:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for reading the type-level bitmaps from previous bitmap layers\nby maintaining an array for each type, where each element in that type's\narray corresponds to one layer's bitmap for that type.\n\nThese fields will be used in a later commit to instantiate the 'struct\newah_or_iterator' for each type.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 51 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1fa101bb33..e1badc7887 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -78,6 +78,24 @@ struct bitmap_index {\n \tstruct ewah_bitmap *blobs;\n \tstruct ewah_bitmap *tags;\n \n+\t/*\n+\t * Type index arrays when this bitmap is associated with an\n+\t * incremental multi-pack index chain.\n+\t *\n+\t * If n is the number of unique layers in the MIDX chain, then\n+\t * commits_all[n-1] is this structs 'commits' field,\n+\t * commits_all[n-2] is the commits field of this bitmap's\n+\t * 'base', and so on.\n+\t *\n+\t * When either associated either with a non-incremental MIDX, or\n+\t * a single packfile, these arrays each contain a single\n+\t * element.\n+\t */\n+\tstruct ewah_bitmap **commits_all;\n+\tstruct ewah_bitmap **trees_all;\n+\tstruct ewah_bitmap **blobs_all;\n+\tstruct ewah_bitmap **tags_all;\n+\n \t/* Map from object ID -> `stored_bitmap` for all the bitmapped commits */\n \tkh_oid_map_t *bitmaps;\n \n@@ -586,7 +604,29 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \treturn load_pack_revindex(r, bitmap_git->pack);\n }\n \n-static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n+static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n+{\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tsize_t i = bitmap_git->base_nr - 1;\n+\n+\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr);\n+\n+\twhile (curr) {\n+\t\tbitmap_git->commits_all[i] = curr->commits;\n+\t\tbitmap_git->trees_all[i] = curr->trees;\n+\t\tbitmap_git->blobs_all[i] = curr->blobs;\n+\t\tbitmap_git->tags_all[i] = curr->tags;\n+\n+\t\tcurr = curr->base;\n+\t\ti -= 1;\n+\t}\n+}\n+\n+static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git,\n+\t\t       int recursing)\n {\n \tassert(bitmap_git->map);\n \n@@ -608,10 +648,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (bitmap_git->base) {\n \t\tif (!bitmap_is_midx(bitmap_git))\n \t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n-\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\tif (load_bitmap(r, bitmap_git->base, 1) < 0)\n \t\t\tgoto failed;\n \t}\n \n+\tif (!recursing)\n+\t\tload_all_type_bitmaps(bitmap_git);\n+\n \treturn 0;\n \n failed:\n@@ -687,7 +730,7 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git, 0))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2042,7 +2085,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (load_bitmap(revs->repo, bitmap_git, 0) < 0)\n \t\tgoto cleanup;\n \n \tif (!use_boundary_traversal)\n@@ -2957,6 +3000,10 @@ void free_bitmap_index(struct bitmap_index *b)\n \tewah_pool_free(b->trees);\n \tewah_pool_free(b->blobs);\n \tewah_pool_free(b->tags);\n+\tfree(b->commits_all);\n+\tfree(b->trees_all);\n+\tfree(b->blobs_all);\n+\tfree(b->tags_all);\n \tif (b->bitmaps) {\n \t\tstruct stored_bitmap *sb;\n \t\tkh_foreach_value(b->bitmaps, sb, {\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501057","messageId":"59a50a2ea24312eb5b643927aac3fa7143b838f5.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:55Z","receivedAt":"2024-08-15T21:01:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have initialized arrays for each bitmap layer's type bitmaps\nin the previous commit, adjust existing callers to use them in\npreparation for multi-layered bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 42 +++++++++++++++++++++++++++---------------\n 1 file changed, 27 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e1badc7887..9fac43749c 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1622,25 +1622,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t}\n }\n \n-static void init_type_iterator(struct ewah_iterator *it,\n+static void init_type_iterator(struct ewah_or_iterator *it,\n \t\t\t       struct bitmap_index *bitmap_git,\n \t\t\t       enum object_type type)\n {\n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tewah_iterator_init(it, bitmap_git->commits);\n+\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_TREE:\n-\t\tewah_iterator_init(it, bitmap_git->trees);\n+\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_BLOB:\n-\t\tewah_iterator_init(it, bitmap_git->blobs);\n+\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_TAG:\n-\t\tewah_iterator_init(it, bitmap_git->tags);\n+\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tdefault:\n@@ -1657,7 +1661,7 @@ static void show_objects_for_type(\n \tsize_t i = 0;\n \tuint32_t offset;\n \n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tstruct bitmap *objects = bitmap_git->result;\n@@ -1665,7 +1669,7 @@ static void show_objects_for_type(\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = objects->words[i] & filter;\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n \n@@ -1707,6 +1711,8 @@ static void show_objects_for_type(\n \t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n+\n+\tewah_or_iterator_free(&it);\n }\n \n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n@@ -1758,7 +1764,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n@@ -1775,7 +1781,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * packfile.\n \t */\n \tfor (i = 0, init_type_iterator(&it, bitmap_git, type);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\tif (i < tips->word_alloc)\n \t\t\tmask &= ~tips->words[i];\n@@ -1795,6 +1801,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -1852,14 +1859,14 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n \ttips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB);\n \n \tfor (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\teword_t word = to_filter->words[i] & mask;\n \t\tunsigned offset;\n@@ -1887,6 +1894,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -2502,12 +2510,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \n \tuint32_t i = 0, count = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tinit_type_iterator(&it, bitmap_git, type);\n \n-\twhile (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) {\n+\twhile (i < objects->word_alloc && ewah_or_iterator_next(&filter, &it)) {\n \t\teword_t word = objects->words[i++] & filter;\n \t\tcount += ewah_bit_popcount64(word);\n \t}\n@@ -2519,6 +2527,8 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \t\t\tcount++;\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn count;\n }\n \n@@ -3046,13 +3056,13 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *result = bitmap_git->result;\n \toff_t total = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \tsize_t i;\n \n \tinit_type_iterator(&it, bitmap_git, object_type);\n \tfor (i = 0; i < result->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = result->words[i] & filter;\n \t\tsize_t base = (i * BITS_IN_EWORD);\n \t\tunsigned offset;\n@@ -3093,6 +3103,8 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn total;\n }\n \n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501058","messageId":"da34cc944126a4dce613cb5f1c4f3f2f82cbfce4.1723755667.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T21:01:58Z","receivedAt":"2024-08-15T21:02:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap machinery has learned how to read and interact\nwith an incremental MIDX bitmap, teach the pack-bitmap-write.c machinery\n(and relevant callers from within the MIDX machinery) to write such\nbitmaps.\n\nThe details for doing so are mostly straightforward. The main changes\nare as follows:\n\n  - find_object_pos() now makes use of an extra MIDX parameter which is\n    used to locate the bit positions of objects which are from previous\n    layers (and thus do not exist in the current layer's pack_order\n    field).\n\n    (Note also that the pack_order field is moved into struct\n    write_midx_context to further simplify the callers for\n    write_midx_bitmap()).\n\n  - bitmap_writer_build_type_index() first determines how many objects\n    precede the current bitmap layer and offsets the bits it sets in\n    each respective type-level bitmap by that amount so they can be OR'd\n    together.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c                  |  3 +-\n midx-write.c                            | 35 +++++++----\n pack-bitmap-write.c                     | 65 ++++++++++++++-----\n pack-bitmap.h                           |  4 +-\n t/t5334-incremental-multi-pack-index.sh | 84 +++++++++++++++++++++++++\n 5 files changed, 161 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e23c4950ed..9e61ff7ca3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1342,7 +1342,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_init(&bitmap_writer,\n-\t\t\t\t\t\t   the_repository, &to_pack);\n+\t\t\t\t\t\t   the_repository, &to_pack,\n+\t\t\t\t\t\t   NULL);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t\t\t       written_list);\ndiff --git a/midx-write.c b/midx-write.c\nindex 81501efdda..bac3b0589a 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -826,20 +826,26 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \treturn cb.commits;\n }\n \n-static int write_midx_bitmap(const char *midx_name,\n+static int write_midx_bitmap(struct write_midx_context *ctx,\n+\t\t\t     const char *object_dir, const char *midx_name,\n \t\t\t     const unsigned char *midx_hash,\n \t\t\t     struct packing_data *pdata,\n \t\t\t     struct commit **commits,\n \t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n \t\t\t     unsigned flags)\n {\n \tint ret, i;\n \tuint16_t options = 0;\n \tstruct bitmap_writer writer;\n \tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t\thash_to_hex(midx_hash));\n+\tstruct strbuf bitmap_name = STRBUF_INIT;\n+\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n+\t\t\t\t      MIDX_EXT_BITMAP);\n \n \ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n \n@@ -858,7 +864,8 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer, the_repository, pdata);\n+\tbitmap_writer_init(&writer, the_repository, pdata,\n+\t\t\t   ctx->incremental ? ctx->base_midx : NULL);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, index);\n \n@@ -876,7 +883,7 @@ static int write_midx_bitmap(const char *midx_name,\n \t * bitmap_writer_finish().\n \t */\n \tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\t\tindex[ctx->pack_order[i]] = &pdata->objects[i].idx;\n \n \tbitmap_writer_select_commits(&writer, commits, commits_nr);\n \tret = bitmap_writer_build(&writer);\n@@ -884,11 +891,11 @@ static int write_midx_bitmap(const char *midx_name,\n \t\tgoto cleanup;\n \n \tbitmap_writer_set_checksum(&writer, midx_hash);\n-\tbitmap_writer_finish(&writer, index, bitmap_name, options);\n+\tbitmap_writer_finish(&writer, index, bitmap_name.buf, options);\n \n cleanup:\n \tfree(index);\n-\tfree(bitmap_name);\n+\tstrbuf_release(&bitmap_name);\n \tbitmap_writer_free(&writer);\n \n \ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n@@ -1072,8 +1079,6 @@ static int write_midx_internal(const char *object_dir,\n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n \n \tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n-\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n-\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1115,6 +1120,12 @@ static int write_midx_internal(const char *object_dir,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n+\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n+\t\t\t\t      hash_to_hex(get_midx_checksum(m)));\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n@@ -1404,8 +1415,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\tif (write_midx_bitmap(&ctx, object_dir, midx_name.buf,\n+\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n \t\t\t\t      flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 923f793cec..8fc979cbc9 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -25,6 +25,8 @@\n #include \"alloc.h\"\n #include \"refs.h\"\n #include \"strmap.h\"\n+#include \"midx.h\"\n+#include \"pack-revindex.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -42,7 +44,8 @@ static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer\n }\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata)\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n \tif (writer->bitmaps)\n@@ -50,6 +53,7 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n \twriter->bitmaps = kh_init_oid_map();\n \twriter->pseudo_merge_commits = kh_init_oid_map();\n \twriter->to_pack = pdata;\n+\twriter->midx = midx;\n \n \tstring_list_init_dup(&writer->pseudo_merge_groups);\n \n@@ -104,6 +108,11 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t    struct pack_idx_entry **index)\n {\n \tuint32_t i;\n+\tuint32_t base_objects = 0;\n+\n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n \n \twriter->commits = ewah_new();\n \twriter->trees = ewah_new();\n@@ -133,19 +142,19 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \n \t\tswitch (real_type) {\n \t\tcase OBJ_COMMIT:\n-\t\t\tewah_set(writer->commits, i);\n+\t\t\tewah_set(writer->commits, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TREE:\n-\t\t\tewah_set(writer->trees, i);\n+\t\t\tewah_set(writer->trees, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_BLOB:\n-\t\t\tewah_set(writer->blobs, i);\n+\t\t\tewah_set(writer->blobs, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TAG:\n-\t\t\tewah_set(writer->tags, i);\n+\t\t\tewah_set(writer->tags, i + base_objects);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -198,19 +207,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n static uint32_t find_object_pos(struct bitmap_writer *writer,\n \t\t\t\tconst struct object_id *oid, int *found)\n {\n-\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n+\tstruct object_entry *entry;\n+\n+\tentry = packlist_find(writer->to_pack, oid);\n+\tif (entry) {\n+\t\tuint32_t base_objects = 0;\n+\t\tif (writer->midx)\n+\t\t\tbase_objects = writer->midx->num_objects +\n+\t\t\t\twriter->midx->num_objects_in_base;\n+\n+\t\tif (found)\n+\t\t\t*found = 1;\n+\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n+\t} else if (writer->midx) {\n+\t\tuint32_t at, pos;\n+\n+\t\tif (!bsearch_midx(oid, writer->midx, &at))\n+\t\t\tgoto missing;\n+\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n+\t\t\tgoto missing;\n \n-\tif (!entry) {\n \t\tif (found)\n-\t\t\t*found = 0;\n-\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n-\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n-\t\treturn 0;\n+\t\t\t*found = 1;\n+\t\treturn pos;\n \t}\n \n+missing:\n \tif (found)\n-\t\t*found = 1;\n-\treturn oe_in_pack_pos(writer->to_pack, entry);\n+\t\t*found = 0;\n+\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\treturn 0;\n }\n \n static void compute_xor_offsets(struct bitmap_writer *writer)\n@@ -577,7 +604,7 @@ int bitmap_writer_build(struct bitmap_writer *writer)\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n-\tuint32_t *mapping;\n+\tuint32_t *mapping = NULL;\n \tint closed = 1; /* until proven otherwise */\n \n \tif (writer->show_progress)\n@@ -1010,7 +1037,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \toff_t *offsets = NULL;\n-\tuint32_t i;\n+\tuint32_t i, base_objects;\n \n \tstruct bitmap_disk_header header;\n \n@@ -1036,6 +1063,12 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, writer->to_pack->nr_objects);\n \n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n+\telse\n+\t\tbase_objects = 0;\n+\n \tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n@@ -1044,7 +1077,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \t\tif (commit_pos < 0)\n \t\t\tBUG(_(\"trying to write commit not in index\"));\n-\t\tstored->commit_pos = commit_pos;\n+\t\tstored->commit_pos = commit_pos + base_objects;\n \t}\n \n \twrite_selected_commits_v1(writer, f, offsets);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex ff0fd815b8..4242458198 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -110,6 +110,7 @@ struct bitmap_writer {\n \n \tkh_oid_map_t *bitmaps;\n \tstruct packing_data *to_pack;\n+\tstruct multi_pack_index *midx; /* if appending to a MIDX chain */\n \n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n@@ -124,7 +125,8 @@ struct bitmap_writer {\n };\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata);\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex c3b08acc73..0b6d45c8fd 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -43,4 +43,88 @@ test_expect_success 'convert incremental to non-incremental' '\n \n compare_results_with_midx 'non-incremental MIDX conversion'\n \n+write_midx_layer () {\n+\tn=1\n+\tif test -f $midx_chain\n+\tthen\n+\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n+\tfi\n+\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit $n.$i &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit multi-pack-index write --bitmap --incremental\n+}\n+\n+test_expect_success 'write initial MIDX layer' '\n+\tgit repack -ad &&\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'read bitmap from first MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'write another MIDX layer' '\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'midx verify with multiple layers' '\n+\tgit multi-pack-index verify\n+'\n+\n+test_expect_success 'read bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 2.2\n+'\n+\n+test_expect_success 'read earlier bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'show object from first pack' '\n+\tgit cat-file -p 1.1\n+'\n+\n+test_expect_success 'show object from second pack' '\n+\tgit cat-file -p 2.2\n+'\n+\n+for reuse in false single multi\n+do\n+\ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n+\t\trm -fr clone.git &&\n+\n+\t\tgit config pack.allowPackReuse $reuse &&\n+\t\tgit clone --no-local --bare . clone.git\n+\t'\n+done\n+\n+test_expect_success 'relink existing MIDX layer' '\n+\trm -fr \"$midxdir\" &&\n+\n+\tGIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n+\n+\tmidx_hash=\"$(test-tool read-midx --checksum $objdir)\" &&\n+\n+\ttest_path_is_file \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_commit another &&\n+\tgit repack -d &&\n+\tgit multi-pack-index write --bitmap --incremental &&\n+\n+\ttest_path_is_missing \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.midx\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.rev\" &&\n+\ttest_line_count = 2 \"$midx_chain\"\n+\n+'\n+\n test_done\n-- \n2.46.0.86.ge766d390f0.dirty\n"},{"id":"501064","messageId":"cover.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH v2 00/13] midx: incremental multi-pack indexes, part two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:28:46Z","receivedAt":"2024-08-15T22:28:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"== Changes since last time\n\n[The first round did not include commit messages for four intermediate\ncommits which could benefit from additional description, which has been\ncorrected here. The code is unchanged from the previous round, and a\nrange-diff is below.]\n\n== Original cover letter\n\nThis series is based on 'master', with an additional merge between\ntb/incremental-midx-part-1[1] and my newer series to fix a handful of\nbugs related to pseudo-merge bitmaps[2].\n\nThis is the second of three series to implement support for incremental\nmulti-pack indexes (MIDXs). This series brings support for bitmaps that\nare tied to incremental MIDXs in addition to regular MIDX bitmaps.\n\nThe details are laid out in the commits themselves, but the high-level\napproach is as follows:\n\n  - Each layer in the incremental MIDX chain has its own corresponding\n    *.bitmap file. Each bitmap contains commits / pseudo-merges which\n    are selected only from the commits in that layer. Likewise, only\n    that layer's objects are written in the type-level bitmaps.\n\n  - The reachability traversal is only conducted on the top-most bitmap\n    corresponding to the most recent layer in the incremental MIDX\n    chain. Earlier layers may be consulted to retrieve commit /\n    pseudo-merge reachability bitmaps, but only the top-most bitmap's\n    \"result\" and \"haves\" fields are used.\n\n  - In essence, the top-most bitmap is the only one that \"matters\", and\n    earlier bitmaps are merely used to look up commit and pseudo-merge\n    bitmaps from that layer.\n\n  - Whenever we need to look at the type-level bitmaps corresponding to\n    the whole incremental MIDX chain, a new \"ewah_or_iterator\" is used.\n    This works in concept like a typical ewah_iterator, except works\n    over many EWAH bitmaps in parallel, OR-ing their results together\n    before returning them to the user.\n\n    In effect, this allows us to treat the union of all type-level\n    bitmaps (each of which only stores information about the objects its\n    corresponding layer within the incremental MIDX chain) as a single\n    type-level bitmap corresponding to all of the objects across every\n    layer of the incremental MIDX chain.\n\nThe sum total of this series is that we are able to append new commits /\npseudo-merges to a repository's reachability bitmaps without having to\nrewrite existing bitmaps, making the operation much cheaper to perform\nin large repositories.\n\nThe series is laid out roughly as follows:\n\n  - The first patch describes the technical details of incremental MIDX\n    bitmaps.\n\n  - The second patch adjusts the pack-revindex internals to prepare for\n    incremental MIDX bitmaps.\n\n  - The next seven patches adjust various components of the pack-bitmap\n    internals to do the same.\n\n  - The next three patches introduce and adjust callers to use the\n    ewah_or_iterator (as above).\n\n  - The final patch implements writing incremental MIDX bitmaps, and\n    introduces tests.\n\nAfter this series, the remaining goals for this project include being\nable to compact contiguous runs of incremental MIDX layers into a single\nlayer to support growing the chain of MIDX layers without the chain\nitself becoming too long.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1722958595.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover.1723743050.git.me@ttaylorr.com/\n\nTaylor Blau (13):\n  Documentation: describe incremental MIDX bitmaps\n  pack-revindex: prepare for incremental MIDX bitmaps\n  pack-bitmap.c: open and store incremental bitmap layers\n  pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n  pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n  pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n  pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n  pack-bitmap.c: compute disk-usage with incremental MIDXs\n  pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n  ewah: implement `struct ewah_or_iterator`\n  pack-bitmap.c: keep track of each layer's type bitmaps\n  pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n  midx: implement writing incremental MIDX bitmaps\n\n Documentation/technical/multi-pack-index.txt |  64 ++++\n builtin/pack-objects.c                       |   3 +-\n ewah/ewah_bitmap.c                           |  33 ++\n ewah/ewok.h                                  |  12 +\n midx-write.c                                 |  35 +-\n pack-bitmap-write.c                          |  65 +++-\n pack-bitmap.c                                | 328 ++++++++++++++-----\n pack-bitmap.h                                |   4 +-\n pack-revindex.c                              |  32 +-\n t/t5334-incremental-multi-pack-index.sh      |  84 +++++\n 10 files changed, 548 insertions(+), 112 deletions(-)\n\nRange-diff against v1:\n -:  ---------- >  1:  d1b8d11b37 Documentation: describe incremental MIDX bitmaps\n -:  ---------- >  2:  f5d0866e5c pack-revindex: prepare for incremental MIDX bitmaps\n -:  ---------- >  3:  43444efc21 pack-bitmap.c: open and store incremental bitmap layers\n -:  ---------- >  4:  4487130648 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n 1:  b7eae5dc61 !  5:  b720fe56da pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n    @@ Metadata\n      ## Commit message ##\n         pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n\n    +    Since we may ask for a pack_id that is in an earlier MIDX layer relative\n    +    to the one corresponding to our bitmap, use nth_midxed_pack() instead of\n    +    accessing the ->packs array directly.\n    +\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n\n      ## pack-bitmap.c ##\n 2:  01b8bd22cd !  6:  9716d022e0 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n    @@ Metadata\n      ## Commit message ##\n         pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n\n    +    In a similar fashion as previous commits in the first phase of\n    +    incremental MIDXs, enumerate not just the packs in the current\n    +    incremental MIDX layer, but previous ones as well.\n    +\n    +    Likewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\n    +    single pack from a MIDX, use the oldest layer's preferred pack as it is\n    +    likely to contain the most amount of reusable sections.\n    +\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n\n      ## pack-bitmap.c ##\n 3:  928a4eabc8 !  7:  6baece3175 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n    @@ Metadata\n      ## Commit message ##\n         pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n\n    +    Implement support for the special `--test-bitmap` mode of `git rev-list`\n    +    when using incremental MIDXs.\n    +\n    +    The bitmap_test_data structure is extended to contain a \"base\" pointer\n    +    that mirrors the structure of the bitmap chain that it is being used to\n    +    test.\n    +\n    +    When we find a commit to test, we first chase down the ->base pointer to\n    +    find the appropriate bitmap_test_data for the bitmap layer that the\n    +    given commit is contained within, and then perform the test on that\n    +    bitmap.\n    +\n    +    In order to implement this, light modifications are made to\n    +    bitmap_for_commit() to reimplement it in terms of a new function,\n    +    find_bitmap_for_commit(), which fills out a pointer which indicates the\n    +    bitmap layer which contains the given commit.\n    +\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n\n      ## pack-bitmap.c ##\n 4:  129f55ac28 !  8:  5c909df38a pack-bitmap.c: compute disk-usage with incremental MIDXs\n    @@ Metadata\n      ## Commit message ##\n         pack-bitmap.c: compute disk-usage with incremental MIDXs\n\n    +    In a similar fashion as previous commits, use nth_midxed_pack() instead\n    +    of accessing the MIDX's ->packs array directly to support incremental\n    +    MIDXs.\n    +\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n\n      ## pack-bitmap.c ##\n 5:  81839fc1d1 =  9:  f9ae10fce9 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n 6:  63019e952a = 10:  04042981c1 ewah: implement `struct ewah_or_iterator`\n 7:  01508e4ff5 = 11:  c4d543d43d pack-bitmap.c: keep track of each layer's type bitmaps\n 8:  59a50a2ea2 = 12:  c6730b4107 pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n 9:  da34cc9441 = 13:  afefb45557 midx: implement writing incremental MIDX bitmaps\n\nbase-commit: f28e4ef872c247b0b35f48b1c3d2c5f77753b908\n--\n2.46.0.86.ge766d390f0.dirty\n"},{"id":"501065","messageId":"d1b8d11b37fa5ef6f031d3070669a213e721bab0.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:28:52Z","receivedAt":"2024-08-15T22:28:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement support for reachability bitmaps for the new\nincremental multi-pack index (MIDX) feature over the following commits.\n\nThis commit begins by first describing the relevant format and usage\ndetails for incremental MIDX bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/technical/multi-pack-index.txt | 64 ++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex cc063b30be..a063262c36 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -164,6 +164,70 @@ objects_nr($H2) + objects_nr($H1) + i\n (in the C implementation, this is often computed as `i +\n m->num_objects_in_base`).\n \n+=== Pseudo-pack order for incremental MIDXs\n+\n+The original implementation of multi-pack reachability bitmaps defined\n+the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n+titled \"multi-pack-index reverse indexes\") roughly as follows:\n+\n+____\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+____\n+\n+In the incremental MIDX design, we extend this definition to include\n+objects from multiple layers of the MIDX chain. The pseudo-pack order\n+for incremental MIDXs is determined by concatenating the pseudo-pack\n+ordering for each layer of the MIDX chain in order. Formally two objects\n+`o1` and `o2` are compared as follows:\n+\n+1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n+  `o1` is considered less than `o2`.\n+2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n+  MIDX layer has no base, then If one of `pack(o1)` and `pack(o2)` is\n+  preferred and the other is not, then the preferred one sorts first. If\n+  there is a base layer (i.e. the MIDX layer is not the first layer in\n+  the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n+  pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n+  appears earlier, than the opposite is true.\n+3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n+  same MIDX layer. Sort `o1` and `o2` by their offset within their\n+  containing packfile.\n+\n+=== Reachability bitmaps and incremental MIDXs\n+\n+Each layer of an incremental MIDX chain may have its objects (and the\n+objects from any previous layer in the same MIDX chain) represented in\n+its own `*.bitmap` file.\n+\n+The structure of a `*.bitmap` file belonging to an incremental MIDX\n+chain is identical to that of a non-incremental MIDX bitmap, or a\n+classic single-pack bitmap. Since objects are added to the end of the\n+incremental MIDX's pseudo-pack order (see: above), it is possible to\n+extend a bitmap when appending to the end of a MIDX chain.\n+\n+(Note: it is possible likewise to compress a contiguous sequence of MIDX\n+incremental layers, and their `*.bitmap`(s) into a single layer and\n+`*.bitmap`, but this is not yet implemented.)\n+\n+The object positions used are global within the pseudo-pack order, so\n+subsequent layers will have, for example, `m->num_objects_in_base`\n+number of `0` bits in each of their four type bitmaps. This follows from\n+the fact that we only write type bitmap entries for objects present in\n+the layer immediately corresponding to the bitmap).\n+\n+Note also that only the bitmap pertaining to the most recent layer in an\n+incremental MIDX chain is used to store reachability information about\n+the interesting and uninteresting objects in a reachability query.\n+Earlier bitmap layers are only used to look up commit and pseudo-merge\n+bitmaps from that layer, as well as the type-level bitmaps for objects\n+in that layer.\n+\n+To simplify the implementation, type-level bitmaps are iterated\n+simultaneously, and their results are OR'd together to avoid recursively\n+calling internal bitmap functions.\n+\n Future Work\n -----------\n \n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501066","messageId":"f5d0866e5cb3f7f5695062e12b10072fa05ecff3.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:28:55Z","receivedAt":"2024-08-15T22:28:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the reverse index machinery to handle object lookups in an\nincremental MIDX bitmap. These changes are broken out across a few\nfunctions:\n\n  - load_midx_revindex() learns to use the appropriate MIDX filename\n    depending on whether the given 'struct multi_pack_index *' is\n    incremental or not.\n\n  - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n    object position in the MIDX pseudo-pack order, and finds the\n    earliest containing MIDX (similar to midx.c::midx_for_object().\n\n  - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n    number of objects in the base (since 'vb - midx->revindx_data' is\n    relative to the containing MIDX, and pack_pos_to_midx() expects a\n    global position).\n\n    Likewise, this function adjusts its output by adding\n    m->num_objects_in_base to return a global position out through the\n    `*pos` pointer.\n\nTogether, these changes are sufficient to use the multi-pack index's\nreverse index format for incremental multi-pack reachability bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c   | 30 ++++++++++++++++++++----------\n pack-revindex.c | 32 +++++++++++++++++++++++---------\n 2 files changed, 43 insertions(+), 19 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2e657a2aa4..0a7039d955 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n }\n \n+static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n+{\n+\tif (index->midx) {\n+\t\tstruct multi_pack_index *m = index->midx;\n+\t\treturn m->num_objects + m->num_objects_in_base;\n+\t}\n+\treturn index->pack->num_objects;\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\n@@ -925,7 +934,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\t\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -993,7 +1002,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -1498,7 +1507,8 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\tif (!bitmap_get(objects,\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -1677,7 +1687,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tsize_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n+\t\tsize_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(to_filter, pos) &&\n \t\t    !bitmap_get(tips, pos))\n@@ -1703,7 +1713,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < bitmap_num_objects(bitmap_git)) {\n+\tif (pos < bitmap_non_extended_bits(bitmap_git)) {\n \t\tstruct packed_git *pack;\n \t\toff_t ofs;\n \n@@ -1726,7 +1736,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -1878,7 +1888,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n \tuint32_t objects_nr;\n \tsize_t i, pos;\n \n-\tobjects_nr = bitmap_num_objects(bitmap_git);\n+\tobjects_nr = bitmap_non_extended_bits(bitmap_git);\n \tpos = objects_nr / BITS_IN_EWORD;\n \n \tif (pos > result->word_alloc)\n@@ -2399,7 +2409,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(objects,\n-\t\t\t       st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t       st_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcount++;\n \t}\n \n@@ -2798,7 +2808,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n \t\t    \"extension\");\n \n-\tnum_objects = bitmap_num_objects(bitmap_git);\n+\tnum_objects = bitmap_non_extended_bits(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -2941,7 +2951,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \t\tstruct object *obj = eindex->objects[i];\n \n \t\tif (!bitmap_get(result,\n-\t\t\t\tst_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 22d3c23464..ce3f7ae214 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -383,8 +383,12 @@ int load_midx_revindex(struct multi_pack_index *m)\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t   \"source\", \"rev\");\n \n-\tget_midx_filename_ext(&revindex_name, m->object_dir,\n-\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n+\tif (m->has_chain)\n+\t\tget_split_midx_filename_ext(&revindex_name, m->object_dir,\n+\t\t\t\t\t    get_midx_checksum(m), MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(&revindex_name, m->object_dir,\n+\t\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n \t\t\t\t      m->num_objects,\n@@ -471,11 +475,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \n uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n {\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n \tif (!m->revindex_data)\n \t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n-\tif (m->num_objects <= pos)\n+\tif (m->num_objects + m->num_objects_in_base <= pos)\n \t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n-\treturn get_be32(m->revindex_data + pos);\n+\treturn get_be32(m->revindex_data + pos - m->num_objects_in_base);\n }\n \n struct midx_pack_key {\n@@ -491,7 +499,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n \tconst struct midx_pack_key *key = va;\n \tstruct multi_pack_index *midx = key->midx;\n \n-\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tsize_t pos = (uint32_t*)vb - (const uint32_t *)midx->revindex_data;\n+\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n \toff_t versus_offset;\n \n@@ -529,9 +538,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n {\n \tuint32_t *found;\n \n-\tif (key->pack >= m->num_packs)\n+\tif (key->pack >= m->num_packs + m->num_packs_in_base)\n \t\tBUG(\"MIDX pack lookup out of bounds (%\"PRIu32\" >= %\"PRIu32\")\",\n-\t\t    key->pack, m->num_packs);\n+\t\t    key->pack, m->num_packs + m->num_packs_in_base);\n \t/*\n \t * The preferred pack sorts first, so determine its identifier by\n \t * looking at the first object in pseudo-pack order.\n@@ -551,7 +560,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n \tif (!found)\n \t\treturn -1;\n \n-\t*pos = found - m->revindex_data;\n+\t*pos = (found - m->revindex_data) + m->num_objects_in_base;\n+\n \treturn 0;\n }\n \n@@ -559,9 +569,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n {\n \tstruct midx_pack_key key;\n \n+\twhile (m && at < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, at);\n \tif (!m->revindex_data)\n \t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n-\tif (m->num_objects <= at)\n+\tif (m->num_objects + m->num_objects_in_base <= at)\n \t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n \n \tkey.pack = nth_midxed_pack_int_id(m, at);\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501067","messageId":"43444efc21415b8817522cbd7b590e1ce38636d5.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:28:58Z","receivedAt":"2024-08-15T22:29:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the pack-bitmap machinery to work with incremental MIDXs by\nadding a new \"base\" field to keep track of the bitmap index associated\nwith the previous MIDX layer.\n\nThe changes in this commit are mostly boilerplate to open the correct\nbitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\nensures that the correct packs and their reverse indexes are loaded\nacross MIDX layers, etc.\n\nWhile we're at it, keep track of a base_nr field to indicate how many\nbitmap layers (including the current bitmap) exist. This will be used in\na future commit to allocate an array of 'struct ewah_bitmap' pointers to\ncollect all of the respective type bitmaps among all layers to\ninitialize a multi-EWAH iterator.\n\nSubsequent commits will teach the functions within the pack-bitmap\nmachinery how to interact with these new fields.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 63 ++++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 50 insertions(+), 13 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 0a7039d955..c27383c027 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -54,6 +54,13 @@ struct bitmap_index {\n \tstruct packed_git *pack;\n \tstruct multi_pack_index *midx;\n \n+\t/*\n+\t * If using a multi-pack index chain, 'base' points to the\n+\t * bitmap index corresponding to this bitmap's midx->base_midx.\n+\t */\n+\tstruct bitmap_index *base;\n+\tuint32_t base_nr;\n+\n \t/* mmapped buffer of the whole bitmap index */\n \tunsigned char *map;\n \tsize_t map_size; /* size of the mmaped buffer */\n@@ -377,8 +384,13 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n char *midx_bitmap_filename(struct multi_pack_index *midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tget_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx),\n-\t\t\t      MIDX_EXT_BITMAP);\n+\tif (midx->has_chain)\n+\t\tget_split_midx_filename_ext(&buf, midx->object_dir,\n+\t\t\t\t\t    get_midx_checksum(midx),\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(&buf, midx->object_dir,\n+\t\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n }\n@@ -397,10 +409,17 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n {\n \tstruct stat st;\n \tchar *bitmap_name = midx_bitmap_filename(midx);\n-\tint fd = git_open(bitmap_name);\n+\tint fd;\n \tuint32_t i, preferred_pack;\n \tstruct packed_git *preferred;\n \n+\tfd = git_open(bitmap_name);\n+\tif (fd < 0 && errno == ENOENT) {\n+\t\tFREE_AND_NULL(bitmap_name);\n+\t\tbitmap_name = midx_bitmap_filename(midx);\n+\t\tfd = git_open(bitmap_name);\n+\t}\n+\n \tif (fd < 0) {\n \t\tif (errno != ENOENT)\n \t\t\twarning_errno(\"cannot open '%s'\", bitmap_name);\n@@ -446,7 +465,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\tfor (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n \t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i)) {\n \t\t\twarning(_(\"could not open pack %s\"),\n \t\t\t\tbitmap_git->midx->pack_names[i]);\n@@ -459,13 +478,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tpreferred = bitmap_git->midx->packs[preferred_pack];\n+\tpreferred = nth_midxed_pack(bitmap_git->midx, preferred_pack);\n \tif (!is_pack_valid(preferred)) {\n \t\twarning(_(\"preferred pack (%s) is invalid\"),\n \t\t\tpreferred->pack_name);\n \t\tgoto cleanup;\n \t}\n \n+\tif (midx->base_midx) {\n+\t\tbitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n+\t\tbitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n+\t} else {\n+\t\tbitmap_git->base_nr = 1;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -516,6 +542,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tbitmap_git->map_pos = 0;\n+\tbitmap_git->base_nr = 1;\n \tclose(fd);\n \n \tif (load_bitmap_header(bitmap_git) < 0) {\n@@ -535,8 +562,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)\n {\n \tif (bitmap_is_midx(bitmap_git)) {\n-\t\tuint32_t i;\n-\t\tint ret;\n+\t\tstruct multi_pack_index *m;\n \n \t\t/*\n \t\t * The multi-pack-index's .rev file is already loaded via\n@@ -545,10 +571,15 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \t\t * But we still need to open the individual pack .rev files,\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tret = load_pack_revindex(r, bitmap_git->midx->packs[i]);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n+\t\tfor (m = bitmap_git->midx; m; m = m->base_midx) {\n+\t\t\tuint32_t i;\n+\t\t\tint ret;\n+\n+\t\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\tret = load_pack_revindex(r, m->packs[i]);\n+\t\t\t\tif (ret)\n+\t\t\t\t\treturn ret;\n+\t\t\t}\n \t\t}\n \t\treturn 0;\n \t}\n@@ -574,6 +605,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n+\tif (bitmap_git->base) {\n+\t\tif (!bitmap_is_midx(bitmap_git))\n+\t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n+\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\t\tgoto failed;\n+\t}\n+\n \treturn 0;\n \n failed:\n@@ -658,10 +696,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n \n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n {\n-\tstruct repository *r = the_repository;\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n+\tif (!open_midx_bitmap_1(bitmap_git, midx))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501068","messageId":"44871306487e5c4c93f619b3c81b651cab3099d6.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:01Z","receivedAt":"2024-08-15T22:29:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\nEWAH-compressed bitmap corresponding to some given commit object.\n\nTeach this function about incremental MIDX bitmaps by teaching it to\nrecur on earlier bitmap layers when it fails to find a given commit in\nthe current layer.\n\nThe changes to do so are as follows:\n\n  - Avoid initializing hash_pos at its declaration, since\n    bitmap_for_commit() is now a recursive function and may receive a\n    NULL bitmap_index pointer as its first argument.\n\n  - In cases where we would previously return NULL (to indicate that a\n    lookup failed and the given bitmap_index does not contain an entry\n    corresponding to the given commit), recursively call the function on\n    the previous bitmap layer.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c27383c027..88623d9e06 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -946,18 +946,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t      struct commit *commit)\n {\n-\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n-\t\t\t\t\t   commit->object.oid);\n+\tkhiter_t hash_pos;\n+\tif (!bitmap_git)\n+\t\treturn NULL;\n+\n+\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501069","messageId":"b720fe56da867bfa560184fea7c5ede01c2426c1.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 05/13] pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:04Z","receivedAt":"2024-08-15T22:29:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since we may ask for a pack_id that is in an earlier MIDX layer relative\nto the one corresponding to our bitmap, use nth_midxed_pack() instead of\naccessing the ->packs array directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 88623d9e06..f91ab1b572 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1631,7 +1631,7 @@ static void show_objects_for_type(\n \t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n \n \t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n-\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\t} else {\n \t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n \t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501070","messageId":"9716d022e0b7a6992dd4630f9719079652b7de35.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:07Z","receivedAt":"2024-08-15T22:29:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits in the first phase of\nincremental MIDXs, enumerate not just the packs in the current\nincremental MIDX layer, but previous ones as well.\n\nLikewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\nsingle pack from a MIDX, use the oldest layer's preferred pack as it is\nlikely to contain the most amount of reusable sections.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f91ab1b572..2b3c53d882 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2320,7 +2320,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tmulti_pack_reuse = 0;\n \n \tif (multi_pack_reuse) {\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\t\tstruct bitmapped_pack pack;\n \t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n@@ -2344,14 +2345,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tuint32_t pack_int_id;\n \n \t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n \t\t\tuint32_t preferred_pack_pos;\n \n-\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n+\t\t\twhile (m->base_midx)\n+\t\t\t\tm = m->base_midx;\n+\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n \t\t\t\treturn;\n \t\t\t}\n \n-\t\t\tpack = bitmap_git->midx->packs[preferred_pack_pos];\n+\t\t\tpack = nth_midxed_pack(m, preferred_pack_pos);\n \t\t\tpack_int_id = preferred_pack_pos;\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501071","messageId":"6baece31750b889a189cf81f6121af162db91354.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:10Z","receivedAt":"2024-08-15T22:29:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement support for the special `--test-bitmap` mode of `git rev-list`\nwhen using incremental MIDXs.\n\nThe bitmap_test_data structure is extended to contain a \"base\" pointer\nthat mirrors the structure of the bitmap chain that it is being used to\ntest.\n\nWhen we find a commit to test, we first chase down the ->base pointer to\nfind the appropriate bitmap_test_data for the bitmap layer that the\ngiven commit is contained within, and then perform the test on that\nbitmap.\n\nIn order to implement this, light modifications are made to\nbitmap_for_commit() to reimplement it in terms of a new function,\nfind_bitmap_for_commit(), which fills out a pointer which indicates the\nbitmap layer which contains the given commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 105 ++++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 84 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2b3c53d882..5fea2714c1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -943,8 +943,9 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n \treturn NULL;\n }\n \n-struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n-\t\t\t\t      struct commit *commit)\n+static struct ewah_bitmap *find_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t  struct commit *commit,\n+\t\t\t\t\t\t  struct bitmap_index **found)\n {\n \tkhiter_t hash_pos;\n \tif (!bitmap_git)\n@@ -954,18 +955,30 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n+\t\tif (found)\n+\t\t\t*found = bitmap_git;\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n+\tif (found)\n+\t\t*found = bitmap_git;\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t      struct commit *commit)\n+{\n+\treturn find_bitmap_for_commit(bitmap_git, commit, NULL);\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n@@ -2489,6 +2502,8 @@ struct bitmap_test_data {\n \tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n+\n+\tstruct bitmap_test_data *base_tdata;\n };\n \n static void test_bitmap_type(struct bitmap_test_data *tdata,\n@@ -2497,6 +2512,11 @@ static void test_bitmap_type(struct bitmap_test_data *tdata,\n \tenum object_type bitmap_type = OBJ_NONE;\n \tint bitmaps_nr = 0;\n \n+\tif (bitmap_is_midx(tdata->bitmap_git)) {\n+\t\twhile (pos < tdata->bitmap_git->midx->num_objects_in_base)\n+\t\t\ttdata = tdata->base_tdata;\n+\t}\n+\n \tif (bitmap_get(tdata->commits, pos)) {\n \t\tbitmap_type = OBJ_COMMIT;\n \t\tbitmaps_nr++;\n@@ -2560,13 +2580,57 @@ static void test_show_commit(struct commit *commit, void *data)\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n+static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t total = 0;\n+\tdo {\n+\t\ttotal = st_add(total, bitmap_git->entry_count);\n+\t\tbitmap_git = bitmap_git->base;\n+\t} while (bitmap_git);\n+\n+\treturn total;\n+}\n+\n+static void prepare_bitmap_test_data(struct bitmap_test_data *tdata,\n+\t\t\t\t     struct bitmap_index *bitmap_git)\n+{\n+\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n+\n+\ttdata->bitmap_git = bitmap_git;\n+\ttdata->base = bitmap_new();\n+\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n+\n+\tif (bitmap_git->base) {\n+\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n+\t\tprepare_bitmap_test_data(tdata->base_tdata, bitmap_git->base);\n+\t}\n+}\n+\n+static void free_bitmap_test_data(struct bitmap_test_data *tdata)\n+{\n+\tif (!tdata)\n+\t\treturn;\n+\n+\tfree_bitmap_test_data(tdata->base_tdata);\n+\tfree(tdata->base_tdata);\n+\n+\tbitmap_free(tdata->base);\n+\tbitmap_free(tdata->commits);\n+\tbitmap_free(tdata->trees);\n+\tbitmap_free(tdata->blobs);\n+\tbitmap_free(tdata->tags);\n+}\n+\n void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n-\tstruct bitmap_index *bitmap_git;\n+\tstruct bitmap_index *bitmap_git, *found;\n \tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n@@ -2575,17 +2639,26 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n \t\tbitmap_git->version,\n \t\tbitmap_git->entry_count,\n-\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n+\t\tbitmap_total_entry_count(bitmap_git));\n \n \troot = revs->pending.objects[0].item;\n-\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n+\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n \n \tif (bm) {\n \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n-\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n+\t\t\toid_to_hex(&root->oid),\n+\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n+\n+\t\tif (bitmap_is_midx(found))\n+\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n+\t\t\t\t   hash_to_hex(get_midx_checksum(found->midx)));\n+\t\telse\n+\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n+\t\t\t\t   hash_to_hex(found->pack->hash));\n \n \t\tresult = ewah_to_bitmap(bm);\n \t}\n@@ -2602,14 +2675,8 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n-\ttdata.bitmap_git = bitmap_git;\n-\ttdata.base = bitmap_new();\n-\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n-\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n-\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n-\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n+\tprepare_bitmap_test_data(&tdata, bitmap_git);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n-\ttdata.seen = 0;\n \n \ttraverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);\n \n@@ -2621,11 +2688,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tdie(_(\"mismatch in bitmap results\"));\n \n \tbitmap_free(result);\n-\tbitmap_free(tdata.base);\n-\tbitmap_free(tdata.commits);\n-\tbitmap_free(tdata.trees);\n-\tbitmap_free(tdata.blobs);\n-\tbitmap_free(tdata.tags);\n+\tfree_bitmap_test_data(&tdata);\n \tfree_bitmap_index(bitmap_git);\n }\n \n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501072","messageId":"5c909df38ad86eb09ca92eda609cd8cda81edca9.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 08/13] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:13Z","receivedAt":"2024-08-15T22:29:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, use nth_midxed_pack() instead\nof accessing the MIDX's ->packs array directly to support incremental\nMIDXs.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5fea2714c1..d3bb78237e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1774,7 +1774,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n \t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n \n-\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n@@ -3020,7 +3020,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \n \t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n-\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tstruct packed_git *pack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \n \t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n \t\t\t\t\tstruct object_id oid;\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501073","messageId":"f9ae10fce907a751ac14add65942f51f24f9f7bb.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 09/13] pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:16Z","receivedAt":"2024-08-15T22:29:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for using pseudo-merges with incremental MIDX bitmaps by\nattempting to apply pseudo-merges from each layer when encountering a\ngiven commit during a walk.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d3bb78237e..1fa101bb33 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1087,10 +1087,15 @@ static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git\n \t\t\t\t\t\t struct commit *commit,\n \t\t\t\t\t\t uint32_t commit_pos)\n {\n-\tint ret;\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tint ret = 0;\n \n-\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t     result, commit, commit_pos);\n+\twhile (curr) {\n+\t\tret += apply_pseudo_merges_for_commit(&curr->pseudo_merges,\n+\t\t\t\t\t\t      result, commit,\n+\t\t\t\t\t\t      commit_pos);\n+\t\tcurr = curr->base;\n+\t}\n \n \tif (ret)\n \t\tpseudo_merges_satisfied_nr += ret;\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501074","messageId":"04042981c1ad0edb6ab768516cdff341ee278786.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:19Z","receivedAt":"2024-08-15T22:29:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"While individual bitmap layers store different commit, type-level, and\npseudo-merge bitmaps, only the top-most layer is used to compute\nreachability traversals.\n\nMany functions which implement the aforementioned traversal rely on\nenumerating the results according to the type-level bitmaps, and so\nwould benefit from a conceptual type-level bitmap that spans multiple\nlayers.\n\nImplement `struct ewah_or_iterator` which is capable of enumerating\nmultiple EWAH bitmaps at once, and OR-ing the results together. When\ninitialized with, for example, all of the commit type bitmaps from each\nlayer, callers can pretend as if they are enumerating a large type-level\nbitmap which contains the commits from *all* bitmap layers.\n\nThere are a couple of alternative approaches which were considered:\n\n  - Decompress each EWAH bitmap and OR them together, enumerating a\n    single (non-EWAH) bitmap. This would work, but has the disadvantage\n    of decompressing a potentially large bitmap, which may not be\n    necessary if the caller does not wish to read all of it.\n\n  - Recursively call bitmap internal functions, reusing the \"result\" and\n    \"haves\" bitmap from the top-most layer. This approach resembles the\n    original implementation of this feature, but is inefficient in that\n    it both (a) requires significant refactoring to implement, and (b)\n    enumerates large sections of later bitmaps which are all zeros (as\n    they pertain to objects in earlier layers).\n\n    (b) is not so bad in and of itself, but can cause significant\n    slow-downs when combined with expensive loop bodies.\n\nThis approach (enumerating an OR'd together version of all of the\ntype-level bitmaps from each layer) produces a significantly more\nstraightforward implementation with significantly less refactoring\nrequired in order to make it work.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ewah/ewah_bitmap.c | 33 +++++++++++++++++++++++++++++++++\n ewah/ewok.h        | 12 ++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex 8785cbc54a..b3a7ada071 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -372,6 +372,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n \t\tread_new_rlw(it);\n }\n \n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr)\n+{\n+\tsize_t i;\n+\n+\tmemset(it, 0, sizeof(*it));\n+\n+\tALLOC_ARRAY(it->its, nr);\n+\tfor (i = 0; i < nr; i++)\n+\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n+}\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n+{\n+\teword_t buf, out = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tfor (i = 0; i < it->nr; i++)\n+\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n+\t\t\tout |= buf;\n+\t\t\tret = 1;\n+\t\t}\n+\n+\t*next = out;\n+\treturn ret;\n+}\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it)\n+{\n+\tfree(it->its);\n+}\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 5e357e2493..4b70641045 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -148,6 +148,18 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent);\n  */\n int ewah_iterator_next(eword_t *next, struct ewah_iterator *it);\n \n+struct ewah_or_iterator {\n+\tstruct ewah_iterator *its;\n+\tsize_t nr;\n+};\n+\n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr);\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it);\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501076","messageId":"c4d543d43dc25ee0036a429bbe962f95f261f80f.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:22Z","receivedAt":"2024-08-15T22:29:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for reading the type-level bitmaps from previous bitmap layers\nby maintaining an array for each type, where each element in that type's\narray corresponds to one layer's bitmap for that type.\n\nThese fields will be used in a later commit to instantiate the 'struct\newah_or_iterator' for each type.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 51 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1fa101bb33..e1badc7887 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -78,6 +78,24 @@ struct bitmap_index {\n \tstruct ewah_bitmap *blobs;\n \tstruct ewah_bitmap *tags;\n \n+\t/*\n+\t * Type index arrays when this bitmap is associated with an\n+\t * incremental multi-pack index chain.\n+\t *\n+\t * If n is the number of unique layers in the MIDX chain, then\n+\t * commits_all[n-1] is this structs 'commits' field,\n+\t * commits_all[n-2] is the commits field of this bitmap's\n+\t * 'base', and so on.\n+\t *\n+\t * When either associated either with a non-incremental MIDX, or\n+\t * a single packfile, these arrays each contain a single\n+\t * element.\n+\t */\n+\tstruct ewah_bitmap **commits_all;\n+\tstruct ewah_bitmap **trees_all;\n+\tstruct ewah_bitmap **blobs_all;\n+\tstruct ewah_bitmap **tags_all;\n+\n \t/* Map from object ID -> `stored_bitmap` for all the bitmapped commits */\n \tkh_oid_map_t *bitmaps;\n \n@@ -586,7 +604,29 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \treturn load_pack_revindex(r, bitmap_git->pack);\n }\n \n-static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n+static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n+{\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tsize_t i = bitmap_git->base_nr - 1;\n+\n+\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr);\n+\n+\twhile (curr) {\n+\t\tbitmap_git->commits_all[i] = curr->commits;\n+\t\tbitmap_git->trees_all[i] = curr->trees;\n+\t\tbitmap_git->blobs_all[i] = curr->blobs;\n+\t\tbitmap_git->tags_all[i] = curr->tags;\n+\n+\t\tcurr = curr->base;\n+\t\ti -= 1;\n+\t}\n+}\n+\n+static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git,\n+\t\t       int recursing)\n {\n \tassert(bitmap_git->map);\n \n@@ -608,10 +648,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (bitmap_git->base) {\n \t\tif (!bitmap_is_midx(bitmap_git))\n \t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n-\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\tif (load_bitmap(r, bitmap_git->base, 1) < 0)\n \t\t\tgoto failed;\n \t}\n \n+\tif (!recursing)\n+\t\tload_all_type_bitmaps(bitmap_git);\n+\n \treturn 0;\n \n failed:\n@@ -687,7 +730,7 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git, 0))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2042,7 +2085,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (load_bitmap(revs->repo, bitmap_git, 0) < 0)\n \t\tgoto cleanup;\n \n \tif (!use_boundary_traversal)\n@@ -2957,6 +3000,10 @@ void free_bitmap_index(struct bitmap_index *b)\n \tewah_pool_free(b->trees);\n \tewah_pool_free(b->blobs);\n \tewah_pool_free(b->tags);\n+\tfree(b->commits_all);\n+\tfree(b->trees_all);\n+\tfree(b->blobs_all);\n+\tfree(b->tags_all);\n \tif (b->bitmaps) {\n \t\tstruct stored_bitmap *sb;\n \t\tkh_foreach_value(b->bitmaps, sb, {\n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501077","messageId":"c6730b4107e8069b3dd60f50368a3975668c09b0.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:25Z","receivedAt":"2024-08-15T22:29:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have initialized arrays for each bitmap layer's type bitmaps\nin the previous commit, adjust existing callers to use them in\npreparation for multi-layered bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 42 +++++++++++++++++++++++++++---------------\n 1 file changed, 27 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e1badc7887..9fac43749c 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1622,25 +1622,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t}\n }\n \n-static void init_type_iterator(struct ewah_iterator *it,\n+static void init_type_iterator(struct ewah_or_iterator *it,\n \t\t\t       struct bitmap_index *bitmap_git,\n \t\t\t       enum object_type type)\n {\n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tewah_iterator_init(it, bitmap_git->commits);\n+\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_TREE:\n-\t\tewah_iterator_init(it, bitmap_git->trees);\n+\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_BLOB:\n-\t\tewah_iterator_init(it, bitmap_git->blobs);\n+\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_TAG:\n-\t\tewah_iterator_init(it, bitmap_git->tags);\n+\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tdefault:\n@@ -1657,7 +1661,7 @@ static void show_objects_for_type(\n \tsize_t i = 0;\n \tuint32_t offset;\n \n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tstruct bitmap *objects = bitmap_git->result;\n@@ -1665,7 +1669,7 @@ static void show_objects_for_type(\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = objects->words[i] & filter;\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n \n@@ -1707,6 +1711,8 @@ static void show_objects_for_type(\n \t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n+\n+\tewah_or_iterator_free(&it);\n }\n \n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n@@ -1758,7 +1764,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n@@ -1775,7 +1781,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * packfile.\n \t */\n \tfor (i = 0, init_type_iterator(&it, bitmap_git, type);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\tif (i < tips->word_alloc)\n \t\t\tmask &= ~tips->words[i];\n@@ -1795,6 +1801,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -1852,14 +1859,14 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n \ttips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB);\n \n \tfor (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\teword_t word = to_filter->words[i] & mask;\n \t\tunsigned offset;\n@@ -1887,6 +1894,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -2502,12 +2510,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \n \tuint32_t i = 0, count = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tinit_type_iterator(&it, bitmap_git, type);\n \n-\twhile (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) {\n+\twhile (i < objects->word_alloc && ewah_or_iterator_next(&filter, &it)) {\n \t\teword_t word = objects->words[i++] & filter;\n \t\tcount += ewah_bit_popcount64(word);\n \t}\n@@ -2519,6 +2527,8 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \t\t\tcount++;\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn count;\n }\n \n@@ -3046,13 +3056,13 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *result = bitmap_git->result;\n \toff_t total = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \tsize_t i;\n \n \tinit_type_iterator(&it, bitmap_git, object_type);\n \tfor (i = 0; i < result->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = result->words[i] & filter;\n \t\tsize_t base = (i * BITS_IN_EWORD);\n \t\tunsigned offset;\n@@ -3093,6 +3103,8 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn total;\n }\n \n-- \n2.46.0.86.ge766d390f0.dirty\n\n"},{"id":"501078","messageId":"afefb4555750661ffd2c573a33d92f8fcb9f435a.1723760847.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723760847.git.me@ttaylorr.com","subject":"[PATCH v2 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-15T22:29:28Z","receivedAt":"2024-08-15T22:29:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap machinery has learned how to read and interact\nwith an incremental MIDX bitmap, teach the pack-bitmap-write.c machinery\n(and relevant callers from within the MIDX machinery) to write such\nbitmaps.\n\nThe details for doing so are mostly straightforward. The main changes\nare as follows:\n\n  - find_object_pos() now makes use of an extra MIDX parameter which is\n    used to locate the bit positions of objects which are from previous\n    layers (and thus do not exist in the current layer's pack_order\n    field).\n\n    (Note also that the pack_order field is moved into struct\n    write_midx_context to further simplify the callers for\n    write_midx_bitmap()).\n\n  - bitmap_writer_build_type_index() first determines how many objects\n    precede the current bitmap layer and offsets the bits it sets in\n    each respective type-level bitmap by that amount so they can be OR'd\n    together.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c                  |  3 +-\n midx-write.c                            | 35 +++++++----\n pack-bitmap-write.c                     | 65 ++++++++++++++-----\n pack-bitmap.h                           |  4 +-\n t/t5334-incremental-multi-pack-index.sh | 84 +++++++++++++++++++++++++\n 5 files changed, 161 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e23c4950ed..9e61ff7ca3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1342,7 +1342,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_init(&bitmap_writer,\n-\t\t\t\t\t\t   the_repository, &to_pack);\n+\t\t\t\t\t\t   the_repository, &to_pack,\n+\t\t\t\t\t\t   NULL);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t\t\t       written_list);\ndiff --git a/midx-write.c b/midx-write.c\nindex 81501efdda..bac3b0589a 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -826,20 +826,26 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \treturn cb.commits;\n }\n \n-static int write_midx_bitmap(const char *midx_name,\n+static int write_midx_bitmap(struct write_midx_context *ctx,\n+\t\t\t     const char *object_dir, const char *midx_name,\n \t\t\t     const unsigned char *midx_hash,\n \t\t\t     struct packing_data *pdata,\n \t\t\t     struct commit **commits,\n \t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n \t\t\t     unsigned flags)\n {\n \tint ret, i;\n \tuint16_t options = 0;\n \tstruct bitmap_writer writer;\n \tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t\thash_to_hex(midx_hash));\n+\tstruct strbuf bitmap_name = STRBUF_INIT;\n+\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n+\t\t\t\t      MIDX_EXT_BITMAP);\n \n \ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n \n@@ -858,7 +864,8 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer, the_repository, pdata);\n+\tbitmap_writer_init(&writer, the_repository, pdata,\n+\t\t\t   ctx->incremental ? ctx->base_midx : NULL);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, index);\n \n@@ -876,7 +883,7 @@ static int write_midx_bitmap(const char *midx_name,\n \t * bitmap_writer_finish().\n \t */\n \tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\t\tindex[ctx->pack_order[i]] = &pdata->objects[i].idx;\n \n \tbitmap_writer_select_commits(&writer, commits, commits_nr);\n \tret = bitmap_writer_build(&writer);\n@@ -884,11 +891,11 @@ static int write_midx_bitmap(const char *midx_name,\n \t\tgoto cleanup;\n \n \tbitmap_writer_set_checksum(&writer, midx_hash);\n-\tbitmap_writer_finish(&writer, index, bitmap_name, options);\n+\tbitmap_writer_finish(&writer, index, bitmap_name.buf, options);\n \n cleanup:\n \tfree(index);\n-\tfree(bitmap_name);\n+\tstrbuf_release(&bitmap_name);\n \tbitmap_writer_free(&writer);\n \n \ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n@@ -1072,8 +1079,6 @@ static int write_midx_internal(const char *object_dir,\n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n \n \tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n-\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n-\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1115,6 +1120,12 @@ static int write_midx_internal(const char *object_dir,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n+\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n+\t\t\t\t      hash_to_hex(get_midx_checksum(m)));\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n@@ -1404,8 +1415,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\tif (write_midx_bitmap(&ctx, object_dir, midx_name.buf,\n+\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n \t\t\t\t      flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 923f793cec..8fc979cbc9 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -25,6 +25,8 @@\n #include \"alloc.h\"\n #include \"refs.h\"\n #include \"strmap.h\"\n+#include \"midx.h\"\n+#include \"pack-revindex.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -42,7 +44,8 @@ static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer\n }\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata)\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n \tif (writer->bitmaps)\n@@ -50,6 +53,7 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n \twriter->bitmaps = kh_init_oid_map();\n \twriter->pseudo_merge_commits = kh_init_oid_map();\n \twriter->to_pack = pdata;\n+\twriter->midx = midx;\n \n \tstring_list_init_dup(&writer->pseudo_merge_groups);\n \n@@ -104,6 +108,11 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t    struct pack_idx_entry **index)\n {\n \tuint32_t i;\n+\tuint32_t base_objects = 0;\n+\n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n \n \twriter->commits = ewah_new();\n \twriter->trees = ewah_new();\n@@ -133,19 +142,19 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \n \t\tswitch (real_type) {\n \t\tcase OBJ_COMMIT:\n-\t\t\tewah_set(writer->commits, i);\n+\t\t\tewah_set(writer->commits, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TREE:\n-\t\t\tewah_set(writer->trees, i);\n+\t\t\tewah_set(writer->trees, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_BLOB:\n-\t\t\tewah_set(writer->blobs, i);\n+\t\t\tewah_set(writer->blobs, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TAG:\n-\t\t\tewah_set(writer->tags, i);\n+\t\t\tewah_set(writer->tags, i + base_objects);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -198,19 +207,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n static uint32_t find_object_pos(struct bitmap_writer *writer,\n \t\t\t\tconst struct object_id *oid, int *found)\n {\n-\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n+\tstruct object_entry *entry;\n+\n+\tentry = packlist_find(writer->to_pack, oid);\n+\tif (entry) {\n+\t\tuint32_t base_objects = 0;\n+\t\tif (writer->midx)\n+\t\t\tbase_objects = writer->midx->num_objects +\n+\t\t\t\twriter->midx->num_objects_in_base;\n+\n+\t\tif (found)\n+\t\t\t*found = 1;\n+\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n+\t} else if (writer->midx) {\n+\t\tuint32_t at, pos;\n+\n+\t\tif (!bsearch_midx(oid, writer->midx, &at))\n+\t\t\tgoto missing;\n+\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n+\t\t\tgoto missing;\n \n-\tif (!entry) {\n \t\tif (found)\n-\t\t\t*found = 0;\n-\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n-\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n-\t\treturn 0;\n+\t\t\t*found = 1;\n+\t\treturn pos;\n \t}\n \n+missing:\n \tif (found)\n-\t\t*found = 1;\n-\treturn oe_in_pack_pos(writer->to_pack, entry);\n+\t\t*found = 0;\n+\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\treturn 0;\n }\n \n static void compute_xor_offsets(struct bitmap_writer *writer)\n@@ -577,7 +604,7 @@ int bitmap_writer_build(struct bitmap_writer *writer)\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n-\tuint32_t *mapping;\n+\tuint32_t *mapping = NULL;\n \tint closed = 1; /* until proven otherwise */\n \n \tif (writer->show_progress)\n@@ -1010,7 +1037,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \toff_t *offsets = NULL;\n-\tuint32_t i;\n+\tuint32_t i, base_objects;\n \n \tstruct bitmap_disk_header header;\n \n@@ -1036,6 +1063,12 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, writer->to_pack->nr_objects);\n \n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n+\telse\n+\t\tbase_objects = 0;\n+\n \tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n@@ -1044,7 +1077,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \t\tif (commit_pos < 0)\n \t\t\tBUG(_(\"trying to write commit not in index\"));\n-\t\tstored->commit_pos = commit_pos;\n+\t\tstored->commit_pos = commit_pos + base_objects;\n \t}\n \n \twrite_selected_commits_v1(writer, f, offsets);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex ff0fd815b8..4242458198 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -110,6 +110,7 @@ struct bitmap_writer {\n \n \tkh_oid_map_t *bitmaps;\n \tstruct packing_data *to_pack;\n+\tstruct multi_pack_index *midx; /* if appending to a MIDX chain */\n \n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n@@ -124,7 +125,8 @@ struct bitmap_writer {\n };\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata);\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex c3b08acc73..0b6d45c8fd 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -43,4 +43,88 @@ test_expect_success 'convert incremental to non-incremental' '\n \n compare_results_with_midx 'non-incremental MIDX conversion'\n \n+write_midx_layer () {\n+\tn=1\n+\tif test -f $midx_chain\n+\tthen\n+\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n+\tfi\n+\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit $n.$i &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit multi-pack-index write --bitmap --incremental\n+}\n+\n+test_expect_success 'write initial MIDX layer' '\n+\tgit repack -ad &&\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'read bitmap from first MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'write another MIDX layer' '\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'midx verify with multiple layers' '\n+\tgit multi-pack-index verify\n+'\n+\n+test_expect_success 'read bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 2.2\n+'\n+\n+test_expect_success 'read earlier bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'show object from first pack' '\n+\tgit cat-file -p 1.1\n+'\n+\n+test_expect_success 'show object from second pack' '\n+\tgit cat-file -p 2.2\n+'\n+\n+for reuse in false single multi\n+do\n+\ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n+\t\trm -fr clone.git &&\n+\n+\t\tgit config pack.allowPackReuse $reuse &&\n+\t\tgit clone --no-local --bare . clone.git\n+\t'\n+done\n+\n+test_expect_success 'relink existing MIDX layer' '\n+\trm -fr \"$midxdir\" &&\n+\n+\tGIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n+\n+\tmidx_hash=\"$(test-tool read-midx --checksum $objdir)\" &&\n+\n+\ttest_path_is_file \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_commit another &&\n+\tgit repack -d &&\n+\tgit multi-pack-index write --bitmap --incremental &&\n+\n+\ttest_path_is_missing \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.midx\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.rev\" &&\n+\ttest_line_count = 2 \"$midx_chain\"\n+\n+'\n+\n test_done\n-- \n2.46.0.86.ge766d390f0.dirty\n"},{"id":"501785","messageId":"xmqqseuozg53.fsf_-_@gitster.g","threadId":"61958","inReplyTo":"afefb4555750661ffd2c573a33d92f8fcb9f435a.1723760847.git.me@ttaylorr.com","subject":"[PATCH] fixup! midx: implement writing incremental MIDX bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-28T17:55:20Z","receivedAt":"2024-08-28T17:55:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"With -Wunused, the compiler notices that the midx_name parameter is\nunused.  In this case, it is truly unused, the function signature is\nnot constrained externally, so we can simply drop the parameter from\nthe definition of the function and its sole caller.\n\nThis comes from 01a2cbab (midx: implement writing incremental MIDX\nbitmaps, 2024-08-15), so I'll squash the following to that commit.\n\n midx-write.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git c/midx-write.c w/midx-write.c\nindex bac3b0589a..0ad9139fdb 100644\n--- c/midx-write.c\n+++ w/midx-write.c\n@@ -827,7 +827,7 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n }\n \n static int write_midx_bitmap(struct write_midx_context *ctx,\n-\t\t\t     const char *object_dir, const char *midx_name,\n+\t\t\t     const char *object_dir,\n \t\t\t     const unsigned char *midx_hash,\n \t\t\t     struct packing_data *pdata,\n \t\t\t     struct commit **commits,\n@@ -1415,7 +1415,7 @@ static int write_midx_internal(const char *object_dir,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(&ctx, object_dir, midx_name.buf,\n+\t\tif (write_midx_bitmap(&ctx, object_dir,\n \t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n \t\t\t\t      flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n\n"},{"id":"501787","messageId":"20240828183356.GA4043247@coredump.intra.peff.net","threadId":"61958","inReplyTo":"xmqqseuozg53.fsf_-_@gitster.g","subject":"Re: [PATCH] fixup! midx: implement writing incremental MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-28T18:33:56Z","receivedAt":"2024-08-28T18:33:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 28, 2024 at 10:55:20AM -0700, Junio C Hamano wrote:\n\n> With -Wunused, the compiler notices that the midx_name parameter is\n> unused.  In this case, it is truly unused, the function signature is\n> not constrained externally, so we can simply drop the parameter from\n> the definition of the function and its sole caller.\n> \n> This comes from 01a2cbab (midx: implement writing incremental MIDX\n> bitmaps, 2024-08-15), so I'll squash the following to that commit.\n\nWell, that didn't take long for this to come up again. :) I've been\nfixing them progressively as they hit 'next' (since I don't usually\nbuild 'seen'), but this one isn't there yet.\n\nI'm always curious in these cases why we have the parameter at all if\nit's unnecessary (i.e., is it a bug or leftover cruft). In this case, it\nwas present before that commit, but refactoring meant that we no longer\nwrite to $name-$hash.bitmap, but instead use get_midx_filename_ext(), or\nget_split_midx_filename_ext() in incremental mode.\n\nIs that right, though? It looks like the caller might pass in a\ntempfile name like .../pack/multi-pack-index.d/tmp_midx_XXXXXX,\nif we're in incremental mode. But we'll write directly to\n\"multi-pack-index-$hash.bitmap\" in the same directory. I'm not sure to\nwhat degree it matters, since that's the name we want in the long run.\nBut would we possibly overwrite an active-in-use file rather than doing\nthe atomic rename-into-place if we happened to generate the same midx?\n\nIt feels like we should still respect the name the caller is using for\ntempfiles, and then rename it into the correct spot at the end.\n\n-Peff\n"},{"id":"501853","messageId":"ZtDEhNRfXth63SJs@nand.local","threadId":"61958","inReplyTo":"20240828183356.GA4043247@coredump.intra.peff.net","subject":"Re: [PATCH] fixup! midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-08-29T18:57:08Z","receivedAt":"2024-08-29T18:57:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 28, 2024 at 02:33:56PM -0400, Jeff King wrote:\n> Is that right, though? It looks like the caller might pass in a\n> tempfile name like .../pack/multi-pack-index.d/tmp_midx_XXXXXX,\n> if we're in incremental mode. But we'll write directly to\n> \"multi-pack-index-$hash.bitmap\" in the same directory. I'm not sure to\n> what degree it matters, since that's the name we want in the long run.\n> But would we possibly overwrite an active-in-use file rather than doing\n> the atomic rename-into-place if we happened to generate the same midx?\n>\n> It feels like we should still respect the name the caller is using for\n> tempfiles, and then rename it into the correct spot at the end.\n\nIn either case, we're going to write to a temporary file initialized by\nthe pack-bitmap machinery and then rename() it into place at the end of\nbitmap_writer_finish().\n\nOn the caller side, in the non-incremental mode, we'll pass\n$GIT_DIR/objects/pack/multi-pack-index-$hash.bitmap as the location,\nwrite its contents into a temporary file, and then rename() it there.\n\nBut in the incremental mode this series introduces, I think it would be\na bug to pass a tmp_midx_XXXXXX file path there, since nobody would move\nit from tmp_midx_XXXXX-$HASH.bitmap into its final location.\n\nSo I think what's written here with the fixup! patch is right (and\nshould be squashed into 13/13 in the next round), but let me know if I'm\nmissing something.\n\nThanks,\nTaylor\n"},{"id":"501859","messageId":"20240829192713.GA423429@coredump.intra.peff.net","threadId":"61958","inReplyTo":"ZtDEhNRfXth63SJs@nand.local","subject":"Re: [PATCH] fixup! midx: implement writing incremental MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-29T19:27:13Z","receivedAt":"2024-08-29T19:27:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 29, 2024 at 02:57:08PM -0400, Taylor Blau wrote:\n\n> On Wed, Aug 28, 2024 at 02:33:56PM -0400, Jeff King wrote:\n> > Is that right, though? It looks like the caller might pass in a\n> > tempfile name like .../pack/multi-pack-index.d/tmp_midx_XXXXXX,\n> > if we're in incremental mode. But we'll write directly to\n> > \"multi-pack-index-$hash.bitmap\" in the same directory. I'm not sure to\n> > what degree it matters, since that's the name we want in the long run.\n> > But would we possibly overwrite an active-in-use file rather than doing\n> > the atomic rename-into-place if we happened to generate the same midx?\n> >\n> > It feels like we should still respect the name the caller is using for\n> > tempfiles, and then rename it into the correct spot at the end.\n> \n> In either case, we're going to write to a temporary file initialized by\n> the pack-bitmap machinery and then rename() it into place at the end of\n> bitmap_writer_finish().\n\nOK, that addresses my worry, if we're always writing to a tempfile (and\nI verified with some recent stracing that this is the case). So renaming\nthat into tmp_midx_XXXXXX.bitmap would just be a pointless extra layer\nof renames.\n\nI do wonder if it's possible for us to generate a new different revindex\nand bitmap pair for the same midx hash, and for a reader to see a\nmismatched set for a moment. But that's an atomicity problem, and an\nextra layer of renames is not going to solve that.\n\n> On the caller side, in the non-incremental mode, we'll pass\n> $GIT_DIR/objects/pack/multi-pack-index-$hash.bitmap as the location,\n> write its contents into a temporary file, and then rename() it there.\n> \n> But in the incremental mode this series introduces, I think it would be\n> a bug to pass a tmp_midx_XXXXXX file path there, since nobody would move\n> it from tmp_midx_XXXXX-$HASH.bitmap into its final location.\n> \n> So I think what's written here with the fixup! patch is right (and\n> should be squashed into 13/13 in the next round), but let me know if I'm\n> missing something.\n\nWhat confused me is that write_midx_reverse_index() _does_ still take\nmidx_name, and respects it. But I think that is a bug!\n\nWe do not usually even call that function, since modern midx's have a\nRIDX chunk inside them instead of a separate file. But if you do this:\n\n  # generate an extra pack\n  git commit --allow-empty -m foo\n  git repack -d\n\n  # make an incremental midx with a .rev file; usually this ends up\n  # as a RIDX chunk, so we have to force it.\n  GIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --incremental --bitmap\n\nthen you'll end up with a tmp_midx_XXXXXX-*.rev file leftover in\nmulti-pack-index.d (since, as you note, nobody is moving those into\nplace).\n\nSo probably write_midx_reverse_index() needs the same treatment to\nderive its own filenames for the incremental case, and to drop the\nmidx_name parameter.\n\nOr I wonder if we could simply drop the code to write a separate .rev\nfile entirely? I don't think there's a reason anybody would want it.\n\n-Peff\n"},{"id":"507644","messageId":"Zzz7jdI7Y+QEbwRw@nand.local","threadId":"61958","inReplyTo":"20240829192713.GA423429@coredump.intra.peff.net","subject":"Re: [PATCH] fixup! midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T20:56:45Z","receivedAt":"2024-11-19T20:56:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 29, 2024 at 03:27:13PM -0400, Jeff King wrote:\n> On Thu, Aug 29, 2024 at 02:57:08PM -0400, Taylor Blau wrote:\n>\n> > On Wed, Aug 28, 2024 at 02:33:56PM -0400, Jeff King wrote:\n> > > Is that right, though? It looks like the caller might pass in a\n> > > tempfile name like .../pack/multi-pack-index.d/tmp_midx_XXXXXX,\n> > > if we're in incremental mode. But we'll write directly to\n> > > \"multi-pack-index-$hash.bitmap\" in the same directory. I'm not sure to\n> > > what degree it matters, since that's the name we want in the long run.\n> > > But would we possibly overwrite an active-in-use file rather than doing\n> > > the atomic rename-into-place if we happened to generate the same midx?\n> > >\n> > > It feels like we should still respect the name the caller is using for\n> > > tempfiles, and then rename it into the correct spot at the end.\n> >\n> > In either case, we're going to write to a temporary file initialized by\n> > the pack-bitmap machinery and then rename() it into place at the end of\n> > bitmap_writer_finish().\n>\n> OK, that addresses my worry, if we're always writing to a tempfile (and\n> I verified with some recent stracing that this is the case). So renaming\n> that into tmp_midx_XXXXXX.bitmap would just be a pointless extra layer\n> of renames.\n\nYeah, I think we are OK here.\n\n> I do wonder if it's possible for us to generate a new different revindex\n> and bitmap pair for the same midx hash, and for a reader to see a\n> mismatched set for a moment. But that's an atomicity problem, and an\n> extra layer of renames is not going to solve that.\n\nWhat you're describing is basically the bug that we fixed in 95e8383bac\n(midx.c: make changing the preferred pack safe, 2022-01-25). That commit\nsought to ensure that there was no way to have a different reverse index\n(IOW, pseudo-pack order) for a given MIDX hash.\n\nIt's possible that there is some case that we're not yet covering, but I\ncan't think of it. The things that we care about are (a) the set of\nobjects, (b) the set of packs those objects came from, and (c) the\npreferred pack. We don't store (c) directly, but we can infer it from\nthe reverse index, which we do write within the RIDX chunk as you note\nbelow.\n\n> > On the caller side, in the non-incremental mode, we'll pass\n> > $GIT_DIR/objects/pack/multi-pack-index-$hash.bitmap as the location,\n> > write its contents into a temporary file, and then rename() it there.\n> >\n> > But in the incremental mode this series introduces, I think it would be\n> > a bug to pass a tmp_midx_XXXXXX file path there, since nobody would move\n> > it from tmp_midx_XXXXX-$HASH.bitmap into its final location.\n> >\n> > So I think what's written here with the fixup! patch is right (and\n> > should be squashed into 13/13 in the next round), but let me know if I'm\n> > missing something.\n>\n> What confused me is that write_midx_reverse_index() _does_ still take\n> midx_name, and respects it. But I think that is a bug!\n>\n> We do not usually even call that function, since modern midx's have a\n> RIDX chunk inside them instead of a separate file. But if you do this:\n>\n>   # generate an extra pack\n>   git commit --allow-empty -m foo\n>   git repack -d\n>\n>   # make an incremental midx with a .rev file; usually this ends up\n>   # as a RIDX chunk, so we have to force it.\n>   GIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --incremental --bitmap\n>\n> then you'll end up with a tmp_midx_XXXXXX-*.rev file leftover in\n> multi-pack-index.d (since, as you note, nobody is moving those into\n> place).\n>\n> So probably write_midx_reverse_index() needs the same treatment to\n> derive its own filenames for the incremental case, and to drop the\n> midx_name parameter.\n>\n> Or I wonder if we could simply drop the code to write a separate .rev\n> file entirely? I don't think there's a reason anybody would want it.\n\nI would kind of like to get rid of it, but we use it in a couple of\nplaces in the test suite:\n\n    $ git grep GIT_TEST_MIDX_WRITE_REV=\n    t/t5327-multi-pack-bitmaps-rev.sh:GIT_TEST_MIDX_WRITE_REV=1\n    t/t5334-incremental-multi-pack-index.sh:        GIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n\nAnd both of those tests are testing the old behavior, which we need an\nout-of-MIDX .rev file in order to do. Alternatively, we could store a\ntest fixture in the repository that contains these files so we don't\nhave to build them from scratch.\n\nBut after the xz incident earlier this year, I am *very* wary of adding\nbinary test fixtures into the tree, since they seem like an easy vector\nfor attack.\n\nSo I'm content to fix the bug here and keep the old code around for a\nwhile longer. The fix is indeed as simple as you described, which is\nnice ;-).\n\nThanks,\nTaylor\n"},{"id":"507648","messageId":"cover.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH v3 00/13] midx: incremental multi-pack indexes, part two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:16Z","receivedAt":"2024-11-19T22:07:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"== Changes since last time\n\nThis round fixes a small issue when writing legacy \".rev\" files outside\nof the MIDX in '--incremental' mode.\n\nThe rest of the series is unchanged, and re-submitted to solicit review\nnow that I have more time to focus on this series.\n\n== Original cover letter\n\nThis series is based on 'master', with an additional merge between\ntb/incremental-midx-part-1[1] and my newer series to fix a handful of\nbugs related to pseudo-merge bitmaps[2].\n\nThis is the second of three series to implement support for incremental\nmulti-pack indexes (MIDXs). This series brings support for bitmaps that\nare tied to incremental MIDXs in addition to regular MIDX bitmaps.\n\nThe details are laid out in the commits themselves, but the high-level\napproach is as follows:\n\n  - Each layer in the incremental MIDX chain has its own corresponding\n    *.bitmap file. Each bitmap contains commits / pseudo-merges which\n    are selected only from the commits in that layer. Likewise, only\n    that layer's objects are written in the type-level bitmaps.\n\n  - The reachability traversal is only conducted on the top-most bitmap\n    corresponding to the most recent layer in the incremental MIDX\n    chain. Earlier layers may be consulted to retrieve commit /\n    pseudo-merge reachability bitmaps, but only the top-most bitmap's\n    \"result\" and \"haves\" fields are used.\n\n  - In essence, the top-most bitmap is the only one that \"matters\", and\n    earlier bitmaps are merely used to look up commit and pseudo-merge\n    bitmaps from that layer.\n\n  - Whenever we need to look at the type-level bitmaps corresponding to\n    the whole incremental MIDX chain, a new \"ewah_or_iterator\" is used.\n    This works in concept like a typical ewah_iterator, except works\n    over many EWAH bitmaps in parallel, OR-ing their results together\n    before returning them to the user.\n\n    In effect, this allows us to treat the union of all type-level\n    bitmaps (each of which only stores information about the objects its\n    corresponding layer within the incremental MIDX chain) as a single\n    type-level bitmap corresponding to all of the objects across every\n    layer of the incremental MIDX chain.\n\nThe sum total of this series is that we are able to append new commits /\npseudo-merges to a repository's reachability bitmaps without having to\nrewrite existing bitmaps, making the operation much cheaper to perform\nin large repositories.\n\nThe series is laid out roughly as follows:\n\n  - The first patch describes the technical details of incremental MIDX\n    bitmaps.\n\n  - The second patch adjusts the pack-revindex internals to prepare for\n    incremental MIDX bitmaps.\n\n  - The next seven patches adjust various components of the pack-bitmap\n    internals to do the same.\n\n  - The next three patches introduce and adjust callers to use the\n    ewah_or_iterator (as above).\n\n  - The final patch implements writing incremental MIDX bitmaps, and\n    introduces tests.\n\nAfter this series, the remaining goals for this project include being\nable to compact contiguous runs of incremental MIDX layers into a single\nlayer to support growing the chain of MIDX layers without the chain\nitself becoming too long.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1722958595.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover.1723743050.git.me@ttaylorr.com/\n\nTaylor Blau (13):\n  Documentation: describe incremental MIDX bitmaps\n  pack-revindex: prepare for incremental MIDX bitmaps\n  pack-bitmap.c: open and store incremental bitmap layers\n  pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n  pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n  pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n  pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n  pack-bitmap.c: compute disk-usage with incremental MIDXs\n  pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n  ewah: implement `struct ewah_or_iterator`\n  pack-bitmap.c: keep track of each layer's type bitmaps\n  pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n  midx: implement writing incremental MIDX bitmaps\n\n Documentation/technical/multi-pack-index.txt |  64 ++++\n builtin/pack-objects.c                       |   3 +-\n ewah/ewah_bitmap.c                           |  33 ++\n ewah/ewok.h                                  |  12 +\n midx-write.c                                 |  49 ++-\n pack-bitmap-write.c                          |  65 +++-\n pack-bitmap.c                                | 329 +++++++++++++++----\n pack-bitmap.h                                |   4 +-\n pack-revindex.c                              |  32 +-\n t/t5334-incremental-multi-pack-index.sh      |  84 +++++\n 10 files changed, 559 insertions(+), 116 deletions(-)\n\nRange-diff against v2:\n 1:  90b21b11ed7 <  -:  ----------- Documentation: describe incremental MIDX format\n 2:  0d3b19c59ff <  -:  ----------- midx: add new fields for incremental MIDX chains\n 3:  5cd742b6776 <  -:  ----------- midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs\n 4:  372104c73de <  -:  ----------- midx: teach `prepare_midx_pack()` about incremental MIDXs\n 5:  e68a3ceff9a <  -:  ----------- midx: teach `nth_midxed_object_oid()` about incremental MIDXs\n 6:  ff2d7bc5ca0 <  -:  ----------- midx: teach `nth_bitmapped_pack()` about incremental MIDXs\n 7:  32c3fceada7 <  -:  ----------- midx: introduce `bsearch_one_midx()`\n 8:  16db6c98cec <  -:  ----------- midx: teach `bsearch_midx()` about incremental MIDXs\n 9:  761c7c59ba1 <  -:  ----------- midx: teach `nth_midxed_offset()` about incremental MIDXs\n10:  8366456d29b <  -:  ----------- midx: teach `fill_midx_entry()` about incremental MIDXs\n11:  909d927c470 <  -:  ----------- midx: remove unused `midx_locate_pack()`\n12:  71127601b5d <  -:  ----------- midx: teach `midx_contains_pack()` about incremental MIDXs\n13:  2f98ebb141e <  -:  ----------- midx: teach `midx_preferred_pack()` about incremental MIDXs\n14:  550ae2dc933 <  -:  ----------- midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs\n15:  9ae1bc415e9 <  -:  ----------- midx: support reading incremental MIDX chains\n16:  3d4181df518 <  -:  ----------- midx: implement verification support for incremental MIDXs\n17:  3b268f91bf3 <  -:  ----------- t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n18:  09d74f89423 <  -:  ----------- t/t5313-pack-bounds-checks.sh: prepare for sub-directories\n19:  5d467d38a8d <  -:  ----------- midx: implement support for writing incremental MIDX chains\n20:  9d322fc5399 <  -:  ----------- pack-bitmap: initialize `bitmap_writer_init()` with packing_data\n21:  238ca46998e <  -:  ----------- pack-bitmap: drop redundant args from `bitmap_writer_build_type_index()`\n22:  5e198489fa8 <  -:  ----------- pack-bitmap: drop redundant args from `bitmap_writer_build()`\n23:  819a0765f38 <  -:  ----------- pack-bitmap: drop redundant args from `bitmap_writer_finish()`\n24:  0fea7803d86 <  -:  ----------- pack-bitmap-write.c: select pseudo-merges even for small bitmaps\n25:  228553e412f <  -:  ----------- t/t5333-pseudo-merge-bitmaps.sh: demonstrate empty pseudo-merge groups\n26:  c7e0ee07120 <  -:  ----------- pseudo-merge.c: do not generate empty pseudo-merge commits\n27:  c9a64b1d2a9 <  -:  ----------- pseudo-merge.c: ensure pseudo-merge groups are closed\n28:  d1b8d11b37f =  1:  caed2c6ec34 Documentation: describe incremental MIDX bitmaps\n29:  f5d0866e5cb =  2:  b902513f436 pack-revindex: prepare for incremental MIDX bitmaps\n30:  43444efc214 !  3:  5b5d625cbe0 pack-bitmap.c: open and store incremental bitmap layers\n    @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_git(struct repository *r)\n      \t\treturn bitmap_git;\n      \n      \tfree_bitmap_index(bitmap_git);\n    +@@ pack-bitmap.c: void free_bitmap_index(struct bitmap_index *b)\n    + \t\tclose_midx_revindex(b->midx);\n    + \t}\n    + \tfree_pseudo_merge_map(&b->pseudo_merges);\n    ++\tfree_bitmap_index(b->base);\n    + \tfree(b);\n    + }\n    + \n31:  44871306487 =  4:  16259667fb4 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n32:  b720fe56da8 =  5:  b7a45d7eff8 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n33:  9716d022e0b =  6:  c8401fa0fbd pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n34:  6baece31750 =  7:  17ab23dd76d pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n35:  5c909df38ad =  8:  75d170ce078 pack-bitmap.c: compute disk-usage with incremental MIDXs\n36:  f9ae10fce90 =  9:  0b4fcfcecb6 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n37:  04042981c1a = 10:  e1b5f6181e3 ewah: implement `struct ewah_or_iterator`\n38:  c4d543d43dc = 11:  9ab8fb472f4 pack-bitmap.c: keep track of each layer's type bitmaps\n39:  c6730b4107e = 12:  87cb011e7fc pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n40:  afefb455575 ! 13:  77ddd1170f9 midx: implement writing incremental MIDX bitmaps\n    @@ builtin/pack-objects.c: static void write_pack_file(void)\n      \t\t\t\t\t\t\t       written_list);\n     \n      ## midx-write.c ##\n    +@@ midx-write.c: static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n    + \treturn pack_order;\n    + }\n    + \n    +-static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n    +-\t\t\t\t     struct write_midx_context *ctx)\n    ++static void write_midx_reverse_index(struct write_midx_context *ctx,\n    ++\t\t\t\t     const char *object_dir,\n    ++\t\t\t\t     unsigned char *midx_hash)\n    + {\n    + \tstruct strbuf buf = STRBUF_INIT;\n    + \tchar *tmp_file;\n    + \n    + \ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n    + \n    +-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n    ++\tif (ctx->incremental)\n    ++\t\tget_split_midx_filename_ext(&buf, object_dir, midx_hash,\n    ++\t\t\t\t\t    MIDX_EXT_REV);\n    ++\telse\n    ++\t\tget_midx_filename_ext(&buf, object_dir, midx_hash,\n    ++\t\t\t\t      MIDX_EXT_REV);\n    + \n    + \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n    + \t\t\t\t\tmidx_hash, WRITE_REV);\n     @@ midx-write.c: static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n      \treturn cb.commits;\n      }\n      \n     -static int write_midx_bitmap(const char *midx_name,\n     +static int write_midx_bitmap(struct write_midx_context *ctx,\n    -+\t\t\t     const char *object_dir, const char *midx_name,\n    ++\t\t\t     const char *object_dir,\n      \t\t\t     const unsigned char *midx_hash,\n      \t\t\t     struct packing_data *pdata,\n      \t\t\t     struct commit **commits,\n    @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n      \t\t\tm = m->base_midx;\n      \t\t}\n     @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n    + \n    + \tif (flags & MIDX_WRITE_REV_INDEX &&\n    + \t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n    +-\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n    ++\t\twrite_midx_reverse_index(&ctx, object_dir, midx_hash);\n    + \n    + \tif (flags & MIDX_WRITE_BITMAP) {\n    + \t\tstruct packing_data pdata;\n    +@@ midx-write.c: static int write_midx_internal(const char *object_dir,\n      \t\tFREE_AND_NULL(ctx.entries);\n      \t\tctx.entries_nr = 0;\n      \n     -\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n     -\t\t\t\t      commits, commits_nr, ctx.pack_order,\n    -+\t\tif (write_midx_bitmap(&ctx, object_dir, midx_name.buf,\n    ++\t\tif (write_midx_bitmap(&ctx, object_dir,\n     +\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n      \t\t\t\t      flags) < 0) {\n      \t\t\terror(_(\"could not write multi-pack bitmap\"));\n\nbase-commit: 090d24e9af6e9f59c3f7bee97c42bb1ae3c7f559\n-- \n2.47.0.301.g77ddd1170f9\n"},{"id":"507649","messageId":"caed2c6ec3483f028f59777bba40480e2661ca80.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:19Z","receivedAt":"2024-11-19T22:07:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement support for reachability bitmaps for the new\nincremental multi-pack index (MIDX) feature over the following commits.\n\nThis commit begins by first describing the relevant format and usage\ndetails for incremental MIDX bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/technical/multi-pack-index.txt | 64 ++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex cc063b30bea..a063262c360 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -164,6 +164,70 @@ objects_nr($H2) + objects_nr($H1) + i\n (in the C implementation, this is often computed as `i +\n m->num_objects_in_base`).\n \n+=== Pseudo-pack order for incremental MIDXs\n+\n+The original implementation of multi-pack reachability bitmaps defined\n+the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n+titled \"multi-pack-index reverse indexes\") roughly as follows:\n+\n+____\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+____\n+\n+In the incremental MIDX design, we extend this definition to include\n+objects from multiple layers of the MIDX chain. The pseudo-pack order\n+for incremental MIDXs is determined by concatenating the pseudo-pack\n+ordering for each layer of the MIDX chain in order. Formally two objects\n+`o1` and `o2` are compared as follows:\n+\n+1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n+  `o1` is considered less than `o2`.\n+2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n+  MIDX layer has no base, then If one of `pack(o1)` and `pack(o2)` is\n+  preferred and the other is not, then the preferred one sorts first. If\n+  there is a base layer (i.e. the MIDX layer is not the first layer in\n+  the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n+  pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n+  appears earlier, than the opposite is true.\n+3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n+  same MIDX layer. Sort `o1` and `o2` by their offset within their\n+  containing packfile.\n+\n+=== Reachability bitmaps and incremental MIDXs\n+\n+Each layer of an incremental MIDX chain may have its objects (and the\n+objects from any previous layer in the same MIDX chain) represented in\n+its own `*.bitmap` file.\n+\n+The structure of a `*.bitmap` file belonging to an incremental MIDX\n+chain is identical to that of a non-incremental MIDX bitmap, or a\n+classic single-pack bitmap. Since objects are added to the end of the\n+incremental MIDX's pseudo-pack order (see: above), it is possible to\n+extend a bitmap when appending to the end of a MIDX chain.\n+\n+(Note: it is possible likewise to compress a contiguous sequence of MIDX\n+incremental layers, and their `*.bitmap`(s) into a single layer and\n+`*.bitmap`, but this is not yet implemented.)\n+\n+The object positions used are global within the pseudo-pack order, so\n+subsequent layers will have, for example, `m->num_objects_in_base`\n+number of `0` bits in each of their four type bitmaps. This follows from\n+the fact that we only write type bitmap entries for objects present in\n+the layer immediately corresponding to the bitmap).\n+\n+Note also that only the bitmap pertaining to the most recent layer in an\n+incremental MIDX chain is used to store reachability information about\n+the interesting and uninteresting objects in a reachability query.\n+Earlier bitmap layers are only used to look up commit and pseudo-merge\n+bitmaps from that layer, as well as the type-level bitmaps for objects\n+in that layer.\n+\n+To simplify the implementation, type-level bitmaps are iterated\n+simultaneously, and their results are OR'd together to avoid recursively\n+calling internal bitmap functions.\n+\n Future Work\n -----------\n \n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507650","messageId":"b902513f43697e94976c1baf5598222fe2e3b847.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:22Z","receivedAt":"2024-11-19T22:07:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the reverse index machinery to handle object lookups in an\nincremental MIDX bitmap. These changes are broken out across a few\nfunctions:\n\n  - load_midx_revindex() learns to use the appropriate MIDX filename\n    depending on whether the given 'struct multi_pack_index *' is\n    incremental or not.\n\n  - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n    object position in the MIDX pseudo-pack order, and finds the\n    earliest containing MIDX (similar to midx.c::midx_for_object().\n\n  - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n    number of objects in the base (since 'vb - midx->revindx_data' is\n    relative to the containing MIDX, and pack_pos_to_midx() expects a\n    global position).\n\n    Likewise, this function adjusts its output by adding\n    m->num_objects_in_base to return a global position out through the\n    `*pos` pointer.\n\nTogether, these changes are sufficient to use the multi-pack index's\nreverse index format for incremental multi-pack reachability bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c   | 30 ++++++++++++++++++++----------\n pack-revindex.c | 32 +++++++++++++++++++++++---------\n 2 files changed, 43 insertions(+), 19 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4fa9dfc771a..bba9c6a905a 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n }\n \n+static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n+{\n+\tif (index->midx) {\n+\t\tstruct multi_pack_index *m = index->midx;\n+\t\treturn m->num_objects + m->num_objects_in_base;\n+\t}\n+\treturn index->pack->num_objects;\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\n@@ -925,7 +934,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\t\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -993,7 +1002,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -1498,7 +1507,8 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\tif (!bitmap_get(objects,\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -1677,7 +1687,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tsize_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n+\t\tsize_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(to_filter, pos) &&\n \t\t    !bitmap_get(tips, pos))\n@@ -1703,7 +1713,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < bitmap_num_objects(bitmap_git)) {\n+\tif (pos < bitmap_non_extended_bits(bitmap_git)) {\n \t\tstruct packed_git *pack;\n \t\toff_t ofs;\n \n@@ -1726,7 +1736,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -1878,7 +1888,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n \tuint32_t objects_nr;\n \tsize_t i, pos;\n \n-\tobjects_nr = bitmap_num_objects(bitmap_git);\n+\tobjects_nr = bitmap_non_extended_bits(bitmap_git);\n \tpos = objects_nr / BITS_IN_EWORD;\n \n \tif (pos > result->word_alloc)\n@@ -2403,7 +2413,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(objects,\n-\t\t\t       st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t       st_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcount++;\n \t}\n \n@@ -2802,7 +2812,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n \t\t    \"extension\");\n \n-\tnum_objects = bitmap_num_objects(bitmap_git);\n+\tnum_objects = bitmap_non_extended_bits(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -2945,7 +2955,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \t\tstruct object *obj = eindex->objects[i];\n \n \t\tif (!bitmap_get(result,\n-\t\t\t\tst_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 22d3c234648..ce3f7ae2149 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -383,8 +383,12 @@ int load_midx_revindex(struct multi_pack_index *m)\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t   \"source\", \"rev\");\n \n-\tget_midx_filename_ext(&revindex_name, m->object_dir,\n-\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n+\tif (m->has_chain)\n+\t\tget_split_midx_filename_ext(&revindex_name, m->object_dir,\n+\t\t\t\t\t    get_midx_checksum(m), MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(&revindex_name, m->object_dir,\n+\t\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n \t\t\t\t      m->num_objects,\n@@ -471,11 +475,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \n uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n {\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n \tif (!m->revindex_data)\n \t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n-\tif (m->num_objects <= pos)\n+\tif (m->num_objects + m->num_objects_in_base <= pos)\n \t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n-\treturn get_be32(m->revindex_data + pos);\n+\treturn get_be32(m->revindex_data + pos - m->num_objects_in_base);\n }\n \n struct midx_pack_key {\n@@ -491,7 +499,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n \tconst struct midx_pack_key *key = va;\n \tstruct multi_pack_index *midx = key->midx;\n \n-\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tsize_t pos = (uint32_t*)vb - (const uint32_t *)midx->revindex_data;\n+\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n \toff_t versus_offset;\n \n@@ -529,9 +538,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n {\n \tuint32_t *found;\n \n-\tif (key->pack >= m->num_packs)\n+\tif (key->pack >= m->num_packs + m->num_packs_in_base)\n \t\tBUG(\"MIDX pack lookup out of bounds (%\"PRIu32\" >= %\"PRIu32\")\",\n-\t\t    key->pack, m->num_packs);\n+\t\t    key->pack, m->num_packs + m->num_packs_in_base);\n \t/*\n \t * The preferred pack sorts first, so determine its identifier by\n \t * looking at the first object in pseudo-pack order.\n@@ -551,7 +560,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n \tif (!found)\n \t\treturn -1;\n \n-\t*pos = found - m->revindex_data;\n+\t*pos = (found - m->revindex_data) + m->num_objects_in_base;\n+\n \treturn 0;\n }\n \n@@ -559,9 +569,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n {\n \tstruct midx_pack_key key;\n \n+\twhile (m && at < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, at);\n \tif (!m->revindex_data)\n \t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n-\tif (m->num_objects <= at)\n+\tif (m->num_objects + m->num_objects_in_base <= at)\n \t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n \n \tkey.pack = nth_midxed_pack_int_id(m, at);\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507651","messageId":"5b5d625cbe02560a20c12b7dd20aeda4979017bb.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:26Z","receivedAt":"2024-11-19T22:07:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the pack-bitmap machinery to work with incremental MIDXs by\nadding a new \"base\" field to keep track of the bitmap index associated\nwith the previous MIDX layer.\n\nThe changes in this commit are mostly boilerplate to open the correct\nbitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\nensures that the correct packs and their reverse indexes are loaded\nacross MIDX layers, etc.\n\nWhile we're at it, keep track of a base_nr field to indicate how many\nbitmap layers (including the current bitmap) exist. This will be used in\na future commit to allocate an array of 'struct ewah_bitmap' pointers to\ncollect all of the respective type bitmaps among all layers to\ninitialize a multi-EWAH iterator.\n\nSubsequent commits will teach the functions within the pack-bitmap\nmachinery how to interact with these new fields.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 64 ++++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 51 insertions(+), 13 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex bba9c6a905a..41675a69f68 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -54,6 +54,13 @@ struct bitmap_index {\n \tstruct packed_git *pack;\n \tstruct multi_pack_index *midx;\n \n+\t/*\n+\t * If using a multi-pack index chain, 'base' points to the\n+\t * bitmap index corresponding to this bitmap's midx->base_midx.\n+\t */\n+\tstruct bitmap_index *base;\n+\tuint32_t base_nr;\n+\n \t/* mmapped buffer of the whole bitmap index */\n \tunsigned char *map;\n \tsize_t map_size; /* size of the mmaped buffer */\n@@ -377,8 +384,13 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n char *midx_bitmap_filename(struct multi_pack_index *midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tget_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx),\n-\t\t\t      MIDX_EXT_BITMAP);\n+\tif (midx->has_chain)\n+\t\tget_split_midx_filename_ext(&buf, midx->object_dir,\n+\t\t\t\t\t    get_midx_checksum(midx),\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(&buf, midx->object_dir,\n+\t\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n }\n@@ -397,10 +409,17 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n {\n \tstruct stat st;\n \tchar *bitmap_name = midx_bitmap_filename(midx);\n-\tint fd = git_open(bitmap_name);\n+\tint fd;\n \tuint32_t i, preferred_pack;\n \tstruct packed_git *preferred;\n \n+\tfd = git_open(bitmap_name);\n+\tif (fd < 0 && errno == ENOENT) {\n+\t\tFREE_AND_NULL(bitmap_name);\n+\t\tbitmap_name = midx_bitmap_filename(midx);\n+\t\tfd = git_open(bitmap_name);\n+\t}\n+\n \tif (fd < 0) {\n \t\tif (errno != ENOENT)\n \t\t\twarning_errno(\"cannot open '%s'\", bitmap_name);\n@@ -446,7 +465,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\tfor (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n \t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i)) {\n \t\t\twarning(_(\"could not open pack %s\"),\n \t\t\t\tbitmap_git->midx->pack_names[i]);\n@@ -459,13 +478,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tpreferred = bitmap_git->midx->packs[preferred_pack];\n+\tpreferred = nth_midxed_pack(bitmap_git->midx, preferred_pack);\n \tif (!is_pack_valid(preferred)) {\n \t\twarning(_(\"preferred pack (%s) is invalid\"),\n \t\t\tpreferred->pack_name);\n \t\tgoto cleanup;\n \t}\n \n+\tif (midx->base_midx) {\n+\t\tbitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n+\t\tbitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n+\t} else {\n+\t\tbitmap_git->base_nr = 1;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -516,6 +542,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tbitmap_git->map_pos = 0;\n+\tbitmap_git->base_nr = 1;\n \tclose(fd);\n \n \tif (load_bitmap_header(bitmap_git) < 0) {\n@@ -535,8 +562,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)\n {\n \tif (bitmap_is_midx(bitmap_git)) {\n-\t\tuint32_t i;\n-\t\tint ret;\n+\t\tstruct multi_pack_index *m;\n \n \t\t/*\n \t\t * The multi-pack-index's .rev file is already loaded via\n@@ -545,10 +571,15 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \t\t * But we still need to open the individual pack .rev files,\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tret = load_pack_revindex(r, bitmap_git->midx->packs[i]);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n+\t\tfor (m = bitmap_git->midx; m; m = m->base_midx) {\n+\t\t\tuint32_t i;\n+\t\t\tint ret;\n+\n+\t\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\tret = load_pack_revindex(r, m->packs[i]);\n+\t\t\t\tif (ret)\n+\t\t\t\t\treturn ret;\n+\t\t\t}\n \t\t}\n \t\treturn 0;\n \t}\n@@ -574,6 +605,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n+\tif (bitmap_git->base) {\n+\t\tif (!bitmap_is_midx(bitmap_git))\n+\t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n+\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\t\tgoto failed;\n+\t}\n+\n \treturn 0;\n \n failed:\n@@ -658,10 +696,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n \n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n {\n-\tstruct repository *r = the_repository;\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n+\tif (!open_midx_bitmap_1(bitmap_git, midx))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2875,6 +2912,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\tclose_midx_revindex(b->midx);\n \t}\n \tfree_pseudo_merge_map(&b->pseudo_merges);\n+\tfree_bitmap_index(b->base);\n \tfree(b);\n }\n \n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507652","messageId":"16259667fb4d7534458bb458afd6cefe032c3b6f.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:29Z","receivedAt":"2024-11-19T22:07:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\nEWAH-compressed bitmap corresponding to some given commit object.\n\nTeach this function about incremental MIDX bitmaps by teaching it to\nrecur on earlier bitmap layers when it fails to find a given commit in\nthe current layer.\n\nThe changes to do so are as follows:\n\n  - Avoid initializing hash_pos at its declaration, since\n    bitmap_for_commit() is now a recursive function and may receive a\n    NULL bitmap_index pointer as its first argument.\n\n  - In cases where we would previously return NULL (to indicate that a\n    lookup failed and the given bitmap_index does not contain an entry\n    corresponding to the given commit), recursively call the function on\n    the previous bitmap layer.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 41675a69f68..e3fdcf8a01a 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -946,18 +946,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t      struct commit *commit)\n {\n-\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n-\t\t\t\t\t   commit->object.oid);\n+\tkhiter_t hash_pos;\n+\tif (!bitmap_git)\n+\t\treturn NULL;\n+\n+\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507653","messageId":"b7a45d7eff8e2c4e6c34b053c682b23a730f4877.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 05/13] pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:32Z","receivedAt":"2024-11-19T22:07:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since we may ask for a pack_id that is in an earlier MIDX layer relative\nto the one corresponding to our bitmap, use nth_midxed_pack() instead of\naccessing the ->packs array directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e3fdcf8a01a..c2c824347a6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1631,7 +1631,7 @@ static void show_objects_for_type(\n \t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n \n \t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n-\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\t} else {\n \t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n \t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507654","messageId":"c8401fa0fbdbf1fe1422cb3105aab2cb8058f331.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:35Z","receivedAt":"2024-11-19T22:07:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits in the first phase of\nincremental MIDXs, enumerate not just the packs in the current\nincremental MIDX layer, but previous ones as well.\n\nLikewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\nsingle pack from a MIDX, use the oldest layer's preferred pack as it is\nlikely to contain the most amount of reusable sections.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c2c824347a6..1dddb242434 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2323,7 +2323,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tmulti_pack_reuse = 0;\n \n \tif (multi_pack_reuse) {\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\t\tstruct bitmapped_pack pack;\n \t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n@@ -2347,14 +2348,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tuint32_t pack_int_id;\n \n \t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n \t\t\tuint32_t preferred_pack_pos;\n \n-\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n+\t\t\twhile (m->base_midx)\n+\t\t\t\tm = m->base_midx;\n+\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n \t\t\t\treturn;\n \t\t\t}\n \n-\t\t\tpack = bitmap_git->midx->packs[preferred_pack_pos];\n+\t\t\tpack = nth_midxed_pack(m, preferred_pack_pos);\n \t\t\tpack_int_id = preferred_pack_pos;\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507655","messageId":"17ab23dd76dce076275873e96991acd2f2b2a994.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:38Z","receivedAt":"2024-11-19T22:07:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement support for the special `--test-bitmap` mode of `git rev-list`\nwhen using incremental MIDXs.\n\nThe bitmap_test_data structure is extended to contain a \"base\" pointer\nthat mirrors the structure of the bitmap chain that it is being used to\ntest.\n\nWhen we find a commit to test, we first chase down the ->base pointer to\nfind the appropriate bitmap_test_data for the bitmap layer that the\ngiven commit is contained within, and then perform the test on that\nbitmap.\n\nIn order to implement this, light modifications are made to\nbitmap_for_commit() to reimplement it in terms of a new function,\nfind_bitmap_for_commit(), which fills out a pointer which indicates the\nbitmap layer which contains the given commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 105 ++++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 84 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1dddb242434..02864a0e1f7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -943,8 +943,9 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n \treturn NULL;\n }\n \n-struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n-\t\t\t\t      struct commit *commit)\n+static struct ewah_bitmap *find_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t  struct commit *commit,\n+\t\t\t\t\t\t  struct bitmap_index **found)\n {\n \tkhiter_t hash_pos;\n \tif (!bitmap_git)\n@@ -954,18 +955,30 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n+\t\tif (found)\n+\t\t\t*found = bitmap_git;\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n+\tif (found)\n+\t\t*found = bitmap_git;\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t      struct commit *commit)\n+{\n+\treturn find_bitmap_for_commit(bitmap_git, commit, NULL);\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n@@ -2493,6 +2506,8 @@ struct bitmap_test_data {\n \tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n+\n+\tstruct bitmap_test_data *base_tdata;\n };\n \n static void test_bitmap_type(struct bitmap_test_data *tdata,\n@@ -2501,6 +2516,11 @@ static void test_bitmap_type(struct bitmap_test_data *tdata,\n \tenum object_type bitmap_type = OBJ_NONE;\n \tint bitmaps_nr = 0;\n \n+\tif (bitmap_is_midx(tdata->bitmap_git)) {\n+\t\twhile (pos < tdata->bitmap_git->midx->num_objects_in_base)\n+\t\t\ttdata = tdata->base_tdata;\n+\t}\n+\n \tif (bitmap_get(tdata->commits, pos)) {\n \t\tbitmap_type = OBJ_COMMIT;\n \t\tbitmaps_nr++;\n@@ -2564,13 +2584,57 @@ static void test_show_commit(struct commit *commit, void *data)\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n+static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t total = 0;\n+\tdo {\n+\t\ttotal = st_add(total, bitmap_git->entry_count);\n+\t\tbitmap_git = bitmap_git->base;\n+\t} while (bitmap_git);\n+\n+\treturn total;\n+}\n+\n+static void prepare_bitmap_test_data(struct bitmap_test_data *tdata,\n+\t\t\t\t     struct bitmap_index *bitmap_git)\n+{\n+\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n+\n+\ttdata->bitmap_git = bitmap_git;\n+\ttdata->base = bitmap_new();\n+\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n+\n+\tif (bitmap_git->base) {\n+\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n+\t\tprepare_bitmap_test_data(tdata->base_tdata, bitmap_git->base);\n+\t}\n+}\n+\n+static void free_bitmap_test_data(struct bitmap_test_data *tdata)\n+{\n+\tif (!tdata)\n+\t\treturn;\n+\n+\tfree_bitmap_test_data(tdata->base_tdata);\n+\tfree(tdata->base_tdata);\n+\n+\tbitmap_free(tdata->base);\n+\tbitmap_free(tdata->commits);\n+\tbitmap_free(tdata->trees);\n+\tbitmap_free(tdata->blobs);\n+\tbitmap_free(tdata->tags);\n+}\n+\n void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n-\tstruct bitmap_index *bitmap_git;\n+\tstruct bitmap_index *bitmap_git, *found;\n \tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n@@ -2579,17 +2643,26 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n \t\tbitmap_git->version,\n \t\tbitmap_git->entry_count,\n-\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n+\t\tbitmap_total_entry_count(bitmap_git));\n \n \troot = revs->pending.objects[0].item;\n-\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n+\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n \n \tif (bm) {\n \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n-\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n+\t\t\toid_to_hex(&root->oid),\n+\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n+\n+\t\tif (bitmap_is_midx(found))\n+\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n+\t\t\t\t   hash_to_hex(get_midx_checksum(found->midx)));\n+\t\telse\n+\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n+\t\t\t\t   hash_to_hex(found->pack->hash));\n \n \t\tresult = ewah_to_bitmap(bm);\n \t}\n@@ -2606,14 +2679,8 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n-\ttdata.bitmap_git = bitmap_git;\n-\ttdata.base = bitmap_new();\n-\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n-\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n-\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n-\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n+\tprepare_bitmap_test_data(&tdata, bitmap_git);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n-\ttdata.seen = 0;\n \n \ttraverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);\n \n@@ -2625,11 +2692,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tdie(_(\"mismatch in bitmap results\"));\n \n \tbitmap_free(result);\n-\tbitmap_free(tdata.base);\n-\tbitmap_free(tdata.commits);\n-\tbitmap_free(tdata.trees);\n-\tbitmap_free(tdata.blobs);\n-\tbitmap_free(tdata.tags);\n+\tfree_bitmap_test_data(&tdata);\n \tfree_bitmap_index(bitmap_git);\n }\n \n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507656","messageId":"75d170ce07832a7f31b85f293044b4b32257162a.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 08/13] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:41Z","receivedAt":"2024-11-19T22:07:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, use nth_midxed_pack() instead\nof accessing the MIDX's ->packs array directly to support incremental\nMIDXs.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 02864a0e1f7..b48d6b144d8 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1774,7 +1774,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n \t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n \n-\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n@@ -3025,7 +3025,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \n \t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n-\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tstruct packed_git *pack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \n \t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n \t\t\t\t\tstruct object_id oid;\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507657","messageId":"0b4fcfcecb6043534424c3c7ffc80a63dfe63f3c.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 09/13] pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:44Z","receivedAt":"2024-11-19T22:07:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for using pseudo-merges with incremental MIDX bitmaps by\nattempting to apply pseudo-merges from each layer when encountering a\ngiven commit during a walk.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex b48d6b144d8..570f6dbdad6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1087,10 +1087,15 @@ static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git\n \t\t\t\t\t\t struct commit *commit,\n \t\t\t\t\t\t uint32_t commit_pos)\n {\n-\tint ret;\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tint ret = 0;\n \n-\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t     result, commit, commit_pos);\n+\twhile (curr) {\n+\t\tret += apply_pseudo_merges_for_commit(&curr->pseudo_merges,\n+\t\t\t\t\t\t      result, commit,\n+\t\t\t\t\t\t      commit_pos);\n+\t\tcurr = curr->base;\n+\t}\n \n \tif (ret)\n \t\tpseudo_merges_satisfied_nr += ret;\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507658","messageId":"e1b5f6181e38a526cee4bd1f08a397d88b776661.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:47Z","receivedAt":"2024-11-19T22:07:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"While individual bitmap layers store different commit, type-level, and\npseudo-merge bitmaps, only the top-most layer is used to compute\nreachability traversals.\n\nMany functions which implement the aforementioned traversal rely on\nenumerating the results according to the type-level bitmaps, and so\nwould benefit from a conceptual type-level bitmap that spans multiple\nlayers.\n\nImplement `struct ewah_or_iterator` which is capable of enumerating\nmultiple EWAH bitmaps at once, and OR-ing the results together. When\ninitialized with, for example, all of the commit type bitmaps from each\nlayer, callers can pretend as if they are enumerating a large type-level\nbitmap which contains the commits from *all* bitmap layers.\n\nThere are a couple of alternative approaches which were considered:\n\n  - Decompress each EWAH bitmap and OR them together, enumerating a\n    single (non-EWAH) bitmap. This would work, but has the disadvantage\n    of decompressing a potentially large bitmap, which may not be\n    necessary if the caller does not wish to read all of it.\n\n  - Recursively call bitmap internal functions, reusing the \"result\" and\n    \"haves\" bitmap from the top-most layer. This approach resembles the\n    original implementation of this feature, but is inefficient in that\n    it both (a) requires significant refactoring to implement, and (b)\n    enumerates large sections of later bitmaps which are all zeros (as\n    they pertain to objects in earlier layers).\n\n    (b) is not so bad in and of itself, but can cause significant\n    slow-downs when combined with expensive loop bodies.\n\nThis approach (enumerating an OR'd together version of all of the\ntype-level bitmaps from each layer) produces a significantly more\nstraightforward implementation with significantly less refactoring\nrequired in order to make it work.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ewah/ewah_bitmap.c | 33 +++++++++++++++++++++++++++++++++\n ewah/ewok.h        | 12 ++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex 8785cbc54a8..b3a7ada0714 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -372,6 +372,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n \t\tread_new_rlw(it);\n }\n \n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr)\n+{\n+\tsize_t i;\n+\n+\tmemset(it, 0, sizeof(*it));\n+\n+\tALLOC_ARRAY(it->its, nr);\n+\tfor (i = 0; i < nr; i++)\n+\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n+}\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n+{\n+\teword_t buf, out = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tfor (i = 0; i < it->nr; i++)\n+\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n+\t\t\tout |= buf;\n+\t\t\tret = 1;\n+\t\t}\n+\n+\t*next = out;\n+\treturn ret;\n+}\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it)\n+{\n+\tfree(it->its);\n+}\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 5e357e24933..4b70641045e 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -148,6 +148,18 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent);\n  */\n int ewah_iterator_next(eword_t *next, struct ewah_iterator *it);\n \n+struct ewah_or_iterator {\n+\tstruct ewah_iterator *its;\n+\tsize_t nr;\n+};\n+\n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr);\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it);\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507659","messageId":"9ab8fb472f48f42f7e0eebc6f0f986c6c74970e9.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:50Z","receivedAt":"2024-11-19T22:07:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for reading the type-level bitmaps from previous bitmap layers\nby maintaining an array for each type, where each element in that type's\narray corresponds to one layer's bitmap for that type.\n\nThese fields will be used in a later commit to instantiate the 'struct\newah_or_iterator' for each type.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 51 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 570f6dbdad6..348488e2d9e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -78,6 +78,24 @@ struct bitmap_index {\n \tstruct ewah_bitmap *blobs;\n \tstruct ewah_bitmap *tags;\n \n+\t/*\n+\t * Type index arrays when this bitmap is associated with an\n+\t * incremental multi-pack index chain.\n+\t *\n+\t * If n is the number of unique layers in the MIDX chain, then\n+\t * commits_all[n-1] is this structs 'commits' field,\n+\t * commits_all[n-2] is the commits field of this bitmap's\n+\t * 'base', and so on.\n+\t *\n+\t * When either associated either with a non-incremental MIDX, or\n+\t * a single packfile, these arrays each contain a single\n+\t * element.\n+\t */\n+\tstruct ewah_bitmap **commits_all;\n+\tstruct ewah_bitmap **trees_all;\n+\tstruct ewah_bitmap **blobs_all;\n+\tstruct ewah_bitmap **tags_all;\n+\n \t/* Map from object ID -> `stored_bitmap` for all the bitmapped commits */\n \tkh_oid_map_t *bitmaps;\n \n@@ -586,7 +604,29 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \treturn load_pack_revindex(r, bitmap_git->pack);\n }\n \n-static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n+static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n+{\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tsize_t i = bitmap_git->base_nr - 1;\n+\n+\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr);\n+\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr);\n+\n+\twhile (curr) {\n+\t\tbitmap_git->commits_all[i] = curr->commits;\n+\t\tbitmap_git->trees_all[i] = curr->trees;\n+\t\tbitmap_git->blobs_all[i] = curr->blobs;\n+\t\tbitmap_git->tags_all[i] = curr->tags;\n+\n+\t\tcurr = curr->base;\n+\t\ti -= 1;\n+\t}\n+}\n+\n+static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git,\n+\t\t       int recursing)\n {\n \tassert(bitmap_git->map);\n \n@@ -608,10 +648,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (bitmap_git->base) {\n \t\tif (!bitmap_is_midx(bitmap_git))\n \t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n-\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\tif (load_bitmap(r, bitmap_git->base, 1) < 0)\n \t\t\tgoto failed;\n \t}\n \n+\tif (!recursing)\n+\t\tload_all_type_bitmaps(bitmap_git);\n+\n \treturn 0;\n \n failed:\n@@ -687,7 +730,7 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git, 0))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2042,7 +2085,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (load_bitmap(revs->repo, bitmap_git, 0) < 0)\n \t\tgoto cleanup;\n \n \tif (!use_boundary_traversal)\n@@ -2961,6 +3004,10 @@ void free_bitmap_index(struct bitmap_index *b)\n \tewah_pool_free(b->trees);\n \tewah_pool_free(b->blobs);\n \tewah_pool_free(b->tags);\n+\tfree(b->commits_all);\n+\tfree(b->trees_all);\n+\tfree(b->blobs_all);\n+\tfree(b->tags_all);\n \tif (b->bitmaps) {\n \t\tstruct stored_bitmap *sb;\n \t\tkh_foreach_value(b->bitmaps, sb, {\n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507660","messageId":"87cb011e7fc283ef34f4554122fb901c1cd87294.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:53Z","receivedAt":"2024-11-19T22:07:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have initialized arrays for each bitmap layer's type bitmaps\nin the previous commit, adjust existing callers to use them in\npreparation for multi-layered bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 42 +++++++++++++++++++++++++++---------------\n 1 file changed, 27 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 348488e2d9e..83696d834f6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1622,25 +1622,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t}\n }\n \n-static void init_type_iterator(struct ewah_iterator *it,\n+static void init_type_iterator(struct ewah_or_iterator *it,\n \t\t\t       struct bitmap_index *bitmap_git,\n \t\t\t       enum object_type type)\n {\n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tewah_iterator_init(it, bitmap_git->commits);\n+\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_TREE:\n-\t\tewah_iterator_init(it, bitmap_git->trees);\n+\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_BLOB:\n-\t\tewah_iterator_init(it, bitmap_git->blobs);\n+\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tcase OBJ_TAG:\n-\t\tewah_iterator_init(it, bitmap_git->tags);\n+\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n+\t\t\t\t      bitmap_git->base_nr);\n \t\tbreak;\n \n \tdefault:\n@@ -1657,7 +1661,7 @@ static void show_objects_for_type(\n \tsize_t i = 0;\n \tuint32_t offset;\n \n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tstruct bitmap *objects = bitmap_git->result;\n@@ -1665,7 +1669,7 @@ static void show_objects_for_type(\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = objects->words[i] & filter;\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n \n@@ -1707,6 +1711,8 @@ static void show_objects_for_type(\n \t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n+\n+\tewah_or_iterator_free(&it);\n }\n \n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n@@ -1758,7 +1764,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n@@ -1775,7 +1781,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * packfile.\n \t */\n \tfor (i = 0, init_type_iterator(&it, bitmap_git, type);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\tif (i < tips->word_alloc)\n \t\t\tmask &= ~tips->words[i];\n@@ -1795,6 +1801,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -1852,14 +1859,14 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n \ttips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB);\n \n \tfor (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\teword_t word = to_filter->words[i] & mask;\n \t\tunsigned offset;\n@@ -1887,6 +1894,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -2506,12 +2514,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \n \tuint32_t i = 0, count = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tinit_type_iterator(&it, bitmap_git, type);\n \n-\twhile (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) {\n+\twhile (i < objects->word_alloc && ewah_or_iterator_next(&filter, &it)) {\n \t\teword_t word = objects->words[i++] & filter;\n \t\tcount += ewah_bit_popcount64(word);\n \t}\n@@ -2523,6 +2531,8 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \t\t\tcount++;\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn count;\n }\n \n@@ -3051,13 +3061,13 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *result = bitmap_git->result;\n \toff_t total = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \tsize_t i;\n \n \tinit_type_iterator(&it, bitmap_git, object_type);\n \tfor (i = 0; i < result->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = result->words[i] & filter;\n \t\tsize_t base = (i * BITS_IN_EWORD);\n \t\tunsigned offset;\n@@ -3098,6 +3108,8 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn total;\n }\n \n-- \n2.47.0.301.g77ddd1170f9\n\n"},{"id":"507661","messageId":"77ddd1170f9178849b5dbfd9cd16a14ae96cfa87.1732054032.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"[PATCH v3 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-19T22:07:56Z","receivedAt":"2024-11-19T22:07:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap machinery has learned how to read and interact\nwith an incremental MIDX bitmap, teach the pack-bitmap-write.c machinery\n(and relevant callers from within the MIDX machinery) to write such\nbitmaps.\n\nThe details for doing so are mostly straightforward. The main changes\nare as follows:\n\n  - find_object_pos() now makes use of an extra MIDX parameter which is\n    used to locate the bit positions of objects which are from previous\n    layers (and thus do not exist in the current layer's pack_order\n    field).\n\n    (Note also that the pack_order field is moved into struct\n    write_midx_context to further simplify the callers for\n    write_midx_bitmap()).\n\n  - bitmap_writer_build_type_index() first determines how many objects\n    precede the current bitmap layer and offsets the bits it sets in\n    each respective type-level bitmap by that amount so they can be OR'd\n    together.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c                  |  3 +-\n midx-write.c                            | 49 ++++++++++-----\n pack-bitmap-write.c                     | 65 ++++++++++++++-----\n pack-bitmap.h                           |  4 +-\n t/t5334-incremental-multi-pack-index.sh | 84 +++++++++++++++++++++++++\n 5 files changed, 171 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 08007142671..09d9ef62055 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1370,7 +1370,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_init(&bitmap_writer,\n-\t\t\t\t\t\t   the_repository, &to_pack);\n+\t\t\t\t\t\t   the_repository, &to_pack,\n+\t\t\t\t\t\t   NULL);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t\t\t       written_list);\ndiff --git a/midx-write.c b/midx-write.c\nindex b3a5f6c5166..6f7a8e045fd 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -645,15 +645,21 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \treturn pack_order;\n }\n \n-static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n-\t\t\t\t     struct write_midx_context *ctx)\n+static void write_midx_reverse_index(struct write_midx_context *ctx,\n+\t\t\t\t     const char *object_dir,\n+\t\t\t\t     unsigned char *midx_hash)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tchar *tmp_file;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n \n-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(&buf, object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(&buf, object_dir, midx_hash,\n+\t\t\t\t      MIDX_EXT_REV);\n \n \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n \t\t\t\t\tmidx_hash, WRITE_REV);\n@@ -827,20 +833,26 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \treturn cb.commits;\n }\n \n-static int write_midx_bitmap(const char *midx_name,\n+static int write_midx_bitmap(struct write_midx_context *ctx,\n+\t\t\t     const char *object_dir,\n \t\t\t     const unsigned char *midx_hash,\n \t\t\t     struct packing_data *pdata,\n \t\t\t     struct commit **commits,\n \t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n \t\t\t     unsigned flags)\n {\n \tint ret, i;\n \tuint16_t options = 0;\n \tstruct bitmap_writer writer;\n \tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t\thash_to_hex(midx_hash));\n+\tstruct strbuf bitmap_name = STRBUF_INIT;\n+\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n+\t\t\t\t      MIDX_EXT_BITMAP);\n \n \ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n \n@@ -859,7 +871,8 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer, the_repository, pdata);\n+\tbitmap_writer_init(&writer, the_repository, pdata,\n+\t\t\t   ctx->incremental ? ctx->base_midx : NULL);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, index);\n \n@@ -877,7 +890,7 @@ static int write_midx_bitmap(const char *midx_name,\n \t * bitmap_writer_finish().\n \t */\n \tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\t\tindex[ctx->pack_order[i]] = &pdata->objects[i].idx;\n \n \tbitmap_writer_select_commits(&writer, commits, commits_nr);\n \tret = bitmap_writer_build(&writer);\n@@ -885,11 +898,11 @@ static int write_midx_bitmap(const char *midx_name,\n \t\tgoto cleanup;\n \n \tbitmap_writer_set_checksum(&writer, midx_hash);\n-\tbitmap_writer_finish(&writer, index, bitmap_name, options);\n+\tbitmap_writer_finish(&writer, index, bitmap_name.buf, options);\n \n cleanup:\n \tfree(index);\n-\tfree(bitmap_name);\n+\tstrbuf_release(&bitmap_name);\n \tbitmap_writer_free(&writer);\n \n \ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n@@ -1073,8 +1086,6 @@ static int write_midx_internal(const char *object_dir,\n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n \n \tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n-\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n-\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1116,6 +1127,12 @@ static int write_midx_internal(const char *object_dir,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n+\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n+\t\t\t\t      hash_to_hex(get_midx_checksum(m)));\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n@@ -1382,7 +1399,7 @@ static int write_midx_internal(const char *object_dir,\n \n \tif (flags & MIDX_WRITE_REV_INDEX &&\n \t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n-\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n+\t\twrite_midx_reverse_index(&ctx, object_dir, midx_hash);\n \n \tif (flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n@@ -1405,8 +1422,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\tif (write_midx_bitmap(&ctx, object_dir,\n+\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n \t\t\t\t      flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 49758e2525f..1fbebe84479 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -25,6 +25,8 @@\n #include \"alloc.h\"\n #include \"refs.h\"\n #include \"strmap.h\"\n+#include \"midx.h\"\n+#include \"pack-revindex.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -42,7 +44,8 @@ static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer\n }\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata)\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n \tif (writer->bitmaps)\n@@ -50,6 +53,7 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n \twriter->bitmaps = kh_init_oid_map();\n \twriter->pseudo_merge_commits = kh_init_oid_map();\n \twriter->to_pack = pdata;\n+\twriter->midx = midx;\n \n \tstring_list_init_dup(&writer->pseudo_merge_groups);\n \n@@ -112,6 +116,11 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t    struct pack_idx_entry **index)\n {\n \tuint32_t i;\n+\tuint32_t base_objects = 0;\n+\n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n \n \twriter->commits = ewah_new();\n \twriter->trees = ewah_new();\n@@ -141,19 +150,19 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \n \t\tswitch (real_type) {\n \t\tcase OBJ_COMMIT:\n-\t\t\tewah_set(writer->commits, i);\n+\t\t\tewah_set(writer->commits, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TREE:\n-\t\t\tewah_set(writer->trees, i);\n+\t\t\tewah_set(writer->trees, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_BLOB:\n-\t\t\tewah_set(writer->blobs, i);\n+\t\t\tewah_set(writer->blobs, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TAG:\n-\t\t\tewah_set(writer->tags, i);\n+\t\t\tewah_set(writer->tags, i + base_objects);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -206,19 +215,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n static uint32_t find_object_pos(struct bitmap_writer *writer,\n \t\t\t\tconst struct object_id *oid, int *found)\n {\n-\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n+\tstruct object_entry *entry;\n+\n+\tentry = packlist_find(writer->to_pack, oid);\n+\tif (entry) {\n+\t\tuint32_t base_objects = 0;\n+\t\tif (writer->midx)\n+\t\t\tbase_objects = writer->midx->num_objects +\n+\t\t\t\twriter->midx->num_objects_in_base;\n+\n+\t\tif (found)\n+\t\t\t*found = 1;\n+\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n+\t} else if (writer->midx) {\n+\t\tuint32_t at, pos;\n+\n+\t\tif (!bsearch_midx(oid, writer->midx, &at))\n+\t\t\tgoto missing;\n+\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n+\t\t\tgoto missing;\n \n-\tif (!entry) {\n \t\tif (found)\n-\t\t\t*found = 0;\n-\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n-\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n-\t\treturn 0;\n+\t\t\t*found = 1;\n+\t\treturn pos;\n \t}\n \n+missing:\n \tif (found)\n-\t\t*found = 1;\n-\treturn oe_in_pack_pos(writer->to_pack, entry);\n+\t\t*found = 0;\n+\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\treturn 0;\n }\n \n static void compute_xor_offsets(struct bitmap_writer *writer)\n@@ -585,7 +612,7 @@ int bitmap_writer_build(struct bitmap_writer *writer)\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n-\tuint32_t *mapping;\n+\tuint32_t *mapping = NULL;\n \tint closed = 1; /* until proven otherwise */\n \n \tif (writer->show_progress)\n@@ -1018,7 +1045,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \toff_t *offsets = NULL;\n-\tuint32_t i;\n+\tuint32_t i, base_objects;\n \n \tstruct bitmap_disk_header header;\n \n@@ -1044,6 +1071,12 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, writer->to_pack->nr_objects);\n \n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n+\telse\n+\t\tbase_objects = 0;\n+\n \tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n@@ -1052,7 +1085,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \t\tif (commit_pos < 0)\n \t\t\tBUG(_(\"trying to write commit not in index\"));\n-\t\tstored->commit_pos = commit_pos;\n+\t\tstored->commit_pos = commit_pos + base_objects;\n \t}\n \n \twrite_selected_commits_v1(writer, f, offsets);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex d7f4b8b8e95..dd0951088f6 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -111,6 +111,7 @@ struct bitmap_writer {\n \n \tkh_oid_map_t *bitmaps;\n \tstruct packing_data *to_pack;\n+\tstruct multi_pack_index *midx; /* if appending to a MIDX chain */\n \n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n@@ -125,7 +126,8 @@ struct bitmap_writer {\n };\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata);\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 471994c4bc8..3aac7ccdfe2 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -45,4 +45,88 @@ test_expect_success 'convert incremental to non-incremental' '\n \n compare_results_with_midx 'non-incremental MIDX conversion'\n \n+write_midx_layer () {\n+\tn=1\n+\tif test -f $midx_chain\n+\tthen\n+\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n+\tfi\n+\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit $n.$i &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit multi-pack-index write --bitmap --incremental\n+}\n+\n+test_expect_success 'write initial MIDX layer' '\n+\tgit repack -ad &&\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'read bitmap from first MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'write another MIDX layer' '\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'midx verify with multiple layers' '\n+\tgit multi-pack-index verify\n+'\n+\n+test_expect_success 'read bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 2.2\n+'\n+\n+test_expect_success 'read earlier bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'show object from first pack' '\n+\tgit cat-file -p 1.1\n+'\n+\n+test_expect_success 'show object from second pack' '\n+\tgit cat-file -p 2.2\n+'\n+\n+for reuse in false single multi\n+do\n+\ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n+\t\trm -fr clone.git &&\n+\n+\t\tgit config pack.allowPackReuse $reuse &&\n+\t\tgit clone --no-local --bare . clone.git\n+\t'\n+done\n+\n+test_expect_success 'relink existing MIDX layer' '\n+\trm -fr \"$midxdir\" &&\n+\n+\tGIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n+\n+\tmidx_hash=\"$(test-tool read-midx --checksum $objdir)\" &&\n+\n+\ttest_path_is_file \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_commit another &&\n+\tgit repack -d &&\n+\tgit multi-pack-index write --bitmap --incremental &&\n+\n+\ttest_path_is_missing \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.midx\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.rev\" &&\n+\ttest_line_count = 2 \"$midx_chain\"\n+\n+'\n+\n test_done\n-- \n2.47.0.301.g77ddd1170f9\n"},{"id":"507699","messageId":"xmqqttc2i92c.fsf@gitster.g","threadId":"61958","inReplyTo":"cover.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/13] midx: incremental multi-pack indexes, part two","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-20T08:49:31Z","receivedAt":"2024-11-20T08:49:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> == Changes since last time\n>\n> This round fixes a small issue when writing legacy \".rev\" files outside\n> of the MIDX in '--incremental' mode.\n>\n> The rest of the series is unchanged, and re-submitted to solicit review\n> now that I have more time to focus on this series.\n>\n> == Original cover letter\n>\n> This series is based on 'master', with an additional merge between\n> tb/incremental-midx-part-1[1] and my newer series to fix a handful of\n> bugs related to pseudo-merge bitmaps[2].\n\nBoth of these prerequisite topics were from August, so we do not\nhave to worry about reconstructing the base anymore ;-)  As I do not\nhave any trace of this topic in my tree anymore (except that I know\nan earlier round that ended with \"fixup! midex: implement writing\"\nexisted in the past), we could queue this on 'maint' afresh, I\nguess?\n\nWhen merged to 'seen', pack-bitmap.c has conflicts with other topics\nin flight and what is annoying is the lines involved in the\nconflicts are rather on the overly long side.\n\nI think I resolved them correctly, but we may want to correct these\noverly long lines if a new iteration is needed in the future.\n\nThanks.\n"},{"id":"513259","messageId":"Z8GJYMbXMZqI5fUL@pks.im","threadId":"61958","inReplyTo":"caed2c6ec3483f028f59777bba40480e2661ca80.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:04Z","receivedAt":"2025-02-28T10:01:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:19PM -0500, Taylor Blau wrote:\n> diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\n> index cc063b30bea..a063262c360 100644\n> --- a/Documentation/technical/multi-pack-index.txt\n> +++ b/Documentation/technical/multi-pack-index.txt\n> @@ -164,6 +164,70 @@ objects_nr($H2) + objects_nr($H1) + i\n>  (in the C implementation, this is often computed as `i +\n>  m->num_objects_in_base`).\n>  \n> +=== Pseudo-pack order for incremental MIDXs\n> +\n> +The original implementation of multi-pack reachability bitmaps defined\n> +the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n> +titled \"multi-pack-index reverse indexes\") roughly as follows:\n> +\n> +____\n> +In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n> +objects in packs stored by the MIDX, laid out in pack order, and the\n> +packs arranged in MIDX order (with the preferred pack coming first).\n> +____\n> +\n> +In the incremental MIDX design, we extend this definition to include\n> +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> +for incremental MIDXs is determined by concatenating the pseudo-pack\n> +ordering for each layer of the MIDX chain in order. Formally two objects\n> +`o1` and `o2` are compared as follows:\n> +\n> +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> +  `o1` is considered less than `o2`.\n\nJust as a refresher for myself: what is the consequence of an object\n`o1` sorting earlier than `o2`? In the case where those refer to\ndifferent objects it is only used to establish the pseudo-pack order so\nthat we know how to interpret the bitmaps. But in the case where those\ntwo objects refer to the same underlying object, e.g. because the object\nis contained in two packs, it also impacts which of both objects would\nbe preferred e.g. during a clone, right?\n\n> +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n> +  MIDX layer has no base, then If one of `pack(o1)` and `pack(o2)` is\n\ns/If/if\n\n> +  preferred and the other is not, then the preferred one sorts first. If\n> +  there is a base layer (i.e. the MIDX layer is not the first layer in\n> +  the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n> +  pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n> +  appears earlier, than the opposite is true.\n\nAnother question for my own understanding: why is it relevant whether we\nhave a base or not? I would have expected that the case where the\nobjects appear in two different layers is already covered by (1), so\nfrom thereon we only need to care about two objects existing in the same\nlayer.\n\n> +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n> +  same MIDX layer. Sort `o1` and `o2` by their offset within their\n> +  containing packfile.\n> +\n> +=== Reachability bitmaps and incremental MIDXs\n> +\n> +Each layer of an incremental MIDX chain may have its objects (and the\n> +objects from any previous layer in the same MIDX chain) represented in\n> +its own `*.bitmap` file.\n> +\n> +The structure of a `*.bitmap` file belonging to an incremental MIDX\n> +chain is identical to that of a non-incremental MIDX bitmap, or a\n> +classic single-pack bitmap. Since objects are added to the end of the\n> +incremental MIDX's pseudo-pack order (see: above), it is possible to\n> +extend a bitmap when appending to the end of a MIDX chain.\n> +\n> +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n> +incremental layers, and their `*.bitmap`(s) into a single layer and\n> +`*.bitmap`, but this is not yet implemented.)\n\nFair. What do we currently do in this context? Do we just keep on\nappending layer after layer?\n\nPatrick\n"},{"id":"513260","messageId":"Z8GJaFPMQtslEEt3@pks.im","threadId":"61958","inReplyTo":"b902513f43697e94976c1baf5598222fe2e3b847.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:12Z","receivedAt":"2025-02-28T10:01:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:22PM -0500, Taylor Blau wrote:\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 4fa9dfc771a..bba9c6a905a 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n>  \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n>  }\n>  \n> +static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n> +{\n> +\tif (index->midx) {\n> +\t\tstruct multi_pack_index *m = index->midx;\n> +\t\treturn m->num_objects + m->num_objects_in_base;\n> +\t}\n> +\treturn index->pack->num_objects;\n> +}\n> \n>  static uint32_t bitmap_num_objects(struct bitmap_index *index)\n>  {\n>  \tif (index->midx)\n\nOkay, despite counting our own objects, we also need to account for any\nobjects that the MIDX layer that we depend on may refer to. I assume\nthat this is basically recursive, and that the base itself would also\naccount for its next layer, if any.\n\nWhat is interesting to see after this commit is what callsites remain\nfor `bitmap_num_objects()`. Most of them are converted, but some still\nexist:\n\n  - `load_bitmap_header()`, where we use it to determine the size of the\n    hash cache. Makes sense.\n\n  - `pseudo_merge_bitmap_for_commit()`, where we use it to compute the\n    merged bitmap of a specific commit. This one feels weird to me, I\n    would have expected to use `bitmap_non_extended_bits()` here.\n\n  - `filter_bitmap_blob_limit()`, where we seem to filter through the\n    bitmap of the current layer. I _think_ it makes sense to retain.\n\n  - `create_bitmap_mapping()`, which feels like it should be converted?\n\nIt would be nice to document in the commit message why those functions\ndon't need to be converted to help guide the reader a bit.\n\n> diff --git a/pack-revindex.c b/pack-revindex.c\n> index 22d3c234648..ce3f7ae2149 100644\n> --- a/pack-revindex.c\n> +++ b/pack-revindex.c\n> @@ -383,8 +383,12 @@ int load_midx_revindex(struct multi_pack_index *m)\n>  \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n>  \t\t\t   \"source\", \"rev\");\n>  \n> -\tget_midx_filename_ext(&revindex_name, m->object_dir,\n> -\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n> +\tif (m->has_chain)\n> +\t\tget_split_midx_filename_ext(&revindex_name, m->object_dir,\n> +\t\t\t\t\t    get_midx_checksum(m), MIDX_EXT_REV);\n> +\telse\n> +\t\tget_midx_filename_ext(&revindex_name, m->object_dir,\n> +\t\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n>  \n>  \tret = load_revindex_from_disk(revindex_name.buf,\n>  \t\t\t\t      m->num_objects,\n\nHere we teach the reverse index to read indices from MIDX chains.\n\n> @@ -471,11 +475,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n>  \n>  uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n>  {\n> +\twhile (m && pos < m->num_objects_in_base)\n> +\t\tm = m->base_midx;\n> +\tif (!m)\n> +\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n>  \tif (!m->revindex_data)\n>  \t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n> -\tif (m->num_objects <= pos)\n> +\tif (m->num_objects + m->num_objects_in_base <= pos)\n>  \t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n> -\treturn get_be32(m->revindex_data + pos);\n> +\treturn get_be32(m->revindex_data + pos - m->num_objects_in_base);\n>  }\n>  \n>  struct midx_pack_key {\n\nHere we teach the reverse index logic to walk the MIDX layers so that we\nfind the one that is supposed to contain a given position.\n\n> @@ -491,7 +499,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n>  \tconst struct midx_pack_key *key = va;\n>  \tstruct multi_pack_index *midx = key->midx;\n>  \n> -\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n> +\tsize_t pos = (uint32_t*)vb - (const uint32_t *)midx->revindex_data;\n\nMicronit: missing space between `uint32_t` and `*`.\n\n> +\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n>  \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n>  \toff_t versus_offset;\n>  \n\nOkay, the calculation to calculate the position is basically the same,\nbut we now also offset the position by the number of objects in\npreceding layers. Makes sense.\n\nPatrick\n"},{"id":"513261","messageId":"Z8GJbOoY9Z3VMSEM@pks.im","threadId":"61958","inReplyTo":"5b5d625cbe02560a20c12b7dd20aeda4979017bb.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:16Z","receivedAt":"2025-02-28T10:01:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:26PM -0500, Taylor Blau wrote:\n> Prepare the pack-bitmap machinery to work with incremental MIDXs by\n> adding a new \"base\" field to keep track of the bitmap index associated\n> with the previous MIDX layer.\n> \n> The changes in this commit are mostly boilerplate to open the correct\n> bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n\ns/bitmap layers/of &/\n\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index bba9c6a905a..41675a69f68 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -54,6 +54,13 @@ struct bitmap_index {\n>  \tstruct packed_git *pack;\n>  \tstruct multi_pack_index *midx;\n>  \n> +\t/*\n> +\t * If using a multi-pack index chain, 'base' points to the\n> +\t * bitmap index corresponding to this bitmap's midx->base_midx.\n> +\t */\n> +\tstruct bitmap_index *base;\n> +\tuint32_t base_nr;\n> +\n\nIt would be nice to point out that `base_nr` is not 0-indexed, but\n1-indexed, which is rather uncommon. Is there any particular reason why\nyou made it 1-indexed?\n\n> @@ -377,8 +384,13 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n>  char *midx_bitmap_filename(struct multi_pack_index *midx)\n>  {\n>  \tstruct strbuf buf = STRBUF_INIT;\n> -\tget_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx),\n> -\t\t\t      MIDX_EXT_BITMAP);\n> +\tif (midx->has_chain)\n> +\t\tget_split_midx_filename_ext(&buf, midx->object_dir,\n> +\t\t\t\t\t    get_midx_checksum(midx),\n> +\t\t\t\t\t    MIDX_EXT_BITMAP);\n> +\telse\n> +\t\tget_midx_filename_ext(&buf, midx->object_dir,\n> +\t\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n>  \n>  \treturn strbuf_detach(&buf, NULL);\n>  }\n\nOkay, this is mostly the same change as in the preceding commit, but for\nbitmaps instead of reverse indices.\n\n> @@ -397,10 +409,17 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n>  {\n>  \tstruct stat st;\n>  \tchar *bitmap_name = midx_bitmap_filename(midx);\n> -\tint fd = git_open(bitmap_name);\n> +\tint fd;\n>  \tuint32_t i, preferred_pack;\n>  \tstruct packed_git *preferred;\n>  \n> +\tfd = git_open(bitmap_name);\n> +\tif (fd < 0 && errno == ENOENT) {\n> +\t\tFREE_AND_NULL(bitmap_name);\n> +\t\tbitmap_name = midx_bitmap_filename(midx);\n> +\t\tfd = git_open(bitmap_name);\n> +\t}\n> +\n\nWait, this looks weird to me. `bitmap_name` already contains the result\nof `midx_bitmap_filename()`, so you're essentially retrying the exact\nsame operation as before?\n\nPatrick\n"},{"id":"513262","messageId":"Z8GJb4y7yui7rvxZ@pks.im","threadId":"61958","inReplyTo":"16259667fb4d7534458bb458afd6cefe032c3b6f.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:19Z","receivedAt":"2025-02-28T10:01:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:29PM -0500, Taylor Blau wrote:\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 41675a69f68..e3fdcf8a01a 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -946,18 +946,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n>  struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n>  \t\t\t\t      struct commit *commit)\n>  {\n> -\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n> -\t\t\t\t\t   commit->object.oid);\n> +\tkhiter_t hash_pos;\n> +\tif (!bitmap_git)\n> +\t\treturn NULL;\n> +\n> +\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n>  \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n>  \t\tstruct stored_bitmap *bitmap = NULL;\n>  \t\tif (!bitmap_git->table_lookup)\n> -\t\t\treturn NULL;\n> +\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n>  \n>  \t\t/* this is a fairly hot codepath - no trace2_region please */\n>  \t\t/* NEEDSWORK: cache misses aren't recorded */\n>  \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n>  \t\tif (!bitmap)\n> -\t\t\treturn NULL;\n> +\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n>  \t\treturn lookup_stored_bitmap(bitmap);\n>  \t}\n>  \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n\nOne of the things that worries me a bit is that by recursing, we\nessentially are bound in the depth of MIDX layers as we may otherwise\nbust the stack. Not that I expect us to typically have thousands of\nlayers, but if there ever was a bug this may fail in bad ways.\n\nI already asked this for a previous commit, but what is the current\nstate regarding compaction of the layers? Do we need to be worried about\nthis or do we already know to keep things limited in general?\n\nPatrick\n"},{"id":"513263","messageId":"Z8GJc6iDqueVgc67@pks.im","threadId":"61958","inReplyTo":"c8401fa0fbdbf1fe1422cb3105aab2cb8058f331.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:23Z","receivedAt":"2025-02-28T10:01:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:35PM -0500, Taylor Blau wrote:\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index c2c824347a6..1dddb242434 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -2347,14 +2348,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n>  \t\tuint32_t pack_int_id;\n>  \n>  \t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n>  \t\t\tuint32_t preferred_pack_pos;\n>  \n> -\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n> +\t\t\twhile (m->base_midx)\n> +\t\t\t\tm = m->base_midx;\n> +\n> +\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n>  \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n>  \t\t\t\treturn;\n>  \t\t\t}\n\nInstead of completely disabling preferred packs, should we maybe fall\nback to the preferred pack of the next-higher layer?\n\nPatrick\n"},{"id":"513264","messageId":"Z8GJdv5FwDP_k1et@pks.im","threadId":"61958","inReplyTo":"17ab23dd76dce076275873e96991acd2f2b2a994.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:26Z","receivedAt":"2025-02-28T10:01:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:38PM -0500, Taylor Blau wrote:\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 1dddb242434..02864a0e1f7 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -2564,13 +2584,57 @@ static void test_show_commit(struct commit *commit, void *data)\n>  \tdisplay_progress(tdata->prg, ++tdata->seen);\n>  }\n>  \n> +static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n> +{\n> +\tuint32_t total = 0;\n> +\tdo {\n> +\t\ttotal = st_add(total, bitmap_git->entry_count);\n> +\t\tbitmap_git = bitmap_git->base;\n> +\t} while (bitmap_git);\n> +\n> +\treturn total;\n> +}\n> +\n> +static void prepare_bitmap_test_data(struct bitmap_test_data *tdata,\n> +\t\t\t\t     struct bitmap_index *bitmap_git)\n\nNit: according to our style guide this should be called\n`bitmap_test_data_prepare()`.\n\n> +{\n> +\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n> +\n> +\ttdata->bitmap_git = bitmap_git;\n> +\ttdata->base = bitmap_new();\n> +\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n> +\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n> +\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n> +\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n> +\n> +\tif (bitmap_git->base) {\n> +\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n> +\t\tprepare_bitmap_test_data(tdata->base_tdata, bitmap_git->base);\n> +\t}\n> +}\n> +\n> +static void free_bitmap_test_data(struct bitmap_test_data *tdata)\n\nSame nit here, this should be called `bitmap_test_data_free()`. In fact,\nit should be called `bitmap_test_data_release()`, because we don't free\n`tadata` itself.\n\n> @@ -2579,17 +2643,26 @@ void test_bitmap_walk(struct rev_info *revs)\n>  \tif (revs->pending.nr != 1)\n>  \t\tdie(_(\"you must specify exactly one commit to test\"));\n>  \n> -\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n> +\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n>  \t\tbitmap_git->version,\n>  \t\tbitmap_git->entry_count,\n> -\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n> +\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n> +\t\tbitmap_total_entry_count(bitmap_git));\n>  \n>  \troot = revs->pending.objects[0].item;\n> -\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n> +\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n>  \n>  \tif (bm) {\n>  \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n> -\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n> +\t\t\toid_to_hex(&root->oid),\n> +\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n> +\n> +\t\tif (bitmap_is_midx(found))\n> +\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n> +\t\t\t\t   hash_to_hex(get_midx_checksum(found->midx)));\n> +\t\telse\n> +\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n> +\t\t\t\t   hash_to_hex(found->pack->hash));\n>  \n>  \t\tresult = ewah_to_bitmap(bm);\n>  \t}\n\nI'm a bit surprised that this doesn't result in any changes required in\nour tests.\n\nPatrick\n"},{"id":"513265","messageId":"Z8GJemN4K99BVT2J@pks.im","threadId":"61958","inReplyTo":"e1b5f6181e38a526cee4bd1f08a397d88b776661.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:30Z","receivedAt":"2025-02-28T10:01:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:47PM -0500, Taylor Blau wrote:\n> diff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\n> index 8785cbc54a8..b3a7ada0714 100644\n> --- a/ewah/ewah_bitmap.c\n> +++ b/ewah/ewah_bitmap.c\n> @@ -372,6 +372,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n>  \t\tread_new_rlw(it);\n>  }\n>  \n> +void ewah_or_iterator_init(struct ewah_or_iterator *it,\n> +\t\t\t   struct ewah_bitmap **parents, size_t nr)\n> +{\n> +\tsize_t i;\n> +\n> +\tmemset(it, 0, sizeof(*it));\n> +\n> +\tALLOC_ARRAY(it->its, nr);\n> +\tfor (i = 0; i < nr; i++)\n> +\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n> +}\n> +\n> +int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n> +{\n> +\teword_t buf, out = 0;\n> +\tsize_t i;\n> +\tint ret = 0;\n> +\n> +\tfor (i = 0; i < it->nr; i++)\n> +\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n> +\t\t\tout |= buf;\n> +\t\t\tret = 1;\n> +\t\t}\n> +\n> +\t*next = out;\n> +\treturn ret;\n> +}\n\nYup, this looks rather straight-forward: we advance each of our\nsubiterators and or their respective results into the accumulated result\nthat we end up returning to the user.\n\nOne thing that surprised me though is that we don't seem to be able to\ntell whether all of the iterators could be moved on to the next result\nor not. But I guess that makes sense: iterators of lower levels will\ncover less objects and will thus eventually be exhausted before the\niterators on the higher levels.\n\n> +void ewah_or_iterator_free(struct ewah_or_iterator *it)\n> +{\n> +\tfree(it->its);\n> +}\n> +\n\nThis should be called `ewar_or_iterator_release()` because we don't\nreleate `it`, as documented by our style guide.\n\nPatrick\n"},{"id":"513266","messageId":"Z8GJfjhGAbXnA1LW@pks.im","threadId":"61958","inReplyTo":"9ab8fb472f48f42f7e0eebc6f0f986c6c74970e9.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:34Z","receivedAt":"2025-02-28T10:01:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:50PM -0500, Taylor Blau wrote:\n> @@ -586,7 +604,29 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n>  \treturn load_pack_revindex(r, bitmap_git->pack);\n>  }\n>  \n> -static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n> +static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n> +{\n> +\tstruct bitmap_index *curr = bitmap_git;\n> +\tsize_t i = bitmap_git->base_nr - 1;\n> +\n> +\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr);\n> +\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr);\n> +\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr);\n> +\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr);\n> +\n> +\twhile (curr) {\n> +\t\tbitmap_git->commits_all[i] = curr->commits;\n> +\t\tbitmap_git->trees_all[i] = curr->trees;\n> +\t\tbitmap_git->blobs_all[i] = curr->blobs;\n> +\t\tbitmap_git->tags_all[i] = curr->tags;\n> +\n> +\t\tcurr = curr->base;\n> +\t\ti -= 1;\n\nDo we want to `BUG()` in case `i == 0` before this statement?\n\nPatrick\n"},{"id":"513267","messageId":"Z8GJg8TMeE8YLrqA@pks.im","threadId":"61958","inReplyTo":"77ddd1170f9178849b5dbfd9cd16a14ae96cfa87.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:39Z","receivedAt":"2025-02-28T10:01:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:56PM -0500, Taylor Blau wrote:\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 49758e2525f..1fbebe84479 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -25,6 +25,8 @@\n>  #include \"alloc.h\"\n>  #include \"refs.h\"\n>  #include \"strmap.h\"\n> +#include \"midx.h\"\n> +#include \"pack-revindex.h\"\n\nNit: let's keep the headers sorted alphabetically.\n\n> @@ -206,19 +215,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n>  static uint32_t find_object_pos(struct bitmap_writer *writer,\n>  \t\t\t\tconst struct object_id *oid, int *found)\n>  {\n> -\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n> +\tstruct object_entry *entry;\n> +\n> +\tentry = packlist_find(writer->to_pack, oid);\n> +\tif (entry) {\n> +\t\tuint32_t base_objects = 0;\n> +\t\tif (writer->midx)\n> +\t\t\tbase_objects = writer->midx->num_objects +\n> +\t\t\t\twriter->midx->num_objects_in_base;\n> +\n> +\t\tif (found)\n> +\t\t\t*found = 1;\n> +\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n> +\t} else if (writer->midx) {\n> +\t\tuint32_t at, pos;\n> +\n> +\t\tif (!bsearch_midx(oid, writer->midx, &at))\n> +\t\t\tgoto missing;\n> +\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n> +\t\t\tgoto missing;\n>  \n> -\tif (!entry) {\n>  \t\tif (found)\n> -\t\t\t*found = 0;\n> -\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> -\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n> -\t\treturn 0;\n> +\t\t\t*found = 1;\n> +\t\treturn pos;\n>  \t}\n>  \n> +missing:\n>  \tif (found)\n> -\t\t*found = 1;\n> -\treturn oe_in_pack_pos(writer->to_pack, entry);\n> +\t\t*found = 0;\n> +\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> +\t\t\"(object %s is missing)\", oid_to_hex(oid));\n\nIs this warning still accurate? I assume that in the MIDX case we don't\nhave to have full closure in a single packfile, as that would otherwise\nmake the whole thing rather moot.\n\nPatrick\n"},{"id":"513268","messageId":"Z8GJhzdUtH5IB1Pr@pks.im","threadId":"61958","inReplyTo":"87cb011e7fc283ef34f4554122fb901c1cd87294.1732054032.git.me@ttaylorr.com","subject":"Re: [PATCH v3 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:01:43Z","receivedAt":"2025-02-28T10:01:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 19, 2024 at 05:07:53PM -0500, Taylor Blau wrote:\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 348488e2d9e..83696d834f6 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -1622,25 +1622,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n>  \t}\n>  }\n>  \n> -static void init_type_iterator(struct ewah_iterator *it,\n> +static void init_type_iterator(struct ewah_or_iterator *it,\n>  \t\t\t       struct bitmap_index *bitmap_git,\n>  \t\t\t       enum object_type type)\n>  {\n>  \tswitch (type) {\n>  \tcase OBJ_COMMIT:\n> -\t\tewah_iterator_init(it, bitmap_git->commits);\n> +\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n> +\t\t\t\t      bitmap_git->base_nr);\n>  \t\tbreak;\n>  \n>  \tcase OBJ_TREE:\n> -\t\tewah_iterator_init(it, bitmap_git->trees);\n> +\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n> +\t\t\t\t      bitmap_git->base_nr);\n>  \t\tbreak;\n>  \n>  \tcase OBJ_BLOB:\n> -\t\tewah_iterator_init(it, bitmap_git->blobs);\n> +\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n> +\t\t\t\t      bitmap_git->base_nr);\n>  \t\tbreak;\n>  \n>  \tcase OBJ_TAG:\n> -\t\tewah_iterator_init(it, bitmap_git->tags);\n> +\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n> +\t\t\t\t      bitmap_git->base_nr);\n>  \t\tbreak;\n>  \n>  \tdefault:\n\nOne thing I wonder here is whether we want to continue using the\nnon-layered iterator in case we don't have an MIDX. But I guess it makes\nit easier to not do it like that because we have less conditionals, and\noverall the or'd logic shouldn't perform significantly workse anyway. So\nas long as we always initialize `base_nr` and the relevant `*_all`\nfields even in the non-MIDX case we're fine.\n\nPatrick\n"},{"id":"513305","messageId":"Z8JGNQZolfs7fm65@nand.local","threadId":"61958","inReplyTo":"Z8GJYMbXMZqI5fUL@pks.im","subject":"Re: [PATCH v3 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-28T23:26:45Z","receivedAt":"2025-02-28T23:26:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:04AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:19PM -0500, Taylor Blau wrote:\n> > diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\n> > index cc063b30bea..a063262c360 100644\n> > --- a/Documentation/technical/multi-pack-index.txt\n> > +++ b/Documentation/technical/multi-pack-index.txt\n> > @@ -164,6 +164,70 @@ objects_nr($H2) + objects_nr($H1) + i\n> >  (in the C implementation, this is often computed as `i +\n> >  m->num_objects_in_base`).\n> >\n> > +=== Pseudo-pack order for incremental MIDXs\n> > +\n> > +The original implementation of multi-pack reachability bitmaps defined\n> > +the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n> > +titled \"multi-pack-index reverse indexes\") roughly as follows:\n> > +\n> > +____\n> > +In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n> > +objects in packs stored by the MIDX, laid out in pack order, and the\n> > +packs arranged in MIDX order (with the preferred pack coming first).\n> > +____\n> > +\n> > +In the incremental MIDX design, we extend this definition to include\n> > +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> > +for incremental MIDXs is determined by concatenating the pseudo-pack\n> > +ordering for each layer of the MIDX chain in order. Formally two objects\n> > +`o1` and `o2` are compared as follows:\n> > +\n> > +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> > +  `o1` is considered less than `o2`.\n>\n> Just as a refresher for myself: what is the consequence of an object\n> `o1` sorting earlier than `o2`? In the case where those refer to\n> different objects it is only used to establish the pseudo-pack order so\n> that we know how to interpret the bitmaps. But in the case where those\n> two objects refer to the same underlying object, e.g. because the object\n> is contained in two packs, it also impacts which of both objects would\n> be preferred e.g. during a clone, right?\n\nGreat question -- the pseudo-pack order here is how we translate the set\nof objects in a MIDX into their corresponding bit positions in the\nbitmap.\n\nSo if \"o1\" sorts ahead of \"o2\", that means that \"o1\" will appear in an\nearlier bit position than \"o2\". But note that we're talking about\nobjects in a MIDX chain here, comprised of objects from each MIDX'd layer of\nthat chain. So by that point the duplicates have already been filtered\nout, since:\n\n  - The MIDX only stores one copy of an object in any given MIDX, and\n\n  - The incremental MIDX design avoids putting objects from earlier\n    layers in later ones.\n\nI tried to get at this a few lines up with \"[...] a MIDX's pseudo-pack\nis the de-duplicated concatenation of [...]\" to make clear that o1 != o2\nhere. But let me know if you think I should clarify or emphasize that\npoint further.\n\n> > +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n> > +  MIDX layer has no base, then If one of `pack(o1)` and `pack(o2)` is\n>\n> s/If/if\n\nGreat catch, thanks!\n\n> > +  preferred and the other is not, then the preferred one sorts first. If\n> > +  there is a base layer (i.e. the MIDX layer is not the first layer in\n> > +  the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n> > +  pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n> > +  appears earlier, than the opposite is true.\n>\n> Another question for my own understanding: why is it relevant whether we\n> have a base or not? I would have expected that the case where the\n> objects appear in two different layers is already covered by (1), so\n> from thereon we only need to care about two objects existing in the same\n> layer.\n\nGood question. Throughout this design I'm trying to get rid of the\nconcept of a \"preferred\" pack with respect to the MIDX. Before\nmulti-pack reuse existed, the idea behind having a preferred pack was\nthat it was a way to indicate which pack we should prioritize for\npack-reuse.\n\nBut now that we can reuse objects from any pack stored in a MIDX, the\nconcept of a preferred pack doesn't need to exist. It still makes some\nsense (if you want to use single-pack reuse for some reason, etc.) but\nI'm trying to push us away from it.\n\nSo to answer your question of \"why does it matter if there is a base\nor not?\" the reason is that this series treats the preferred pack as a\nproperty of the chain instead of the individual layers. And the\nmention of it here is to differentiate between how we compare packs in\nthe base (favoring the preferred pack) versus subsequent layers\n(reflecting the pack order within that layer).\n\n> > +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n> > +  same MIDX layer. Sort `o1` and `o2` by their offset within their\n> > +  containing packfile.\n> > +\n> > +=== Reachability bitmaps and incremental MIDXs\n> > +\n> > +Each layer of an incremental MIDX chain may have its objects (and the\n> > +objects from any previous layer in the same MIDX chain) represented in\n> > +its own `*.bitmap` file.\n> > +\n> > +The structure of a `*.bitmap` file belonging to an incremental MIDX\n> > +chain is identical to that of a non-incremental MIDX bitmap, or a\n> > +classic single-pack bitmap. Since objects are added to the end of the\n> > +incremental MIDX's pseudo-pack order (see: above), it is possible to\n> > +extend a bitmap when appending to the end of a MIDX chain.\n> > +\n> > +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n> > +incremental layers, and their `*.bitmap`(s) into a single layer and\n> > +`*.bitmap`, but this is not yet implemented.)\n>\n> Fair. What do we currently do in this context? Do we just keep on\n> appending layer after layer?\n\nThat's right. At this point in the project we only know how to append\nlayers and compress the whole chain into a single layer. But\nfundamentally it is possible to compress any contiguous sub-sequence of\nthe chain into a single layer, and part three of this project will do\njust that.\n\nThanks,\nTaylor\n"},{"id":"513306","messageId":"Z8JJG5EX1MUtGvss@nand.local","threadId":"61958","inReplyTo":"Z8GJaFPMQtslEEt3@pks.im","subject":"Re: [PATCH v3 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-28T23:39:07Z","receivedAt":"2025-02-28T23:39:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:12AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:22PM -0500, Taylor Blau wrote:\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index 4fa9dfc771a..bba9c6a905a 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n> >  \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n> >  }\n> >\n> > +static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n> > +{\n> > +\tif (index->midx) {\n> > +\t\tstruct multi_pack_index *m = index->midx;\n> > +\t\treturn m->num_objects + m->num_objects_in_base;\n> > +\t}\n> > +\treturn index->pack->num_objects;\n> > +}\n> >\n> >  static uint32_t bitmap_num_objects(struct bitmap_index *index)\n> >  {\n> >  \tif (index->midx)\n>\n> Okay, despite counting our own objects, we also need to account for any\n> objects that the MIDX layer that we depend on may refer to. I assume\n> that this is basically recursive, and that the base itself would also\n> account for its next layer, if any.\n>\n> What is interesting to see after this commit is what callsites remain\n> for `bitmap_num_objects()`. Most of them are converted, but some still\n> exist:\n>\n>   - `load_bitmap_header()`, where we use it to determine the size of the\n>     hash cache. Makes sense.\n>\n>   - `pseudo_merge_bitmap_for_commit()`, where we use it to compute the\n>     merged bitmap of a specific commit. This one feels weird to me, I\n>     would have expected to use `bitmap_non_extended_bits()` here.\n\nGreat question. The reason is that this function determines a bitmap\nwhich identifies the parents of a given commit, and that bitmap is\ncompared against the set of pseudo-merge commits we know about in a\ngiven layer to determine whether or not we have a matching pseudo-merge.\n\nI left a comment to that effect nearby since this is far from obvious\n(including to me -- I had to take a few minutes to remember how all of\nthis works!).\n\n>   - `filter_bitmap_blob_limit()`, where we seem to filter through the\n>     bitmap of the current layer. I _think_ it makes sense to retain.\n>\n>   - `create_bitmap_mapping()`, which feels like it should be converted?\n\nI *think* that this is OK because we are only remapping one layer at a\ntime, but I'll have to double check. I'd do so now, but I'm trying to\nrespond as much as I can before my week is over ;-).\n\n> It would be nice to document in the commit message why those functions\n> don't need to be converted to help guide the reader a bit.\n\nThe conversions here are case-by-case, so I lean towards documenting any\nnon-obvious ones inline with a brief comment (like the adjustment I made\nabove for pseudo_merge_bitmap_for_commit()).\n\n> > @@ -491,7 +499,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n> >  \tconst struct midx_pack_key *key = va;\n> >  \tstruct multi_pack_index *midx = key->midx;\n> >\n> > -\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n> > +\tsize_t pos = (uint32_t*)vb - (const uint32_t *)midx->revindex_data;\n>\n> Micronit: missing space between `uint32_t` and `*`.\n\nGreat catch, thanks!\n\n> > +\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n> >  \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n> >  \toff_t versus_offset;\n> >\n>\n> Okay, the calculation to calculate the position is basically the same,\n> but we now also offset the position by the number of objects in\n> preceding layers. Makes sense.\n\nExactly.\n\nThanks,\nTaylor\n"},{"id":"513307","messageId":"Z8JLbxBQh7XUpplz@nand.local","threadId":"61958","inReplyTo":"Z8GJbOoY9Z3VMSEM@pks.im","subject":"Re: [PATCH v3 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-28T23:49:03Z","receivedAt":"2025-02-28T23:49:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:16AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:26PM -0500, Taylor Blau wrote:\n> > Prepare the pack-bitmap machinery to work with incremental MIDXs by\n> > adding a new \"base\" field to keep track of the bitmap index associated\n> > with the previous MIDX layer.\n> >\n> > The changes in this commit are mostly boilerplate to open the correct\n> > bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n>\n> s/bitmap layers/of &/\n>\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index bba9c6a905a..41675a69f68 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -54,6 +54,13 @@ struct bitmap_index {\n> >  \tstruct packed_git *pack;\n> >  \tstruct multi_pack_index *midx;\n> >\n> > +\t/*\n> > +\t * If using a multi-pack index chain, 'base' points to the\n> > +\t * bitmap index corresponding to this bitmap's midx->base_midx.\n> > +\t */\n> > +\tstruct bitmap_index *base;\n> > +\tuint32_t base_nr;\n> > +\n>\n> It would be nice to point out that `base_nr` is not 0-indexed, but\n> 1-indexed, which is rather uncommon. Is there any particular reason why\n> you made it 1-indexed?\n\nHah, I have no idea! If I remember correctly, it's because it makes it\n(slightly) more convenient to do:\n\n    ewah_or_iterator_init(it, bitmap_git->commits_all,\n                          bitmap_git->base_nr);\n\n, instead of incrementing 'base_nr' by 1 to determine the number of\nsub-iterators to allocate.\n\nSo I think there are a couple of options here. Short of doing nothing,\nwe could:\n\n 1. Rename 'base_nr' to 'layers_nr' which would make it clearer that the\n    count includes the current layer, thus making it 1-indexed.\n\n 2. Leave 'base_nr' named as-is, but make it 0-indexed, and have callers add\n    1 when they need to know the number of layers.\n\nI prefer the explicitness of (2), which is how I adjusted things\nlocally. But if you prefer (1) or some yet-unknown (3), I'm happy to\nadjust it further!\n\n> > @@ -397,10 +409,17 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> >  {\n> >  \tstruct stat st;\n> >  \tchar *bitmap_name = midx_bitmap_filename(midx);\n> > -\tint fd = git_open(bitmap_name);\n> > +\tint fd;\n> >  \tuint32_t i, preferred_pack;\n> >  \tstruct packed_git *preferred;\n> >\n> > +\tfd = git_open(bitmap_name);\n> > +\tif (fd < 0 && errno == ENOENT) {\n> > +\t\tFREE_AND_NULL(bitmap_name);\n> > +\t\tbitmap_name = midx_bitmap_filename(midx);\n> > +\t\tfd = git_open(bitmap_name);\n> > +\t}\n> > +\n>\n> Wait, this looks weird to me. `bitmap_name` already contains the result\n> of `midx_bitmap_filename()`, so you're essentially retrying the exact\n> same operation as before?\n\nHmm. I have no idea, but you're exactly right. I dropped it from my\nlocal copy.\n\nThanks,\nTaylor\n"},{"id":"513309","messageId":"Z8JQ+1mg/iYe7OCw@nand.local","threadId":"61958","inReplyTo":"Z8GJb4y7yui7rvxZ@pks.im","subject":"Re: [PATCH v3 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:12:43Z","receivedAt":"2025-03-01T00:12:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:19AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:29PM -0500, Taylor Blau wrote:\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index 41675a69f68..e3fdcf8a01a 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -946,18 +946,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n> >  struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n> >  \t\t\t\t      struct commit *commit)\n> >  {\n> > -\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n> > -\t\t\t\t\t   commit->object.oid);\n> > +\tkhiter_t hash_pos;\n> > +\tif (!bitmap_git)\n> > +\t\treturn NULL;\n> > +\n> > +\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n> >  \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n> >  \t\tstruct stored_bitmap *bitmap = NULL;\n> >  \t\tif (!bitmap_git->table_lookup)\n> > -\t\t\treturn NULL;\n> > +\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n> >\n> >  \t\t/* this is a fairly hot codepath - no trace2_region please */\n> >  \t\t/* NEEDSWORK: cache misses aren't recorded */\n> >  \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n> >  \t\tif (!bitmap)\n> > -\t\t\treturn NULL;\n> > +\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n> >  \t\treturn lookup_stored_bitmap(bitmap);\n> >  \t}\n> >  \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n>\n> One of the things that worries me a bit is that by recursing, we\n> essentially are bound in the depth of MIDX layers as we may otherwise\n> bust the stack. Not that I expect us to typically have thousands of\n> layers, but if there ever was a bug this may fail in bad ways.\n\nI think it's a valid concern, but in practice I suspect we are unlikely\nto run into it. If we have enough MIDX layers to blow the stack, we\nprobably have much bigger problems to worry about ;-).\n\n> I already asked this for a previous commit, but what is the current\n> state regarding compaction of the layers? Do we need to be worried about\n> this or do we already know to keep things limited in general?\n\nI mentioned upthread, but briefly: we don't compact MIDX layers today,\nbut the design is such that it is possible (and planned) to do in the\nfuture (part three of this multi-series thing).\n\nThanks,\nTaylor\n"},{"id":"513310","messageId":"Z8JRx1eY0YrTvNEJ@nand.local","threadId":"61958","inReplyTo":"Z8GJc6iDqueVgc67@pks.im","subject":"Re: [PATCH v3 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:16:07Z","receivedAt":"2025-03-01T00:16:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:23AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:35PM -0500, Taylor Blau wrote:\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index c2c824347a6..1dddb242434 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -2347,14 +2348,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n> >  \t\tuint32_t pack_int_id;\n> >\n> >  \t\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n> >  \t\t\tuint32_t preferred_pack_pos;\n> >\n> > -\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n> > +\t\t\twhile (m->base_midx)\n> > +\t\t\t\tm = m->base_midx;\n> > +\n> > +\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n> >  \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n> >  \t\t\t\treturn;\n> >  \t\t\t}\n>\n> Instead of completely disabling preferred packs, should we maybe fall\n> back to the preferred pack of the next-higher layer?\n\nThe upper layers aren't really supposed to have a notion of a preferred\npack, just whatever pack happens to be first in that layer's order\n(which by definition of the pseudo-pack order makes it preferred, so to\nspeak).\n\nBut the base layer definitely can have a preferred pack, and failing to\nfind it usually means that there is other corruption or something else\nwrong.\n\nThanks,\nTaylor\n"},{"id":"513311","messageId":"Z8JSreTnEFlocYQ9@nand.local","threadId":"61958","inReplyTo":"Z8GJdv5FwDP_k1et@pks.im","subject":"Re: [PATCH v3 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:19:57Z","receivedAt":"2025-03-01T00:19:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:26AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:38PM -0500, Taylor Blau wrote:\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index 1dddb242434..02864a0e1f7 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -2564,13 +2584,57 @@ static void test_show_commit(struct commit *commit, void *data)\n> >  \tdisplay_progress(tdata->prg, ++tdata->seen);\n> >  }\n> >\n> > +static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n> > +{\n> > +\tuint32_t total = 0;\n> > +\tdo {\n> > +\t\ttotal = st_add(total, bitmap_git->entry_count);\n> > +\t\tbitmap_git = bitmap_git->base;\n> > +\t} while (bitmap_git);\n> > +\n> > +\treturn total;\n> > +}\n> > +\n> > +static void prepare_bitmap_test_data(struct bitmap_test_data *tdata,\n> > +\t\t\t\t     struct bitmap_index *bitmap_git)\n>\n> Nit: according to our style guide this should be called\n> `bitmap_test_data_prepare()`.\n\nAh, I didn't know about that part of the style guide. It looks like it\ncomes from your 10f0723c8d (Documentation: document idiomatic function\nnames, 2024-07-30), which is semi-recent ;-).\n\n> > +{\n> > +\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n> > +\n> > +\ttdata->bitmap_git = bitmap_git;\n> > +\ttdata->base = bitmap_new();\n> > +\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n> > +\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n> > +\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n> > +\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n> > +\n> > +\tif (bitmap_git->base) {\n> > +\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n> > +\t\tprepare_bitmap_test_data(tdata->base_tdata, bitmap_git->base);\n> > +\t}\n> > +}\n> > +\n> > +static void free_bitmap_test_data(struct bitmap_test_data *tdata)\n>\n> Same nit here, this should be called `bitmap_test_data_free()`. In fact,\n> it should be called `bitmap_test_data_release()`, because we don't free\n> `tadata` itself.\n\nNoted, and adjusted!\n\n> > @@ -2579,17 +2643,26 @@ void test_bitmap_walk(struct rev_info *revs)\n> >  \tif (revs->pending.nr != 1)\n> >  \t\tdie(_(\"you must specify exactly one commit to test\"));\n> >\n> > -\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n> > +\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n> >  \t\tbitmap_git->version,\n> >  \t\tbitmap_git->entry_count,\n> > -\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n> > +\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n> > +\t\tbitmap_total_entry_count(bitmap_git));\n> >\n> >  \troot = revs->pending.objects[0].item;\n> > -\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n> > +\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n> >\n> >  \tif (bm) {\n> >  \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n> > -\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n> > +\t\t\toid_to_hex(&root->oid),\n> > +\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n> > +\n> > +\t\tif (bitmap_is_midx(found))\n> > +\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n> > +\t\t\t\t   hash_to_hex(get_midx_checksum(found->midx)));\n> > +\t\telse\n> > +\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n> > +\t\t\t\t   hash_to_hex(found->pack->hash));\n> >\n> >  \t\tresult = ewah_to_bitmap(bm);\n> >  \t}\n>\n> I'm a bit surprised that this doesn't result in any changes required in\n> our tests.\n\nThat's intentional, the aim of this commit is really just to get us\nready for the eventuality of having incremental MIDXs. In the meantime,\nwe don't disrupt the behavior of --test-bitmap for single-layer MIDX\nbitmaps.\n\nThanks,\nTaylor\n"},{"id":"513312","messageId":"Z8JTLmgOQLNshnVS@nand.local","threadId":"61958","inReplyTo":"Z8GJemN4K99BVT2J@pks.im","subject":"Re: [PATCH v3 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:22:06Z","receivedAt":"2025-03-01T00:22:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:30AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:47PM -0500, Taylor Blau wrote:\n> > diff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\n> > index 8785cbc54a8..b3a7ada0714 100644\n> > --- a/ewah/ewah_bitmap.c\n> > +++ b/ewah/ewah_bitmap.c\n> > @@ -372,6 +372,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n> >  \t\tread_new_rlw(it);\n> >  }\n> >\n> > +void ewah_or_iterator_init(struct ewah_or_iterator *it,\n> > +\t\t\t   struct ewah_bitmap **parents, size_t nr)\n> > +{\n> > +\tsize_t i;\n> > +\n> > +\tmemset(it, 0, sizeof(*it));\n> > +\n> > +\tALLOC_ARRAY(it->its, nr);\n> > +\tfor (i = 0; i < nr; i++)\n> > +\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n> > +}\n> > +\n> > +int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n> > +{\n> > +\teword_t buf, out = 0;\n> > +\tsize_t i;\n> > +\tint ret = 0;\n> > +\n> > +\tfor (i = 0; i < it->nr; i++)\n> > +\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n> > +\t\t\tout |= buf;\n> > +\t\t\tret = 1;\n> > +\t\t}\n> > +\n> > +\t*next = out;\n> > +\treturn ret;\n> > +}\n>\n> Yup, this looks rather straight-forward: we advance each of our\n> subiterators and or their respective results into the accumulated result\n> that we end up returning to the user.\n>\n> One thing that surprised me though is that we don't seem to be able to\n> tell whether all of the iterators could be moved on to the next result\n> or not. But I guess that makes sense: iterators of lower levels will\n> cover less objects and will thus eventually be exhausted before the\n> iterators on the higher levels.\n\nExactly. Conceptually each iterator ends with an infinite number of\nzero'd bits, and indeed they are expected to be different lengths for\nthe reason you mention.\n\n> > +void ewah_or_iterator_free(struct ewah_or_iterator *it)\n> > +{\n> > +\tfree(it->its);\n> > +}\n> > +\n>\n> This should be called `ewar_or_iterator_release()` because we don't\n> releate `it`, as documented by our style guide.\n\nGreat call, thanks.\n\nThanks,\nTaylor\n"},{"id":"513313","messageId":"Z8JUPlXSZ7tEKWET@nand.local","threadId":"61958","inReplyTo":"Z8GJfjhGAbXnA1LW@pks.im","subject":"Re: [PATCH v3 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:26:38Z","receivedAt":"2025-03-01T00:26:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:34AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:50PM -0500, Taylor Blau wrote:\n> > @@ -586,7 +604,29 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n> >  \treturn load_pack_revindex(r, bitmap_git->pack);\n> >  }\n> >\n> > -static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n> > +static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n> > +{\n> > +\tstruct bitmap_index *curr = bitmap_git;\n> > +\tsize_t i = bitmap_git->base_nr - 1;\n> > +\n> > +\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr);\n> > +\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr);\n> > +\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr);\n> > +\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr);\n> > +\n> > +\twhile (curr) {\n> > +\t\tbitmap_git->commits_all[i] = curr->commits;\n> > +\t\tbitmap_git->trees_all[i] = curr->trees;\n> > +\t\tbitmap_git->blobs_all[i] = curr->blobs;\n> > +\t\tbitmap_git->tags_all[i] = curr->tags;\n> > +\n> > +\t\tcurr = curr->base;\n> > +\t\ti -= 1;\n>\n> Do we want to `BUG()` in case `i == 0` before this statement?\n\nGood idea, but I think we only want to do this when curr is non-NULL\nafter assigning it to curr->base, so something like:\n\n--- 8< ---\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 48da3f3b5b..7cea838f58 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -616,6 +616,9 @@ static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n \t\tbitmap_git->tags_all[i] = curr->tags;\n\n \t\tcurr = curr->base;\n+\t\tif (curr && !i)\n+\t\t\tBUG(\"unexpected number of bitmap layers, expected %lu\",\n+\t\t\t    bitmap_git->base_nr);\n \t\ti -= 1;\n \t}\n }\n--- >8 ---\n\n(though note that this is on top of an adjustment I made earlier to make\nbase_nr 0-indexed as it always should have been, though I don't think it\nis relevant to the portion of the change shown here).\n\nThanks,\nTaylor\n"},{"id":"513314","messageId":"Z8JUwZFZ3yAv+AV2@nand.local","threadId":"61958","inReplyTo":"Z8GJhzdUtH5IB1Pr@pks.im","subject":"Re: [PATCH v3 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:28:49Z","receivedAt":"2025-03-01T00:28:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:43AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:53PM -0500, Taylor Blau wrote:\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index 348488e2d9e..83696d834f6 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -1622,25 +1622,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n> >  \t}\n> >  }\n> >\n> > -static void init_type_iterator(struct ewah_iterator *it,\n> > +static void init_type_iterator(struct ewah_or_iterator *it,\n> >  \t\t\t       struct bitmap_index *bitmap_git,\n> >  \t\t\t       enum object_type type)\n> >  {\n> >  \tswitch (type) {\n> >  \tcase OBJ_COMMIT:\n> > -\t\tewah_iterator_init(it, bitmap_git->commits);\n> > +\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n> > +\t\t\t\t      bitmap_git->base_nr);\n> >  \t\tbreak;\n> >\n> >  \tcase OBJ_TREE:\n> > -\t\tewah_iterator_init(it, bitmap_git->trees);\n> > +\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n> > +\t\t\t\t      bitmap_git->base_nr);\n> >  \t\tbreak;\n> >\n> >  \tcase OBJ_BLOB:\n> > -\t\tewah_iterator_init(it, bitmap_git->blobs);\n> > +\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n> > +\t\t\t\t      bitmap_git->base_nr);\n> >  \t\tbreak;\n> >\n> >  \tcase OBJ_TAG:\n> > -\t\tewah_iterator_init(it, bitmap_git->tags);\n> > +\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n> > +\t\t\t\t      bitmap_git->base_nr);\n> >  \t\tbreak;\n> >\n> >  \tdefault:\n>\n> One thing I wonder here is whether we want to continue using the\n> non-layered iterator in case we don't have an MIDX. But I guess it makes\n> it easier to not do it like that because we have less conditionals, and\n> overall the or'd logic shouldn't perform significantly workse anyway. So\n> as long as we always initialize `base_nr` and the relevant `*_all`\n> fields even in the non-MIDX case we're fine.\n\nI had a similar thought when writing this and agree with you. We do set\nup the _all iterators properly at the top of a chain (see the \"if\n(!recursing)\" conditional inside of load_bitmap()).\n\nSince we call that function for single-pack bitmaps as well (naturally\nwe are not recursing in that case), we'll call load_all_type_bitmaps()\nin that case too.\n\nThanks,\nTaylor\n"},{"id":"513315","messageId":"Z8JVc8+frlXwTghi@nand.local","threadId":"61958","inReplyTo":"Z8GJg8TMeE8YLrqA@pks.im","subject":"Re: [PATCH v3 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-01T00:31:47Z","receivedAt":"2025-03-01T00:31:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 28, 2025 at 11:01:39AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 19, 2024 at 05:07:56PM -0500, Taylor Blau wrote:\n> > diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> > index 49758e2525f..1fbebe84479 100644\n> > --- a/pack-bitmap-write.c\n> > +++ b/pack-bitmap-write.c\n> > @@ -25,6 +25,8 @@\n> >  #include \"alloc.h\"\n> >  #include \"refs.h\"\n> >  #include \"strmap.h\"\n> > +#include \"midx.h\"\n> > +#include \"pack-revindex.h\"\n>\n> Nit: let's keep the headers sorted alphabetically.\n>\n> > @@ -206,19 +215,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n> >  static uint32_t find_object_pos(struct bitmap_writer *writer,\n> >  \t\t\t\tconst struct object_id *oid, int *found)\n> >  {\n> > -\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n> > +\tstruct object_entry *entry;\n> > +\n> > +\tentry = packlist_find(writer->to_pack, oid);\n> > +\tif (entry) {\n> > +\t\tuint32_t base_objects = 0;\n> > +\t\tif (writer->midx)\n> > +\t\t\tbase_objects = writer->midx->num_objects +\n> > +\t\t\t\twriter->midx->num_objects_in_base;\n> > +\n> > +\t\tif (found)\n> > +\t\t\t*found = 1;\n> > +\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n> > +\t} else if (writer->midx) {\n> > +\t\tuint32_t at, pos;\n> > +\n> > +\t\tif (!bsearch_midx(oid, writer->midx, &at))\n> > +\t\t\tgoto missing;\n> > +\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n> > +\t\t\tgoto missing;\n> >\n> > -\tif (!entry) {\n> >  \t\tif (found)\n> > -\t\t\t*found = 0;\n> > -\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> > -\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n> > -\t\treturn 0;\n> > +\t\t\t*found = 1;\n> > +\t\treturn pos;\n> >  \t}\n> >\n> > +missing:\n> >  \tif (found)\n> > -\t\t*found = 1;\n> > -\treturn oe_in_pack_pos(writer->to_pack, entry);\n> > +\t\t*found = 0;\n> > +\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> > +\t\t\"(object %s is missing)\", oid_to_hex(oid));\n>\n> Is this warning still accurate? I assume that in the MIDX case we don't\n> have to have full closure in a single packfile, as that would otherwise\n> make the whole thing rather moot.\n\nGood question -- the warning is still \"accurate\" in the sense that the\nthing we're generating the bitmap against has to have full object\nclosure under reachability.\n\nIt's important to note that even though in any individual layer we are\nonly selecting bitmaps from the set of commits in the union of packs in\nthe corresponding MIDX layer, they may well reference objects from\nearlier layers. And there it isn't important that we find some reachable\nobject in the same MIDX layer, but rather that we find it in an earlier\nlayer or the current layer.\n\nIt's less than accurate in the sense that it says \"Packfile doesn't have\nfull closure\" and isn't marked for translation, but I punted on it here.\n(This is definitely not the first time I've had the thought to change\nit, so maybe I just should have done it here.)\n\nThanks,\nTaylor\n"},{"id":"513397","messageId":"Z8WKgisnKb5zc1xO@pks.im","threadId":"61958","inReplyTo":"Z8JGNQZolfs7fm65@nand.local","subject":"Re: [PATCH v3 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-03T10:54:58Z","receivedAt":"2025-03-03T10:55:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 28, 2025 at 06:26:45PM -0500, Taylor Blau wrote:\n> On Fri, Feb 28, 2025 at 11:01:04AM +0100, Patrick Steinhardt wrote:\n> > On Tue, Nov 19, 2024 at 05:07:19PM -0500, Taylor Blau wrote:\n> > > diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\n> > > index cc063b30bea..a063262c360 100644\n> > > --- a/Documentation/technical/multi-pack-index.txt\n> > > +++ b/Documentation/technical/multi-pack-index.txt\n> > > @@ -164,6 +164,70 @@ objects_nr($H2) + objects_nr($H1) + i\n> > >  (in the C implementation, this is often computed as `i +\n> > >  m->num_objects_in_base`).\n> > >\n> > > +=== Pseudo-pack order for incremental MIDXs\n> > > +\n> > > +The original implementation of multi-pack reachability bitmaps defined\n> > > +the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n> > > +titled \"multi-pack-index reverse indexes\") roughly as follows:\n> > > +\n> > > +____\n> > > +In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n> > > +objects in packs stored by the MIDX, laid out in pack order, and the\n> > > +packs arranged in MIDX order (with the preferred pack coming first).\n> > > +____\n> > > +\n> > > +In the incremental MIDX design, we extend this definition to include\n> > > +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> > > +for incremental MIDXs is determined by concatenating the pseudo-pack\n> > > +ordering for each layer of the MIDX chain in order. Formally two objects\n> > > +`o1` and `o2` are compared as follows:\n> > > +\n> > > +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> > > +  `o1` is considered less than `o2`.\n> >\n> > Just as a refresher for myself: what is the consequence of an object\n> > `o1` sorting earlier than `o2`? In the case where those refer to\n> > different objects it is only used to establish the pseudo-pack order so\n> > that we know how to interpret the bitmaps. But in the case where those\n> > two objects refer to the same underlying object, e.g. because the object\n> > is contained in two packs, it also impacts which of both objects would\n> > be preferred e.g. during a clone, right?\n> \n> Great question -- the pseudo-pack order here is how we translate the set\n> of objects in a MIDX into their corresponding bit positions in the\n> bitmap.\n> \n> So if \"o1\" sorts ahead of \"o2\", that means that \"o1\" will appear in an\n> earlier bit position than \"o2\". But note that we're talking about\n> objects in a MIDX chain here, comprised of objects from each MIDX'd layer of\n> that chain. So by that point the duplicates have already been filtered\n> out, since:\n> \n>   - The MIDX only stores one copy of an object in any given MIDX, and\n> \n>   - The incremental MIDX design avoids putting objects from earlier\n>     layers in later ones.\n> \n> I tried to get at this a few lines up with \"[...] a MIDX's pseudo-pack\n> is the de-duplicated concatenation of [...]\" to make clear that o1 != o2\n> here. But let me know if you think I should clarify or emphasize that\n> point further.\n\nOkay, the deduplication bit was a bit subtle, so I missed that part. And\nonce one has learned about it my question makes less sense, as I was\nexpecting that an object may appear in the same MIDX chain multiple\ntimes.\n\nPatrick\n"},{"id":"513398","messageId":"Z8WKjegeTs9aAsBr@pks.im","threadId":"61958","inReplyTo":"Z8JLbxBQh7XUpplz@nand.local","subject":"Re: [PATCH v3 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-03T10:55:09Z","receivedAt":"2025-03-03T10:55:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 28, 2025 at 06:49:03PM -0500, Taylor Blau wrote:\n> On Fri, Feb 28, 2025 at 11:01:16AM +0100, Patrick Steinhardt wrote:\n> > On Tue, Nov 19, 2024 at 05:07:26PM -0500, Taylor Blau wrote:\n> > > Prepare the pack-bitmap machinery to work with incremental MIDXs by\n> > > adding a new \"base\" field to keep track of the bitmap index associated\n> > > with the previous MIDX layer.\n> > >\n> > > The changes in this commit are mostly boilerplate to open the correct\n> > > bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n> >\n> > s/bitmap layers/of &/\n> >\n> > > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > > index bba9c6a905a..41675a69f68 100644\n> > > --- a/pack-bitmap.c\n> > > +++ b/pack-bitmap.c\n> > > @@ -54,6 +54,13 @@ struct bitmap_index {\n> > >  \tstruct packed_git *pack;\n> > >  \tstruct multi_pack_index *midx;\n> > >\n> > > +\t/*\n> > > +\t * If using a multi-pack index chain, 'base' points to the\n> > > +\t * bitmap index corresponding to this bitmap's midx->base_midx.\n> > > +\t */\n> > > +\tstruct bitmap_index *base;\n> > > +\tuint32_t base_nr;\n> > > +\n> >\n> > It would be nice to point out that `base_nr` is not 0-indexed, but\n> > 1-indexed, which is rather uncommon. Is there any particular reason why\n> > you made it 1-indexed?\n> \n> Hah, I have no idea! If I remember correctly, it's because it makes it\n> (slightly) more convenient to do:\n> \n>     ewah_or_iterator_init(it, bitmap_git->commits_all,\n>                           bitmap_git->base_nr);\n> \n> , instead of incrementing 'base_nr' by 1 to determine the number of\n> sub-iterators to allocate.\n> \n> So I think there are a couple of options here. Short of doing nothing,\n> we could:\n> \n>  1. Rename 'base_nr' to 'layers_nr' which would make it clearer that the\n>     count includes the current layer, thus making it 1-indexed.\n> \n>  2. Leave 'base_nr' named as-is, but make it 0-indexed, and have callers add\n>     1 when they need to know the number of layers.\n> \n> I prefer the explicitness of (2), which is how I adjusted things\n> locally. But if you prefer (1) or some yet-unknown (3), I'm happy to\n> adjust it further!\n\nYup, I also favor (2) here as it is the least surprising option to me.\n\nPatrick\n"},{"id":"514303","messageId":"cover.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH v4 00/13] midx: incremental multi-pack indexes, part two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:12Z","receivedAt":"2025-03-14T20:18:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This is a new round of my series to implement bitmap support for\nincremental multi-pack indexes (MIDXs). It has been rebased on current\n'master', which is 683c54c999 (Git 2.49, 2025-03-14) at the time of\nwriting.\n\n== Changes since last time\n\nThis round addresses helpful review from Patrick Steinhardt. It\nclarifies a number of concepts which were previously less than clear,\nand makes a number of cosmetic changes to align more closely with our\nCodingGuidelines in certain areas.\n\nThere are a few technical changes which you can see in the range-diff\nbelow, but they are relatively minor. The series is functionally\nunchanged since last time.\n\n== Original cover letter\n\nThis series is based on 'master', with an additional merge between\ntb/incremental-midx-part-1[1] and my newer series to fix a handful of\nbugs related to pseudo-merge bitmaps[2].\n\nThis is the second of three series to implement support for incremental\nmulti-pack indexes (MIDXs). This series brings support for bitmaps that\nare tied to incremental MIDXs in addition to regular MIDX bitmaps.\n\nThe details are laid out in the commits themselves, but the high-level\napproach is as follows:\n\n  - Each layer in the incremental MIDX chain has its own corresponding\n    *.bitmap file. Each bitmap contains commits / pseudo-merges which\n    are selected only from the commits in that layer. Likewise, only\n    that layer's objects are written in the type-level bitmaps.\n\n  - The reachability traversal is only conducted on the top-most bitmap\n    corresponding to the most recent layer in the incremental MIDX\n    chain. Earlier layers may be consulted to retrieve commit /\n    pseudo-merge reachability bitmaps, but only the top-most bitmap's\n    \"result\" and \"haves\" fields are used.\n\n  - In essence, the top-most bitmap is the only one that \"matters\", and\n    earlier bitmaps are merely used to look up commit and pseudo-merge\n    bitmaps from that layer.\n\n  - Whenever we need to look at the type-level bitmaps corresponding to\n    the whole incremental MIDX chain, a new \"ewah_or_iterator\" is used.\n    This works in concept like a typical ewah_iterator, except works\n    over many EWAH bitmaps in parallel, OR-ing their results together\n    before returning them to the user.\n\n    In effect, this allows us to treat the union of all type-level\n    bitmaps (each of which only stores information about the objects its\n    corresponding layer within the incremental MIDX chain) as a single\n    type-level bitmap corresponding to all of the objects across every\n    layer of the incremental MIDX chain.\n\nThe sum total of this series is that we are able to append new commits /\npseudo-merges to a repository's reachability bitmaps without having to\nrewrite existing bitmaps, making the operation much cheaper to perform\nin large repositories.\n\nThe series is laid out roughly as follows:\n\n  - The first patch describes the technical details of incremental MIDX\n    bitmaps.\n\n  - The second patch adjusts the pack-revindex internals to prepare for\n    incremental MIDX bitmaps.\n\n  - The next seven patches adjust various components of the pack-bitmap\n    internals to do the same.\n\n  - The next three patches introduce and adjust callers to use the\n    ewah_or_iterator (as above).\n\n  - The final patch implements writing incremental MIDX bitmaps, and\n    introduces tests.\n\nAfter this series, the remaining goals for this project include being\nable to compact contiguous runs of incremental MIDX layers into a single\nlayer to support growing the chain of MIDX layers without the chain\nitself becoming too long.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1722958595.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover.1723743050.git.me@ttaylorr.com/\n\nTaylor Blau (13):\n  Documentation: describe incremental MIDX bitmaps\n  pack-revindex: prepare for incremental MIDX bitmaps\n  pack-bitmap.c: open and store incremental bitmap layers\n  pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n  pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n  pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n  pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n  pack-bitmap.c: compute disk-usage with incremental MIDXs\n  pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n  ewah: implement `struct ewah_or_iterator`\n  pack-bitmap.c: keep track of each layer's type bitmaps\n  pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n  midx: implement writing incremental MIDX bitmaps\n\n Documentation/technical/multi-pack-index.adoc |  71 ++++\n builtin/pack-objects.c                        |   3 +-\n ewah/ewah_bitmap.c                            |  33 ++\n ewah/ewok.h                                   |  12 +\n midx-write.c                                  |  57 ++-\n pack-bitmap-write.c                           |  65 +++-\n pack-bitmap.c                                 | 342 ++++++++++++++----\n pack-bitmap.h                                 |   4 +-\n pack-revindex.c                               |  34 +-\n t/t5334-incremental-multi-pack-index.sh       |  84 +++++\n 10 files changed, 583 insertions(+), 122 deletions(-)\n\nRange-diff against v3:\n 1:  caed2c6ec3 !  1:  f565f2fff1 Documentation: describe incremental MIDX bitmaps\n    @@ Commit message\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    - ## Documentation/technical/multi-pack-index.txt ##\n    -@@ Documentation/technical/multi-pack-index.txt: objects_nr($H2) + objects_nr($H1) + i\n    + ## Documentation/technical/multi-pack-index.adoc ##\n    +@@ Documentation/technical/multi-pack-index.adoc: objects_nr($H2) + objects_nr($H1) + i\n      (in the C implementation, this is often computed as `i +\n      m->num_objects_in_base`).\n      \n    @@ Documentation/technical/multi-pack-index.txt: objects_nr($H2) + objects_nr($H1)\n     +\n     +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n     +  `o1` is considered less than `o2`.\n    ++\n     +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n    -+  MIDX layer has no base, then If one of `pack(o1)` and `pack(o2)` is\n    -+  preferred and the other is not, then the preferred one sorts first. If\n    -+  there is a base layer (i.e. the MIDX layer is not the first layer in\n    -+  the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n    -+  pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n    -+  appears earlier, than the opposite is true.\n    ++   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n    ++   preferred and the other is not, then the preferred one sorts first. If\n    ++   there is a base layer (i.e. the MIDX layer is not the first layer in\n    ++   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n    ++   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n    ++   appears earlier, than the opposite is true.\n    ++\n     +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n    -+  same MIDX layer. Sort `o1` and `o2` by their offset within their\n    -+  containing packfile.\n    ++   same MIDX layer. Sort `o1` and `o2` by their offset within their\n    ++   containing packfile.\n    ++\n    ++Note that the preferred pack is a property of the MIDX chain, not the\n    ++individual layers themselves. Fundamentally we could introduce a\n    ++per-layer preferred pack, but this is less relevant now that we can\n    ++perform multi-pack reuse across the set of packs in a MIDX.\n     +\n     +=== Reachability bitmaps and incremental MIDXs\n     +\n 2:  b902513f43 !  2:  f2a232e556 pack-revindex: prepare for incremental MIDX bitmaps\n    @@ pack-bitmap.c: static int ext_index_add_object(struct bitmap_index *bitmap_git,\n      }\n      \n      struct bitmap_show_data {\n    +@@ pack-bitmap.c: struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_g\n    + \t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n    + \t\t\tgoto done;\n    + \n    ++\t\t/*\n    ++\t\t * Use bitmap-relative positions instead of offsetting\n    ++\t\t * by bitmap_git->num_objects_in_base because we use\n    ++\t\t * this to find a match in pseudo_merge_for_parents(),\n    ++\t\t * and pseudo-merge groups cannot span multiple bitmap\n    ++\t\t * layers.\n    ++\t\t */\n    + \t\tbitmap_set(parents, pos);\n    + \t}\n    + \n    +-\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n    +-\t\t\t\t\t\tparents);\n    ++\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges, parents);\n    + \n    + done:\n    + \tbitmap_free(parents);\n     @@ pack-bitmap.c: static void show_extended_objects(struct bitmap_index *bitmap_git,\n      \tfor (i = 0; i < eindex->count; ++i) {\n      \t\tstruct object *obj;\n    @@ pack-bitmap.c: static unsigned long get_size_by_pos(struct bitmap_index *bitmap_\n      \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n     -\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n     +\t\tstruct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n    - \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n    + \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n    + \t\t\t\t\t     &oi, 0) < 0)\n      \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n    - \t}\n     @@ pack-bitmap.c: static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n      \tuint32_t objects_nr;\n      \tsize_t i, pos;\n    @@ pack-bitmap.c: static off_t get_disk_usage_for_extended(struct bitmap_index *bit\n     +\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n      \t\t\tcontinue;\n      \n    - \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n    + \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n     \n      ## pack-revindex.c ##\n     @@ pack-revindex.c: int load_midx_revindex(struct multi_pack_index *m)\n      \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n      \t\t\t   \"source\", \"rev\");\n      \n    --\tget_midx_filename_ext(&revindex_name, m->object_dir,\n    +-\tget_midx_filename_ext(m->repo->hash_algo, &revindex_name, m->object_dir,\n     -\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n     +\tif (m->has_chain)\n    -+\t\tget_split_midx_filename_ext(&revindex_name, m->object_dir,\n    -+\t\t\t\t\t    get_midx_checksum(m), MIDX_EXT_REV);\n    ++\t\tget_split_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n    ++\t\t\t\t\t    m->object_dir, get_midx_checksum(m),\n    ++\t\t\t\t\t    MIDX_EXT_REV);\n     +\telse\n    -+\t\tget_midx_filename_ext(&revindex_name, m->object_dir,\n    -+\t\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n    ++\t\tget_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n    ++\t\t\t\t      m->object_dir, get_midx_checksum(m),\n    ++\t\t\t\t      MIDX_EXT_REV);\n      \n      \tret = load_revindex_from_disk(revindex_name.buf,\n      \t\t\t\t      m->num_objects,\n    @@ pack-revindex.c: static int midx_pack_order_cmp(const void *va, const void *vb)\n      \tstruct multi_pack_index *midx = key->midx;\n      \n     -\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n    -+\tsize_t pos = (uint32_t*)vb - (const uint32_t *)midx->revindex_data;\n    ++\tsize_t pos = (uint32_t *)vb - (const uint32_t *)midx->revindex_data;\n     +\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n      \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n      \toff_t versus_offset;\n 3:  5b5d625cbe !  3:  aca0318fb1 pack-bitmap.c: open and store incremental bitmap layers\n    @@ pack-bitmap.c: struct bitmap_index {\n     +\t/*\n     +\t * If using a multi-pack index chain, 'base' points to the\n     +\t * bitmap index corresponding to this bitmap's midx->base_midx.\n    ++\t *\n    ++\t * base_nr indicates how many layers precede this one, and is\n    ++\t * zero when base is NULL.\n     +\t */\n     +\tstruct bitmap_index *base;\n     +\tuint32_t base_nr;\n    @@ pack-bitmap.c: static int load_bitmap_entries_v1(struct bitmap_index *index)\n      char *midx_bitmap_filename(struct multi_pack_index *midx)\n      {\n      \tstruct strbuf buf = STRBUF_INIT;\n    --\tget_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx),\n    --\t\t\t      MIDX_EXT_BITMAP);\n    +-\tget_midx_filename_ext(midx->repo->hash_algo, &buf, midx->object_dir,\n    +-\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n     +\tif (midx->has_chain)\n    -+\t\tget_split_midx_filename_ext(&buf, midx->object_dir,\n    ++\t\tget_split_midx_filename_ext(midx->repo->hash_algo, &buf,\n    ++\t\t\t\t\t    midx->object_dir,\n     +\t\t\t\t\t    get_midx_checksum(midx),\n     +\t\t\t\t\t    MIDX_EXT_BITMAP);\n     +\telse\n    -+\t\tget_midx_filename_ext(&buf, midx->object_dir,\n    -+\t\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n    ++\t\tget_midx_filename_ext(midx->repo->hash_algo, &buf,\n    ++\t\t\t\t      midx->object_dir, get_midx_checksum(midx),\n    ++\t\t\t\t      MIDX_EXT_BITMAP);\n      \n      \treturn strbuf_detach(&buf, NULL);\n      }\n    -@@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n    - {\n    - \tstruct stat st;\n    - \tchar *bitmap_name = midx_bitmap_filename(midx);\n    --\tint fd = git_open(bitmap_name);\n    -+\tint fd;\n    - \tuint32_t i, preferred_pack;\n    - \tstruct packed_git *preferred;\n    - \n    -+\tfd = git_open(bitmap_name);\n    -+\tif (fd < 0 && errno == ENOENT) {\n    -+\t\tFREE_AND_NULL(bitmap_name);\n    -+\t\tbitmap_name = midx_bitmap_filename(midx);\n    -+\t\tfd = git_open(bitmap_name);\n    -+\t}\n    -+\n    - \tif (fd < 0) {\n    - \t\tif (errno != ENOENT)\n    - \t\t\twarning_errno(\"cannot open '%s'\", bitmap_name);\n     @@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n      \t\tgoto cleanup;\n      \t}\n      \n     -\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n    +-\t\tif (prepare_midx_pack(bitmap_repo(bitmap_git),\n    +-\t\t\t\t      bitmap_git->midx,\n    +-\t\t\t\t      i)) {\n     +\tfor (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n    - \t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i)) {\n    ++\t\tif (prepare_midx_pack(bitmap_repo(bitmap_git), bitmap_git->midx, i)) {\n      \t\t\twarning(_(\"could not open pack %s\"),\n      \t\t\t\tbitmap_git->midx->pack_names[i]);\n    -@@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n    - \t\tgoto cleanup;\n    - \t}\n    - \n    --\tpreferred = bitmap_git->midx->packs[preferred_pack];\n    -+\tpreferred = nth_midxed_pack(bitmap_git->midx, preferred_pack);\n    - \tif (!is_pack_valid(preferred)) {\n    - \t\twarning(_(\"preferred pack (%s) is invalid\"),\n    - \t\t\tpreferred->pack_name);\n    - \t\tgoto cleanup;\n    + \t\t\tgoto cleanup;\n    + \t\t}\n      \t}\n      \n     +\tif (midx->base_midx) {\n     +\t\tbitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n     +\t\tbitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n     +\t} else {\n    -+\t\tbitmap_git->base_nr = 1;\n    ++\t\tbitmap_git->base_nr = 0;\n     +\t}\n     +\n      \treturn 0;\n    @@ pack-bitmap.c: static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, st\n      \tbitmap_git->map_size = xsize_t(st.st_size);\n      \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n      \tbitmap_git->map_pos = 0;\n    -+\tbitmap_git->base_nr = 1;\n    ++\tbitmap_git->base_nr = 0;\n      \tclose(fd);\n      \n      \tif (load_bitmap_header(bitmap_git) < 0) {\n    @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_git(struct repository *r)\n      \n      struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n      {\n    --\tstruct repository *r = the_repository;\n    +-\tstruct repository *r = midx->repo;\n      \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n      \n     -\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n 4:  16259667fb =  4:  832fd0e8dc pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n 5:  b7a45d7eff =  5:  c7c9f89956 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n 6:  c8401fa0fb =  6:  14d3d80c3d pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n 7:  17ab23dd76 !  7:  b45a9ccbc2 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n    @@ pack-bitmap.c: static void test_show_commit(struct commit *commit, void *data)\n     +\treturn total;\n     +}\n     +\n    -+static void prepare_bitmap_test_data(struct bitmap_test_data *tdata,\n    ++static void bitmap_test_data_prepare(struct bitmap_test_data *tdata,\n     +\t\t\t\t     struct bitmap_index *bitmap_git)\n     +{\n     +\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n    @@ pack-bitmap.c: static void test_show_commit(struct commit *commit, void *data)\n     +\n     +\tif (bitmap_git->base) {\n     +\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n    -+\t\tprepare_bitmap_test_data(tdata->base_tdata, bitmap_git->base);\n    ++\t\tbitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n     +\t}\n     +}\n     +\n    -+static void free_bitmap_test_data(struct bitmap_test_data *tdata)\n    ++static void bitmap_test_data_release(struct bitmap_test_data *tdata)\n     +{\n     +\tif (!tdata)\n     +\t\treturn;\n     +\n    -+\tfree_bitmap_test_data(tdata->base_tdata);\n    ++\tbitmap_test_data_release(tdata->base_tdata);\n     +\tfree(tdata->base_tdata);\n     +\n     +\tbitmap_free(tdata->base);\n    @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n     +\n     +\t\tif (bitmap_is_midx(found))\n     +\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n    -+\t\t\t\t   hash_to_hex(get_midx_checksum(found->midx)));\n    ++\t\t\t\t   hash_to_hex_algop(get_midx_checksum(found->midx),\n    ++\t\t\t\t\t\t     revs->repo->hash_algo));\n     +\t\telse\n     +\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n    -+\t\t\t\t   hash_to_hex(found->pack->hash));\n    ++\t\t\t\t   hash_to_hex_algop(found->pack->hash,\n    ++\t\t\t\t\t\t     revs->repo->hash_algo));\n      \n      \t\tresult = ewah_to_bitmap(bm);\n      \t}\n    @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n     -\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n     -\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n     -\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n    -+\tprepare_bitmap_test_data(&tdata, bitmap_git);\n    - \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n    ++\tbitmap_test_data_prepare(&tdata, bitmap_git);\n    + \ttdata.prg = start_progress(revs->repo,\n    + \t\t\t\t   \"Verifying bitmap entries\",\n    + \t\t\t\t   result_popcnt);\n     -\ttdata.seen = 0;\n      \n      \ttraverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);\n    @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n     -\tbitmap_free(tdata.trees);\n     -\tbitmap_free(tdata.blobs);\n     -\tbitmap_free(tdata.tags);\n    -+\tfree_bitmap_test_data(&tdata);\n    ++\tbitmap_test_data_release(&tdata);\n      \tfree_bitmap_index(bitmap_git);\n      }\n      \n 8:  75d170ce07 =  8:  c1eefeae99 pack-bitmap.c: compute disk-usage with incremental MIDXs\n 9:  0b4fcfcecb =  9:  11c4b7b949 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n10:  e1b5f6181e = 10:  cb08ad6a62 ewah: implement `struct ewah_or_iterator`\n11:  9ab8fb472f ! 11:  a29f4ee60d pack-bitmap.c: keep track of each layer's type bitmaps\n    @@ pack-bitmap.c: static int load_reverse_index(struct repository *r, struct bitmap\n     +static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n     +{\n     +\tstruct bitmap_index *curr = bitmap_git;\n    -+\tsize_t i = bitmap_git->base_nr - 1;\n    ++\tsize_t i = bitmap_git->base_nr;\n     +\n    -+\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr);\n    -+\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr);\n    -+\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr);\n    -+\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr);\n    ++\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);\n    ++\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);\n    ++\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);\n    ++\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr + 1);\n     +\n     +\twhile (curr) {\n     +\t\tbitmap_git->commits_all[i] = curr->commits;\n    @@ pack-bitmap.c: static int load_reverse_index(struct repository *r, struct bitmap\n     +\t\tbitmap_git->tags_all[i] = curr->tags;\n     +\n     +\t\tcurr = curr->base;\n    ++\t\tif (curr && !i)\n    ++\t\t\tBUG(\"unexpected number of bitmap layers, expected %\"PRIu32,\n    ++\t\t\t    bitmap_git->base_nr + 1);\n     +\t\ti -= 1;\n     +\t}\n     +}\n12:  87cb011e7f ! 12:  a1cf65bedc pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n    @@ pack-bitmap.c: static void show_extended_objects(struct bitmap_index *bitmap_git\n      \tcase OBJ_COMMIT:\n     -\t\tewah_iterator_init(it, bitmap_git->commits);\n     +\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n    -+\t\t\t\t      bitmap_git->base_nr);\n    ++\t\t\t\t      bitmap_git->base_nr + 1);\n      \t\tbreak;\n      \n      \tcase OBJ_TREE:\n     -\t\tewah_iterator_init(it, bitmap_git->trees);\n     +\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n    -+\t\t\t\t      bitmap_git->base_nr);\n    ++\t\t\t\t      bitmap_git->base_nr + 1);\n      \t\tbreak;\n      \n      \tcase OBJ_BLOB:\n     -\t\tewah_iterator_init(it, bitmap_git->blobs);\n     +\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n    -+\t\t\t\t      bitmap_git->base_nr);\n    ++\t\t\t\t      bitmap_git->base_nr + 1);\n      \t\tbreak;\n      \n      \tcase OBJ_TAG:\n     -\t\tewah_iterator_init(it, bitmap_git->tags);\n     +\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n    -+\t\t\t\t      bitmap_git->base_nr);\n    ++\t\t\t\t      bitmap_git->base_nr + 1);\n      \t\tbreak;\n      \n      \tdefault:\n13:  77ddd1170f ! 13:  d0d564685b midx: implement writing incremental MIDX bitmaps\n    @@ builtin/pack-objects.c: static void write_pack_file(void)\n      \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n      \t\t\t\t\t\t\t       written_list);\n     \n    + ## ewah/ewah_bitmap.c ##\n    +@@ ewah/ewah_bitmap.c: int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n    + \treturn ret;\n    + }\n    + \n    +-void ewah_or_iterator_free(struct ewah_or_iterator *it)\n    ++void ewah_or_iterator_release(struct ewah_or_iterator *it)\n    + {\n    + \tfree(it->its);\n    + }\n    +\n    + ## ewah/ewok.h ##\n    +@@ ewah/ewok.h: void ewah_or_iterator_init(struct ewah_or_iterator *it,\n    + \n    + int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n    + \n    +-void ewah_or_iterator_free(struct ewah_or_iterator *it);\n    ++void ewah_or_iterator_release(struct ewah_or_iterator *it);\n    + \n    + void ewah_xor(\n    + \tstruct ewah_bitmap *ewah_i,\n    +\n      ## midx-write.c ##\n     @@ midx-write.c: static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n      \treturn pack_order;\n    @@ midx-write.c: static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n      \tstruct strbuf buf = STRBUF_INIT;\n      \tchar *tmp_file;\n      \n    - \ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n    + \ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", ctx->repo);\n      \n    --\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n    +-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex_algop(midx_hash,\n    +-\t\t\t\t\t\t\t\t    ctx->repo->hash_algo));\n     +\tif (ctx->incremental)\n    -+\t\tget_split_midx_filename_ext(&buf, object_dir, midx_hash,\n    ++\t\tget_split_midx_filename_ext(ctx->repo->hash_algo, &buf,\n    ++\t\t\t\t\t    object_dir, midx_hash,\n     +\t\t\t\t\t    MIDX_EXT_REV);\n     +\telse\n    -+\t\tget_midx_filename_ext(&buf, object_dir, midx_hash,\n    -+\t\t\t\t      MIDX_EXT_REV);\n    ++\t\tget_midx_filename_ext(ctx->repo->hash_algo, &buf, object_dir,\n    ++\t\t\t\t      midx_hash, MIDX_EXT_REV);\n      \n    - \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n    - \t\t\t\t\tmidx_hash, WRITE_REV);\n    + \ttmp_file = write_rev_file_order(ctx->repo->hash_algo, NULL, ctx->pack_order,\n    + \t\t\t\t\tctx->entries_nr, midx_hash, WRITE_REV);\n     @@ midx-write.c: static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n      \treturn cb.commits;\n      }\n      \n    --static int write_midx_bitmap(const char *midx_name,\n    +-static int write_midx_bitmap(struct repository *r, const char *midx_name,\n     +static int write_midx_bitmap(struct write_midx_context *ctx,\n     +\t\t\t     const char *object_dir,\n      \t\t\t     const unsigned char *midx_hash,\n    @@ midx-write.c: static struct commit **find_commits_for_midx_bitmap(uint32_t *inde\n      \tstruct bitmap_writer writer;\n      \tstruct pack_idx_entry **index;\n     -\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n    --\t\t\t\t\thash_to_hex(midx_hash));\n    +-\t\t\t\t    hash_to_hex_algop(midx_hash, r->hash_algo));\n     +\tstruct strbuf bitmap_name = STRBUF_INIT;\n    + \n    +-\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", r);\n    ++\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", ctx->repo);\n     +\n     +\tif (ctx->incremental)\n    -+\t\tget_split_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n    ++\t\tget_split_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,\n    ++\t\t\t\t\t    object_dir, midx_hash,\n     +\t\t\t\t\t    MIDX_EXT_BITMAP);\n     +\telse\n    -+\t\tget_midx_filename_ext(&bitmap_name, object_dir, midx_hash,\n    -+\t\t\t\t      MIDX_EXT_BITMAP);\n    ++\t\tget_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,\n    ++\t\t\t\t      object_dir, midx_hash, MIDX_EXT_BITMAP);\n      \n    - \ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n    - \n    -@@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n    + \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n    + \t\toptions |= BITMAP_OPT_HASH_CACHE;\n    +@@ midx-write.c: static int write_midx_bitmap(struct repository *r, const char *midx_name,\n      \tfor (i = 0; i < pdata->nr_objects; i++)\n      \t\tindex[i] = &pdata->objects[i].idx;\n      \n    --\tbitmap_writer_init(&writer, the_repository, pdata);\n    -+\tbitmap_writer_init(&writer, the_repository, pdata,\n    +-\tbitmap_writer_init(&writer, r, pdata);\n    ++\tbitmap_writer_init(&writer, ctx->repo, pdata,\n     +\t\t\t   ctx->incremental ? ctx->base_midx : NULL);\n      \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n      \tbitmap_writer_build_type_index(&writer, index);\n      \n    -@@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n    +@@ midx-write.c: static int write_midx_bitmap(struct repository *r, const char *midx_name,\n      \t * bitmap_writer_finish().\n      \t */\n      \tfor (i = 0; i < pdata->nr_objects; i++)\n    @@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n      \n      \tbitmap_writer_select_commits(&writer, commits, commits_nr);\n      \tret = bitmap_writer_build(&writer);\n    -@@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n    +@@ midx-write.c: static int write_midx_bitmap(struct repository *r, const char *midx_name,\n      \t\tgoto cleanup;\n      \n      \tbitmap_writer_set_checksum(&writer, midx_hash);\n    @@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n     +\tstrbuf_release(&bitmap_name);\n      \tbitmap_writer_free(&writer);\n      \n    - \ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n    -@@ midx-write.c: static int write_midx_internal(const char *object_dir,\n    - \ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n    +-\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", r);\n    ++\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", ctx->repo);\n    + \n    + \treturn ret;\n    + }\n    +@@ midx-write.c: static int write_midx_internal(struct repository *r, const char *object_dir,\n    + \tctx.repo = r;\n      \n      \tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n     -\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n    @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n      \n      \tif (ctx.incremental)\n      \t\tstrbuf_addf(&midx_name,\n    -@@ midx-write.c: static int write_midx_internal(const char *object_dir,\n    +@@ midx-write.c: static int write_midx_internal(struct repository *r, const char *object_dir,\n      \tif (ctx.incremental) {\n      \t\tstruct multi_pack_index *m = ctx.base_midx;\n      \t\twhile (m) {\n     +\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n     +\t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n    -+\t\t\t\t      hash_to_hex(get_midx_checksum(m)));\n    ++\t\t\t\t      hash_to_hex_algop(get_midx_checksum(m),\n    ++\t\t\t\t\t\t\tm->repo->hash_algo));\n     +\t\t\t\tresult = 1;\n     +\t\t\t\tgoto cleanup;\n     +\t\t\t}\n      \t\t\tctx.num_multi_pack_indexes_before++;\n      \t\t\tm = m->base_midx;\n      \t\t}\n    -@@ midx-write.c: static int write_midx_internal(const char *object_dir,\n    +@@ midx-write.c: static int write_midx_internal(struct repository *r, const char *object_dir,\n      \n      \tif (flags & MIDX_WRITE_REV_INDEX &&\n      \t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n    @@ midx-write.c: static int write_midx_internal(const char *object_dir,\n      \n      \tif (flags & MIDX_WRITE_BITMAP) {\n      \t\tstruct packing_data pdata;\n    -@@ midx-write.c: static int write_midx_internal(const char *object_dir,\n    +@@ midx-write.c: static int write_midx_internal(struct repository *r, const char *object_dir,\n      \t\tFREE_AND_NULL(ctx.entries);\n      \t\tctx.entries_nr = 0;\n      \n    --\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n    +-\t\tif (write_midx_bitmap(r, midx_name.buf, midx_hash, &pdata,\n     -\t\t\t\t      commits, commits_nr, ctx.pack_order,\n     +\t\tif (write_midx_bitmap(&ctx, object_dir,\n     +\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n    @@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n      \n      \twrite_selected_commits_v1(writer, f, offsets);\n     \n    + ## pack-bitmap.c ##\n    +@@ pack-bitmap.c: static void show_objects_for_type(\n    + \t\t}\n    + \t}\n    + \n    +-\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n    + }\n    + \n    + static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n    +@@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n    + \t\t\tbitmap_unset(to_filter, pos);\n    + \t}\n    + \n    +-\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n    + \tbitmap_free(tips);\n    + }\n    + \n    +@@ pack-bitmap.c: static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n    + \t\t\tbitmap_unset(to_filter, pos);\n    + \t}\n    + \n    +-\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n    + \tbitmap_free(tips);\n    + }\n    + \n    +@@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n    + \t\t\tcount++;\n    + \t}\n    + \n    +-\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n    + \n    + \treturn count;\n    + }\n    +@@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n    + \t\t}\n    + \t}\n    + \n    +-\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n    + \n    + \treturn total;\n    + }\n    +\n      ## pack-bitmap.h ##\n     @@ pack-bitmap.h: struct bitmap_writer {\n      \n\nbase-commit: 683c54c999c301c2cd6f715c411407c413b1d84e\n-- \n2.49.0.13.gd0d564685b\n"},{"id":"514304","messageId":"f565f2fff166bdf4bb2505f4a8853215a5365b17.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:20Z","receivedAt":"2025-03-14T20:18:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement support for reachability bitmaps for the new\nincremental multi-pack index (MIDX) feature over the following commits.\n\nThis commit begins by first describing the relevant format and usage\ndetails for incremental MIDX bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/technical/multi-pack-index.adoc | 71 +++++++++++++++++++\n 1 file changed, 71 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.adoc b/Documentation/technical/multi-pack-index.adoc\nindex cc063b30be..ab98ecfeb9 100644\n--- a/Documentation/technical/multi-pack-index.adoc\n+++ b/Documentation/technical/multi-pack-index.adoc\n@@ -164,6 +164,77 @@ objects_nr($H2) + objects_nr($H1) + i\n (in the C implementation, this is often computed as `i +\n m->num_objects_in_base`).\n \n+=== Pseudo-pack order for incremental MIDXs\n+\n+The original implementation of multi-pack reachability bitmaps defined\n+the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n+titled \"multi-pack-index reverse indexes\") roughly as follows:\n+\n+____\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+____\n+\n+In the incremental MIDX design, we extend this definition to include\n+objects from multiple layers of the MIDX chain. The pseudo-pack order\n+for incremental MIDXs is determined by concatenating the pseudo-pack\n+ordering for each layer of the MIDX chain in order. Formally two objects\n+`o1` and `o2` are compared as follows:\n+\n+1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n+  `o1` is considered less than `o2`.\n+\n+2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n+   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n+   preferred and the other is not, then the preferred one sorts first. If\n+   there is a base layer (i.e. the MIDX layer is not the first layer in\n+   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n+   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n+   appears earlier, than the opposite is true.\n+\n+3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n+   same MIDX layer. Sort `o1` and `o2` by their offset within their\n+   containing packfile.\n+\n+Note that the preferred pack is a property of the MIDX chain, not the\n+individual layers themselves. Fundamentally we could introduce a\n+per-layer preferred pack, but this is less relevant now that we can\n+perform multi-pack reuse across the set of packs in a MIDX.\n+\n+=== Reachability bitmaps and incremental MIDXs\n+\n+Each layer of an incremental MIDX chain may have its objects (and the\n+objects from any previous layer in the same MIDX chain) represented in\n+its own `*.bitmap` file.\n+\n+The structure of a `*.bitmap` file belonging to an incremental MIDX\n+chain is identical to that of a non-incremental MIDX bitmap, or a\n+classic single-pack bitmap. Since objects are added to the end of the\n+incremental MIDX's pseudo-pack order (see: above), it is possible to\n+extend a bitmap when appending to the end of a MIDX chain.\n+\n+(Note: it is possible likewise to compress a contiguous sequence of MIDX\n+incremental layers, and their `*.bitmap`(s) into a single layer and\n+`*.bitmap`, but this is not yet implemented.)\n+\n+The object positions used are global within the pseudo-pack order, so\n+subsequent layers will have, for example, `m->num_objects_in_base`\n+number of `0` bits in each of their four type bitmaps. This follows from\n+the fact that we only write type bitmap entries for objects present in\n+the layer immediately corresponding to the bitmap).\n+\n+Note also that only the bitmap pertaining to the most recent layer in an\n+incremental MIDX chain is used to store reachability information about\n+the interesting and uninteresting objects in a reachability query.\n+Earlier bitmap layers are only used to look up commit and pseudo-merge\n+bitmaps from that layer, as well as the type-level bitmaps for objects\n+in that layer.\n+\n+To simplify the implementation, type-level bitmaps are iterated\n+simultaneously, and their results are OR'd together to avoid recursively\n+calling internal bitmap functions.\n+\n Future Work\n -----------\n \n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514305","messageId":"f2a232e556a066c1a5083f02584ddc3637ecfc48.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:24Z","receivedAt":"2025-03-14T20:18:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the reverse index machinery to handle object lookups in an\nincremental MIDX bitmap. These changes are broken out across a few\nfunctions:\n\n  - load_midx_revindex() learns to use the appropriate MIDX filename\n    depending on whether the given 'struct multi_pack_index *' is\n    incremental or not.\n\n  - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n    object position in the MIDX pseudo-pack order, and finds the\n    earliest containing MIDX (similar to midx.c::midx_for_object().\n\n  - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n    number of objects in the base (since 'vb - midx->revindx_data' is\n    relative to the containing MIDX, and pack_pos_to_midx() expects a\n    global position).\n\n    Likewise, this function adjusts its output by adding\n    m->num_objects_in_base to return a global position out through the\n    `*pos` pointer.\n\nTogether, these changes are sufficient to use the multi-pack index's\nreverse index format for incremental multi-pack reachability bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c   | 40 ++++++++++++++++++++++++++++------------\n pack-revindex.c | 34 +++++++++++++++++++++++++---------\n 2 files changed, 53 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6406953d32..c26d85b5db 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n }\n \n+static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n+{\n+\tif (index->midx) {\n+\t\tstruct multi_pack_index *m = index->midx;\n+\t\treturn m->num_objects + m->num_objects_in_base;\n+\t}\n+\treturn index->pack->num_objects;\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\n@@ -924,7 +933,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\t\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -992,7 +1001,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -1342,11 +1351,17 @@ struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_g\n \t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n \t\t\tgoto done;\n \n+\t\t/*\n+\t\t * Use bitmap-relative positions instead of offsetting\n+\t\t * by bitmap_git->num_objects_in_base because we use\n+\t\t * this to find a match in pseudo_merge_for_parents(),\n+\t\t * and pseudo-merge groups cannot span multiple bitmap\n+\t\t * layers.\n+\t\t */\n \t\tbitmap_set(parents, pos);\n \t}\n \n-\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t\tparents);\n+\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges, parents);\n \n done:\n \tbitmap_free(parents);\n@@ -1500,7 +1515,8 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\tif (!bitmap_get(objects,\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -1679,7 +1695,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tsize_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n+\t\tsize_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(to_filter, pos) &&\n \t\t    !bitmap_get(tips, pos))\n@@ -1705,7 +1721,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < bitmap_num_objects(bitmap_git)) {\n+\tif (pos < bitmap_non_extended_bits(bitmap_git)) {\n \t\tstruct packed_git *pack;\n \t\toff_t ofs;\n \n@@ -1729,7 +1745,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n \t\t\t\t\t     &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n@@ -1882,7 +1898,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n \tuint32_t objects_nr;\n \tsize_t i, pos;\n \n-\tobjects_nr = bitmap_num_objects(bitmap_git);\n+\tobjects_nr = bitmap_non_extended_bits(bitmap_git);\n \tpos = objects_nr / BITS_IN_EWORD;\n \n \tif (pos > result->word_alloc)\n@@ -2419,7 +2435,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(objects,\n-\t\t\t       st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t       st_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcount++;\n \t}\n \n@@ -2820,7 +2836,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n \t\t    \"extension\");\n \n-\tnum_objects = bitmap_num_objects(bitmap_git);\n+\tnum_objects = bitmap_non_extended_bits(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -2963,7 +2979,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \t\tstruct object *obj = eindex->objects[i];\n \n \t\tif (!bitmap_get(result,\n-\t\t\t\tst_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex d3832478d9..d3faab6a37 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -383,8 +383,14 @@ int load_midx_revindex(struct multi_pack_index *m)\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t   \"source\", \"rev\");\n \n-\tget_midx_filename_ext(m->repo->hash_algo, &revindex_name, m->object_dir,\n-\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n+\tif (m->has_chain)\n+\t\tget_split_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n+\t\t\t\t\t    m->object_dir, get_midx_checksum(m),\n+\t\t\t\t\t    MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n+\t\t\t\t      m->object_dir, get_midx_checksum(m),\n+\t\t\t\t      MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n \t\t\t\t      m->num_objects,\n@@ -471,11 +477,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \n uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n {\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n \tif (!m->revindex_data)\n \t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n-\tif (m->num_objects <= pos)\n+\tif (m->num_objects + m->num_objects_in_base <= pos)\n \t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n-\treturn get_be32(m->revindex_data + pos);\n+\treturn get_be32(m->revindex_data + pos - m->num_objects_in_base);\n }\n \n struct midx_pack_key {\n@@ -491,7 +501,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n \tconst struct midx_pack_key *key = va;\n \tstruct multi_pack_index *midx = key->midx;\n \n-\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tsize_t pos = (uint32_t *)vb - (const uint32_t *)midx->revindex_data;\n+\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n \toff_t versus_offset;\n \n@@ -529,9 +540,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n {\n \tuint32_t *found;\n \n-\tif (key->pack >= m->num_packs)\n+\tif (key->pack >= m->num_packs + m->num_packs_in_base)\n \t\tBUG(\"MIDX pack lookup out of bounds (%\"PRIu32\" >= %\"PRIu32\")\",\n-\t\t    key->pack, m->num_packs);\n+\t\t    key->pack, m->num_packs + m->num_packs_in_base);\n \t/*\n \t * The preferred pack sorts first, so determine its identifier by\n \t * looking at the first object in pseudo-pack order.\n@@ -551,7 +562,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n \tif (!found)\n \t\treturn -1;\n \n-\t*pos = found - m->revindex_data;\n+\t*pos = (found - m->revindex_data) + m->num_objects_in_base;\n+\n \treturn 0;\n }\n \n@@ -559,9 +571,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n {\n \tstruct midx_pack_key key;\n \n+\twhile (m && at < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, at);\n \tif (!m->revindex_data)\n \t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n-\tif (m->num_objects <= at)\n+\tif (m->num_objects + m->num_objects_in_base <= at)\n \t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n \n \tkey.pack = nth_midxed_pack_int_id(m, at);\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514306","messageId":"aca0318fb12499ea810d03e66cc3145d682f5098.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:27Z","receivedAt":"2025-03-14T20:18:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the pack-bitmap machinery to work with incremental MIDXs by\nadding a new \"base\" field to keep track of the bitmap index associated\nwith the previous MIDX layer.\n\nThe changes in this commit are mostly boilerplate to open the correct\nbitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\nensures that the correct packs and their reverse indexes are loaded\nacross MIDX layers, etc.\n\nWhile we're at it, keep track of a base_nr field to indicate how many\nbitmap layers (including the current bitmap) exist. This will be used in\na future commit to allocate an array of 'struct ewah_bitmap' pointers to\ncollect all of the respective type bitmaps among all layers to\ninitialize a multi-EWAH iterator.\n\nSubsequent commits will teach the functions within the pack-bitmap\nmachinery how to interact with these new fields.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 62 +++++++++++++++++++++++++++++++++++++++------------\n 1 file changed, 48 insertions(+), 14 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c26d85b5db..72fb11d014 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -54,6 +54,16 @@ struct bitmap_index {\n \tstruct packed_git *pack;\n \tstruct multi_pack_index *midx;\n \n+\t/*\n+\t * If using a multi-pack index chain, 'base' points to the\n+\t * bitmap index corresponding to this bitmap's midx->base_midx.\n+\t *\n+\t * base_nr indicates how many layers precede this one, and is\n+\t * zero when base is NULL.\n+\t */\n+\tstruct bitmap_index *base;\n+\tuint32_t base_nr;\n+\n \t/* mmapped buffer of the whole bitmap index */\n \tunsigned char *map;\n \tsize_t map_size; /* size of the mmaped buffer */\n@@ -386,8 +396,15 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n char *midx_bitmap_filename(struct multi_pack_index *midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tget_midx_filename_ext(midx->repo->hash_algo, &buf, midx->object_dir,\n-\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n+\tif (midx->has_chain)\n+\t\tget_split_midx_filename_ext(midx->repo->hash_algo, &buf,\n+\t\t\t\t\t    midx->object_dir,\n+\t\t\t\t\t    get_midx_checksum(midx),\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(midx->repo->hash_algo, &buf,\n+\t\t\t\t      midx->object_dir, get_midx_checksum(midx),\n+\t\t\t\t      MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n }\n@@ -454,16 +471,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\tif (prepare_midx_pack(bitmap_repo(bitmap_git),\n-\t\t\t\t      bitmap_git->midx,\n-\t\t\t\t      i)) {\n+\tfor (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(bitmap_repo(bitmap_git), bitmap_git->midx, i)) {\n \t\t\twarning(_(\"could not open pack %s\"),\n \t\t\t\tbitmap_git->midx->pack_names[i]);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n+\tif (midx->base_midx) {\n+\t\tbitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n+\t\tbitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n+\t} else {\n+\t\tbitmap_git->base_nr = 0;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -515,6 +537,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tbitmap_git->map_pos = 0;\n+\tbitmap_git->base_nr = 0;\n \tclose(fd);\n \n \tif (load_bitmap_header(bitmap_git) < 0) {\n@@ -534,8 +557,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)\n {\n \tif (bitmap_is_midx(bitmap_git)) {\n-\t\tuint32_t i;\n-\t\tint ret;\n+\t\tstruct multi_pack_index *m;\n \n \t\t/*\n \t\t * The multi-pack-index's .rev file is already loaded via\n@@ -544,10 +566,15 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \t\t * But we still need to open the individual pack .rev files,\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tret = load_pack_revindex(r, bitmap_git->midx->packs[i]);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n+\t\tfor (m = bitmap_git->midx; m; m = m->base_midx) {\n+\t\t\tuint32_t i;\n+\t\t\tint ret;\n+\n+\t\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\tret = load_pack_revindex(r, m->packs[i]);\n+\t\t\t\tif (ret)\n+\t\t\t\t\treturn ret;\n+\t\t\t}\n \t\t}\n \t\treturn 0;\n \t}\n@@ -573,6 +600,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n+\tif (bitmap_git->base) {\n+\t\tif (!bitmap_is_midx(bitmap_git))\n+\t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n+\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\t\tgoto failed;\n+\t}\n+\n \treturn 0;\n \n failed:\n@@ -657,10 +691,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n \n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n {\n-\tstruct repository *r = midx->repo;\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n+\tif (!open_midx_bitmap_1(bitmap_git, midx))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2899,6 +2932,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\tclose_midx_revindex(b->midx);\n \t}\n \tfree_pseudo_merge_map(&b->pseudo_merges);\n+\tfree_bitmap_index(b->base);\n \tfree(b);\n }\n \n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514307","messageId":"832fd0e8dc3a37e36b3d59085e448f8de84ce4b4.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:31Z","receivedAt":"2025-03-14T20:18:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\nEWAH-compressed bitmap corresponding to some given commit object.\n\nTeach this function about incremental MIDX bitmaps by teaching it to\nrecur on earlier bitmap layers when it fails to find a given commit in\nthe current layer.\n\nThe changes to do so are as follows:\n\n  - Avoid initializing hash_pos at its declaration, since\n    bitmap_for_commit() is now a recursive function and may receive a\n    NULL bitmap_index pointer as its first argument.\n\n  - In cases where we would previously return NULL (to indicate that a\n    lookup failed and the given bitmap_index does not contain an entry\n    corresponding to the given commit), recursively call the function on\n    the previous bitmap layer.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 72fb11d014..615d5de85e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -941,18 +941,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t      struct commit *commit)\n {\n-\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n-\t\t\t\t\t   commit->object.oid);\n+\tkhiter_t hash_pos;\n+\tif (!bitmap_git)\n+\t\treturn NULL;\n+\n+\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514308","messageId":"c7c9f8995628f0b1de6c802688be934319508103.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 05/13] pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:34Z","receivedAt":"2025-03-14T20:18:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since we may ask for a pack_id that is in an earlier MIDX layer relative\nto the one corresponding to our bitmap, use nth_midxed_pack() instead of\naccessing the ->packs array directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 615d5de85e..1b4fec0033 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1635,7 +1635,7 @@ static void show_objects_for_type(\n \t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n \n \t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n-\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\t} else {\n \t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n \t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514309","messageId":"14d3d80c3d301233048004fe9de18154aa597782.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:37Z","receivedAt":"2025-03-14T20:18:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits in the first phase of\nincremental MIDXs, enumerate not just the packs in the current\nincremental MIDX layer, but previous ones as well.\n\nLikewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\nsingle pack from a MIDX, use the oldest layer's preferred pack as it is\nlikely to contain the most amount of reusable sections.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1b4fec0033..7a41535425 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2333,7 +2333,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tmulti_pack_reuse = 0;\n \n \tif (multi_pack_reuse) {\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\t\tstruct bitmapped_pack pack;\n \t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n@@ -2359,14 +2360,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tuint32_t pack_int_id;\n \n \t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n \t\t\tuint32_t preferred_pack_pos;\n \n-\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n+\t\t\twhile (m->base_midx)\n+\t\t\t\tm = m->base_midx;\n+\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n \t\t\t\treturn;\n \t\t\t}\n \n-\t\t\tpack = bitmap_git->midx->packs[preferred_pack_pos];\n+\t\t\tpack = nth_midxed_pack(m, preferred_pack_pos);\n \t\t\tpack_int_id = preferred_pack_pos;\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514310","messageId":"b45a9ccbc20180e3358e314b4fd5e46bfa566241.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:41Z","receivedAt":"2025-03-14T20:18:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement support for the special `--test-bitmap` mode of `git rev-list`\nwhen using incremental MIDXs.\n\nThe bitmap_test_data structure is extended to contain a \"base\" pointer\nthat mirrors the structure of the bitmap chain that it is being used to\ntest.\n\nWhen we find a commit to test, we first chase down the ->base pointer to\nfind the appropriate bitmap_test_data for the bitmap layer that the\ngiven commit is contained within, and then perform the test on that\nbitmap.\n\nIn order to implement this, light modifications are made to\nbitmap_for_commit() to reimplement it in terms of a new function,\nfind_bitmap_for_commit(), which fills out a pointer which indicates the\nbitmap layer which contains the given commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 107 ++++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 86 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 7a41535425..bb09ce3cf5 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -938,8 +938,9 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n \treturn NULL;\n }\n \n-struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n-\t\t\t\t      struct commit *commit)\n+static struct ewah_bitmap *find_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t  struct commit *commit,\n+\t\t\t\t\t\t  struct bitmap_index **found)\n {\n \tkhiter_t hash_pos;\n \tif (!bitmap_git)\n@@ -949,18 +950,30 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n+\t\tif (found)\n+\t\t\t*found = bitmap_git;\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n+\tif (found)\n+\t\t*found = bitmap_git;\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t      struct commit *commit)\n+{\n+\treturn find_bitmap_for_commit(bitmap_git, commit, NULL);\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n@@ -2511,6 +2524,8 @@ struct bitmap_test_data {\n \tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n+\n+\tstruct bitmap_test_data *base_tdata;\n };\n \n static void test_bitmap_type(struct bitmap_test_data *tdata,\n@@ -2519,6 +2534,11 @@ static void test_bitmap_type(struct bitmap_test_data *tdata,\n \tenum object_type bitmap_type = OBJ_NONE;\n \tint bitmaps_nr = 0;\n \n+\tif (bitmap_is_midx(tdata->bitmap_git)) {\n+\t\twhile (pos < tdata->bitmap_git->midx->num_objects_in_base)\n+\t\t\ttdata = tdata->base_tdata;\n+\t}\n+\n \tif (bitmap_get(tdata->commits, pos)) {\n \t\tbitmap_type = OBJ_COMMIT;\n \t\tbitmaps_nr++;\n@@ -2582,13 +2602,57 @@ static void test_show_commit(struct commit *commit, void *data)\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n+static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t total = 0;\n+\tdo {\n+\t\ttotal = st_add(total, bitmap_git->entry_count);\n+\t\tbitmap_git = bitmap_git->base;\n+\t} while (bitmap_git);\n+\n+\treturn total;\n+}\n+\n+static void bitmap_test_data_prepare(struct bitmap_test_data *tdata,\n+\t\t\t\t     struct bitmap_index *bitmap_git)\n+{\n+\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n+\n+\ttdata->bitmap_git = bitmap_git;\n+\ttdata->base = bitmap_new();\n+\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n+\n+\tif (bitmap_git->base) {\n+\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n+\t\tbitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n+\t}\n+}\n+\n+static void bitmap_test_data_release(struct bitmap_test_data *tdata)\n+{\n+\tif (!tdata)\n+\t\treturn;\n+\n+\tbitmap_test_data_release(tdata->base_tdata);\n+\tfree(tdata->base_tdata);\n+\n+\tbitmap_free(tdata->base);\n+\tbitmap_free(tdata->commits);\n+\tbitmap_free(tdata->trees);\n+\tbitmap_free(tdata->blobs);\n+\tbitmap_free(tdata->tags);\n+}\n+\n void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n-\tstruct bitmap_index *bitmap_git;\n+\tstruct bitmap_index *bitmap_git, *found;\n \tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n@@ -2597,17 +2661,28 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n \t\tbitmap_git->version,\n \t\tbitmap_git->entry_count,\n-\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n+\t\tbitmap_total_entry_count(bitmap_git));\n \n \troot = revs->pending.objects[0].item;\n-\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n+\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n \n \tif (bm) {\n \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n-\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n+\t\t\toid_to_hex(&root->oid),\n+\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n+\n+\t\tif (bitmap_is_midx(found))\n+\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n+\t\t\t\t   hash_to_hex_algop(get_midx_checksum(found->midx),\n+\t\t\t\t\t\t     revs->repo->hash_algo));\n+\t\telse\n+\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n+\t\t\t\t   hash_to_hex_algop(found->pack->hash,\n+\t\t\t\t\t\t     revs->repo->hash_algo));\n \n \t\tresult = ewah_to_bitmap(bm);\n \t}\n@@ -2624,16 +2699,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n-\ttdata.bitmap_git = bitmap_git;\n-\ttdata.base = bitmap_new();\n-\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n-\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n-\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n-\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n+\tbitmap_test_data_prepare(&tdata, bitmap_git);\n \ttdata.prg = start_progress(revs->repo,\n \t\t\t\t   \"Verifying bitmap entries\",\n \t\t\t\t   result_popcnt);\n-\ttdata.seen = 0;\n \n \ttraverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);\n \n@@ -2645,11 +2714,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tdie(_(\"mismatch in bitmap results\"));\n \n \tbitmap_free(result);\n-\tbitmap_free(tdata.base);\n-\tbitmap_free(tdata.commits);\n-\tbitmap_free(tdata.trees);\n-\tbitmap_free(tdata.blobs);\n-\tbitmap_free(tdata.tags);\n+\tbitmap_test_data_release(&tdata);\n \tfree_bitmap_index(bitmap_git);\n }\n \n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514311","messageId":"c1eefeae993f2c8778dd0ec9626f977dbb2011a2.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 08/13] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:44Z","receivedAt":"2025-03-14T20:18:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, use nth_midxed_pack() instead\nof accessing the MIDX's ->packs array directly to support incremental\nMIDXs.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex bb09ce3cf5..8442f8e55f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1778,7 +1778,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n \t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n \n-\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n@@ -3047,7 +3047,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \n \t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n-\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tstruct packed_git *pack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \n \t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n \t\t\t\t\tstruct object_id oid;\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514312","messageId":"11c4b7b9491c9b34a441b941902a6c29fde912dc.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 09/13] pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:47Z","receivedAt":"2025-03-14T20:18:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for using pseudo-merges with incremental MIDX bitmaps by\nattempting to apply pseudo-merges from each layer when encountering a\ngiven commit during a walk.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 8442f8e55f..00acf5ec73 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1082,10 +1082,15 @@ static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git\n \t\t\t\t\t\t struct commit *commit,\n \t\t\t\t\t\t uint32_t commit_pos)\n {\n-\tint ret;\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tint ret = 0;\n \n-\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t     result, commit, commit_pos);\n+\twhile (curr) {\n+\t\tret += apply_pseudo_merges_for_commit(&curr->pseudo_merges,\n+\t\t\t\t\t\t      result, commit,\n+\t\t\t\t\t\t      commit_pos);\n+\t\tcurr = curr->base;\n+\t}\n \n \tif (ret)\n \t\tpseudo_merges_satisfied_nr += ret;\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514313","messageId":"cb08ad6a62092b9521a11c0524f23bd8831c0409.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:50Z","receivedAt":"2025-03-14T20:18:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"While individual bitmap layers store different commit, type-level, and\npseudo-merge bitmaps, only the top-most layer is used to compute\nreachability traversals.\n\nMany functions which implement the aforementioned traversal rely on\nenumerating the results according to the type-level bitmaps, and so\nwould benefit from a conceptual type-level bitmap that spans multiple\nlayers.\n\nImplement `struct ewah_or_iterator` which is capable of enumerating\nmultiple EWAH bitmaps at once, and OR-ing the results together. When\ninitialized with, for example, all of the commit type bitmaps from each\nlayer, callers can pretend as if they are enumerating a large type-level\nbitmap which contains the commits from *all* bitmap layers.\n\nThere are a couple of alternative approaches which were considered:\n\n  - Decompress each EWAH bitmap and OR them together, enumerating a\n    single (non-EWAH) bitmap. This would work, but has the disadvantage\n    of decompressing a potentially large bitmap, which may not be\n    necessary if the caller does not wish to read all of it.\n\n  - Recursively call bitmap internal functions, reusing the \"result\" and\n    \"haves\" bitmap from the top-most layer. This approach resembles the\n    original implementation of this feature, but is inefficient in that\n    it both (a) requires significant refactoring to implement, and (b)\n    enumerates large sections of later bitmaps which are all zeros (as\n    they pertain to objects in earlier layers).\n\n    (b) is not so bad in and of itself, but can cause significant\n    slow-downs when combined with expensive loop bodies.\n\nThis approach (enumerating an OR'd together version of all of the\ntype-level bitmaps from each layer) produces a significantly more\nstraightforward implementation with significantly less refactoring\nrequired in order to make it work.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ewah/ewah_bitmap.c | 33 +++++++++++++++++++++++++++++++++\n ewah/ewok.h        | 12 ++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex 67f8f588e0..e92341b8fa 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -371,6 +371,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n \t\tread_new_rlw(it);\n }\n \n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr)\n+{\n+\tsize_t i;\n+\n+\tmemset(it, 0, sizeof(*it));\n+\n+\tALLOC_ARRAY(it->its, nr);\n+\tfor (i = 0; i < nr; i++)\n+\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n+}\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n+{\n+\teword_t buf, out = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tfor (i = 0; i < it->nr; i++)\n+\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n+\t\t\tout |= buf;\n+\t\t\tret = 1;\n+\t\t}\n+\n+\t*next = out;\n+\treturn ret;\n+}\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it)\n+{\n+\tfree(it->its);\n+}\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 5e357e2493..4b70641045 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -148,6 +148,18 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent);\n  */\n int ewah_iterator_next(eword_t *next, struct ewah_iterator *it);\n \n+struct ewah_or_iterator {\n+\tstruct ewah_iterator *its;\n+\tsize_t nr;\n+};\n+\n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr);\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n+\n+void ewah_or_iterator_free(struct ewah_or_iterator *it);\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514314","messageId":"a29f4ee60d519318d36a8d3c812b4bad039b891e.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:53Z","receivedAt":"2025-03-14T20:18:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for reading the type-level bitmaps from previous bitmap layers\nby maintaining an array for each type, where each element in that type's\narray corresponds to one layer's bitmap for that type.\n\nThese fields will be used in a later commit to instantiate the 'struct\newah_or_iterator' for each type.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 54 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 00acf5ec73..3517972892 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -81,6 +81,24 @@ struct bitmap_index {\n \tstruct ewah_bitmap *blobs;\n \tstruct ewah_bitmap *tags;\n \n+\t/*\n+\t * Type index arrays when this bitmap is associated with an\n+\t * incremental multi-pack index chain.\n+\t *\n+\t * If n is the number of unique layers in the MIDX chain, then\n+\t * commits_all[n-1] is this structs 'commits' field,\n+\t * commits_all[n-2] is the commits field of this bitmap's\n+\t * 'base', and so on.\n+\t *\n+\t * When either associated either with a non-incremental MIDX, or\n+\t * a single packfile, these arrays each contain a single\n+\t * element.\n+\t */\n+\tstruct ewah_bitmap **commits_all;\n+\tstruct ewah_bitmap **trees_all;\n+\tstruct ewah_bitmap **blobs_all;\n+\tstruct ewah_bitmap **tags_all;\n+\n \t/* Map from object ID -> `stored_bitmap` for all the bitmapped commits */\n \tkh_oid_map_t *bitmaps;\n \n@@ -581,7 +599,32 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \treturn load_pack_revindex(r, bitmap_git->pack);\n }\n \n-static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n+static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n+{\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tsize_t i = bitmap_git->base_nr;\n+\n+\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);\n+\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);\n+\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);\n+\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr + 1);\n+\n+\twhile (curr) {\n+\t\tbitmap_git->commits_all[i] = curr->commits;\n+\t\tbitmap_git->trees_all[i] = curr->trees;\n+\t\tbitmap_git->blobs_all[i] = curr->blobs;\n+\t\tbitmap_git->tags_all[i] = curr->tags;\n+\n+\t\tcurr = curr->base;\n+\t\tif (curr && !i)\n+\t\t\tBUG(\"unexpected number of bitmap layers, expected %\"PRIu32,\n+\t\t\t    bitmap_git->base_nr + 1);\n+\t\ti -= 1;\n+\t}\n+}\n+\n+static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git,\n+\t\t       int recursing)\n {\n \tassert(bitmap_git->map);\n \n@@ -603,10 +646,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (bitmap_git->base) {\n \t\tif (!bitmap_is_midx(bitmap_git))\n \t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n-\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\tif (load_bitmap(r, bitmap_git->base, 1) < 0)\n \t\t\tgoto failed;\n \t}\n \n+\tif (!recursing)\n+\t\tload_all_type_bitmaps(bitmap_git);\n+\n \treturn 0;\n \n failed:\n@@ -682,7 +728,7 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git, 0))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2050,7 +2096,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (load_bitmap(revs->repo, bitmap_git, 0) < 0)\n \t\tgoto cleanup;\n \n \tif (!use_boundary_traversal)\n@@ -2983,6 +3029,10 @@ void free_bitmap_index(struct bitmap_index *b)\n \tewah_pool_free(b->trees);\n \tewah_pool_free(b->blobs);\n \tewah_pool_free(b->tags);\n+\tfree(b->commits_all);\n+\tfree(b->trees_all);\n+\tfree(b->blobs_all);\n+\tfree(b->tags_all);\n \tif (b->bitmaps) {\n \t\tstruct stored_bitmap *sb;\n \t\tkh_foreach_value(b->bitmaps, sb, {\n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514315","messageId":"a1cf65bedc94ab6e318ff81a6d48eb30b6fc7868.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:18:56Z","receivedAt":"2025-03-14T20:18:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have initialized arrays for each bitmap layer's type bitmaps\nin the previous commit, adjust existing callers to use them in\npreparation for multi-layered bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 42 +++++++++++++++++++++++++++---------------\n 1 file changed, 27 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3517972892..5e6d4ace58 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1629,25 +1629,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t}\n }\n \n-static void init_type_iterator(struct ewah_iterator *it,\n+static void init_type_iterator(struct ewah_or_iterator *it,\n \t\t\t       struct bitmap_index *bitmap_git,\n \t\t\t       enum object_type type)\n {\n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tewah_iterator_init(it, bitmap_git->commits);\n+\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tcase OBJ_TREE:\n-\t\tewah_iterator_init(it, bitmap_git->trees);\n+\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tcase OBJ_BLOB:\n-\t\tewah_iterator_init(it, bitmap_git->blobs);\n+\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tcase OBJ_TAG:\n-\t\tewah_iterator_init(it, bitmap_git->tags);\n+\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tdefault:\n@@ -1664,7 +1668,7 @@ static void show_objects_for_type(\n \tsize_t i = 0;\n \tuint32_t offset;\n \n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tstruct bitmap *objects = bitmap_git->result;\n@@ -1672,7 +1676,7 @@ static void show_objects_for_type(\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = objects->words[i] & filter;\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n \n@@ -1714,6 +1718,8 @@ static void show_objects_for_type(\n \t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n+\n+\tewah_or_iterator_free(&it);\n }\n \n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n@@ -1765,7 +1771,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n@@ -1782,7 +1788,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * packfile.\n \t */\n \tfor (i = 0, init_type_iterator(&it, bitmap_git, type);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\tif (i < tips->word_alloc)\n \t\t\tmask &= ~tips->words[i];\n@@ -1802,6 +1808,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -1861,14 +1868,14 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n \ttips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB);\n \n \tfor (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\teword_t word = to_filter->words[i] & mask;\n \t\tunsigned offset;\n@@ -1896,6 +1903,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_free(&it);\n \tbitmap_free(tips);\n }\n \n@@ -2527,12 +2535,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \n \tuint32_t i = 0, count = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tinit_type_iterator(&it, bitmap_git, type);\n \n-\twhile (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) {\n+\twhile (i < objects->word_alloc && ewah_or_iterator_next(&filter, &it)) {\n \t\teword_t word = objects->words[i++] & filter;\n \t\tcount += ewah_bit_popcount64(word);\n \t}\n@@ -2544,6 +2552,8 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \t\t\tcount++;\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn count;\n }\n \n@@ -3076,13 +3086,13 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *result = bitmap_git->result;\n \toff_t total = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \tsize_t i;\n \n \tinit_type_iterator(&it, bitmap_git, object_type);\n \tfor (i = 0; i < result->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = result->words[i] & filter;\n \t\tsize_t base = (i * BITS_IN_EWORD);\n \t\tunsigned offset;\n@@ -3123,6 +3133,8 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n+\tewah_or_iterator_free(&it);\n+\n \treturn total;\n }\n \n-- \n2.49.0.13.gd0d564685b\n\n"},{"id":"514316","messageId":"d0d564685bc66df71b4c3ed1093452e58e0f54fd.1741983492.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"[PATCH v4 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-14T20:19:00Z","receivedAt":"2025-03-14T20:19:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap machinery has learned how to read and interact\nwith an incremental MIDX bitmap, teach the pack-bitmap-write.c machinery\n(and relevant callers from within the MIDX machinery) to write such\nbitmaps.\n\nThe details for doing so are mostly straightforward. The main changes\nare as follows:\n\n  - find_object_pos() now makes use of an extra MIDX parameter which is\n    used to locate the bit positions of objects which are from previous\n    layers (and thus do not exist in the current layer's pack_order\n    field).\n\n    (Note also that the pack_order field is moved into struct\n    write_midx_context to further simplify the callers for\n    write_midx_bitmap()).\n\n  - bitmap_writer_build_type_index() first determines how many objects\n    precede the current bitmap layer and offsets the bits it sets in\n    each respective type-level bitmap by that amount so they can be OR'd\n    together.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c                  |  3 +-\n ewah/ewah_bitmap.c                      |  2 +-\n ewah/ewok.h                             |  2 +-\n midx-write.c                            | 57 +++++++++++------\n pack-bitmap-write.c                     | 65 ++++++++++++++-----\n pack-bitmap.c                           | 10 +--\n pack-bitmap.h                           |  4 +-\n t/t5334-incremental-multi-pack-index.sh | 84 +++++++++++++++++++++++++\n 8 files changed, 183 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 58a9b16126..a7e4bb7904 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1397,7 +1397,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_init(&bitmap_writer,\n-\t\t\t\t\t\t   the_repository, &to_pack);\n+\t\t\t\t\t\t   the_repository, &to_pack,\n+\t\t\t\t\t\t   NULL);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t\t\t       written_list);\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex e92341b8fa..056c410efb 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -399,7 +399,7 @@ int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n \treturn ret;\n }\n \n-void ewah_or_iterator_free(struct ewah_or_iterator *it)\n+void ewah_or_iterator_release(struct ewah_or_iterator *it)\n {\n \tfree(it->its);\n }\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 4b70641045..c29d354236 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -158,7 +158,7 @@ void ewah_or_iterator_init(struct ewah_or_iterator *it,\n \n int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n \n-void ewah_or_iterator_free(struct ewah_or_iterator *it);\n+void ewah_or_iterator_release(struct ewah_or_iterator *it);\n \n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\ndiff --git a/midx-write.c b/midx-write.c\nindex 48d6558253..0897cbd829 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -647,16 +647,22 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \treturn pack_order;\n }\n \n-static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n-\t\t\t\t     struct write_midx_context *ctx)\n+static void write_midx_reverse_index(struct write_midx_context *ctx,\n+\t\t\t\t     const char *object_dir,\n+\t\t\t\t     unsigned char *midx_hash)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tchar *tmp_file;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", ctx->repo);\n \n-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t\t    ctx->repo->hash_algo));\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(ctx->repo->hash_algo, &buf,\n+\t\t\t\t\t    object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(ctx->repo->hash_algo, &buf, object_dir,\n+\t\t\t\t      midx_hash, MIDX_EXT_REV);\n \n \ttmp_file = write_rev_file_order(ctx->repo->hash_algo, NULL, ctx->pack_order,\n \t\t\t\t\tctx->entries_nr, midx_hash, WRITE_REV);\n@@ -829,22 +835,29 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \treturn cb.commits;\n }\n \n-static int write_midx_bitmap(struct repository *r, const char *midx_name,\n+static int write_midx_bitmap(struct write_midx_context *ctx,\n+\t\t\t     const char *object_dir,\n \t\t\t     const unsigned char *midx_hash,\n \t\t\t     struct packing_data *pdata,\n \t\t\t     struct commit **commits,\n \t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n \t\t\t     unsigned flags)\n {\n \tint ret, i;\n \tuint16_t options = 0;\n \tstruct bitmap_writer writer;\n \tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t    hash_to_hex_algop(midx_hash, r->hash_algo));\n+\tstruct strbuf bitmap_name = STRBUF_INIT;\n \n-\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", r);\n+\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", ctx->repo);\n+\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,\n+\t\t\t\t\t    object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,\n+\t\t\t\t      object_dir, midx_hash, MIDX_EXT_BITMAP);\n \n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n@@ -861,7 +874,8 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer, r, pdata);\n+\tbitmap_writer_init(&writer, ctx->repo, pdata,\n+\t\t\t   ctx->incremental ? ctx->base_midx : NULL);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, index);\n \n@@ -879,7 +893,7 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,\n \t * bitmap_writer_finish().\n \t */\n \tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\t\tindex[ctx->pack_order[i]] = &pdata->objects[i].idx;\n \n \tbitmap_writer_select_commits(&writer, commits, commits_nr);\n \tret = bitmap_writer_build(&writer);\n@@ -887,14 +901,14 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,\n \t\tgoto cleanup;\n \n \tbitmap_writer_set_checksum(&writer, midx_hash);\n-\tbitmap_writer_finish(&writer, index, bitmap_name, options);\n+\tbitmap_writer_finish(&writer, index, bitmap_name.buf, options);\n \n cleanup:\n \tfree(index);\n-\tfree(bitmap_name);\n+\tstrbuf_release(&bitmap_name);\n \tbitmap_writer_free(&writer);\n \n-\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", r);\n+\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", ctx->repo);\n \n \treturn ret;\n }\n@@ -1077,8 +1091,6 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \tctx.repo = r;\n \n \tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n-\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n-\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1119,6 +1131,13 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n+\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n+\t\t\t\t      hash_to_hex_algop(get_midx_checksum(m),\n+\t\t\t\t\t\t\tm->repo->hash_algo));\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n@@ -1387,7 +1406,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \n \tif (flags & MIDX_WRITE_REV_INDEX &&\n \t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n-\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n+\t\twrite_midx_reverse_index(&ctx, object_dir, midx_hash);\n \n \tif (flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n@@ -1410,8 +1429,8 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(r, midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\tif (write_midx_bitmap(&ctx, object_dir,\n+\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n \t\t\t\t      flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 34e86d4994..8a30853d2e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -26,6 +26,8 @@\n #include \"alloc.h\"\n #include \"refs.h\"\n #include \"strmap.h\"\n+#include \"midx.h\"\n+#include \"pack-revindex.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -43,7 +45,8 @@ static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer\n }\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata)\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n \tif (writer->bitmaps)\n@@ -51,6 +54,7 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n \twriter->bitmaps = kh_init_oid_map();\n \twriter->pseudo_merge_commits = kh_init_oid_map();\n \twriter->to_pack = pdata;\n+\twriter->midx = midx;\n \n \tstring_list_init_dup(&writer->pseudo_merge_groups);\n \n@@ -113,6 +117,11 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t    struct pack_idx_entry **index)\n {\n \tuint32_t i;\n+\tuint32_t base_objects = 0;\n+\n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n \n \twriter->commits = ewah_new();\n \twriter->trees = ewah_new();\n@@ -142,19 +151,19 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \n \t\tswitch (real_type) {\n \t\tcase OBJ_COMMIT:\n-\t\t\tewah_set(writer->commits, i);\n+\t\t\tewah_set(writer->commits, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TREE:\n-\t\t\tewah_set(writer->trees, i);\n+\t\t\tewah_set(writer->trees, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_BLOB:\n-\t\t\tewah_set(writer->blobs, i);\n+\t\t\tewah_set(writer->blobs, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TAG:\n-\t\t\tewah_set(writer->tags, i);\n+\t\t\tewah_set(writer->tags, i + base_objects);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -207,19 +216,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n static uint32_t find_object_pos(struct bitmap_writer *writer,\n \t\t\t\tconst struct object_id *oid, int *found)\n {\n-\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n+\tstruct object_entry *entry;\n+\n+\tentry = packlist_find(writer->to_pack, oid);\n+\tif (entry) {\n+\t\tuint32_t base_objects = 0;\n+\t\tif (writer->midx)\n+\t\t\tbase_objects = writer->midx->num_objects +\n+\t\t\t\twriter->midx->num_objects_in_base;\n+\n+\t\tif (found)\n+\t\t\t*found = 1;\n+\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n+\t} else if (writer->midx) {\n+\t\tuint32_t at, pos;\n+\n+\t\tif (!bsearch_midx(oid, writer->midx, &at))\n+\t\t\tgoto missing;\n+\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n+\t\t\tgoto missing;\n \n-\tif (!entry) {\n \t\tif (found)\n-\t\t\t*found = 0;\n-\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n-\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n-\t\treturn 0;\n+\t\t\t*found = 1;\n+\t\treturn pos;\n \t}\n \n+missing:\n \tif (found)\n-\t\t*found = 1;\n-\treturn oe_in_pack_pos(writer->to_pack, entry);\n+\t\t*found = 0;\n+\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\treturn 0;\n }\n \n static void compute_xor_offsets(struct bitmap_writer *writer)\n@@ -586,7 +613,7 @@ int bitmap_writer_build(struct bitmap_writer *writer)\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n-\tuint32_t *mapping;\n+\tuint32_t *mapping = NULL;\n \tint closed = 1; /* until proven otherwise */\n \n \tif (writer->show_progress)\n@@ -1021,7 +1048,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \toff_t *offsets = NULL;\n-\tuint32_t i;\n+\tuint32_t i, base_objects;\n \n \tstruct bitmap_disk_header header;\n \n@@ -1047,6 +1074,12 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, writer->to_pack->nr_objects);\n \n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n+\telse\n+\t\tbase_objects = 0;\n+\n \tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n@@ -1055,7 +1088,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \t\tif (commit_pos < 0)\n \t\t\tBUG(_(\"trying to write commit not in index\"));\n-\t\tstored->commit_pos = commit_pos;\n+\t\tstored->commit_pos = commit_pos + base_objects;\n \t}\n \n \twrite_selected_commits_v1(writer, f, offsets);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5e6d4ace58..94d1e8474a 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1719,7 +1719,7 @@ static void show_objects_for_type(\n \t\t}\n \t}\n \n-\tewah_or_iterator_free(&it);\n+\tewah_or_iterator_release(&it);\n }\n \n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n@@ -1808,7 +1808,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n-\tewah_or_iterator_free(&it);\n+\tewah_or_iterator_release(&it);\n \tbitmap_free(tips);\n }\n \n@@ -1903,7 +1903,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n-\tewah_or_iterator_free(&it);\n+\tewah_or_iterator_release(&it);\n \tbitmap_free(tips);\n }\n \n@@ -2552,7 +2552,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \t\t\tcount++;\n \t}\n \n-\tewah_or_iterator_free(&it);\n+\tewah_or_iterator_release(&it);\n \n \treturn count;\n }\n@@ -3133,7 +3133,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n-\tewah_or_iterator_free(&it);\n+\tewah_or_iterator_release(&it);\n \n \treturn total;\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex d7f4b8b8e9..dd0951088f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -111,6 +111,7 @@ struct bitmap_writer {\n \n \tkh_oid_map_t *bitmaps;\n \tstruct packing_data *to_pack;\n+\tstruct multi_pack_index *midx; /* if appending to a MIDX chain */\n \n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n@@ -125,7 +126,8 @@ struct bitmap_writer {\n };\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata);\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 26257e5660..46d1f0b864 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -44,4 +44,88 @@ test_expect_success 'convert incremental to non-incremental' '\n \n compare_results_with_midx 'non-incremental MIDX conversion'\n \n+write_midx_layer () {\n+\tn=1\n+\tif test -f $midx_chain\n+\tthen\n+\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n+\tfi\n+\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit $n.$i &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit multi-pack-index write --bitmap --incremental\n+}\n+\n+test_expect_success 'write initial MIDX layer' '\n+\tgit repack -ad &&\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'read bitmap from first MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'write another MIDX layer' '\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'midx verify with multiple layers' '\n+\tgit multi-pack-index verify\n+'\n+\n+test_expect_success 'read bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 2.2\n+'\n+\n+test_expect_success 'read earlier bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'show object from first pack' '\n+\tgit cat-file -p 1.1\n+'\n+\n+test_expect_success 'show object from second pack' '\n+\tgit cat-file -p 2.2\n+'\n+\n+for reuse in false single multi\n+do\n+\ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n+\t\trm -fr clone.git &&\n+\n+\t\tgit config pack.allowPackReuse $reuse &&\n+\t\tgit clone --no-local --bare . clone.git\n+\t'\n+done\n+\n+test_expect_success 'relink existing MIDX layer' '\n+\trm -fr \"$midxdir\" &&\n+\n+\tGIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n+\n+\tmidx_hash=\"$(test-tool read-midx --checksum $objdir)\" &&\n+\n+\ttest_path_is_file \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_commit another &&\n+\tgit repack -d &&\n+\tgit multi-pack-index write --bitmap --incremental &&\n+\n+\ttest_path_is_missing \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.midx\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.rev\" &&\n+\ttest_line_count = 2 \"$midx_chain\"\n+\n+'\n+\n test_done\n-- \n2.49.0.13.gd0d564685b\n"},{"id":"514479","messageId":"20250318011618.GA1471939@coredump.intra.peff.net","threadId":"61958","inReplyTo":"f565f2fff166bdf4bb2505f4a8853215a5365b17.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T01:16:18Z","receivedAt":"2025-03-18T01:16:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:20PM -0400, Taylor Blau wrote:\n\n> +In the incremental MIDX design, we extend this definition to include\n> +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> +for incremental MIDXs is determined by concatenating the pseudo-pack\n> +ordering for each layer of the MIDX chain in order. Formally two objects\n> +`o1` and `o2` are compared as follows:\n> +\n> +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> +  `o1` is considered less than `o2`.\n> +\n> +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n> +   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n> +   preferred and the other is not, then the preferred one sorts first. If\n> +   there is a base layer (i.e. the MIDX layer is not the first layer in\n> +   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n> +   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n> +   appears earlier, than the opposite is true.\n> +\n> +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n> +   same MIDX layer. Sort `o1` and `o2` by their offset within their\n> +   containing packfile.\n\nOK, I think this ordering makes sense. I had to read this description\nover several times to make sure I wasn't missing something. The earlier\npart that says \"it's just concatenating the pack order of the layers\" is\na much more intuitive way of looking at it (modulo that you might need\nto remove duplicates found in earlier layers).\n\nBut I think an even more basic way of thinking about it is that it's the\nsame as the pseudo-pack order you would get if you had a single midx of\nall of the packs in all of the layers (in their layer order). We already\nhave to deal with (and have documented) duplicates in that case.\n\nNot really suggesting any wording change here, just making sure I\ngrokked it all.\n\n> +Note that the preferred pack is a property of the MIDX chain, not the\n> +individual layers themselves. Fundamentally we could introduce a\n> +per-layer preferred pack, but this is less relevant now that we can\n> +perform multi-pack reuse across the set of packs in a MIDX.\n\nCalling this out explicitly is good, since it's an obvious question\nfor somebody to have.\n\n> +=== Reachability bitmaps and incremental MIDXs\n> +\n> +Each layer of an incremental MIDX chain may have its objects (and the\n> +objects from any previous layer in the same MIDX chain) represented in\n> +its own `*.bitmap` file.\n> +\n> +The structure of a `*.bitmap` file belonging to an incremental MIDX\n> +chain is identical to that of a non-incremental MIDX bitmap, or a\n> +classic single-pack bitmap. Since objects are added to the end of the\n> +incremental MIDX's pseudo-pack order (see: above), it is possible to\n> +extend a bitmap when appending to the end of a MIDX chain.\n> +\n> +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n> +incremental layers, and their `*.bitmap`(s) into a single layer and\n> +`*.bitmap`, but this is not yet implemented.)\n> +\n> +The object positions used are global within the pseudo-pack order, so\n> +subsequent layers will have, for example, `m->num_objects_in_base`\n> +number of `0` bits in each of their four type bitmaps. This follows from\n> +the fact that we only write type bitmap entries for objects present in\n> +the layer immediately corresponding to the bitmap).\n\nOK, so each layer's bitmap does depend on the layers above/before it.\nThat obviously needs to happen because each incremental midx is not\nlikely to be a complete reachability set anyway.\n\nBut I also wondered what would happen with a situation like this:\n\n  A -- B\n   \\\n    -- C\n\nstored like this:\n\n  base midx:\n    - pack 1:\n      - object A\n      - object B, which can reach A\n  incremental midx:\n    - pack 2:\n      - object A\n      - object C, which can reach A\n\nThat is, two objects B and C both depend on A, which is duplicated in\ntwo midx layers. Even if the incremental midx is complete in the sense\nthat C only depends on A, its bitmap cannot just be \"11\". Because the\nbit position for object A in the incremental midx does not exist in the\npseudo-pack order at all! It must refer to the copy of \"A\" in the base\nmidx, so it's correct bitmap is \"101\" (A and C, but not B).\n\nAgain, just talking through it here.\n\n> +Note also that only the bitmap pertaining to the most recent layer in an\n> +incremental MIDX chain is used to store reachability information about\n> +the interesting and uninteresting objects in a reachability query.\n> +Earlier bitmap layers are only used to look up commit and pseudo-merge\n> +bitmaps from that layer, as well as the type-level bitmaps for objects\n> +in that layer.\n\nI'm not quite sure what this means, but I guess you're saying that\ninternally as we produce a bitmap, we'll always use the complete bitmap\nover all of the layers?\n\n> +To simplify the implementation, type-level bitmaps are iterated\n> +simultaneously, and their results are OR'd together to avoid recursively\n> +calling internal bitmap functions.\n\nOK, I guess we'll see what this means in the patches. ;)\n\nThe general rules for the data structure make sense to me, though.\n\n-Peff\n"},{"id":"514480","messageId":"20250318012726.GB1471939@coredump.intra.peff.net","threadId":"61958","inReplyTo":"f2a232e556a066c1a5083f02584ddc3637ecfc48.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T01:27:26Z","receivedAt":"2025-03-18T01:27:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:24PM -0400, Taylor Blau wrote:\n\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 6406953d32..c26d85b5db 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n>  \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n>  }\n>  \n> +static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n> +{\n> +\tif (index->midx) {\n> +\t\tstruct multi_pack_index *m = index->midx;\n> +\t\treturn m->num_objects + m->num_objects_in_base;\n> +\t}\n> +\treturn index->pack->num_objects;\n> +}\n\nI understand why we need to account for the objects in the base to\noffset our total size.\n\nSimilar to Patrick's comments on v3, I wondered about why we couldn't\njust modify bitmap_num_objects() here, and why some callers would be\nleft with the other.\n\nI guess sometimes we still need to consider a single layer. We can't\nquite just access m->num_objects there, because we still need the midx\nvs pack abstraction layer. I just thought there'd be more discussion\nhere, but it looks the same as v3.\n\nI wonder if it is worth renaming bitmap_num_objects() to indicate that\nit is a single layer (and make sure other callers are examined). I\ndunno.\n\nI also suspect from previous forays into bitmap indexing that it will be\neasy to mix up positions in various units (local to the layer vs in the\nglobal pseudo-pack ordering, for example). In theory we could use types\nto help us with this, but they're kind of weak in C (unless we wrap all\nof the ints in structs). Maybe not worth it.\n\n-Peff\n"},{"id":"514481","messageId":"20250318013823.GC1471939@coredump.intra.peff.net","threadId":"61958","inReplyTo":"832fd0e8dc3a37e36b3d59085e448f8de84ce4b4.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T01:38:23Z","receivedAt":"2025-03-18T01:38:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:31PM -0400, Taylor Blau wrote:\n\n> The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\n> EWAH-compressed bitmap corresponding to some given commit object.\n> \n> Teach this function about incremental MIDX bitmaps by teaching it to\n> recur on earlier bitmap layers when it fails to find a given commit in\n> the current layer.\n> \n> The changes to do so are as follows:\n> \n>   - Avoid initializing hash_pos at its declaration, since\n>     bitmap_for_commit() is now a recursive function and may receive a\n>     NULL bitmap_index pointer as its first argument.\n> \n>   - In cases where we would previously return NULL (to indicate that a\n>     lookup failed and the given bitmap_index does not contain an entry\n>     corresponding to the given commit), recursively call the function on\n>     the previous bitmap layer.\n\nThis makes sense, though it does make me wonder if we could/should store\na (midx/pack,pos) pair. I.e., a master hash table stored once for the\nwhole midx stack. And then you wouldn't need to recurse; it would just\nbe a single lookup.\n\nOr would that work badly with the lazy nature? You'd need to load all of\nthe layers to fill it (rather than doing each incrementally). OTOH, if\nyou ask for the bitmap for commit X you're eventually going to have to\nfigure out what's in all of the layers as soon as you have a miss and\nhave to check them all. And I think the lookup table extension is what's\nsupposed to make that cheap-ish.\n\nI dunno. It has been a long time since I dug into this (and the whole\nkhash-of-commits thing is just a hack around the lack of the lookup\ntable in the first place, I think?).\n\n-Peff\n"},{"id":"514482","messageId":"20250318014139.GD1471939@coredump.intra.peff.net","threadId":"61958","inReplyTo":"c1eefeae993f2c8778dd0ec9626f977dbb2011a2.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 08/13] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T01:41:39Z","receivedAt":"2025-03-18T01:41:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:44PM -0400, Taylor Blau wrote:\n\n> In a similar fashion as previous commits, use nth_midxed_pack() instead\n> of accessing the MIDX's ->packs array directly to support incremental\n> MIDXs.\n\nProbably not worth it to change it in an actual patch, but is it worth\nrenaming midx->packs to something else to make sure we catch all of the\nspots that need to be considered? Or maybe you already did that, which\nis how you found all of these. :)\n\n-Peff\n"},{"id":"514483","messageId":"20250318014417.GE1471939@coredump.intra.peff.net","threadId":"61958","inReplyTo":"cb08ad6a62092b9521a11c0524f23bd8831c0409.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T01:44:17Z","receivedAt":"2025-03-18T01:44:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:50PM -0400, Taylor Blau wrote:\n\n> +void ewah_or_iterator_free(struct ewah_or_iterator *it)\n> +{\n> +\tfree(it->its);\n> +}\n\nHmm, I thought this was going to be come \"_release()\" based on the last\nround?\n\n-Peff\n"},{"id":"514484","messageId":"20250318020113.GA1473033@coredump.intra.peff.net","threadId":"61958","inReplyTo":"a29f4ee60d519318d36a8d3c812b4bad039b891e.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T02:01:13Z","receivedAt":"2025-03-18T02:01:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:53PM -0400, Taylor Blau wrote:\n\n> @@ -81,6 +81,24 @@ struct bitmap_index {\n>  \tstruct ewah_bitmap *blobs;\n>  \tstruct ewah_bitmap *tags;\n>  \n> +\t/*\n> +\t * Type index arrays when this bitmap is associated with an\n> +\t * incremental multi-pack index chain.\n> +\t *\n> +\t * If n is the number of unique layers in the MIDX chain, then\n> +\t * commits_all[n-1] is this structs 'commits' field,\n> +\t * commits_all[n-2] is the commits field of this bitmap's\n> +\t * 'base', and so on.\n> +\t *\n> +\t * When either associated either with a non-incremental MIDX, or\n> +\t * a single packfile, these arrays each contain a single\n> +\t * element.\n> +\t */\n> +\tstruct ewah_bitmap **commits_all;\n> +\tstruct ewah_bitmap **trees_all;\n> +\tstruct ewah_bitmap **blobs_all;\n> +\tstruct ewah_bitmap **tags_all;\n\nOK, so these are valid only for the top-level of the chain? I guess\nthere would not be much point in having the lower levels know about\ntheir incremental versions.\n\n> -static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n> +static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n> +{\n> +\tstruct bitmap_index *curr = bitmap_git;\n> +\tsize_t i = bitmap_git->base_nr;\n> +\n> +\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);\n> +\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);\n> +\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);\n> +\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr + 1);\n> +\n> +\twhile (curr) {\n> +\t\tbitmap_git->commits_all[i] = curr->commits;\n> +\t\tbitmap_git->trees_all[i] = curr->trees;\n> +\t\tbitmap_git->blobs_all[i] = curr->blobs;\n> +\t\tbitmap_git->tags_all[i] = curr->tags;\n> +\n> +\t\tcurr = curr->base;\n> +\t\tif (curr && !i)\n> +\t\t\tBUG(\"unexpected number of bitmap layers, expected %\"PRIu32,\n> +\t\t\t    bitmap_git->base_nr + 1);\n> +\t\ti -= 1;\n> +\t}\n> +}\n\nIt looks like we always allocate these. For the non-incremental case, I\nthink you could just do:\n\n  bitmap_git->commits_all = &bitmap_git->commits;\n\nand so forth. But I doubt that micro-optimization really matters, and it\nintroduces complications when you have to decide whether to free them or\nnot.\n\n(And if you really cared about micro-optimizing, probably trying to\nprevent the extra pointer-chase in the first place would be a more\nproductive path).\n\n-Peff\n"},{"id":"514486","messageId":"20250318020526.GB1473033@coredump.intra.peff.net","threadId":"61958","inReplyTo":"a1cf65bedc94ab6e318ff81a6d48eb30b6fc7868.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T02:05:26Z","receivedAt":"2025-03-18T02:05:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:56PM -0400, Taylor Blau wrote:\n\n> -static void init_type_iterator(struct ewah_iterator *it,\n> +static void init_type_iterator(struct ewah_or_iterator *it,\n>  \t\t\t       struct bitmap_index *bitmap_git,\n>  \t\t\t       enum object_type type)\n>  {\n>  \tswitch (type) {\n>  \tcase OBJ_COMMIT:\n> -\t\tewah_iterator_init(it, bitmap_git->commits);\n> +\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n> +\t\t\t\t      bitmap_git->base_nr + 1);\n\nThis certainly makes sense. It looks like we now use the or_iterator\nunconditionally, even for non-layered queries. It's probably a little\nslower in practice, just because it's an extra layer of indirection. But\nI don't know if trying to micro-optimize here is worth it. In general\nI'd say no, but sometimes there are surprising tight loops with bitmaps.\n\nI dunno. I guess it would be easy enough to do a simple before/after\nbenchmark on a single packfile with this series. I wouldn't expect it to\nfind anything, but might not hurt to double check.\n\n-Peff\n"},{"id":"514487","messageId":"20250318021605.GC1473033@coredump.intra.peff.net","threadId":"61958","inReplyTo":"d0d564685bc66df71b4c3ed1093452e58e0f54fd.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T02:16:05Z","receivedAt":"2025-03-18T02:16:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:19:00PM -0400, Taylor Blau wrote:\n\n> +write_midx_layer () {\n> +\tn=1\n> +\tif test -f $midx_chain\n> +\tthen\n> +\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n> +\tfi\n> +\n> +\tfor i in 1 2\n> +\tdo\n> +\t\ttest_commit $n.$i &&\n> +\t\tgit repack -d || return 1\n> +\tdone &&\n> +\tgit multi-pack-index write --bitmap --incremental\n> +}\n> +\n> +test_expect_success 'write initial MIDX layer' '\n> +\tgit repack -ad &&\n> +\twrite_midx_layer\n> +'\n> +\n> +test_expect_success 'read bitmap from first MIDX layer' '\n> +\tgit rev-list --test-bitmap 1.2\n> +'\n> +\n> +test_expect_success 'write another MIDX layer' '\n> +\twrite_midx_layer\n> +'\n> +\n> +test_expect_success 'midx verify with multiple layers' '\n> +\tgit multi-pack-index verify\n> +'\n\nPerhaps a silly suggestion, but do you want to confirm in one of these\ntests that there are in fact multiple layers of bitmaps? (I expect it to\nbe true, but just trying to cover all bases in the test).\n\nI guess that happens somewhat here:\n\n> +test_expect_success 'relink existing MIDX layer' '\n> +\trm -fr \"$midxdir\" &&\n> +\n> +\tGIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n> +\n> +\tmidx_hash=\"$(test-tool read-midx --checksum $objdir)\" &&\n> +\n> +\ttest_path_is_file \"$packdir/multi-pack-index\" &&\n> +\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n> +\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n> +\n> +\ttest_commit another &&\n> +\tgit repack -d &&\n> +\tgit multi-pack-index write --bitmap --incremental &&\n> +\n> +\ttest_path_is_missing \"$packdir/multi-pack-index\" &&\n> +\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n> +\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n> +\n> +\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.midx\" &&\n> +\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.bitmap\" &&\n> +\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.rev\" &&\n> +\ttest_line_count = 2 \"$midx_chain\"\n\nwhere we check that we switched to $midxdir.\n\n-Peff\n"},{"id":"514488","messageId":"20250318022134.GD1473033@coredump.intra.peff.net","threadId":"61958","inReplyTo":"cover.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 00/13] midx: incremental multi-pack indexes, part two","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-18T02:21:34Z","receivedAt":"2025-03-18T02:21:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 14, 2025 at 04:18:12PM -0400, Taylor Blau wrote:\n\n> This is a new round of my series to implement bitmap support for\n> incremental multi-pack indexes (MIDXs). It has been rebased on current\n> 'master', which is 683c54c999 (Git 2.49, 2025-03-14) at the time of\n> writing.\n\nI read over this and didn't find anything objectionable (I left a few\ncomments here and there). I think I've said this before with big\nbitmap/midx series: the biggest issue is that it's hard to know what you\nmight have missed. Especially in terms of corner cases. So it all looks\nreasonable to me (including the overall design), but ultimately I think\nit's more fruitful to put it through the paces on real-looking data than\nit is to try to go over every inch of the midx code with a fine-tooth\ncomb. And I'd guess the eventual fate here is for this code to get\nexercise on GitHub, which would help with that shaking out.\n\nSo mainly I tried to look for things that might hurt the non-incremental\ncases, and didn't see anything (modulo one or two questions about\nmicro-optimizations, though I expect the answer there is \"nothing big\nenough to measure\"). So if this can progress towards the \"shaking out\"\nphase, and has the potential to hurt only people who turn on the new\nfeature, that seems like a good path to me.\n\n-Peff\n"},{"id":"514489","messageId":"CABPp-BHMSs7R1kufM6f0PQDBAoPrOkKwnKK+hA+sK72LFLzCCg@mail.gmail.com","threadId":"61958","inReplyTo":"f565f2fff166bdf4bb2505f4a8853215a5365b17.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T02:42:54Z","receivedAt":"2025-03-18T02:43:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> Prepare to implement support for reachability bitmaps for the new\n> incremental multi-pack index (MIDX) feature over the following commits.\n>\n> This commit begins by first describing the relevant format and usage\n> details for incremental MIDX bitmaps.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  Documentation/technical/multi-pack-index.adoc | 71 +++++++++++++++++++\n>  1 file changed, 71 insertions(+)\n>\n> diff --git a/Documentation/technical/multi-pack-index.adoc b/Documentation/technical/multi-pack-index.adoc\n> index cc063b30be..ab98ecfeb9 100644\n> --- a/Documentation/technical/multi-pack-index.adoc\n> +++ b/Documentation/technical/multi-pack-index.adoc\n> @@ -164,6 +164,77 @@ objects_nr($H2) + objects_nr($H1) + i\n>  (in the C implementation, this is often computed as `i +\n>  m->num_objects_in_base`).\n>\n> +=== Pseudo-pack order for incremental MIDXs\n> +\n> +The original implementation of multi-pack reachability bitmaps defined\n> +the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n> +titled \"multi-pack-index reverse indexes\") roughly as follows:\n> +\n> +____\n> +In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n> +objects in packs stored by the MIDX, laid out in pack order, and the\n> +packs arranged in MIDX order (with the preferred pack coming first).\n> +____\n> +\n> +In the incremental MIDX design, we extend this definition to include\n> +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> +for incremental MIDXs is determined by concatenating the pseudo-pack\n> +ordering for each layer of the MIDX chain in order. Formally two objects\n> +`o1` and `o2` are compared as follows:\n> +\n> +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> +  `o1` is considered less than `o2`.\n\nFor sorting order, 'less than' doesn't tell us if you are sorting\nsmallest to greatest or greatest to smallest.  Maybe \"less than (so\nits order is earlier than) `o2'\" ?\n\n> +\n> +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n> +   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n> +   preferred and the other is not, then the preferred one sorts first. If\n> +   there is a base layer (i.e. the MIDX layer is not the first layer in\n> +   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n> +   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n\ns/than/then/\n\n> +   appears earlier, than the opposite is true.\n\ns/than/then/\n\n> +\n> +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n> +   same MIDX layer. Sort `o1` and `o2` by their offset within their\n> +   containing packfile.\n> +\n> +Note that the preferred pack is a property of the MIDX chain, not the\n> +individual layers themselves. Fundamentally we could introduce a\n> +per-layer preferred pack, but this is less relevant now that we can\n> +perform multi-pack reuse across the set of packs in a MIDX.\n> +\n> +=== Reachability bitmaps and incremental MIDXs\n> +\n> +Each layer of an incremental MIDX chain may have its objects (and the\n> +objects from any previous layer in the same MIDX chain) represented in\n> +its own `*.bitmap` file.\n> +\n> +The structure of a `*.bitmap` file belonging to an incremental MIDX\n> +chain is identical to that of a non-incremental MIDX bitmap, or a\n> +classic single-pack bitmap. Since objects are added to the end of the\n> +incremental MIDX's pseudo-pack order (see: above), it is possible to\n\ndrop the colon?\n\n> +extend a bitmap when appending to the end of a MIDX chain.\n> +\n> +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n> +incremental layers, and their `*.bitmap`(s) into a single layer and\n> +`*.bitmap`, but this is not yet implemented.)\n\n\"`*.bitmap`(s)\" feels slightly awkward and only saves 2 characters.\nMaybe just \"`*.bitmap` files\"?\n\n> +\n> +The object positions used are global within the pseudo-pack order, so\n> +subsequent layers will have, for example, `m->num_objects_in_base`\n> +number of `0` bits in each of their four type bitmaps. This follows from\n> +the fact that we only write type bitmap entries for objects present in\n> +the layer immediately corresponding to the bitmap).\n> +\n> +Note also that only the bitmap pertaining to the most recent layer in an\n> +incremental MIDX chain is used to store reachability information about\n> +the interesting and uninteresting objects in a reachability query.\n> +Earlier bitmap layers are only used to look up commit and pseudo-merge\n> +bitmaps from that layer, as well as the type-level bitmaps for objects\n> +in that layer.\n> +\n> +To simplify the implementation, type-level bitmaps are iterated\n> +simultaneously, and their results are OR'd together to avoid recursively\n> +calling internal bitmap functions.\n> +\n>  Future Work\n>  -----------\n\nShould the patch also remove the first item from Future Work, since\nthis series is implementing it?\n\n\n> --\n> 2.49.0.13.gd0d564685b\n"},{"id":"514490","messageId":"CABPp-BGbw+MOq9=cQzO5sc3XkakZQhXXg3r0iKeY5-p-4QR-3g@mail.gmail.com","threadId":"61958","inReplyTo":"f2a232e556a066c1a5083f02584ddc3637ecfc48.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T02:43:07Z","receivedAt":"2025-03-18T02:43:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> Prepare the reverse index machinery to handle object lookups in an\n> incremental MIDX bitmap. These changes are broken out across a few\n> functions:\n>\n>   - load_midx_revindex() learns to use the appropriate MIDX filename\n>     depending on whether the given 'struct multi_pack_index *' is\n>     incremental or not.\n>\n>   - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n>     object position in the MIDX pseudo-pack order, and finds the\n>     earliest containing MIDX (similar to midx.c::midx_for_object().\n\ns/finds/find/ ?\n\n\n>\n>   - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n>     number of objects in the base (since 'vb - midx->revindx_data' is\n>     relative to the containing MIDX, and pack_pos_to_midx() expects a\n>     global position).\n>\n>     Likewise, this function adjusts its output by adding\n>     m->num_objects_in_base to return a global position out through the\n>     `*pos` pointer.\n>\n> Together, these changes are sufficient to use the multi-pack index's\n> reverse index format for incremental multi-pack reachability bitmaps.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  pack-bitmap.c   | 40 ++++++++++++++++++++++++++++------------\n>  pack-revindex.c | 34 +++++++++++++++++++++++++---------\n>  2 files changed, 53 insertions(+), 21 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 6406953d32..c26d85b5db 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n>         return read_bitmap(index->map, index->map_size, &index->map_pos);\n>  }\n>\n> +static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n> +{\n> +       if (index->midx) {\n> +               struct multi_pack_index *m = index->midx;\n> +               return m->num_objects + m->num_objects_in_base;\n> +       }\n> +       return index->pack->num_objects;\n> +}\n> +\n>  static uint32_t bitmap_num_objects(struct bitmap_index *index)\n>  {\n>         if (index->midx)\n> @@ -924,7 +933,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n>\n>         if (pos < kh_end(positions)) {\n>                 int bitmap_pos = kh_value(positions, pos);\n> -               return bitmap_pos + bitmap_num_objects(bitmap_git);\n> +               return bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n>         }\n>\n>         return -1;\n> @@ -992,7 +1001,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n>                 bitmap_pos = kh_value(eindex->positions, hash_pos);\n>         }\n>\n> -       return bitmap_pos + bitmap_num_objects(bitmap_git);\n> +       return bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n>  }\n>\n>  struct bitmap_show_data {\n> @@ -1342,11 +1351,17 @@ struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_g\n>                 if (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n>                         goto done;\n>\n> +               /*\n> +                * Use bitmap-relative positions instead of offsetting\n> +                * by bitmap_git->num_objects_in_base because we use\n> +                * this to find a match in pseudo_merge_for_parents(),\n> +                * and pseudo-merge groups cannot span multiple bitmap\n> +                * layers.\n> +                */\n>                 bitmap_set(parents, pos);\n>         }\n>\n> -       match = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n> -                                               parents);\n> +       match = pseudo_merge_for_parents(&bitmap_git->pseudo_merges, parents);\n>\n>  done:\n>         bitmap_free(parents);\n> @@ -1500,7 +1515,8 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n>         for (i = 0; i < eindex->count; ++i) {\n>                 struct object *obj;\n>\n> -               if (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n> +               if (!bitmap_get(objects,\n> +                               st_add(bitmap_non_extended_bits(bitmap_git), i)))\n>                         continue;\n>\n>                 obj = eindex->objects[i];\n> @@ -1679,7 +1695,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n>          * them individually.\n>          */\n>         for (i = 0; i < eindex->count; i++) {\n> -               size_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n> +               size_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n>                 if (eindex->objects[i]->type == type &&\n>                     bitmap_get(to_filter, pos) &&\n>                     !bitmap_get(tips, pos))\n> @@ -1705,7 +1721,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n>\n>         oi.sizep = &size;\n>\n> -       if (pos < bitmap_num_objects(bitmap_git)) {\n> +       if (pos < bitmap_non_extended_bits(bitmap_git)) {\n>                 struct packed_git *pack;\n>                 off_t ofs;\n>\n> @@ -1729,7 +1745,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n>                 }\n>         } else {\n>                 struct eindex *eindex = &bitmap_git->ext_index;\n> -               struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n> +               struct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n>                 if (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n>                                              &oi, 0) < 0)\n>                         die(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n> @@ -1882,7 +1898,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n>         uint32_t objects_nr;\n>         size_t i, pos;\n>\n> -       objects_nr = bitmap_num_objects(bitmap_git);\n> +       objects_nr = bitmap_non_extended_bits(bitmap_git);\n>         pos = objects_nr / BITS_IN_EWORD;\n>\n>         if (pos > result->word_alloc)\n> @@ -2419,7 +2435,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n>         for (i = 0; i < eindex->count; ++i) {\n>                 if (eindex->objects[i]->type == type &&\n>                     bitmap_get(objects,\n> -                              st_add(bitmap_num_objects(bitmap_git), i)))\n> +                              st_add(bitmap_non_extended_bits(bitmap_git), i)))\n>                         count++;\n>         }\n>\n> @@ -2820,7 +2836,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n>                 BUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n>                     \"extension\");\n>\n> -       num_objects = bitmap_num_objects(bitmap_git);\n> +       num_objects = bitmap_non_extended_bits(bitmap_git);\n>         CALLOC_ARRAY(reposition, num_objects);\n>\n>         for (i = 0; i < num_objects; ++i) {\n> @@ -2963,7 +2979,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n>                 struct object *obj = eindex->objects[i];\n>\n>                 if (!bitmap_get(result,\n> -                               st_add(bitmap_num_objects(bitmap_git), i)))\n> +                               st_add(bitmap_non_extended_bits(bitmap_git), i)))\n>                         continue;\n>\n>                 if (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n> diff --git a/pack-revindex.c b/pack-revindex.c\n> index d3832478d9..d3faab6a37 100644\n> --- a/pack-revindex.c\n> +++ b/pack-revindex.c\n> @@ -383,8 +383,14 @@ int load_midx_revindex(struct multi_pack_index *m)\n>         trace2_data_string(\"load_midx_revindex\", the_repository,\n>                            \"source\", \"rev\");\n>\n> -       get_midx_filename_ext(m->repo->hash_algo, &revindex_name, m->object_dir,\n> -                             get_midx_checksum(m), MIDX_EXT_REV);\n> +       if (m->has_chain)\n> +               get_split_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n> +                                           m->object_dir, get_midx_checksum(m),\n> +                                           MIDX_EXT_REV);\n> +       else\n> +               get_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n> +                                     m->object_dir, get_midx_checksum(m),\n> +                                     MIDX_EXT_REV);\n>\n>         ret = load_revindex_from_disk(revindex_name.buf,\n>                                       m->num_objects,\n> @@ -471,11 +477,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n>\n>  uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n>  {\n> +       while (m && pos < m->num_objects_in_base)\n> +               m = m->base_midx;\n> +       if (!m)\n> +               BUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n>         if (!m->revindex_data)\n>                 BUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n> -       if (m->num_objects <= pos)\n> +       if (m->num_objects + m->num_objects_in_base <= pos)\n>                 BUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n> -       return get_be32(m->revindex_data + pos);\n> +       return get_be32(m->revindex_data + pos - m->num_objects_in_base);\n>  }\n>\n>  struct midx_pack_key {\n> @@ -491,7 +501,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n>         const struct midx_pack_key *key = va;\n>         struct multi_pack_index *midx = key->midx;\n>\n> -       uint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n> +       size_t pos = (uint32_t *)vb - (const uint32_t *)midx->revindex_data;\n> +       uint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n>         uint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n>         off_t versus_offset;\n>\n> @@ -529,9 +540,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n>  {\n>         uint32_t *found;\n>\n> -       if (key->pack >= m->num_packs)\n> +       if (key->pack >= m->num_packs + m->num_packs_in_base)\n>                 BUG(\"MIDX pack lookup out of bounds (%\"PRIu32\" >= %\"PRIu32\")\",\n> -                   key->pack, m->num_packs);\n> +                   key->pack, m->num_packs + m->num_packs_in_base);\n>         /*\n>          * The preferred pack sorts first, so determine its identifier by\n>          * looking at the first object in pseudo-pack order.\n> @@ -551,7 +562,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n>         if (!found)\n>                 return -1;\n>\n> -       *pos = found - m->revindex_data;\n> +       *pos = (found - m->revindex_data) + m->num_objects_in_base;\n> +\n>         return 0;\n>  }\n>\n> @@ -559,9 +571,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n>  {\n>         struct midx_pack_key key;\n>\n> +       while (m && at < m->num_objects_in_base)\n> +               m = m->base_midx;\n> +       if (!m)\n> +               BUG(\"NULL multi-pack-index for object position: %\"PRIu32, at);\n>         if (!m->revindex_data)\n>                 BUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n> -       if (m->num_objects <= at)\n> +       if (m->num_objects + m->num_objects_in_base <= at)\n>                 BUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n>\n>         key.pack = nth_midxed_pack_int_id(m, at);\n> --\n> 2.49.0.13.gd0d564685b\n>\n"},{"id":"514492","messageId":"CABPp-BHXYzMR61C=aX4yNn4QdgZ8uMtivg33oAN_eC3=J1U8kw@mail.gmail.com","threadId":"61958","inReplyTo":"aca0318fb12499ea810d03e66cc3145d682f5098.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T04:13:45Z","receivedAt":"2025-03-18T04:13:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> Prepare the pack-bitmap machinery to work with incremental MIDXs by\n> adding a new \"base\" field to keep track of the bitmap index associated\n> with the previous MIDX layer.\n>\n> The changes in this commit are mostly boilerplate to open the correct\n> bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n\ns/chain/chain of/ ?\n\n> ensures that the correct packs and their reverse indexes are loaded\n\ns/ensures/ensure/ ?\n\n\n> across MIDX layers, etc.\n>\n> While we're at it, keep track of a base_nr field to indicate how many\n> bitmap layers (including the current bitmap) exist. This will be used in\n> a future commit to allocate an array of 'struct ewah_bitmap' pointers to\n> collect all of the respective type bitmaps among all layers to\n> initialize a multi-EWAH iterator.\n>\n> Subsequent commits will teach the functions within the pack-bitmap\n> machinery how to interact with these new fields.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  pack-bitmap.c | 62 +++++++++++++++++++++++++++++++++++++++------------\n>  1 file changed, 48 insertions(+), 14 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index c26d85b5db..72fb11d014 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -54,6 +54,16 @@ struct bitmap_index {\n>         struct packed_git *pack;\n>         struct multi_pack_index *midx;\n>\n> +       /*\n> +        * If using a multi-pack index chain, 'base' points to the\n> +        * bitmap index corresponding to this bitmap's midx->base_midx.\n> +        *\n> +        * base_nr indicates how many layers precede this one, and is\n> +        * zero when base is NULL.\n> +        */\n> +       struct bitmap_index *base;\n> +       uint32_t base_nr;\n> +\n>         /* mmapped buffer of the whole bitmap index */\n>         unsigned char *map;\n>         size_t map_size; /* size of the mmaped buffer */\n> @@ -386,8 +396,15 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n>  char *midx_bitmap_filename(struct multi_pack_index *midx)\n>  {\n>         struct strbuf buf = STRBUF_INIT;\n> -       get_midx_filename_ext(midx->repo->hash_algo, &buf, midx->object_dir,\n> -                             get_midx_checksum(midx), MIDX_EXT_BITMAP);\n> +       if (midx->has_chain)\n> +               get_split_midx_filename_ext(midx->repo->hash_algo, &buf,\n> +                                           midx->object_dir,\n> +                                           get_midx_checksum(midx),\n> +                                           MIDX_EXT_BITMAP);\n> +       else\n> +               get_midx_filename_ext(midx->repo->hash_algo, &buf,\n> +                                     midx->object_dir, get_midx_checksum(midx),\n> +                                     MIDX_EXT_BITMAP);\n>\n>         return strbuf_detach(&buf, NULL);\n>  }\n> @@ -454,16 +471,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n>                 goto cleanup;\n>         }\n>\n> -       for (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> -               if (prepare_midx_pack(bitmap_repo(bitmap_git),\n> -                                     bitmap_git->midx,\n> -                                     i)) {\n> +       for (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n> +               if (prepare_midx_pack(bitmap_repo(bitmap_git), bitmap_git->midx, i)) {\n>                         warning(_(\"could not open pack %s\"),\n>                                 bitmap_git->midx->pack_names[i]);\n>                         goto cleanup;\n>                 }\n>         }\n>\n> +       if (midx->base_midx) {\n> +               bitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n> +               bitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n> +       } else {\n> +               bitmap_git->base_nr = 0;\n> +       }\n> +\n>         return 0;\n>\n>  cleanup:\n> @@ -515,6 +537,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n>         bitmap_git->map_size = xsize_t(st.st_size);\n>         bitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n>         bitmap_git->map_pos = 0;\n> +       bitmap_git->base_nr = 0;\n>         close(fd);\n>\n>         if (load_bitmap_header(bitmap_git) < 0) {\n> @@ -534,8 +557,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n>  static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)\n>  {\n>         if (bitmap_is_midx(bitmap_git)) {\n> -               uint32_t i;\n> -               int ret;\n> +               struct multi_pack_index *m;\n>\n>                 /*\n>                  * The multi-pack-index's .rev file is already loaded via\n> @@ -544,10 +566,15 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n>                  * But we still need to open the individual pack .rev files,\n>                  * since we will need to make use of them in pack-objects.\n>                  */\n> -               for (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> -                       ret = load_pack_revindex(r, bitmap_git->midx->packs[i]);\n> -                       if (ret)\n> -                               return ret;\n> +               for (m = bitmap_git->midx; m; m = m->base_midx) {\n> +                       uint32_t i;\n> +                       int ret;\n> +\n> +                       for (i = 0; i < m->num_packs; i++) {\n> +                               ret = load_pack_revindex(r, m->packs[i]);\n> +                               if (ret)\n> +                                       return ret;\n> +                       }\n>                 }\n>                 return 0;\n>         }\n> @@ -573,6 +600,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n>         if (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n>                 goto failed;\n>\n> +       if (bitmap_git->base) {\n> +               if (!bitmap_is_midx(bitmap_git))\n> +                       BUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n> +               if (load_bitmap(r, bitmap_git->base) < 0)\n> +                       goto failed;\n> +       }\n> +\n>         return 0;\n>\n>  failed:\n> @@ -657,10 +691,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n>\n>  struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n>  {\n> -       struct repository *r = midx->repo;\n>         struct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n>\n> -       if (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n> +       if (!open_midx_bitmap_1(bitmap_git, midx))\n>                 return bitmap_git;\n>\n>         free_bitmap_index(bitmap_git);\n> @@ -2899,6 +2932,7 @@ void free_bitmap_index(struct bitmap_index *b)\n>                 close_midx_revindex(b->midx);\n>         }\n>         free_pseudo_merge_map(&b->pseudo_merges);\n> +       free_bitmap_index(b->base);\n>         free(b);\n>  }\n>\n> --\n> 2.49.0.13.gd0d564685b\n>\n"},{"id":"514493","messageId":"CABPp-BH3h_XY-+K1uB5SWWKrr7UrN1JHpQX+fxgJ=pTaLwPZ9Q@mail.gmail.com","threadId":"61958","inReplyTo":"14d3d80c3d301233048004fe9de18154aa597782.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T04:13:52Z","receivedAt":"2025-03-18T04:14:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> In a similar fashion as previous commits in the first phase of\n> incremental MIDXs, enumerate not just the packs in the current\n> incremental MIDX layer, but previous ones as well.\n>\n> Likewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\n> single pack from a MIDX, use the oldest layer's preferred pack as it is\n> likely to contain the most amount of reusable sections.\n\n\"most amount\" => \"largest number\" or \"largest size\" ?\n\n\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  pack-bitmap.c | 11 ++++++++---\n>  1 file changed, 8 insertions(+), 3 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 1b4fec0033..7a41535425 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -2333,7 +2333,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n>                 multi_pack_reuse = 0;\n>\n>         if (multi_pack_reuse) {\n> -               for (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> +               struct multi_pack_index *m = bitmap_git->midx;\n> +               for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n>                         struct bitmapped_pack pack;\n>                         if (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n>                                 warning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n> @@ -2359,14 +2360,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n>                 uint32_t pack_int_id;\n>\n>                 if (bitmap_is_midx(bitmap_git)) {\n> +                       struct multi_pack_index *m = bitmap_git->midx;\n>                         uint32_t preferred_pack_pos;\n>\n> -                       if (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n> +                       while (m->base_midx)\n> +                               m = m->base_midx;\n> +\n> +                       if (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n>                                 warning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n>                                 return;\n>                         }\n>\n> -                       pack = bitmap_git->midx->packs[preferred_pack_pos];\n> +                       pack = nth_midxed_pack(m, preferred_pack_pos);\n>                         pack_int_id = preferred_pack_pos;\n>                 } else {\n>                         pack = bitmap_git->pack;\n> --\n> 2.49.0.13.gd0d564685b\n>\n"},{"id":"514496","messageId":"CABPp-BEyKkbVMs=7AgSf1P9CTXqA5CY5epG-piuSzppAWfPe4g@mail.gmail.com","threadId":"61958","inReplyTo":"b45a9ccbc20180e3358e314b4fd5e46bfa566241.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T05:31:06Z","receivedAt":"2025-03-18T05:31:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> Implement support for the special `--test-bitmap` mode of `git rev-list`\n> when using incremental MIDXs.\n>\n> The bitmap_test_data structure is extended to contain a \"base\" pointer\n> that mirrors the structure of the bitmap chain that it is being used to\n> test.\n>\n> When we find a commit to test, we first chase down the ->base pointer to\n> find the appropriate bitmap_test_data for the bitmap layer that the\n> given commit is contained within, and then perform the test on that\n> bitmap.\n>\n> In order to implement this, light modifications are made to\n> bitmap_for_commit() to reimplement it in terms of a new function,\n> find_bitmap_for_commit(), which fills out a pointer which indicates the\n> bitmap layer which contains the given commit.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  pack-bitmap.c | 107 ++++++++++++++++++++++++++++++++++++++++----------\n>  1 file changed, 86 insertions(+), 21 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 7a41535425..bb09ce3cf5 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n[...]\n> +\n> +static void bitmap_test_data_prepare(struct bitmap_test_data *tdata,\n> +                                    struct bitmap_index *bitmap_git)\n> +{\n> +       memset(tdata, 0, sizeof(struct bitmap_test_data));\n\nSo, the first thing this function does is 0 out tdata.\n\n> +\n> +       tdata->bitmap_git = bitmap_git;\n> +       tdata->base = bitmap_new();\n> +       tdata->commits = ewah_to_bitmap(bitmap_git->commits);\n> +       tdata->trees = ewah_to_bitmap(bitmap_git->trees);\n> +       tdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n> +       tdata->tags = ewah_to_bitmap(bitmap_git->tags);\n> +\n> +       if (bitmap_git->base) {\n> +               CALLOC_ARRAY(tdata->base_tdata, 1);\n\nWe use CALLOC to both allocate the array and set it all to 0...\n\n> +               bitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n\nand then call bitmap_test_data_prepare() which will re-zero it all out.\n\nShould we either ditch the zeroing at the beginning of the function,\nor use xmalloc instead of CALLOC_ARRAY, to avoid duplicate zeroing?\n\n> +       }\n> +}\n\nDidn't spot anything else.\n"},{"id":"514500","messageId":"CABPp-BE3dfXa8xda2M2zMH_Qhf_RPggBpC=tx8qaDpCmbr9hWA@mail.gmail.com","threadId":"61958","inReplyTo":"a29f4ee60d519318d36a8d3c812b4bad039b891e.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T06:43:47Z","receivedAt":"2025-03-18T06:43:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> Prepare for reading the type-level bitmaps from previous bitmap layers\n> by maintaining an array for each type, where each element in that type's\n> array corresponds to one layer's bitmap for that type.\n>\n> These fields will be used in a later commit to instantiate the 'struct\n> ewah_or_iterator' for each type.\n>\n\nAll I spotted was some possible wording fixups...\n\n> +        *\n> +        * When either associated either with a non-incremental MIDX, or\n> +        * a single packfile, these arrays each contain a single\n> +        * element.\n> +        */\n\nDrop the first \"either\", and the first comma?\n"},{"id":"514537","messageId":"CABPp-BF4msjkhV=aQYt8GrK8ptKaciE28OqQx0y64xrjWKrPMA@mail.gmail.com","threadId":"61958","inReplyTo":"d0d564685bc66df71b4c3ed1093452e58e0f54fd.1741983492.git.me@ttaylorr.com","subject":"Re: [PATCH v4 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-18T17:13:03Z","receivedAt":"2025-03-18T17:13:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 14, 2025 at 1:19 PM Taylor Blau <me@ttaylorr.com> wrote:\n\n[...]\n> diff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\n> index e92341b8fa..056c410efb 100644\n> --- a/ewah/ewah_bitmap.c\n> +++ b/ewah/ewah_bitmap.c\n> @@ -399,7 +399,7 @@ int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n>         return ret;\n>  }\n>\n> -void ewah_or_iterator_free(struct ewah_or_iterator *it)\n> +void ewah_or_iterator_release(struct ewah_or_iterator *it)\n>  {\n>         free(it->its);\n>  }\n> diff --git a/ewah/ewok.h b/ewah/ewok.h\n> index 4b70641045..c29d354236 100644\n> --- a/ewah/ewok.h\n> +++ b/ewah/ewok.h\n> @@ -158,7 +158,7 @@ void ewah_or_iterator_init(struct ewah_or_iterator *it,\n>\n>  int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n>\n> -void ewah_or_iterator_free(struct ewah_or_iterator *it);\n> +void ewah_or_iterator_release(struct ewah_or_iterator *it);\n\nWas the rename from these last two hunks squashed into the wrong\npatch?  Since you're not changing its definition, I'm assuming the\nupdated name should have been applied to when it was introduced.\n"},{"id":"514577","messageId":"Z9n9jf7GDBMPyM2R@nand.local","threadId":"61958","inReplyTo":"20250318011618.GA1471939@coredump.intra.peff.net","subject":"Re: [PATCH v4 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-18T23:11:09Z","receivedAt":"2025-03-18T23:11:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:16:18PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:20PM -0400, Taylor Blau wrote:\n>\n> > +In the incremental MIDX design, we extend this definition to include\n> > +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> > +for incremental MIDXs is determined by concatenating the pseudo-pack\n> > +ordering for each layer of the MIDX chain in order. Formally two objects\n> > +`o1` and `o2` are compared as follows:\n> > +\n> > +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> > +  `o1` is considered less than `o2`.\n> > +\n> > +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n> > +   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n> > +   preferred and the other is not, then the preferred one sorts first. If\n> > +   there is a base layer (i.e. the MIDX layer is not the first layer in\n> > +   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n> > +   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n> > +   appears earlier, than the opposite is true.\n> > +\n> > +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n> > +   same MIDX layer. Sort `o1` and `o2` by their offset within their\n> > +   containing packfile.\n>\n> OK, I think this ordering makes sense. I had to read this description\n> over several times to make sure I wasn't missing something. The earlier\n> part that says \"it's just concatenating the pack order of the layers\" is\n> a much more intuitive way of looking at it (modulo that you might need\n> to remove duplicates found in earlier layers).\n>\n> But I think an even more basic way of thinking about it is that it's the\n> same as the pseudo-pack order you would get if you had a single midx of\n> all of the packs in all of the layers (in their layer order). We already\n> have to deal with (and have documented) duplicates in that case.\n>\n> Not really suggesting any wording change here, just making sure I\n> grokked it all.\n\nYeah, those are both excellent ways to think about it. I hadn't\nconsidered the \"the new ordering is the same as the pseudo-pack order\nyou'd get if you had a single MIDX of all the packs in layer order\"\nthing before, but it's quite intuitive.\n\nAs a side note, it's somewhat hilarious to me that we could really\nwrite:\n\n    \"The new ordering is the same as the pseudo-pack order you'd get if\n    you had a single MIDX of all the packs in layer order, which is the\n    same order you'd get if you had a single pack containing all of the\n    objects in MIDX order.\"\n\n;-)\n\n> > +Note that the preferred pack is a property of the MIDX chain, not the\n> > +individual layers themselves. Fundamentally we could introduce a\n> > +per-layer preferred pack, but this is less relevant now that we can\n> > +perform multi-pack reuse across the set of packs in a MIDX.\n>\n> Calling this out explicitly is good, since it's an obvious question\n> for somebody to have.\n\nThanks, I think this was an addition from Patrick's earlier review of\nthe series.\n\n> OK, so each layer's bitmap does depend on the layers above/before it.\n> That obviously needs to happen because each incremental midx is not\n> likely to be a complete reachability set anyway.\n>\n> But I also wondered what would happen with a situation like this:\n>\n>   A -- B\n>    \\\n>     -- C\n>\n> stored like this:\n>\n>   base midx:\n>     - pack 1:\n>       - object A\n>       - object B, which can reach A\n>   incremental midx:\n>     - pack 2:\n>       - object A\n>       - object C, which can reach A\n>\n> That is, two objects B and C both depend on A, which is duplicated in\n> two midx layers. Even if the incremental midx is complete in the sense\n> that C only depends on A, its bitmap cannot just be \"11\". Because the\n> bit position for object A in the incremental midx does not exist in the\n> pseudo-pack order at all! It must refer to the copy of \"A\" in the base\n> midx, so it's correct bitmap is \"101\" (A and C, but not B).\n>\n\nRight. Since the base MIDX has objects A and B, B's bitmap here would be\n\"11\". C's bit position in the subsequent layer is a function of where it\nsits not just in that MIDX layer, but how many (de-duplicated) objects\nexist in all prior layers. There are two, so the earliest bit position\npossible to allocate towards C is the third bit. And since C reaches A,\nits bitmap would indeed be \"101\".\n\n> Again, just talking through it here.\n\nHeh, thanks for saying so. It's good to know when we're just talking\nthrough examples versus asking for changes. (Of course, the mere fact of\ntalking through an example is sometimes enough to suggest a change by\nvirtue of that example being confusing enough to need to be talked\nthrough in the first place).\n\n> > +Note also that only the bitmap pertaining to the most recent layer in an\n> > +incremental MIDX chain is used to store reachability information about\n> > +the interesting and uninteresting objects in a reachability query.\n> > +Earlier bitmap layers are only used to look up commit and pseudo-merge\n> > +bitmaps from that layer, as well as the type-level bitmaps for objects\n> > +in that layer.\n>\n> I'm not quite sure what this means, but I guess you're saying that\n> internally as we produce a bitmap, we'll always use the complete bitmap\n> over all of the layers?\n\nThat's exactly right.\n\n> > +To simplify the implementation, type-level bitmaps are iterated\n> > +simultaneously, and their results are OR'd together to avoid recursively\n> > +calling internal bitmap functions.\n>\n> OK, I guess we'll see what this means in the patches. ;)\n>\n> The general rules for the data structure make sense to me, though.\n\nGreat, and thanks in advance for the review as I work through the rest\nof your emails :-).\n\nThanks,\nTaylor\n"},{"id":"514578","messageId":"Z9n/dX0HSjkC9PMd@nand.local","threadId":"61958","inReplyTo":"CABPp-BHMSs7R1kufM6f0PQDBAoPrOkKwnKK+hA+sK72LFLzCCg@mail.gmail.com","subject":"Re: [PATCH v4 01/13] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-18T23:19:17Z","receivedAt":"2025-03-18T23:19:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 07:42:54PM -0700, Elijah Newren wrote:\n> > +In the incremental MIDX design, we extend this definition to include\n> > +objects from multiple layers of the MIDX chain. The pseudo-pack order\n> > +for incremental MIDXs is determined by concatenating the pseudo-pack\n> > +ordering for each layer of the MIDX chain in order. Formally two objects\n> > +`o1` and `o2` are compared as follows:\n> > +\n> > +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n> > +  `o1` is considered less than `o2`.\n>\n> For sorting order, 'less than' doesn't tell us if you are sorting\n> smallest to greatest or greatest to smallest.  Maybe \"less than (so\n> its order is earlier than) `o2'\" ?\n\nOh, good suggestion. I found the alternative a little verbose, but went\nwith \"sorts ahead of\" instead of \"less than\".\n\n> > +\n> > +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n> > +   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n> > +   preferred and the other is not, then the preferred one sorts first. If\n> > +   there is a base layer (i.e. the MIDX layer is not the first layer in\n> > +   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n> > +   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n>\n> s/than/then/\n>\n> > +   appears earlier, than the opposite is true.\n>\n> s/than/then/\n\nGood catch on both accounts ;-).\n\n> > +The structure of a `*.bitmap` file belonging to an incremental MIDX\n> > +chain is identical to that of a non-incremental MIDX bitmap, or a\n> > +classic single-pack bitmap. Since objects are added to the end of the\n> > +incremental MIDX's pseudo-pack order (see: above), it is possible to\n>\n> drop the colon?\n\nYep, dropped.\n\n> > +extend a bitmap when appending to the end of a MIDX chain.\n> > +\n> > +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n> > +incremental layers, and their `*.bitmap`(s) into a single layer and\n> > +`*.bitmap`, but this is not yet implemented.)\n>\n> \"`*.bitmap`(s)\" feels slightly awkward and only saves 2 characters.\n> Maybe just \"`*.bitmap` files\"?\n\nFair suggestion, sure!\n\n> >  Future Work\n> >  -----------\n>\n> Should the patch also remove the first item from Future Work, since\n> this series is implementing it?\n\nHah, that was quite satisfying to do. I moved that to its own commit,\nthough, since this series doesn't implement incremental MIDXs, but\nbitmap support for them. Incremental MIDXs were \"done\" as of b9497848df\n(Merge branch 'tb/incremental-midx-part-1', 2024-08-19).\n\nThanks,\nTaylor\n"},{"id":"514587","messageId":"Z9oJoZdfyA4eAbeD@nand.local","threadId":"61958","inReplyTo":"20250318012726.GB1471939@coredump.intra.peff.net","subject":"Re: [PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:02:41Z","receivedAt":"2025-03-19T00:02:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:27:26PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:24PM -0400, Taylor Blau wrote:\n>\n> > diff --git a/pack-bitmap.c b/pack-bitmap.c\n> > index 6406953d32..c26d85b5db 100644\n> > --- a/pack-bitmap.c\n> > +++ b/pack-bitmap.c\n> > @@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n> >  \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n> >  }\n> >\n> > +static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n> > +{\n> > +\tif (index->midx) {\n> > +\t\tstruct multi_pack_index *m = index->midx;\n> > +\t\treturn m->num_objects + m->num_objects_in_base;\n> > +\t}\n> > +\treturn index->pack->num_objects;\n> > +}\n>\n> I understand why we need to account for the objects in the base to\n> offset our total size.\n>\n> Similar to Patrick's comments on v3, I wondered about why we couldn't\n> just modify bitmap_num_objects() here, and why some callers would be\n> left with the other.\n>\n> I guess sometimes we still need to consider a single layer. We can't\n> quite just access m->num_objects there, because we still need the midx\n> vs pack abstraction layer. I just thought there'd be more discussion\n> here, but it looks the same as v3.\n\nRight; some callers care about the number of objects in *their* layer,\nlike computing the size of some bitmap extensions, bounds-checking\npseudo-merge commit lookups, or generating positions for objects in the\nextended index.\n\nI'm happy to include that discussion somewhere in the commit message or\nas a comment nearby bitmap_non_extended_bits(), but I'm not sure which\nis better. If you have thoughts, LMK.\n\n> I wonder if it is worth renaming bitmap_num_objects() to indicate that\n> it is a single layer (and make sure other callers are examined). I\n> dunno.\n>\n> I also suspect from previous forays into bitmap indexing that it will be\n> easy to mix up positions in various units (local to the layer vs in the\n> global pseudo-pack ordering, for example). In theory we could use types\n> to help us with this, but they're kind of weak in C (unless we wrap all\n> of the ints in structs). Maybe not worth it.\n\nPerhaps. I do like the idea of using types to help with all of this, but\nlike you I suspect they may be more trouble than they're worth.\n\nThanks,\nTaylor\n"},{"id":"514588","messageId":"Z9oJ1veRTy1+UxfB@nand.local","threadId":"61958","inReplyTo":"CABPp-BGbw+MOq9=cQzO5sc3XkakZQhXXg3r0iKeY5-p-4QR-3g@mail.gmail.com","subject":"Re: [PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:03:34Z","receivedAt":"2025-03-19T00:03:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 07:43:07PM -0700, Elijah Newren wrote:\n> On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n> >\n> > Prepare the reverse index machinery to handle object lookups in an\n> > incremental MIDX bitmap. These changes are broken out across a few\n> > functions:\n> >\n> >   - load_midx_revindex() learns to use the appropriate MIDX filename\n> >     depending on whether the given 'struct multi_pack_index *' is\n> >     incremental or not.\n> >\n> >   - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n> >     object position in the MIDX pseudo-pack order, and finds the\n> >     earliest containing MIDX (similar to midx.c::midx_for_object().\n>\n> s/finds/find/ ?\n\nGood eyes, fixed.\n\nThanks,\nTaylor\n"},{"id":"514589","messageId":"Z9oK2XBF9plTBbnl@nand.local","threadId":"61958","inReplyTo":"Z9oJoZdfyA4eAbeD@nand.local","subject":"Re: [PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:07:53Z","receivedAt":"2025-03-19T00:07:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 18, 2025 at 08:02:41PM -0400, Taylor Blau wrote:\n> > I understand why we need to account for the objects in the base to\n> > offset our total size.\n> >\n> > Similar to Patrick's comments on v3, I wondered about why we couldn't\n> > just modify bitmap_num_objects() here, and why some callers would be\n> > left with the other.\n> >\n> > I guess sometimes we still need to consider a single layer. We can't\n> > quite just access m->num_objects there, because we still need the midx\n> > vs pack abstraction layer. I just thought there'd be more discussion\n> > here, but it looks the same as v3.\n>\n> Right; some callers care about the number of objects in *their* layer,\n> like computing the size of some bitmap extensions, bounds-checking\n> pseudo-merge commit lookups, or generating positions for objects in the\n> extended index.\n>\n> I'm happy to include that discussion somewhere in the commit message or\n> as a comment nearby bitmap_non_extended_bits(), but I'm not sure which\n> is better. If you have thoughts, LMK.\n\nI renamed this function to bitmap_num_objects_total(), which I think\nmore clearly distinguishes it from bitmap_num_objects(). If you have\nother thoughts or things you think I should do in addition to that, LMK.\n\nThanks,\nTaylor\n"},{"id":"514590","messageId":"Z9oK/7sn0f4XtQUb@nand.local","threadId":"61958","inReplyTo":"CABPp-BHXYzMR61C=aX4yNn4QdgZ8uMtivg33oAN_eC3=J1U8kw@mail.gmail.com","subject":"Re: [PATCH v4 03/13] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:08:31Z","receivedAt":"2025-03-19T00:08:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:13:45PM -0700, Elijah Newren wrote:\n> On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n> >\n> > Prepare the pack-bitmap machinery to work with incremental MIDXs by\n> > adding a new \"base\" field to keep track of the bitmap index associated\n> > with the previous MIDX layer.\n> >\n> > The changes in this commit are mostly boilerplate to open the correct\n> > bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n>\n> s/chain/chain of/ ?\n\nGood eyes again!\n\n> > ensures that the correct packs and their reverse indexes are loaded\n>\n> s/ensures/ensure/ ?\n\n...and again!\n\nThanks,\nTaylor\n"},{"id":"514591","messageId":"Z9oMQSvqfD6R9T3a@nand.local","threadId":"61958","inReplyTo":"20250318013823.GC1471939@coredump.intra.peff.net","subject":"Re: [PATCH v4 04/13] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:13:53Z","receivedAt":"2025-03-19T00:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:38:23PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:31PM -0400, Taylor Blau wrote:\n>\n> > The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\n> > EWAH-compressed bitmap corresponding to some given commit object.\n> >\n> > Teach this function about incremental MIDX bitmaps by teaching it to\n> > recur on earlier bitmap layers when it fails to find a given commit in\n> > the current layer.\n> >\n> > The changes to do so are as follows:\n> >\n> >   - Avoid initializing hash_pos at its declaration, since\n> >     bitmap_for_commit() is now a recursive function and may receive a\n> >     NULL bitmap_index pointer as its first argument.\n> >\n> >   - In cases where we would previously return NULL (to indicate that a\n> >     lookup failed and the given bitmap_index does not contain an entry\n> >     corresponding to the given commit), recursively call the function on\n> >     the previous bitmap layer.\n>\n> This makes sense, though it does make me wonder if we could/should store\n> a (midx/pack,pos) pair. I.e., a master hash table stored once for the\n> whole midx stack. And then you wouldn't need to recurse; it would just\n> be a single lookup.\n>\n> Or would that work badly with the lazy nature? You'd need to load all of\n> the layers to fill it (rather than doing each incrementally). OTOH, if\n> you ask for the bitmap for commit X you're eventually going to have to\n> figure out what's in all of the layers as soon as you have a miss and\n> have to check them all. And I think the lookup table extension is what's\n> supposed to make that cheap-ish.\n\nI think that it's a good idea, though TBH I think there is even more\nroom for improvement there, like recording cache misses. I suspect the\ndetails are fiddly enough that I'd rather tackle them outside of this\nalready-fiddly series, though ;-).\n\nThanks,\nTaylor\n"},{"id":"514592","messageId":"Z9oNF7+xuQKqYNPX@nand.local","threadId":"61958","inReplyTo":"CABPp-BH3h_XY-+K1uB5SWWKrr7UrN1JHpQX+fxgJ=pTaLwPZ9Q@mail.gmail.com","subject":"Re: [PATCH v4 06/13] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:17:27Z","receivedAt":"2025-03-19T00:17:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:13:52PM -0700, Elijah Newren wrote:\n> On Fri, Mar 14, 2025 at 1:18 PM Taylor Blau <me@ttaylorr.com> wrote:\n> >\n> > In a similar fashion as previous commits in the first phase of\n> > incremental MIDXs, enumerate not just the packs in the current\n> > incremental MIDX layer, but previous ones as well.\n> >\n> > Likewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\n> > single pack from a MIDX, use the oldest layer's preferred pack as it is\n> > likely to contain the most amount of reusable sections.\n>\n> \"most amount\" => \"largest number\" or \"largest size\" ?\n\nGood call; between the two I prefer \"largest number\".\n\nThanks,\nTaylor\n"},{"id":"514594","messageId":"Z9oQDbPpieD0JfHS@nand.local","threadId":"61958","inReplyTo":"CABPp-BEyKkbVMs=7AgSf1P9CTXqA5CY5epG-piuSzppAWfPe4g@mail.gmail.com","subject":"Re: [PATCH v4 07/13] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:30:05Z","receivedAt":"2025-03-19T00:30:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 10:31:06PM -0700, Elijah Newren wrote:\n> > +static void bitmap_test_data_prepare(struct bitmap_test_data *tdata,\n> > +                                    struct bitmap_index *bitmap_git)\n> > +{\n> > +       memset(tdata, 0, sizeof(struct bitmap_test_data));\n>\n> So, the first thing this function does is 0 out tdata.\n>\n> > +\n> > +       tdata->bitmap_git = bitmap_git;\n> > +       tdata->base = bitmap_new();\n> > +       tdata->commits = ewah_to_bitmap(bitmap_git->commits);\n> > +       tdata->trees = ewah_to_bitmap(bitmap_git->trees);\n> > +       tdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n> > +       tdata->tags = ewah_to_bitmap(bitmap_git->tags);\n> > +\n> > +       if (bitmap_git->base) {\n> > +               CALLOC_ARRAY(tdata->base_tdata, 1);\n>\n> We use CALLOC to both allocate the array and set it all to 0...\n>\n> > +               bitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n>\n> and then call bitmap_test_data_prepare() which will re-zero it all out.\n>\n> Should we either ditch the zeroing at the beginning of the function,\n> or use xmalloc instead of CALLOC_ARRAY, to avoid duplicate zeroing?\n\nAh... good point. I think between the two we should drop the\nCALLOC_ARRAY() and just xmalloc() it.\n\nThanks,\nTaylor\n"},{"id":"514595","messageId":"Z9oQPlUbeMI2IbV3@nand.local","threadId":"61958","inReplyTo":"20250318014139.GD1471939@coredump.intra.peff.net","subject":"Re: [PATCH v4 08/13] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:30:54Z","receivedAt":"2025-03-19T00:30:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:41:39PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:44PM -0400, Taylor Blau wrote:\n>\n> > In a similar fashion as previous commits, use nth_midxed_pack() instead\n> > of accessing the MIDX's ->packs array directly to support incremental\n> > MIDXs.\n>\n> Probably not worth it to change it in an actual patch, but is it worth\n> renaming midx->packs to something else to make sure we catch all of the\n> spots that need to be considered? Or maybe you already did that, which\n> is how you found all of these. :)\n\nThat's how I found them originally ;-).\n\nThanks,\nTaylor\n"},{"id":"514596","messageId":"Z9oQ4moLVKh3+vul@nand.local","threadId":"61958","inReplyTo":"20250318014417.GE1471939@coredump.intra.peff.net","subject":"Re: [PATCH v4 10/13] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:33:38Z","receivedAt":"2025-03-19T00:33:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 09:44:17PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:50PM -0400, Taylor Blau wrote:\n>\n> > +void ewah_or_iterator_free(struct ewah_or_iterator *it)\n> > +{\n> > +\tfree(it->its);\n> > +}\n>\n> Hmm, I thought this was going to be come \"_release()\" based on the last\n> round?\n\nOops, yes -- it should have been. I could have sworn I made that change,\nbut I must be hallucinating. In either case, it's fixed up in the round\nthat I'll send shortly.\n\nThanks,\nTaylor\n"},{"id":"514597","messageId":"Z9oSCmTtyDM6JIi9@nand.local","threadId":"61958","inReplyTo":"20250318020113.GA1473033@coredump.intra.peff.net","subject":"Re: [PATCH v4 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:38:34Z","receivedAt":"2025-03-19T00:38:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 10:01:13PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:53PM -0400, Taylor Blau wrote:\n>\n> > @@ -81,6 +81,24 @@ struct bitmap_index {\n> >  \tstruct ewah_bitmap *blobs;\n> >  \tstruct ewah_bitmap *tags;\n> >\n> > +\t/*\n> > +\t * Type index arrays when this bitmap is associated with an\n> > +\t * incremental multi-pack index chain.\n> > +\t *\n> > +\t * If n is the number of unique layers in the MIDX chain, then\n> > +\t * commits_all[n-1] is this structs 'commits' field,\n> > +\t * commits_all[n-2] is the commits field of this bitmap's\n> > +\t * 'base', and so on.\n> > +\t *\n> > +\t * When either associated either with a non-incremental MIDX, or\n> > +\t * a single packfile, these arrays each contain a single\n> > +\t * element.\n> > +\t */\n> > +\tstruct ewah_bitmap **commits_all;\n> > +\tstruct ewah_bitmap **trees_all;\n> > +\tstruct ewah_bitmap **blobs_all;\n> > +\tstruct ewah_bitmap **tags_all;\n>\n> OK, so these are valid only for the top-level of the chain? I guess\n> there would not be much point in having the lower levels know about\n> their incremental versions.\n\nRight; all of the \"useful\" computation like counting, traversing,\nfiltering, etc. is all done at the top-most layer in any chain, so these\nhave no meaning/purpose for lower layers.\n\n> > -static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n> > +static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n> > +{\n> > +\tstruct bitmap_index *curr = bitmap_git;\n> > +\tsize_t i = bitmap_git->base_nr;\n> > +\n> > +\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);\n> > +\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);\n> > +\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);\n> > +\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr + 1);\n> > +\n> > +\twhile (curr) {\n> > +\t\tbitmap_git->commits_all[i] = curr->commits;\n> > +\t\tbitmap_git->trees_all[i] = curr->trees;\n> > +\t\tbitmap_git->blobs_all[i] = curr->blobs;\n> > +\t\tbitmap_git->tags_all[i] = curr->tags;\n> > +\n> > +\t\tcurr = curr->base;\n> > +\t\tif (curr && !i)\n> > +\t\t\tBUG(\"unexpected number of bitmap layers, expected %\"PRIu32,\n> > +\t\t\t    bitmap_git->base_nr + 1);\n> > +\t\ti -= 1;\n> > +\t}\n> > +}\n>\n> It looks like we always allocate these. For the non-incremental case, I\n> think you could just do:\n>\n>   bitmap_git->commits_all = &bitmap_git->commits;\n>\n> and so forth. But I doubt that micro-optimization really matters, and it\n> introduces complications when you have to decide whether to free them or\n> not.\n\nThe complications aren't actually too bad... I think you just have to\navoid free()-ing them when you have a non-NULL 'base' pointer. I think\nit would look something like:\n\n--- 8< ---\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2270a646f6..8a530fa7d8 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -604,6 +604,15 @@ static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n \tstruct bitmap_index *curr = bitmap_git;\n \tsize_t i = bitmap_git->base_nr;\n\n+\tif (!bitmap_git->base) {\n+\t\tbitmap_git->commits_all = &bitmap_git->commits;\n+\t\tbitmap_git->trees_all = &bitmap_git->trees;\n+\t\tbitmap_git->blobs_all = &bitmap_git->blobs;\n+\t\tbitmap_git->tags_all = &bitmap_git->tags;\n+\n+\t\treturn;\n+\t}\n+\n \tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);\n \tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);\n \tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);\n@@ -3031,10 +3040,12 @@ void free_bitmap_index(struct bitmap_index *b)\n \tewah_pool_free(b->trees);\n \tewah_pool_free(b->blobs);\n \tewah_pool_free(b->tags);\n-\tfree(b->commits_all);\n-\tfree(b->trees_all);\n-\tfree(b->blobs_all);\n-\tfree(b->tags_all);\n+\tif (b->base) {\n+\t\tfree(b->commits_all);\n+\t\tfree(b->trees_all);\n+\t\tfree(b->blobs_all);\n+\t\tfree(b->tags_all);\n+\t}\n \tif (b->bitmaps) {\n \t\tstruct stored_bitmap *sb;\n \t\tkh_foreach_value(b->bitmaps, sb, {\n--- >8 ---\n\n, but it leaves a bad taste in my mouth tying the NULL-ness of\n'bitmap_git->base' to the allocation/freeing of these arrays. Maybe I'm\nbeing paranoid, but it feels like a potential landmine.\n\n> (And if you really cared about micro-optimizing, probably trying to\n> prevent the extra pointer-chase in the first place would be a more\n> productive path).\n\nYup.\n\nThanks,\nTaylor\n"},{"id":"514598","messageId":"Z9oSOCtt4ENAskbz@nand.local","threadId":"61958","inReplyTo":"CABPp-BE3dfXa8xda2M2zMH_Qhf_RPggBpC=tx8qaDpCmbr9hWA@mail.gmail.com","subject":"Re: [PATCH v4 11/13] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T00:39:20Z","receivedAt":"2025-03-19T00:39:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 11:43:47PM -0700, Elijah Newren wrote:\n> > +        *\n> > +        * When either associated either with a non-incremental MIDX, or\n> > +        * a single packfile, these arrays each contain a single\n> > +        * element.\n> > +        */\n>\n> Drop the first \"either\", and the first comma?\n\nGood catch, thanks!\n\nThanks,\nTaylor\n"},{"id":"514700","messageId":"Z9tNA/2cZI5ssHnr@nand.local","threadId":"61958","inReplyTo":"20250318020526.GB1473033@coredump.intra.peff.net","subject":"Re: [PATCH v4 12/13] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-19T23:02:27Z","receivedAt":"2025-03-19T23:02:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 10:05:26PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:56PM -0400, Taylor Blau wrote:\n>\n> > -static void init_type_iterator(struct ewah_iterator *it,\n> > +static void init_type_iterator(struct ewah_or_iterator *it,\n> >  \t\t\t       struct bitmap_index *bitmap_git,\n> >  \t\t\t       enum object_type type)\n> >  {\n> >  \tswitch (type) {\n> >  \tcase OBJ_COMMIT:\n> > -\t\tewah_iterator_init(it, bitmap_git->commits);\n> > +\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n> > +\t\t\t\t      bitmap_git->base_nr + 1);\n>\n> This certainly makes sense. It looks like we now use the or_iterator\n> unconditionally, even for non-layered queries. It's probably a little\n> slower in practice, just because it's an extra layer of indirection. But\n> I don't know if trying to micro-optimize here is worth it. In general\n> I'd say no, but sometimes there are surprising tight loops with bitmaps.\n>\n> I dunno. I guess it would be easy enough to do a simple before/after\n> benchmark on a single packfile with this series. I wouldn't expect it to\n> find anything, but might not hurt to double check.\n\nShould be OK. We're adding one extra allocation, and one extra\nfunction call on each _next() iteration. So I think we should be OK\nhere, and indeed...\n\n    $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in\n    $ hyperfine -L v ,.compile 'git{v} pack-objects --stdout --delta-base-offset --use-bitmap-index --revs <in >/dev/null'\n    Benchmark 1: git pack-objects --stdout --delta-base-offset --use-bitmap-index --revs <in >/dev/null\n      Time (mean ± σ):      1.715 s ±  0.026 s    [User: 4.353 s, System: 0.206 s]\n      Range (min … max):    1.692 s …  1.785 s    10 runs\n\n    Benchmark 2: git.compile pack-objects --stdout --delta-base-offset --use-bitmap-index --revs <in >/dev/null\n      Time (mean ± σ):      1.712 s ±  0.021 s    [User: 4.401 s, System: 0.223 s]\n      Range (min … max):    1.676 s …  1.749 s    10 runs\n\n    Summary\n      git.compile pack-objects --stdout --delta-base-offset --use-bitmap-index --revs <in >/dev/null ran\n        1.00 ± 0.02 times faster than git pack-objects --stdout --delta-base-offset --use-bitmap-index --revs <in >/dev/null\n\nLooks like we're about the same. 'git' here is the latest release, and\n'git.compile' is this commit compiled with -O2 for an apples-to-apples\ncomparison.\n\nIn this benchmark it looks like this build is maybe ~1ms faster than the\nstock build, but we're well within the noise range here, so I don't\nthink there's any statistical significance to the results.\n\nThanks,\nTaylor\n"},{"id":"514701","messageId":"Z9td64AzrXcRZmzq@nand.local","threadId":"61958","inReplyTo":"20250318021605.GC1473033@coredump.intra.peff.net","subject":"Re: [PATCH v4 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T00:14:35Z","receivedAt":"2025-03-20T00:14:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 10:16:05PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:19:00PM -0400, Taylor Blau wrote:\n>\n> > +write_midx_layer () {\n> > +\tn=1\n> > +\tif test -f $midx_chain\n> > +\tthen\n> > +\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n> > +\tfi\n> > +\n> > +\tfor i in 1 2\n> > +\tdo\n> > +\t\ttest_commit $n.$i &&\n> > +\t\tgit repack -d || return 1\n> > +\tdone &&\n> > +\tgit multi-pack-index write --bitmap --incremental\n> > +}\n> > +\n> > +test_expect_success 'write initial MIDX layer' '\n> > +\tgit repack -ad &&\n> > +\twrite_midx_layer\n> > +'\n> > +\n> > +test_expect_success 'read bitmap from first MIDX layer' '\n> > +\tgit rev-list --test-bitmap 1.2\n> > +'\n> > +\n> > +test_expect_success 'write another MIDX layer' '\n> > +\twrite_midx_layer\n> > +'\n> > +\n> > +test_expect_success 'midx verify with multiple layers' '\n> > +\tgit multi-pack-index verify\n> > +'\n>\n> Perhaps a silly suggestion, but do you want to confirm in one of these\n> tests that there are in fact multiple layers of bitmaps? (I expect it to\n> be true, but just trying to cover all bases in the test).\n\nI don't think it's a silly suggestion. As you note, we do implicitly\ncheck it further down, but doing something like the following\n\n    test_path_is_dir \"$midx_chain\" &&\n    test_line_count = 2 \"$midx_chain\" &&\n\nexplicitly before calling 'git multi-pack-index verify' would be nice to\nhave.\n\nThanks,\nTaylor\n"},{"id":"514702","messageId":"Z9teUzEvSmK6V2XF@nand.local","threadId":"61958","inReplyTo":"CABPp-BF4msjkhV=aQYt8GrK8ptKaciE28OqQx0y64xrjWKrPMA@mail.gmail.com","subject":"Re: [PATCH v4 13/13] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T00:16:19Z","receivedAt":"2025-03-20T00:16:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 18, 2025 at 10:13:03AM -0700, Elijah Newren wrote:\n> On Fri, Mar 14, 2025 at 1:19 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> [...]\n> > diff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\n> > index e92341b8fa..056c410efb 100644\n> > --- a/ewah/ewah_bitmap.c\n> > +++ b/ewah/ewah_bitmap.c\n> > @@ -399,7 +399,7 @@ int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n> >         return ret;\n> >  }\n> >\n> > -void ewah_or_iterator_free(struct ewah_or_iterator *it)\n> > +void ewah_or_iterator_release(struct ewah_or_iterator *it)\n> >  {\n> >         free(it->its);\n> >  }\n> > diff --git a/ewah/ewok.h b/ewah/ewok.h\n> > index 4b70641045..c29d354236 100644\n> > --- a/ewah/ewok.h\n> > +++ b/ewah/ewok.h\n> > @@ -158,7 +158,7 @@ void ewah_or_iterator_init(struct ewah_or_iterator *it,\n> >\n> >  int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n> >\n> > -void ewah_or_iterator_free(struct ewah_or_iterator *it);\n> > +void ewah_or_iterator_release(struct ewah_or_iterator *it);\n>\n> Was the rename from these last two hunks squashed into the wrong\n> patch?  Since you're not changing its definition, I'm assuming the\n> updated name should have been applied to when it was introduced.\n\nHah! I knew that I made this change, so I was confused in\n\n    https://lore.kernel.org/git/Z9oQ4moLVKh3+vul@nand.local/\n\nwhen it didn't show up in that patch.\n\nIt got rebased out of my local copy of this patch automatically since I\nhad manually applied the rename to the earlier patch while responding to\nthat review comment.\n\nThanks,\nTaylor\n"},{"id":"514703","messageId":"Z9te7BVGeU6VCo8Y@nand.local","threadId":"61958","inReplyTo":"20250318022134.GD1473033@coredump.intra.peff.net","subject":"Re: [PATCH v4 00/13] midx: incremental multi-pack indexes, part two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T00:18:52Z","receivedAt":"2025-03-20T00:18:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 17, 2025 at 10:21:34PM -0400, Jeff King wrote:\n> On Fri, Mar 14, 2025 at 04:18:12PM -0400, Taylor Blau wrote:\n>\n> > This is a new round of my series to implement bitmap support for\n> > incremental multi-pack indexes (MIDXs). It has been rebased on current\n> > 'master', which is 683c54c999 (Git 2.49, 2025-03-14) at the time of\n> > writing.\n>\n> I read over this and didn't find anything objectionable (I left a few\n> comments here and there). I think I've said this before with big\n> bitmap/midx series: the biggest issue is that it's hard to know what you\n> might have missed. Especially in terms of corner cases. So it all looks\n> reasonable to me (including the overall design), but ultimately I think\n> it's more fruitful to put it through the paces on real-looking data than\n> it is to try to go over every inch of the midx code with a fine-tooth\n> comb. And I'd guess the eventual fate here is for this code to get\n> exercise on GitHub, which would help with that shaking out.\n\nThanks for reviewing it, and sorry that this series is so dense to begin\nwith.\n\nI agree that the proof is really in the pudding here, and the best way\nto confirm that we squashed everything is by putting real usage through\nthe new paths and seeing what shakes out.\n\nI think the main thing that I was hoping for here were two things:\n\n  1. That others thought the overall design and approach are sane, and\n     that we're not painting ourselves into a corner.\n\n  2. That we are unlikely to regress non-incremental bitmap usage.\n\n> So mainly I tried to look for things that might hurt the non-incremental\n> cases, and didn't see anything (modulo one or two questions about\n> micro-optimizations, though I expect the answer there is \"nothing big\n> enough to measure\"). So if this can progress towards the \"shaking out\"\n> phase, and has the potential to hurt only people who turn on the new\n> feature, that seems like a good path to me.\n\n...and that's exactly what I got :-).\n\nI'll send a small reroll shortly that addresses the comments from you\nand Elijah (thanks, Elijah!) and I'll look forward to hearing what you\nthink of that round.\n\nThanks,\nTaylor\n"},{"id":"514782","messageId":"cover.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1723755667.git.me@ttaylorr.com","subject":"[PATCH v5 00/14] midx: incremental multi-pack indexes, part two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:21Z","receivedAt":"2025-03-20T17:56:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This is another new round of my series to implement bitmap support for\nincremental multi-pack indexes (MIDXs). It is still based on 683c54c999\n(Git 2.49, 2025-03-14).\n\n== Changes since last time\n\nThis round addresses thorough review from Elijah and Peff. The series\nsubstantively is unchanged, but there are lots of little quality-of-life\nand commit message readability improvements throughout. As usual, there\nis a range-diff below for convenience.\n\n== Original cover letter\n\nThis series is based on 'master', with an additional merge between\ntb/incremental-midx-part-1[1] and my newer series to fix a handful of\nbugs related to pseudo-merge bitmaps[2].\n\nThis is the second of three series to implement support for incremental\nmulti-pack indexes (MIDXs). This series brings support for bitmaps that\nare tied to incremental MIDXs in addition to regular MIDX bitmaps.\n\nThe details are laid out in the commits themselves, but the high-level\napproach is as follows:\n\n  - Each layer in the incremental MIDX chain has its own corresponding\n    *.bitmap file. Each bitmap contains commits / pseudo-merges which\n    are selected only from the commits in that layer. Likewise, only\n    that layer's objects are written in the type-level bitmaps.\n\n  - The reachability traversal is only conducted on the top-most bitmap\n    corresponding to the most recent layer in the incremental MIDX\n    chain. Earlier layers may be consulted to retrieve commit /\n    pseudo-merge reachability bitmaps, but only the top-most bitmap's\n    \"result\" and \"haves\" fields are used.\n\n  - In essence, the top-most bitmap is the only one that \"matters\", and\n    earlier bitmaps are merely used to look up commit and pseudo-merge\n    bitmaps from that layer.\n\n  - Whenever we need to look at the type-level bitmaps corresponding to\n    the whole incremental MIDX chain, a new \"ewah_or_iterator\" is used.\n    This works in concept like a typical ewah_iterator, except works\n    over many EWAH bitmaps in parallel, OR-ing their results together\n    before returning them to the user.\n\n    In effect, this allows us to treat the union of all type-level\n    bitmaps (each of which only stores information about the objects its\n    corresponding layer within the incremental MIDX chain) as a single\n    type-level bitmap corresponding to all of the objects across every\n    layer of the incremental MIDX chain.\n\nThe sum total of this series is that we are able to append new commits /\npseudo-merges to a repository's reachability bitmaps without having to\nrewrite existing bitmaps, making the operation much cheaper to perform\nin large repositories.\n\nThe series is laid out roughly as follows:\n\n  - The first patch describes the technical details of incremental MIDX\n    bitmaps.\n\n  - The second patch adjusts the pack-revindex internals to prepare for\n    incremental MIDX bitmaps.\n\n  - The next seven patches adjust various components of the pack-bitmap\n    internals to do the same.\n\n  - The next three patches introduce and adjust callers to use the\n    ewah_or_iterator (as above).\n\n  - The final patch implements writing incremental MIDX bitmaps, and\n    introduces tests.\n\nAfter this series, the remaining goals for this project include being\nable to compact contiguous runs of incremental MIDX layers into a single\nlayer to support growing the chain of MIDX layers without the chain\nitself becoming too long.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1722958595.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover.1723743050.git.me@ttaylorr.com/\n\nTaylor Blau (14):\n  Documentation: remove a \"future work\" item from the MIDX docs\n  Documentation: describe incremental MIDX bitmaps\n  pack-revindex: prepare for incremental MIDX bitmaps\n  pack-bitmap.c: open and store incremental bitmap layers\n  pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n  pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n  pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n  pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n  pack-bitmap.c: compute disk-usage with incremental MIDXs\n  pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n  ewah: implement `struct ewah_or_iterator`\n  pack-bitmap.c: keep track of each layer's type bitmaps\n  pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n  midx: implement writing incremental MIDX bitmaps\n\n Documentation/technical/multi-pack-index.adoc |  82 ++++-\n builtin/pack-objects.c                        |   3 +-\n ewah/ewah_bitmap.c                            |  33 ++\n ewah/ewok.h                                   |  12 +\n midx-write.c                                  |  57 ++-\n pack-bitmap-write.c                           |  65 +++-\n pack-bitmap.c                                 | 344 ++++++++++++++----\n pack-bitmap.h                                 |   4 +-\n pack-revindex.c                               |  34 +-\n t/t5334-incremental-multi-pack-index.sh       |  87 +++++\n 10 files changed, 589 insertions(+), 132 deletions(-)\n\nRange-diff against v4:\n -:  ---------- >  1:  6af65fdaac Documentation: remove a \"future work\" item from the MIDX docs\n 1:  f565f2fff1 !  2:  0897359506 Documentation: describe incremental MIDX bitmaps\n    @@ Documentation/technical/multi-pack-index.adoc: objects_nr($H2) + objects_nr($H1)\n     +`o1` and `o2` are compared as follows:\n     +\n     +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n    -+  `o1` is considered less than `o2`.\n    ++  `o1` sorts ahead of `o2`.\n     +\n     +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n     +   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n    -+   preferred and the other is not, then the preferred one sorts first. If\n    -+   there is a base layer (i.e. the MIDX layer is not the first layer in\n    -+   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n    -+   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n    -+   appears earlier, than the opposite is true.\n    ++   preferred and the other is not, then the preferred one sorts ahead of\n    ++   the non-preferred one. If there is a base layer (i.e. the MIDX layer\n    ++   is not the first layer in the chain), then if `pack(o1)` appears\n    ++   earlier in that MIDX layer's pack order, then `o1` sorts ahead of\n    ++   `o2`. Likewise if `pack(o2)` appears earlier, then the opposite is\n    ++   true.\n     +\n     +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n     +   same MIDX layer. Sort `o1` and `o2` by their offset within their\n    @@ Documentation/technical/multi-pack-index.adoc: objects_nr($H2) + objects_nr($H1)\n     +The structure of a `*.bitmap` file belonging to an incremental MIDX\n     +chain is identical to that of a non-incremental MIDX bitmap, or a\n     +classic single-pack bitmap. Since objects are added to the end of the\n    -+incremental MIDX's pseudo-pack order (see: above), it is possible to\n    ++incremental MIDX's pseudo-pack order (see above), it is possible to\n     +extend a bitmap when appending to the end of a MIDX chain.\n     +\n     +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n    -+incremental layers, and their `*.bitmap`(s) into a single layer and\n    ++incremental layers, and their `*.bitmap` files into a single layer and\n     +`*.bitmap`, but this is not yet implemented.)\n     +\n     +The object positions used are global within the pseudo-pack order, so\n 2:  f2a232e556 !  3:  5eac0d1485 pack-revindex: prepare for incremental MIDX bitmaps\n    @@ Commit message\n             incremental or not.\n     \n           - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n    -        object position in the MIDX pseudo-pack order, and finds the\n    +        object position in the MIDX pseudo-pack order, and find the\n             earliest containing MIDX (similar to midx.c::midx_for_object().\n     \n           - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n    @@ pack-bitmap.c: static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *ind\n      \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n      }\n      \n    -+static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n    ++static uint32_t bitmap_num_objects_total(struct bitmap_index *index)\n     +{\n     +\tif (index->midx) {\n     +\t\tstruct multi_pack_index *m = index->midx;\n    @@ pack-bitmap.c: static inline int bitmap_position_extended(struct bitmap_index *b\n      \tif (pos < kh_end(positions)) {\n      \t\tint bitmap_pos = kh_value(positions, pos);\n     -\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n    -+\t\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n    ++\t\treturn bitmap_pos + bitmap_num_objects_total(bitmap_git);\n      \t}\n      \n      \treturn -1;\n    @@ pack-bitmap.c: static int ext_index_add_object(struct bitmap_index *bitmap_git,\n      \t}\n      \n     -\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n    -+\treturn bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n    ++\treturn bitmap_pos + bitmap_num_objects_total(bitmap_git);\n      }\n      \n      struct bitmap_show_data {\n    @@ pack-bitmap.c: static void show_extended_objects(struct bitmap_index *bitmap_git\n      \n     -\t\tif (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n     +\t\tif (!bitmap_get(objects,\n    -+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n    ++\t\t\t\tst_add(bitmap_num_objects_total(bitmap_git),\n    ++\t\t\t\t       i)))\n      \t\t\tcontinue;\n      \n      \t\tobj = eindex->objects[i];\n    @@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitma\n      \t */\n      \tfor (i = 0; i < eindex->count; i++) {\n     -\t\tsize_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n    -+\t\tsize_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n    ++\t\tsize_t pos = st_add(i, bitmap_num_objects_total(bitmap_git));\n      \t\tif (eindex->objects[i]->type == type &&\n      \t\t    bitmap_get(to_filter, pos) &&\n      \t\t    !bitmap_get(tips, pos))\n    @@ pack-bitmap.c: static unsigned long get_size_by_pos(struct bitmap_index *bitmap_\n      \toi.sizep = &size;\n      \n     -\tif (pos < bitmap_num_objects(bitmap_git)) {\n    -+\tif (pos < bitmap_non_extended_bits(bitmap_git)) {\n    ++\tif (pos < bitmap_num_objects_total(bitmap_git)) {\n      \t\tstruct packed_git *pack;\n      \t\toff_t ofs;\n      \n     @@ pack-bitmap.c: static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n    + \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n      \t\t}\n      \t} else {\n    ++\t\tsize_t eindex_pos = pos - bitmap_num_objects_total(bitmap_git);\n      \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n     -\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n    -+\t\tstruct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n    ++\t\tstruct object *obj = eindex->objects[eindex_pos];\n      \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n      \t\t\t\t\t     &oi, 0) < 0)\n      \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n    @@ pack-bitmap.c: static void filter_packed_objects_from_bitmap(struct bitmap_index\n      \tsize_t i, pos;\n      \n     -\tobjects_nr = bitmap_num_objects(bitmap_git);\n    -+\tobjects_nr = bitmap_non_extended_bits(bitmap_git);\n    ++\tobjects_nr = bitmap_num_objects_total(bitmap_git);\n      \tpos = objects_nr / BITS_IN_EWORD;\n      \n      \tif (pos > result->word_alloc)\n    @@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git\n      \t\tif (eindex->objects[i]->type == type &&\n      \t\t    bitmap_get(objects,\n     -\t\t\t       st_add(bitmap_num_objects(bitmap_git), i)))\n    -+\t\t\t       st_add(bitmap_non_extended_bits(bitmap_git), i)))\n    ++\t\t\t       st_add(bitmap_num_objects_total(bitmap_git), i)))\n      \t\t\tcount++;\n      \t}\n      \n    @@ pack-bitmap.c: uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n      \t\t    \"extension\");\n      \n     -\tnum_objects = bitmap_num_objects(bitmap_git);\n    -+\tnum_objects = bitmap_non_extended_bits(bitmap_git);\n    ++\tnum_objects = bitmap_num_objects_total(bitmap_git);\n      \tCALLOC_ARRAY(reposition, num_objects);\n      \n      \tfor (i = 0; i < num_objects; ++i) {\n    @@ pack-bitmap.c: static off_t get_disk_usage_for_extended(struct bitmap_index *bit\n      \n      \t\tif (!bitmap_get(result,\n     -\t\t\t\tst_add(bitmap_num_objects(bitmap_git), i)))\n    -+\t\t\t\tst_add(bitmap_non_extended_bits(bitmap_git), i)))\n    ++\t\t\t\tst_add(bitmap_num_objects_total(bitmap_git),\n    ++\t\t\t\t       i)))\n      \t\t\tcontinue;\n      \n      \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n 3:  aca0318fb1 !  4:  922ea2f607 pack-bitmap.c: open and store incremental bitmap layers\n    @@ Commit message\n         with the previous MIDX layer.\n     \n         The changes in this commit are mostly boilerplate to open the correct\n    -    bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n    -    ensures that the correct packs and their reverse indexes are loaded\n    -    across MIDX layers, etc.\n    +    bitmap(s), add them to the chain of bitmap layers along the \"base\"\n    +    pointer, ensure that the correct packs and their reverse indexes are\n    +    loaded across MIDX layers, etc.\n     \n         While we're at it, keep track of a base_nr field to indicate how many\n         bitmap layers (including the current bitmap) exist. This will be used in\n 4:  832fd0e8dc =  5:  8fedd96614 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n 5:  c7c9f89956 =  6:  dccc1b2d2e pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n 6:  14d3d80c3d !  7:  e31bddd240 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n    @@ Commit message\n     \n         Likewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\n         single pack from a MIDX, use the oldest layer's preferred pack as it is\n    -    likely to contain the most amount of reusable sections.\n    +    likely to contain the largest number of reusable sections.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n 7:  b45a9ccbc2 !  8:  d9dfcb5a1b pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n    @@ pack-bitmap.c: static void test_show_commit(struct commit *commit, void *data)\n     +\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n     +\n     +\tif (bitmap_git->base) {\n    -+\t\tCALLOC_ARRAY(tdata->base_tdata, 1);\n    ++\t\ttdata->base_tdata = xmalloc(sizeof(struct bitmap_test_data));\n     +\t\tbitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n     +\t}\n     +}\n 8:  c1eefeae99 =  9:  b1bd60d25d pack-bitmap.c: compute disk-usage with incremental MIDXs\n 9:  11c4b7b949 = 10:  7477a8ac03 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n10:  cb08ad6a62 ! 11:  0fbef17acc ewah: implement `struct ewah_or_iterator`\n    @@ ewah/ewah_bitmap.c: void ewah_iterator_init(struct ewah_iterator *it, struct ewa\n     +\treturn ret;\n     +}\n     +\n    -+void ewah_or_iterator_free(struct ewah_or_iterator *it)\n    ++void ewah_or_iterator_release(struct ewah_or_iterator *it)\n     +{\n     +\tfree(it->its);\n     +}\n    @@ ewah/ewok.h: void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitma\n     +\n     +int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n     +\n    -+void ewah_or_iterator_free(struct ewah_or_iterator *it);\n    ++void ewah_or_iterator_release(struct ewah_or_iterator *it);\n     +\n      void ewah_xor(\n      \tstruct ewah_bitmap *ewah_i,\n11:  a29f4ee60d ! 12:  439e743fd5 pack-bitmap.c: keep track of each layer's type bitmaps\n    @@ pack-bitmap.c: struct bitmap_index {\n     +\t * commits_all[n-2] is the commits field of this bitmap's\n     +\t * 'base', and so on.\n     +\t *\n    -+\t * When either associated either with a non-incremental MIDX, or\n    -+\t * a single packfile, these arrays each contain a single\n    -+\t * element.\n    ++\t * When associated either with a non-incremental MIDX or a\n    ++\t * single packfile, these arrays each contain a single element.\n     +\t */\n     +\tstruct ewah_bitmap **commits_all;\n     +\tstruct ewah_bitmap **trees_all;\n12:  a1cf65bedc ! 13:  dcb45e349e pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n    @@ pack-bitmap.c: static void show_objects_for_type(\n      \t\t}\n      \t}\n     +\n    -+\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n      }\n      \n      static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n    @@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitma\n      \t\t\tbitmap_unset(to_filter, pos);\n      \t}\n      \n    -+\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n      \tbitmap_free(tips);\n      }\n      \n    @@ pack-bitmap.c: static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_\n      \t\t\tbitmap_unset(to_filter, pos);\n      \t}\n      \n    -+\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n      \tbitmap_free(tips);\n      }\n      \n    @@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git\n      \t\t\tcount++;\n      \t}\n      \n    -+\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n     +\n      \treturn count;\n      }\n    @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_\n      \t\t}\n      \t}\n      \n    -+\tewah_or_iterator_free(&it);\n    ++\tewah_or_iterator_release(&it);\n     +\n      \treturn total;\n      }\n13:  d0d564685b ! 14:  13568cfa3b midx: implement writing incremental MIDX bitmaps\n    @@ builtin/pack-objects.c: static void write_pack_file(void)\n      \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n      \t\t\t\t\t\t\t       written_list);\n     \n    - ## ewah/ewah_bitmap.c ##\n    -@@ ewah/ewah_bitmap.c: int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n    - \treturn ret;\n    - }\n    - \n    --void ewah_or_iterator_free(struct ewah_or_iterator *it)\n    -+void ewah_or_iterator_release(struct ewah_or_iterator *it)\n    - {\n    - \tfree(it->its);\n    - }\n    -\n    - ## ewah/ewok.h ##\n    -@@ ewah/ewok.h: void ewah_or_iterator_init(struct ewah_or_iterator *it,\n    - \n    - int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n    - \n    --void ewah_or_iterator_free(struct ewah_or_iterator *it);\n    -+void ewah_or_iterator_release(struct ewah_or_iterator *it);\n    - \n    - void ewah_xor(\n    - \tstruct ewah_bitmap *ewah_i,\n    -\n      ## midx-write.c ##\n     @@ midx-write.c: static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n      \treturn pack_order;\n    @@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n      \n      \twrite_selected_commits_v1(writer, f, offsets);\n     \n    - ## pack-bitmap.c ##\n    -@@ pack-bitmap.c: static void show_objects_for_type(\n    - \t\t}\n    - \t}\n    - \n    --\tewah_or_iterator_free(&it);\n    -+\tewah_or_iterator_release(&it);\n    - }\n    - \n    - static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n    -@@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n    - \t\t\tbitmap_unset(to_filter, pos);\n    - \t}\n    - \n    --\tewah_or_iterator_free(&it);\n    -+\tewah_or_iterator_release(&it);\n    - \tbitmap_free(tips);\n    - }\n    - \n    -@@ pack-bitmap.c: static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n    - \t\t\tbitmap_unset(to_filter, pos);\n    - \t}\n    - \n    --\tewah_or_iterator_free(&it);\n    -+\tewah_or_iterator_release(&it);\n    - \tbitmap_free(tips);\n    - }\n    - \n    -@@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n    - \t\t\tcount++;\n    - \t}\n    - \n    --\tewah_or_iterator_free(&it);\n    -+\tewah_or_iterator_release(&it);\n    - \n    - \treturn count;\n    - }\n    -@@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n    - \t\t}\n    - \t}\n    - \n    --\tewah_or_iterator_free(&it);\n    -+\tewah_or_iterator_release(&it);\n    - \n    - \treturn total;\n    - }\n    -\n      ## pack-bitmap.h ##\n     @@ pack-bitmap.h: struct bitmap_writer {\n      \n    @@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'convert incrementa\n     +'\n     +\n     +test_expect_success 'midx verify with multiple layers' '\n    ++\ttest_path_is_file \"$midx_chain\" &&\n    ++\ttest_line_count = 2 \"$midx_chain\" &&\n    ++\n     +\tgit multi-pack-index verify\n     +'\n     +\n\nbase-commit: 683c54c999c301c2cd6f715c411407c413b1d84e\n-- \n2.49.0.14.g88b49c1b34\n"},{"id":"514783","messageId":"6af65fdaacd5e50cc2e3cc0611d1cf08ef3aa452.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 01/14] Documentation: remove a \"future work\" item from the MIDX docs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:24Z","receivedAt":"2025-03-20T17:56:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"One of the items listed as \"future work\" in the MIDX's technical\ndocumentation is to extend the format to allow MIDXs to be written\nincrementally across multiple layers.\n\nThis was suggested all the way back in ceab693d1f (multi-pack-index: add\ndesign document, 2018-07-12), and implemented in b9497848df (Merge\nbranch 'tb/incremental-midx-part-1', 2024-08-19). Let's remove it\naccordingly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/technical/multi-pack-index.adoc | 10 ----------\n 1 file changed, 10 deletions(-)\n\ndiff --git a/Documentation/technical/multi-pack-index.adoc b/Documentation/technical/multi-pack-index.adoc\nindex cc063b30be..dea6486f88 100644\n--- a/Documentation/technical/multi-pack-index.adoc\n+++ b/Documentation/technical/multi-pack-index.adoc\n@@ -167,16 +167,6 @@ m->num_objects_in_base`).\n Future Work\n -----------\n \n-- The multi-pack-index allows many packfiles, especially in a context\n-  where repacking is expensive (such as a very large repo), or\n-  unexpected maintenance time is unacceptable (such as a high-demand\n-  build machine). However, the multi-pack-index needs to be rewritten\n-  in full every time. We can extend the format to be incremental, so\n-  writes are fast. By storing a small \"tip\" multi-pack-index that\n-  points to large \"base\" MIDX files, we can keep writes fast while\n-  still reducing the number of binary searches required for object\n-  lookups.\n-\n - If the multi-pack-index is extended to store a \"stable object order\"\n   (a function Order(hash) = integer that is constant for a given hash,\n   even as the multi-pack-index is updated) then MIDX bitmaps could be\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514784","messageId":"0897359506211b407c173e50007555e7de083342.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 02/14] Documentation: describe incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:28Z","receivedAt":"2025-03-20T17:56:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement support for reachability bitmaps for the new\nincremental multi-pack index (MIDX) feature over the following commits.\n\nThis commit begins by first describing the relevant format and usage\ndetails for incremental MIDX bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/technical/multi-pack-index.adoc | 72 +++++++++++++++++++\n 1 file changed, 72 insertions(+)\n\ndiff --git a/Documentation/technical/multi-pack-index.adoc b/Documentation/technical/multi-pack-index.adoc\nindex dea6486f88..ffda70aa13 100644\n--- a/Documentation/technical/multi-pack-index.adoc\n+++ b/Documentation/technical/multi-pack-index.adoc\n@@ -164,6 +164,78 @@ objects_nr($H2) + objects_nr($H1) + i\n (in the C implementation, this is often computed as `i +\n m->num_objects_in_base`).\n \n+=== Pseudo-pack order for incremental MIDXs\n+\n+The original implementation of multi-pack reachability bitmaps defined\n+the pseudo-pack order in linkgit:gitformat-pack[5] (see the section\n+titled \"multi-pack-index reverse indexes\") roughly as follows:\n+\n+____\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+____\n+\n+In the incremental MIDX design, we extend this definition to include\n+objects from multiple layers of the MIDX chain. The pseudo-pack order\n+for incremental MIDXs is determined by concatenating the pseudo-pack\n+ordering for each layer of the MIDX chain in order. Formally two objects\n+`o1` and `o2` are compared as follows:\n+\n+1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n+  `o1` sorts ahead of `o2`.\n+\n+2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n+   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n+   preferred and the other is not, then the preferred one sorts ahead of\n+   the non-preferred one. If there is a base layer (i.e. the MIDX layer\n+   is not the first layer in the chain), then if `pack(o1)` appears\n+   earlier in that MIDX layer's pack order, then `o1` sorts ahead of\n+   `o2`. Likewise if `pack(o2)` appears earlier, then the opposite is\n+   true.\n+\n+3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n+   same MIDX layer. Sort `o1` and `o2` by their offset within their\n+   containing packfile.\n+\n+Note that the preferred pack is a property of the MIDX chain, not the\n+individual layers themselves. Fundamentally we could introduce a\n+per-layer preferred pack, but this is less relevant now that we can\n+perform multi-pack reuse across the set of packs in a MIDX.\n+\n+=== Reachability bitmaps and incremental MIDXs\n+\n+Each layer of an incremental MIDX chain may have its objects (and the\n+objects from any previous layer in the same MIDX chain) represented in\n+its own `*.bitmap` file.\n+\n+The structure of a `*.bitmap` file belonging to an incremental MIDX\n+chain is identical to that of a non-incremental MIDX bitmap, or a\n+classic single-pack bitmap. Since objects are added to the end of the\n+incremental MIDX's pseudo-pack order (see above), it is possible to\n+extend a bitmap when appending to the end of a MIDX chain.\n+\n+(Note: it is possible likewise to compress a contiguous sequence of MIDX\n+incremental layers, and their `*.bitmap` files into a single layer and\n+`*.bitmap`, but this is not yet implemented.)\n+\n+The object positions used are global within the pseudo-pack order, so\n+subsequent layers will have, for example, `m->num_objects_in_base`\n+number of `0` bits in each of their four type bitmaps. This follows from\n+the fact that we only write type bitmap entries for objects present in\n+the layer immediately corresponding to the bitmap).\n+\n+Note also that only the bitmap pertaining to the most recent layer in an\n+incremental MIDX chain is used to store reachability information about\n+the interesting and uninteresting objects in a reachability query.\n+Earlier bitmap layers are only used to look up commit and pseudo-merge\n+bitmaps from that layer, as well as the type-level bitmaps for objects\n+in that layer.\n+\n+To simplify the implementation, type-level bitmaps are iterated\n+simultaneously, and their results are OR'd together to avoid recursively\n+calling internal bitmap functions.\n+\n Future Work\n -----------\n \n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514785","messageId":"5eac0d1485019179cde4f8b010ac64ee7eb3cd3f.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 03/14] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:31Z","receivedAt":"2025-03-20T17:56:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the reverse index machinery to handle object lookups in an\nincremental MIDX bitmap. These changes are broken out across a few\nfunctions:\n\n  - load_midx_revindex() learns to use the appropriate MIDX filename\n    depending on whether the given 'struct multi_pack_index *' is\n    incremental or not.\n\n  - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n    object position in the MIDX pseudo-pack order, and find the\n    earliest containing MIDX (similar to midx.c::midx_for_object().\n\n  - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n    number of objects in the base (since 'vb - midx->revindx_data' is\n    relative to the containing MIDX, and pack_pos_to_midx() expects a\n    global position).\n\n    Likewise, this function adjusts its output by adding\n    m->num_objects_in_base to return a global position out through the\n    `*pos` pointer.\n\nTogether, these changes are sufficient to use the multi-pack index's\nreverse index format for incremental multi-pack reachability bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c   | 43 +++++++++++++++++++++++++++++++------------\n pack-revindex.c | 34 +++++++++++++++++++++++++---------\n 2 files changed, 56 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6406953d32..87f3b5cf4d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -170,6 +170,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n }\n \n+static uint32_t bitmap_num_objects_total(struct bitmap_index *index)\n+{\n+\tif (index->midx) {\n+\t\tstruct multi_pack_index *m = index->midx;\n+\t\treturn m->num_objects + m->num_objects_in_base;\n+\t}\n+\treturn index->pack->num_objects;\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\n@@ -924,7 +933,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\t\treturn bitmap_pos + bitmap_num_objects_total(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -992,7 +1001,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n+\treturn bitmap_pos + bitmap_num_objects_total(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -1342,11 +1351,17 @@ struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_g\n \t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n \t\t\tgoto done;\n \n+\t\t/*\n+\t\t * Use bitmap-relative positions instead of offsetting\n+\t\t * by bitmap_git->num_objects_in_base because we use\n+\t\t * this to find a match in pseudo_merge_for_parents(),\n+\t\t * and pseudo-merge groups cannot span multiple bitmap\n+\t\t * layers.\n+\t\t */\n \t\tbitmap_set(parents, pos);\n \t}\n \n-\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t\tparents);\n+\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges, parents);\n \n done:\n \tbitmap_free(parents);\n@@ -1500,7 +1515,9 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\tif (!bitmap_get(objects,\n+\t\t\t\tst_add(bitmap_num_objects_total(bitmap_git),\n+\t\t\t\t       i)))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -1679,7 +1696,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tsize_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n+\t\tsize_t pos = st_add(i, bitmap_num_objects_total(bitmap_git));\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(to_filter, pos) &&\n \t\t    !bitmap_get(tips, pos))\n@@ -1705,7 +1722,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < bitmap_num_objects(bitmap_git)) {\n+\tif (pos < bitmap_num_objects_total(bitmap_git)) {\n \t\tstruct packed_git *pack;\n \t\toff_t ofs;\n \n@@ -1728,8 +1745,9 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n+\t\tsize_t eindex_pos = pos - bitmap_num_objects_total(bitmap_git);\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n+\t\tstruct object *obj = eindex->objects[eindex_pos];\n \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n \t\t\t\t\t     &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n@@ -1882,7 +1900,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n \tuint32_t objects_nr;\n \tsize_t i, pos;\n \n-\tobjects_nr = bitmap_num_objects(bitmap_git);\n+\tobjects_nr = bitmap_num_objects_total(bitmap_git);\n \tpos = objects_nr / BITS_IN_EWORD;\n \n \tif (pos > result->word_alloc)\n@@ -2419,7 +2437,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n \t\t    bitmap_get(objects,\n-\t\t\t       st_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t       st_add(bitmap_num_objects_total(bitmap_git), i)))\n \t\t\tcount++;\n \t}\n \n@@ -2820,7 +2838,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n \t\t    \"extension\");\n \n-\tnum_objects = bitmap_num_objects(bitmap_git);\n+\tnum_objects = bitmap_num_objects_total(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -2963,7 +2981,8 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \t\tstruct object *obj = eindex->objects[i];\n \n \t\tif (!bitmap_get(result,\n-\t\t\t\tst_add(bitmap_num_objects(bitmap_git), i)))\n+\t\t\t\tst_add(bitmap_num_objects_total(bitmap_git),\n+\t\t\t\t       i)))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex d3832478d9..d3faab6a37 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -383,8 +383,14 @@ int load_midx_revindex(struct multi_pack_index *m)\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t   \"source\", \"rev\");\n \n-\tget_midx_filename_ext(m->repo->hash_algo, &revindex_name, m->object_dir,\n-\t\t\t      get_midx_checksum(m), MIDX_EXT_REV);\n+\tif (m->has_chain)\n+\t\tget_split_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n+\t\t\t\t\t    m->object_dir, get_midx_checksum(m),\n+\t\t\t\t\t    MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(m->repo->hash_algo, &revindex_name,\n+\t\t\t\t      m->object_dir, get_midx_checksum(m),\n+\t\t\t\t      MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n \t\t\t\t      m->num_objects,\n@@ -471,11 +477,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \n uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n {\n+\twhile (m && pos < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, pos);\n \tif (!m->revindex_data)\n \t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n-\tif (m->num_objects <= pos)\n+\tif (m->num_objects + m->num_objects_in_base <= pos)\n \t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n-\treturn get_be32(m->revindex_data + pos);\n+\treturn get_be32(m->revindex_data + pos - m->num_objects_in_base);\n }\n \n struct midx_pack_key {\n@@ -491,7 +501,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n \tconst struct midx_pack_key *key = va;\n \tstruct multi_pack_index *midx = key->midx;\n \n-\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tsize_t pos = (uint32_t *)vb - (const uint32_t *)midx->revindex_data;\n+\tuint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);\n \tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n \toff_t versus_offset;\n \n@@ -529,9 +540,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n {\n \tuint32_t *found;\n \n-\tif (key->pack >= m->num_packs)\n+\tif (key->pack >= m->num_packs + m->num_packs_in_base)\n \t\tBUG(\"MIDX pack lookup out of bounds (%\"PRIu32\" >= %\"PRIu32\")\",\n-\t\t    key->pack, m->num_packs);\n+\t\t    key->pack, m->num_packs + m->num_packs_in_base);\n \t/*\n \t * The preferred pack sorts first, so determine its identifier by\n \t * looking at the first object in pseudo-pack order.\n@@ -551,7 +562,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,\n \tif (!found)\n \t\treturn -1;\n \n-\t*pos = found - m->revindex_data;\n+\t*pos = (found - m->revindex_data) + m->num_objects_in_base;\n+\n \treturn 0;\n }\n \n@@ -559,9 +571,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n {\n \tstruct midx_pack_key key;\n \n+\twhile (m && at < m->num_objects_in_base)\n+\t\tm = m->base_midx;\n+\tif (!m)\n+\t\tBUG(\"NULL multi-pack-index for object position: %\"PRIu32, at);\n \tif (!m->revindex_data)\n \t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n-\tif (m->num_objects <= at)\n+\tif (m->num_objects + m->num_objects_in_base <= at)\n \t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n \n \tkey.pack = nth_midxed_pack_int_id(m, at);\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514786","messageId":"922ea2f6075b47419a07e881efb2557ac73ba0c9.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 04/14] pack-bitmap.c: open and store incremental bitmap layers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:34Z","receivedAt":"2025-03-20T17:56:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare the pack-bitmap machinery to work with incremental MIDXs by\nadding a new \"base\" field to keep track of the bitmap index associated\nwith the previous MIDX layer.\n\nThe changes in this commit are mostly boilerplate to open the correct\nbitmap(s), add them to the chain of bitmap layers along the \"base\"\npointer, ensure that the correct packs and their reverse indexes are\nloaded across MIDX layers, etc.\n\nWhile we're at it, keep track of a base_nr field to indicate how many\nbitmap layers (including the current bitmap) exist. This will be used in\na future commit to allocate an array of 'struct ewah_bitmap' pointers to\ncollect all of the respective type bitmaps among all layers to\ninitialize a multi-EWAH iterator.\n\nSubsequent commits will teach the functions within the pack-bitmap\nmachinery how to interact with these new fields.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 62 +++++++++++++++++++++++++++++++++++++++------------\n 1 file changed, 48 insertions(+), 14 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 87f3b5cf4d..e84211de15 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -54,6 +54,16 @@ struct bitmap_index {\n \tstruct packed_git *pack;\n \tstruct multi_pack_index *midx;\n \n+\t/*\n+\t * If using a multi-pack index chain, 'base' points to the\n+\t * bitmap index corresponding to this bitmap's midx->base_midx.\n+\t *\n+\t * base_nr indicates how many layers precede this one, and is\n+\t * zero when base is NULL.\n+\t */\n+\tstruct bitmap_index *base;\n+\tuint32_t base_nr;\n+\n \t/* mmapped buffer of the whole bitmap index */\n \tunsigned char *map;\n \tsize_t map_size; /* size of the mmaped buffer */\n@@ -386,8 +396,15 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n char *midx_bitmap_filename(struct multi_pack_index *midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tget_midx_filename_ext(midx->repo->hash_algo, &buf, midx->object_dir,\n-\t\t\t      get_midx_checksum(midx), MIDX_EXT_BITMAP);\n+\tif (midx->has_chain)\n+\t\tget_split_midx_filename_ext(midx->repo->hash_algo, &buf,\n+\t\t\t\t\t    midx->object_dir,\n+\t\t\t\t\t    get_midx_checksum(midx),\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(midx->repo->hash_algo, &buf,\n+\t\t\t\t      midx->object_dir, get_midx_checksum(midx),\n+\t\t\t\t      MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n }\n@@ -454,16 +471,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \t}\n \n-\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\tif (prepare_midx_pack(bitmap_repo(bitmap_git),\n-\t\t\t\t      bitmap_git->midx,\n-\t\t\t\t      i)) {\n+\tfor (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(bitmap_repo(bitmap_git), bitmap_git->midx, i)) {\n \t\t\twarning(_(\"could not open pack %s\"),\n \t\t\t\tbitmap_git->midx->pack_names[i]);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n+\tif (midx->base_midx) {\n+\t\tbitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);\n+\t\tbitmap_git->base_nr = bitmap_git->base->base_nr + 1;\n+\t} else {\n+\t\tbitmap_git->base_nr = 0;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -515,6 +537,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tbitmap_git->map_pos = 0;\n+\tbitmap_git->base_nr = 0;\n \tclose(fd);\n \n \tif (load_bitmap_header(bitmap_git) < 0) {\n@@ -534,8 +557,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)\n {\n \tif (bitmap_is_midx(bitmap_git)) {\n-\t\tuint32_t i;\n-\t\tint ret;\n+\t\tstruct multi_pack_index *m;\n \n \t\t/*\n \t\t * The multi-pack-index's .rev file is already loaded via\n@@ -544,10 +566,15 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \t\t * But we still need to open the individual pack .rev files,\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tret = load_pack_revindex(r, bitmap_git->midx->packs[i]);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n+\t\tfor (m = bitmap_git->midx; m; m = m->base_midx) {\n+\t\t\tuint32_t i;\n+\t\t\tint ret;\n+\n+\t\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\t\tret = load_pack_revindex(r, m->packs[i]);\n+\t\t\t\tif (ret)\n+\t\t\t\t\treturn ret;\n+\t\t\t}\n \t\t}\n \t\treturn 0;\n \t}\n@@ -573,6 +600,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n+\tif (bitmap_git->base) {\n+\t\tif (!bitmap_is_midx(bitmap_git))\n+\t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n+\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\t\tgoto failed;\n+\t}\n+\n \treturn 0;\n \n failed:\n@@ -657,10 +691,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n \n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n {\n-\tstruct repository *r = midx->repo;\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))\n+\tif (!open_midx_bitmap_1(bitmap_git, midx))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2901,6 +2934,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\tclose_midx_revindex(b->midx);\n \t}\n \tfree_pseudo_merge_map(&b->pseudo_merges);\n+\tfree_bitmap_index(b->base);\n \tfree(b);\n }\n \n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514787","messageId":"8fedd9661497c4f74f061f55438014f2726d199d.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 05/14] pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:37Z","receivedAt":"2025-03-20T17:56:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses `bitmap_for_commit()` to locate the\nEWAH-compressed bitmap corresponding to some given commit object.\n\nTeach this function about incremental MIDX bitmaps by teaching it to\nrecur on earlier bitmap layers when it fails to find a given commit in\nthe current layer.\n\nThe changes to do so are as follows:\n\n  - Avoid initializing hash_pos at its declaration, since\n    bitmap_for_commit() is now a recursive function and may receive a\n    NULL bitmap_index pointer as its first argument.\n\n  - In cases where we would previously return NULL (to indicate that a\n    lookup failed and the given bitmap_index does not contain an entry\n    corresponding to the given commit), recursively call the function on\n    the previous bitmap layer.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e84211de15..17f1087fba 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -941,18 +941,21 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t      struct commit *commit)\n {\n-\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n-\t\t\t\t\t   commit->object.oid);\n+\tkhiter_t hash_pos;\n+\tif (!bitmap_git)\n+\t\treturn NULL;\n+\n+\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn NULL;\n+\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514788","messageId":"dccc1b2d2ecf99abc88fa454ed55b36025500c60.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 06/14] pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:40Z","receivedAt":"2025-03-20T17:56:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since we may ask for a pack_id that is in an earlier MIDX layer relative\nto the one corresponding to our bitmap, use nth_midxed_pack() instead of\naccessing the ->packs array directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 17f1087fba..f3ef9e43ef 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1636,7 +1636,7 @@ static void show_objects_for_type(\n \t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n \n \t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n-\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\t} else {\n \t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n \t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514789","messageId":"e31bddd24055c9c5b04e14d16d20274dc4f87134.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 07/14] pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:43Z","receivedAt":"2025-03-20T17:56:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits in the first phase of\nincremental MIDXs, enumerate not just the packs in the current\nincremental MIDX layer, but previous ones as well.\n\nLikewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\nsingle pack from a MIDX, use the oldest layer's preferred pack as it is\nlikely to contain the largest number of reusable sections.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f3ef9e43ef..5ff1bbfd54 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2335,7 +2335,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tmulti_pack_reuse = 0;\n \n \tif (multi_pack_reuse) {\n-\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {\n \t\t\tstruct bitmapped_pack pack;\n \t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n@@ -2361,14 +2362,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tuint32_t pack_int_id;\n \n \t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n \t\t\tuint32_t preferred_pack_pos;\n \n-\t\t\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {\n+\t\t\twhile (m->base_midx)\n+\t\t\t\tm = m->base_midx;\n+\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_pos) < 0) {\n \t\t\t\twarning(_(\"unable to compute preferred pack, disabling pack-reuse\"));\n \t\t\t\treturn;\n \t\t\t}\n \n-\t\t\tpack = bitmap_git->midx->packs[preferred_pack_pos];\n+\t\t\tpack = nth_midxed_pack(m, preferred_pack_pos);\n \t\t\tpack_int_id = preferred_pack_pos;\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514790","messageId":"d9dfcb5a1b479013e94d636ceb11a33398323e18.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 08/14] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:46Z","receivedAt":"2025-03-20T17:56:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement support for the special `--test-bitmap` mode of `git rev-list`\nwhen using incremental MIDXs.\n\nThe bitmap_test_data structure is extended to contain a \"base\" pointer\nthat mirrors the structure of the bitmap chain that it is being used to\ntest.\n\nWhen we find a commit to test, we first chase down the ->base pointer to\nfind the appropriate bitmap_test_data for the bitmap layer that the\ngiven commit is contained within, and then perform the test on that\nbitmap.\n\nIn order to implement this, light modifications are made to\nbitmap_for_commit() to reimplement it in terms of a new function,\nfind_bitmap_for_commit(), which fills out a pointer which indicates the\nbitmap layer which contains the given commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 107 ++++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 86 insertions(+), 21 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5ff1bbfd54..65ad631ce1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -938,8 +938,9 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_\n \treturn NULL;\n }\n \n-struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n-\t\t\t\t      struct commit *commit)\n+static struct ewah_bitmap *find_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t  struct commit *commit,\n+\t\t\t\t\t\t  struct bitmap_index **found)\n {\n \tkhiter_t hash_pos;\n \tif (!bitmap_git)\n@@ -949,18 +950,30 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n \t\tstruct stored_bitmap *bitmap = NULL;\n \t\tif (!bitmap_git->table_lookup)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n \n \t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\n-\t\t\treturn bitmap_for_commit(bitmap_git->base, commit);\n+\t\t\treturn find_bitmap_for_commit(bitmap_git->base, commit,\n+\t\t\t\t\t\t      found);\n+\t\tif (found)\n+\t\t\t*found = bitmap_git;\n \t\treturn lookup_stored_bitmap(bitmap);\n \t}\n+\tif (found)\n+\t\t*found = bitmap_git;\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t      struct commit *commit)\n+{\n+\treturn find_bitmap_for_commit(bitmap_git, commit, NULL);\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n@@ -2513,6 +2526,8 @@ struct bitmap_test_data {\n \tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n+\n+\tstruct bitmap_test_data *base_tdata;\n };\n \n static void test_bitmap_type(struct bitmap_test_data *tdata,\n@@ -2521,6 +2536,11 @@ static void test_bitmap_type(struct bitmap_test_data *tdata,\n \tenum object_type bitmap_type = OBJ_NONE;\n \tint bitmaps_nr = 0;\n \n+\tif (bitmap_is_midx(tdata->bitmap_git)) {\n+\t\twhile (pos < tdata->bitmap_git->midx->num_objects_in_base)\n+\t\t\ttdata = tdata->base_tdata;\n+\t}\n+\n \tif (bitmap_get(tdata->commits, pos)) {\n \t\tbitmap_type = OBJ_COMMIT;\n \t\tbitmaps_nr++;\n@@ -2584,13 +2604,57 @@ static void test_show_commit(struct commit *commit, void *data)\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n+static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t total = 0;\n+\tdo {\n+\t\ttotal = st_add(total, bitmap_git->entry_count);\n+\t\tbitmap_git = bitmap_git->base;\n+\t} while (bitmap_git);\n+\n+\treturn total;\n+}\n+\n+static void bitmap_test_data_prepare(struct bitmap_test_data *tdata,\n+\t\t\t\t     struct bitmap_index *bitmap_git)\n+{\n+\tmemset(tdata, 0, sizeof(struct bitmap_test_data));\n+\n+\ttdata->bitmap_git = bitmap_git;\n+\ttdata->base = bitmap_new();\n+\ttdata->commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata->trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata->blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata->tags = ewah_to_bitmap(bitmap_git->tags);\n+\n+\tif (bitmap_git->base) {\n+\t\ttdata->base_tdata = xmalloc(sizeof(struct bitmap_test_data));\n+\t\tbitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n+\t}\n+}\n+\n+static void bitmap_test_data_release(struct bitmap_test_data *tdata)\n+{\n+\tif (!tdata)\n+\t\treturn;\n+\n+\tbitmap_test_data_release(tdata->base_tdata);\n+\tfree(tdata->base_tdata);\n+\n+\tbitmap_free(tdata->base);\n+\tbitmap_free(tdata->commits);\n+\tbitmap_free(tdata->trees);\n+\tbitmap_free(tdata->blobs);\n+\tbitmap_free(tdata->tags);\n+}\n+\n void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n-\tstruct bitmap_index *bitmap_git;\n+\tstruct bitmap_index *bitmap_git, *found;\n \tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n@@ -2599,17 +2663,28 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s, %d total)\",\n \t\tbitmap_git->version,\n \t\tbitmap_git->entry_count,\n-\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\",\n+\t\tbitmap_total_entry_count(bitmap_git));\n \n \troot = revs->pending.objects[0].item;\n-\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n+\tbm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);\n \n \tif (bm) {\n \t\tfprintf_ln(stderr, \"Found bitmap for '%s'. %d bits / %08x checksum\",\n-\t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n+\t\t\toid_to_hex(&root->oid),\n+\t\t\t(int)bm->bit_size, ewah_checksum(bm));\n+\n+\t\tif (bitmap_is_midx(found))\n+\t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n+\t\t\t\t   hash_to_hex_algop(get_midx_checksum(found->midx),\n+\t\t\t\t\t\t     revs->repo->hash_algo));\n+\t\telse\n+\t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n+\t\t\t\t   hash_to_hex_algop(found->pack->hash,\n+\t\t\t\t\t\t     revs->repo->hash_algo));\n \n \t\tresult = ewah_to_bitmap(bm);\n \t}\n@@ -2626,16 +2701,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n-\ttdata.bitmap_git = bitmap_git;\n-\ttdata.base = bitmap_new();\n-\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n-\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n-\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n-\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n+\tbitmap_test_data_prepare(&tdata, bitmap_git);\n \ttdata.prg = start_progress(revs->repo,\n \t\t\t\t   \"Verifying bitmap entries\",\n \t\t\t\t   result_popcnt);\n-\ttdata.seen = 0;\n \n \ttraverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);\n \n@@ -2647,11 +2716,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tdie(_(\"mismatch in bitmap results\"));\n \n \tbitmap_free(result);\n-\tbitmap_free(tdata.base);\n-\tbitmap_free(tdata.commits);\n-\tbitmap_free(tdata.trees);\n-\tbitmap_free(tdata.blobs);\n-\tbitmap_free(tdata.tags);\n+\tbitmap_test_data_release(&tdata);\n \tfree_bitmap_index(bitmap_git);\n }\n \n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514791","messageId":"b1bd60d25ddb86721b64eccc8bca75a50aaf9ff9.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 09/14] pack-bitmap.c: compute disk-usage with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:49Z","receivedAt":"2025-03-20T17:56:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as previous commits, use nth_midxed_pack() instead\nof accessing the MIDX's ->packs array directly to support incremental\nMIDXs.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 65ad631ce1..4086277de8 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1779,7 +1779,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n \t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n \n-\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tpack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \t\t} else {\n \t\t\tpack = bitmap_git->pack;\n@@ -3049,7 +3049,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n \n \t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n-\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\t\t\t\tstruct packed_git *pack = nth_midxed_pack(bitmap_git->midx, pack_id);\n \n \t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n \t\t\t\t\tstruct object_id oid;\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514792","messageId":"7477a8ac03737afcb42c54739931311539eaaf8e.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 10/14] pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:56Z","receivedAt":"2025-03-20T17:56:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for using pseudo-merges with incremental MIDX bitmaps by\nattempting to apply pseudo-merges from each layer when encountering a\ngiven commit during a walk.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4086277de8..1d1e1a65ca 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1082,10 +1082,15 @@ static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git\n \t\t\t\t\t\t struct commit *commit,\n \t\t\t\t\t\t uint32_t commit_pos)\n {\n-\tint ret;\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tint ret = 0;\n \n-\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n-\t\t\t\t\t     result, commit, commit_pos);\n+\twhile (curr) {\n+\t\tret += apply_pseudo_merges_for_commit(&curr->pseudo_merges,\n+\t\t\t\t\t\t      result, commit,\n+\t\t\t\t\t\t      commit_pos);\n+\t\tcurr = curr->base;\n+\t}\n \n \tif (ret)\n \t\tpseudo_merges_satisfied_nr += ret;\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514793","messageId":"0fbef17acc7fbf49fdb8b61efa2320bab94e3499.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 11/14] ewah: implement `struct ewah_or_iterator`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:56:59Z","receivedAt":"2025-03-20T17:57:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"While individual bitmap layers store different commit, type-level, and\npseudo-merge bitmaps, only the top-most layer is used to compute\nreachability traversals.\n\nMany functions which implement the aforementioned traversal rely on\nenumerating the results according to the type-level bitmaps, and so\nwould benefit from a conceptual type-level bitmap that spans multiple\nlayers.\n\nImplement `struct ewah_or_iterator` which is capable of enumerating\nmultiple EWAH bitmaps at once, and OR-ing the results together. When\ninitialized with, for example, all of the commit type bitmaps from each\nlayer, callers can pretend as if they are enumerating a large type-level\nbitmap which contains the commits from *all* bitmap layers.\n\nThere are a couple of alternative approaches which were considered:\n\n  - Decompress each EWAH bitmap and OR them together, enumerating a\n    single (non-EWAH) bitmap. This would work, but has the disadvantage\n    of decompressing a potentially large bitmap, which may not be\n    necessary if the caller does not wish to read all of it.\n\n  - Recursively call bitmap internal functions, reusing the \"result\" and\n    \"haves\" bitmap from the top-most layer. This approach resembles the\n    original implementation of this feature, but is inefficient in that\n    it both (a) requires significant refactoring to implement, and (b)\n    enumerates large sections of later bitmaps which are all zeros (as\n    they pertain to objects in earlier layers).\n\n    (b) is not so bad in and of itself, but can cause significant\n    slow-downs when combined with expensive loop bodies.\n\nThis approach (enumerating an OR'd together version of all of the\ntype-level bitmaps from each layer) produces a significantly more\nstraightforward implementation with significantly less refactoring\nrequired in order to make it work.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ewah/ewah_bitmap.c | 33 +++++++++++++++++++++++++++++++++\n ewah/ewok.h        | 12 ++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex 67f8f588e0..056c410efb 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -371,6 +371,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)\n \t\tread_new_rlw(it);\n }\n \n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr)\n+{\n+\tsize_t i;\n+\n+\tmemset(it, 0, sizeof(*it));\n+\n+\tALLOC_ARRAY(it->its, nr);\n+\tfor (i = 0; i < nr; i++)\n+\t\tewah_iterator_init(&it->its[it->nr++], parents[i]);\n+}\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n+{\n+\teword_t buf, out = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tfor (i = 0; i < it->nr; i++)\n+\t\tif (ewah_iterator_next(&buf, &it->its[i])) {\n+\t\t\tout |= buf;\n+\t\t\tret = 1;\n+\t\t}\n+\n+\t*next = out;\n+\treturn ret;\n+}\n+\n+void ewah_or_iterator_release(struct ewah_or_iterator *it)\n+{\n+\tfree(it->its);\n+}\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 5e357e2493..c29d354236 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -148,6 +148,18 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent);\n  */\n int ewah_iterator_next(eword_t *next, struct ewah_iterator *it);\n \n+struct ewah_or_iterator {\n+\tstruct ewah_iterator *its;\n+\tsize_t nr;\n+};\n+\n+void ewah_or_iterator_init(struct ewah_or_iterator *it,\n+\t\t\t   struct ewah_bitmap **parents, size_t nr);\n+\n+int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n+\n+void ewah_or_iterator_release(struct ewah_or_iterator *it);\n+\n void ewah_xor(\n \tstruct ewah_bitmap *ewah_i,\n \tstruct ewah_bitmap *ewah_j,\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514794","messageId":"439e743fd52e2023a2a215ce00703f6de1dea856.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 12/14] pack-bitmap.c: keep track of each layer's type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:57:02Z","receivedAt":"2025-03-20T17:57:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for reading the type-level bitmaps from previous bitmap layers\nby maintaining an array for each type, where each element in that type's\narray corresponds to one layer's bitmap for that type.\n\nThese fields will be used in a later commit to instantiate the 'struct\newah_or_iterator' for each type.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 53 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1d1e1a65ca..5721fa7a0f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -81,6 +81,23 @@ struct bitmap_index {\n \tstruct ewah_bitmap *blobs;\n \tstruct ewah_bitmap *tags;\n \n+\t/*\n+\t * Type index arrays when this bitmap is associated with an\n+\t * incremental multi-pack index chain.\n+\t *\n+\t * If n is the number of unique layers in the MIDX chain, then\n+\t * commits_all[n-1] is this structs 'commits' field,\n+\t * commits_all[n-2] is the commits field of this bitmap's\n+\t * 'base', and so on.\n+\t *\n+\t * When associated either with a non-incremental MIDX or a\n+\t * single packfile, these arrays each contain a single element.\n+\t */\n+\tstruct ewah_bitmap **commits_all;\n+\tstruct ewah_bitmap **trees_all;\n+\tstruct ewah_bitmap **blobs_all;\n+\tstruct ewah_bitmap **tags_all;\n+\n \t/* Map from object ID -> `stored_bitmap` for all the bitmapped commits */\n \tkh_oid_map_t *bitmaps;\n \n@@ -581,7 +598,32 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_\n \treturn load_pack_revindex(r, bitmap_git->pack);\n }\n \n-static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n+static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)\n+{\n+\tstruct bitmap_index *curr = bitmap_git;\n+\tsize_t i = bitmap_git->base_nr;\n+\n+\tALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);\n+\tALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);\n+\tALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);\n+\tALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr + 1);\n+\n+\twhile (curr) {\n+\t\tbitmap_git->commits_all[i] = curr->commits;\n+\t\tbitmap_git->trees_all[i] = curr->trees;\n+\t\tbitmap_git->blobs_all[i] = curr->blobs;\n+\t\tbitmap_git->tags_all[i] = curr->tags;\n+\n+\t\tcurr = curr->base;\n+\t\tif (curr && !i)\n+\t\t\tBUG(\"unexpected number of bitmap layers, expected %\"PRIu32,\n+\t\t\t    bitmap_git->base_nr + 1);\n+\t\ti -= 1;\n+\t}\n+}\n+\n+static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git,\n+\t\t       int recursing)\n {\n \tassert(bitmap_git->map);\n \n@@ -603,10 +645,13 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)\n \tif (bitmap_git->base) {\n \t\tif (!bitmap_is_midx(bitmap_git))\n \t\t\tBUG(\"non-MIDX bitmap has non-NULL base bitmap index\");\n-\t\tif (load_bitmap(r, bitmap_git->base) < 0)\n+\t\tif (load_bitmap(r, bitmap_git->base, 1) < 0)\n \t\t\tgoto failed;\n \t}\n \n+\tif (!recursing)\n+\t\tload_all_type_bitmaps(bitmap_git);\n+\n \treturn 0;\n \n failed:\n@@ -682,7 +727,7 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git, 0))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -2052,7 +2097,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (load_bitmap(revs->repo, bitmap_git, 0) < 0)\n \t\tgoto cleanup;\n \n \tif (!use_boundary_traversal)\n@@ -2985,6 +3030,10 @@ void free_bitmap_index(struct bitmap_index *b)\n \tewah_pool_free(b->trees);\n \tewah_pool_free(b->blobs);\n \tewah_pool_free(b->tags);\n+\tfree(b->commits_all);\n+\tfree(b->trees_all);\n+\tfree(b->blobs_all);\n+\tfree(b->tags_all);\n \tif (b->bitmaps) {\n \t\tstruct stored_bitmap *sb;\n \t\tkh_foreach_value(b->bitmaps, sb, {\n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514795","messageId":"dcb45e349e16c12ab4a1be39fcdf8ba214b713fa.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 13/14] pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:57:05Z","receivedAt":"2025-03-20T17:57:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have initialized arrays for each bitmap layer's type bitmaps\nin the previous commit, adjust existing callers to use them in\npreparation for multi-layered bitmaps.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 42 +++++++++++++++++++++++++++---------------\n 1 file changed, 27 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5721fa7a0f..6f7fd94c36 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1629,25 +1629,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t}\n }\n \n-static void init_type_iterator(struct ewah_iterator *it,\n+static void init_type_iterator(struct ewah_or_iterator *it,\n \t\t\t       struct bitmap_index *bitmap_git,\n \t\t\t       enum object_type type)\n {\n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tewah_iterator_init(it, bitmap_git->commits);\n+\t\tewah_or_iterator_init(it, bitmap_git->commits_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tcase OBJ_TREE:\n-\t\tewah_iterator_init(it, bitmap_git->trees);\n+\t\tewah_or_iterator_init(it, bitmap_git->trees_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tcase OBJ_BLOB:\n-\t\tewah_iterator_init(it, bitmap_git->blobs);\n+\t\tewah_or_iterator_init(it, bitmap_git->blobs_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tcase OBJ_TAG:\n-\t\tewah_iterator_init(it, bitmap_git->tags);\n+\t\tewah_or_iterator_init(it, bitmap_git->tags_all,\n+\t\t\t\t      bitmap_git->base_nr + 1);\n \t\tbreak;\n \n \tdefault:\n@@ -1664,7 +1668,7 @@ static void show_objects_for_type(\n \tsize_t i = 0;\n \tuint32_t offset;\n \n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tstruct bitmap *objects = bitmap_git->result;\n@@ -1672,7 +1676,7 @@ static void show_objects_for_type(\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = objects->words[i] & filter;\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n \n@@ -1714,6 +1718,8 @@ static void show_objects_for_type(\n \t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n+\n+\tewah_or_iterator_release(&it);\n }\n \n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n@@ -1765,7 +1771,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n@@ -1782,7 +1788,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * packfile.\n \t */\n \tfor (i = 0, init_type_iterator(&it, bitmap_git, type);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\tif (i < tips->word_alloc)\n \t\t\tmask &= ~tips->words[i];\n@@ -1802,6 +1808,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_release(&it);\n \tbitmap_free(tips);\n }\n \n@@ -1862,14 +1869,14 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n {\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \tstruct bitmap *tips;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t mask;\n \tuint32_t i;\n \n \ttips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB);\n \n \tfor (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB);\n-\t     i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);\n+\t     i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);\n \t     i++) {\n \t\teword_t word = to_filter->words[i] & mask;\n \t\tunsigned offset;\n@@ -1897,6 +1904,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t\t\tbitmap_unset(to_filter, pos);\n \t}\n \n+\tewah_or_iterator_release(&it);\n \tbitmap_free(tips);\n }\n \n@@ -2528,12 +2536,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \n \tuint32_t i = 0, count = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \n \tinit_type_iterator(&it, bitmap_git, type);\n \n-\twhile (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) {\n+\twhile (i < objects->word_alloc && ewah_or_iterator_next(&filter, &it)) {\n \t\teword_t word = objects->words[i++] & filter;\n \t\tcount += ewah_bit_popcount64(word);\n \t}\n@@ -2545,6 +2553,8 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \t\t\tcount++;\n \t}\n \n+\tewah_or_iterator_release(&it);\n+\n \treturn count;\n }\n \n@@ -3077,13 +3087,13 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *result = bitmap_git->result;\n \toff_t total = 0;\n-\tstruct ewah_iterator it;\n+\tstruct ewah_or_iterator it;\n \teword_t filter;\n \tsize_t i;\n \n \tinit_type_iterator(&it, bitmap_git, object_type);\n \tfor (i = 0; i < result->word_alloc &&\n-\t\t\tewah_iterator_next(&filter, &it); i++) {\n+\t\t\tewah_or_iterator_next(&filter, &it); i++) {\n \t\teword_t word = result->words[i] & filter;\n \t\tsize_t base = (i * BITS_IN_EWORD);\n \t\tunsigned offset;\n@@ -3124,6 +3134,8 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n+\tewah_or_iterator_release(&it);\n+\n \treturn total;\n }\n \n-- \n2.49.0.14.g88b49c1b34\n\n"},{"id":"514796","messageId":"13568cfa3b8a9efcc9238d08b9f940e7c0650681.1742493373.git.me@ttaylorr.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"[PATCH v5 14/14] midx: implement writing incremental MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:57:08Z","receivedAt":"2025-03-20T17:57:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap machinery has learned how to read and interact\nwith an incremental MIDX bitmap, teach the pack-bitmap-write.c machinery\n(and relevant callers from within the MIDX machinery) to write such\nbitmaps.\n\nThe details for doing so are mostly straightforward. The main changes\nare as follows:\n\n  - find_object_pos() now makes use of an extra MIDX parameter which is\n    used to locate the bit positions of objects which are from previous\n    layers (and thus do not exist in the current layer's pack_order\n    field).\n\n    (Note also that the pack_order field is moved into struct\n    write_midx_context to further simplify the callers for\n    write_midx_bitmap()).\n\n  - bitmap_writer_build_type_index() first determines how many objects\n    precede the current bitmap layer and offsets the bits it sets in\n    each respective type-level bitmap by that amount so they can be OR'd\n    together.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c                  |  3 +-\n midx-write.c                            | 57 ++++++++++------\n pack-bitmap-write.c                     | 65 +++++++++++++-----\n pack-bitmap.h                           |  4 +-\n t/t5334-incremental-multi-pack-index.sh | 87 +++++++++++++++++++++++++\n 5 files changed, 179 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 58a9b16126..a7e4bb7904 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1397,7 +1397,8 @@ static void write_pack_file(void)\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_init(&bitmap_writer,\n-\t\t\t\t\t\t   the_repository, &to_pack);\n+\t\t\t\t\t\t   the_repository, &to_pack,\n+\t\t\t\t\t\t   NULL);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t\t\t       written_list);\ndiff --git a/midx-write.c b/midx-write.c\nindex 48d6558253..0897cbd829 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -647,16 +647,22 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \treturn pack_order;\n }\n \n-static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n-\t\t\t\t     struct write_midx_context *ctx)\n+static void write_midx_reverse_index(struct write_midx_context *ctx,\n+\t\t\t\t     const char *object_dir,\n+\t\t\t\t     unsigned char *midx_hash)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tchar *tmp_file;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", ctx->repo);\n \n-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t\t    ctx->repo->hash_algo));\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(ctx->repo->hash_algo, &buf,\n+\t\t\t\t\t    object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_REV);\n+\telse\n+\t\tget_midx_filename_ext(ctx->repo->hash_algo, &buf, object_dir,\n+\t\t\t\t      midx_hash, MIDX_EXT_REV);\n \n \ttmp_file = write_rev_file_order(ctx->repo->hash_algo, NULL, ctx->pack_order,\n \t\t\t\t\tctx->entries_nr, midx_hash, WRITE_REV);\n@@ -829,22 +835,29 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \treturn cb.commits;\n }\n \n-static int write_midx_bitmap(struct repository *r, const char *midx_name,\n+static int write_midx_bitmap(struct write_midx_context *ctx,\n+\t\t\t     const char *object_dir,\n \t\t\t     const unsigned char *midx_hash,\n \t\t\t     struct packing_data *pdata,\n \t\t\t     struct commit **commits,\n \t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n \t\t\t     unsigned flags)\n {\n \tint ret, i;\n \tuint16_t options = 0;\n \tstruct bitmap_writer writer;\n \tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t    hash_to_hex_algop(midx_hash, r->hash_algo));\n+\tstruct strbuf bitmap_name = STRBUF_INIT;\n \n-\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", r);\n+\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", ctx->repo);\n+\n+\tif (ctx->incremental)\n+\t\tget_split_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,\n+\t\t\t\t\t    object_dir, midx_hash,\n+\t\t\t\t\t    MIDX_EXT_BITMAP);\n+\telse\n+\t\tget_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,\n+\t\t\t\t      object_dir, midx_hash, MIDX_EXT_BITMAP);\n \n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n@@ -861,7 +874,8 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer, r, pdata);\n+\tbitmap_writer_init(&writer, ctx->repo, pdata,\n+\t\t\t   ctx->incremental ? ctx->base_midx : NULL);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, index);\n \n@@ -879,7 +893,7 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,\n \t * bitmap_writer_finish().\n \t */\n \tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\t\tindex[ctx->pack_order[i]] = &pdata->objects[i].idx;\n \n \tbitmap_writer_select_commits(&writer, commits, commits_nr);\n \tret = bitmap_writer_build(&writer);\n@@ -887,14 +901,14 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,\n \t\tgoto cleanup;\n \n \tbitmap_writer_set_checksum(&writer, midx_hash);\n-\tbitmap_writer_finish(&writer, index, bitmap_name, options);\n+\tbitmap_writer_finish(&writer, index, bitmap_name.buf, options);\n \n cleanup:\n \tfree(index);\n-\tfree(bitmap_name);\n+\tstrbuf_release(&bitmap_name);\n \tbitmap_writer_free(&writer);\n \n-\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", r);\n+\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", ctx->repo);\n \n \treturn ret;\n }\n@@ -1077,8 +1091,6 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \tctx.repo = r;\n \n \tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n-\tif (ctx.incremental && (flags & MIDX_WRITE_BITMAP))\n-\t\tdie(_(\"cannot write incremental MIDX with bitmap\"));\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1119,6 +1131,13 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n+\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n+\t\t\t\t      hash_to_hex_algop(get_midx_checksum(m),\n+\t\t\t\t\t\t\tm->repo->hash_algo));\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n@@ -1387,7 +1406,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \n \tif (flags & MIDX_WRITE_REV_INDEX &&\n \t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n-\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n+\t\twrite_midx_reverse_index(&ctx, object_dir, midx_hash);\n \n \tif (flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n@@ -1410,8 +1429,8 @@ static int write_midx_internal(struct repository *r, const char *object_dir,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(r, midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\tif (write_midx_bitmap(&ctx, object_dir,\n+\t\t\t\t      midx_hash, &pdata, commits, commits_nr,\n \t\t\t\t      flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 34e86d4994..8a30853d2e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -26,6 +26,8 @@\n #include \"alloc.h\"\n #include \"refs.h\"\n #include \"strmap.h\"\n+#include \"midx.h\"\n+#include \"pack-revindex.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -43,7 +45,8 @@ static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer\n }\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata)\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n \tif (writer->bitmaps)\n@@ -51,6 +54,7 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n \twriter->bitmaps = kh_init_oid_map();\n \twriter->pseudo_merge_commits = kh_init_oid_map();\n \twriter->to_pack = pdata;\n+\twriter->midx = midx;\n \n \tstring_list_init_dup(&writer->pseudo_merge_groups);\n \n@@ -113,6 +117,11 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t    struct pack_idx_entry **index)\n {\n \tuint32_t i;\n+\tuint32_t base_objects = 0;\n+\n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n \n \twriter->commits = ewah_new();\n \twriter->trees = ewah_new();\n@@ -142,19 +151,19 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \n \t\tswitch (real_type) {\n \t\tcase OBJ_COMMIT:\n-\t\t\tewah_set(writer->commits, i);\n+\t\t\tewah_set(writer->commits, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TREE:\n-\t\t\tewah_set(writer->trees, i);\n+\t\t\tewah_set(writer->trees, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_BLOB:\n-\t\t\tewah_set(writer->blobs, i);\n+\t\t\tewah_set(writer->blobs, i + base_objects);\n \t\t\tbreak;\n \n \t\tcase OBJ_TAG:\n-\t\t\tewah_set(writer->tags, i);\n+\t\t\tewah_set(writer->tags, i + base_objects);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -207,19 +216,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,\n static uint32_t find_object_pos(struct bitmap_writer *writer,\n \t\t\t\tconst struct object_id *oid, int *found)\n {\n-\tstruct object_entry *entry = packlist_find(writer->to_pack, oid);\n+\tstruct object_entry *entry;\n+\n+\tentry = packlist_find(writer->to_pack, oid);\n+\tif (entry) {\n+\t\tuint32_t base_objects = 0;\n+\t\tif (writer->midx)\n+\t\t\tbase_objects = writer->midx->num_objects +\n+\t\t\t\twriter->midx->num_objects_in_base;\n+\n+\t\tif (found)\n+\t\t\t*found = 1;\n+\t\treturn oe_in_pack_pos(writer->to_pack, entry) + base_objects;\n+\t} else if (writer->midx) {\n+\t\tuint32_t at, pos;\n+\n+\t\tif (!bsearch_midx(oid, writer->midx, &at))\n+\t\t\tgoto missing;\n+\t\tif (midx_to_pack_pos(writer->midx, at, &pos) < 0)\n+\t\t\tgoto missing;\n \n-\tif (!entry) {\n \t\tif (found)\n-\t\t\t*found = 0;\n-\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n-\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n-\t\treturn 0;\n+\t\t\t*found = 1;\n+\t\treturn pos;\n \t}\n \n+missing:\n \tif (found)\n-\t\t*found = 1;\n-\treturn oe_in_pack_pos(writer->to_pack, entry);\n+\t\t*found = 0;\n+\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\treturn 0;\n }\n \n static void compute_xor_offsets(struct bitmap_writer *writer)\n@@ -586,7 +613,7 @@ int bitmap_writer_build(struct bitmap_writer *writer)\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n-\tuint32_t *mapping;\n+\tuint32_t *mapping = NULL;\n \tint closed = 1; /* until proven otherwise */\n \n \tif (writer->show_progress)\n@@ -1021,7 +1048,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \toff_t *offsets = NULL;\n-\tuint32_t i;\n+\tuint32_t i, base_objects;\n \n \tstruct bitmap_disk_header header;\n \n@@ -1047,6 +1074,12 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, writer->to_pack->nr_objects);\n \n+\tif (writer->midx)\n+\t\tbase_objects = writer->midx->num_objects +\n+\t\t\twriter->midx->num_objects_in_base;\n+\telse\n+\t\tbase_objects = 0;\n+\n \tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n@@ -1055,7 +1088,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \t\tif (commit_pos < 0)\n \t\t\tBUG(_(\"trying to write commit not in index\"));\n-\t\tstored->commit_pos = commit_pos;\n+\t\tstored->commit_pos = commit_pos + base_objects;\n \t}\n \n \twrite_selected_commits_v1(writer, f, offsets);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex d7f4b8b8e9..dd0951088f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -111,6 +111,7 @@ struct bitmap_writer {\n \n \tkh_oid_map_t *bitmaps;\n \tstruct packing_data *to_pack;\n+\tstruct multi_pack_index *midx; /* if appending to a MIDX chain */\n \n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n@@ -125,7 +126,8 @@ struct bitmap_writer {\n };\n \n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,\n-\t\t\tstruct packing_data *pdata);\n+\t\t\tstruct packing_data *pdata,\n+\t\t\tstruct multi_pack_index *midx);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 26257e5660..d30d7253d6 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -44,4 +44,91 @@ test_expect_success 'convert incremental to non-incremental' '\n \n compare_results_with_midx 'non-incremental MIDX conversion'\n \n+write_midx_layer () {\n+\tn=1\n+\tif test -f $midx_chain\n+\tthen\n+\t\tn=\"$(($(wc -l <$midx_chain) + 1))\"\n+\tfi\n+\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit $n.$i &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit multi-pack-index write --bitmap --incremental\n+}\n+\n+test_expect_success 'write initial MIDX layer' '\n+\tgit repack -ad &&\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'read bitmap from first MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'write another MIDX layer' '\n+\twrite_midx_layer\n+'\n+\n+test_expect_success 'midx verify with multiple layers' '\n+\ttest_path_is_file \"$midx_chain\" &&\n+\ttest_line_count = 2 \"$midx_chain\" &&\n+\n+\tgit multi-pack-index verify\n+'\n+\n+test_expect_success 'read bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 2.2\n+'\n+\n+test_expect_success 'read earlier bitmap from second MIDX layer' '\n+\tgit rev-list --test-bitmap 1.2\n+'\n+\n+test_expect_success 'show object from first pack' '\n+\tgit cat-file -p 1.1\n+'\n+\n+test_expect_success 'show object from second pack' '\n+\tgit cat-file -p 2.2\n+'\n+\n+for reuse in false single multi\n+do\n+\ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n+\t\trm -fr clone.git &&\n+\n+\t\tgit config pack.allowPackReuse $reuse &&\n+\t\tgit clone --no-local --bare . clone.git\n+\t'\n+done\n+\n+test_expect_success 'relink existing MIDX layer' '\n+\trm -fr \"$midxdir\" &&\n+\n+\tGIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&\n+\n+\tmidx_hash=\"$(test-tool read-midx --checksum $objdir)\" &&\n+\n+\ttest_path_is_file \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_commit another &&\n+\tgit repack -d &&\n+\tgit multi-pack-index write --bitmap --incremental &&\n+\n+\ttest_path_is_missing \"$packdir/multi-pack-index\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_missing \"$packdir/multi-pack-index-$midx_hash.rev\" &&\n+\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.midx\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.bitmap\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$midx_hash.rev\" &&\n+\ttest_line_count = 2 \"$midx_chain\"\n+\n+'\n+\n test_done\n-- \n2.49.0.14.g88b49c1b34\n"},{"id":"514797","messageId":"Z9xXM2DB+3cllQQA@nand.local","threadId":"61958","inReplyTo":"20250320175652.huJnykbhBJ9dLM6drZm3omfLTisaJX-eq1TQ5FN52h4@z","subject":"Re: [PATCH v5 08/14] pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-20T17:58:11Z","receivedAt":"2025-03-20T17:58:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Mar 20, 2025 at 01:56:52PM -0400, Taylor Blau wrote:\n> ---\n>  pack-bitmap.c | 107 ++++++++++++++++++++++++++++++++++++++++----------\n>  1 file changed, 86 insertions(+), 21 deletions(-)\n\nOops. I moved the wrong direction from 9->8 instead of 9->10 when\nsubmitting, hence the duplicate patch here.\n\nPlease disregard this one, though the rest of the round is fine.\n\nThanks,\nTaylor\n"},{"id":"514806","messageId":"CABPp-BGv_KDdviT0C6khs0gaoHLSt+ZfmbAVrAkTyT9PC1=Unw@mail.gmail.com","threadId":"61958","inReplyTo":"cover.1742493373.git.me@ttaylorr.com","subject":"Re: [PATCH v5 00/14] midx: incremental multi-pack indexes, part two","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-03-20T20:00:00Z","receivedAt":"2025-03-20T20:00:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Mar 20, 2025 at 10:56 AM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> This is another new round of my series to implement bitmap support for\n> incremental multi-pack indexes (MIDXs). It is still based on 683c54c999\n> (Git 2.49, 2025-03-14).\n>\n> == Changes since last time\n>\n> This round addresses thorough review from Elijah and Peff. The series\n> substantively is unchanged, but there are lots of little quality-of-life\n> and commit message readability improvements throughout. As usual, there\n> is a range-diff below for convenience.\n>\n[...]\n> Range-diff against v4:\n>  -:  ---------- >  1:  6af65fdaac Documentation: remove a \"future work\" item from the MIDX docs\n>  1:  f565f2fff1 !  2:  0897359506 Documentation: describe incremental MIDX bitmaps\n>     @@ Documentation/technical/multi-pack-index.adoc: objects_nr($H2) + objects_nr($H1)\n>      +`o1` and `o2` are compared as follows:\n>      +\n>      +1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then\n>     -+  `o1` is considered less than `o2`.\n>     ++  `o1` sorts ahead of `o2`.\n>      +\n>      +2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that\n>      +   MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is\n>     -+   preferred and the other is not, then the preferred one sorts first. If\n>     -+   there is a base layer (i.e. the MIDX layer is not the first layer in\n>     -+   the chain), then if `pack(o1)` appears earlier in that MIDX layer's\n>     -+   pack order, than `o1` is less than `o2`. Likewise if `pack(o2)`\n>     -+   appears earlier, than the opposite is true.\n>     ++   preferred and the other is not, then the preferred one sorts ahead of\n>     ++   the non-preferred one. If there is a base layer (i.e. the MIDX layer\n>     ++   is not the first layer in the chain), then if `pack(o1)` appears\n>     ++   earlier in that MIDX layer's pack order, then `o1` sorts ahead of\n>     ++   `o2`. Likewise if `pack(o2)` appears earlier, then the opposite is\n>     ++   true.\n>      +\n>      +3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the\n>      +   same MIDX layer. Sort `o1` and `o2` by their offset within their\n>     @@ Documentation/technical/multi-pack-index.adoc: objects_nr($H2) + objects_nr($H1)\n>      +The structure of a `*.bitmap` file belonging to an incremental MIDX\n>      +chain is identical to that of a non-incremental MIDX bitmap, or a\n>      +classic single-pack bitmap. Since objects are added to the end of the\n>     -+incremental MIDX's pseudo-pack order (see: above), it is possible to\n>     ++incremental MIDX's pseudo-pack order (see above), it is possible to\n>      +extend a bitmap when appending to the end of a MIDX chain.\n>      +\n>      +(Note: it is possible likewise to compress a contiguous sequence of MIDX\n>     -+incremental layers, and their `*.bitmap`(s) into a single layer and\n>     ++incremental layers, and their `*.bitmap` files into a single layer and\n>      +`*.bitmap`, but this is not yet implemented.)\n>      +\n>      +The object positions used are global within the pseudo-pack order, so\n>  2:  f2a232e556 !  3:  5eac0d1485 pack-revindex: prepare for incremental MIDX bitmaps\n>     @@ Commit message\n>              incremental or not.\n>\n>            - pack_pos_to_midx() and midx_to_pack_pos() now both take in a global\n>     -        object position in the MIDX pseudo-pack order, and finds the\n>     +        object position in the MIDX pseudo-pack order, and find the\n>              earliest containing MIDX (similar to midx.c::midx_for_object().\n>\n>            - midx_pack_order_cmp() adjusts its call to pack_pos_to_midx() by the\n>     @@ pack-bitmap.c: static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *ind\n>         return read_bitmap(index->map, index->map_size, &index->map_pos);\n>       }\n>\n>     -+static uint32_t bitmap_non_extended_bits(struct bitmap_index *index)\n>     ++static uint32_t bitmap_num_objects_total(struct bitmap_index *index)\n>      +{\n>      +  if (index->midx) {\n>      +          struct multi_pack_index *m = index->midx;\n>     @@ pack-bitmap.c: static inline int bitmap_position_extended(struct bitmap_index *b\n>         if (pos < kh_end(positions)) {\n>                 int bitmap_pos = kh_value(positions, pos);\n>      -          return bitmap_pos + bitmap_num_objects(bitmap_git);\n>     -+          return bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n>     ++          return bitmap_pos + bitmap_num_objects_total(bitmap_git);\n>         }\n>\n>         return -1;\n>     @@ pack-bitmap.c: static int ext_index_add_object(struct bitmap_index *bitmap_git,\n>         }\n>\n>      -  return bitmap_pos + bitmap_num_objects(bitmap_git);\n>     -+  return bitmap_pos + bitmap_non_extended_bits(bitmap_git);\n>     ++  return bitmap_pos + bitmap_num_objects_total(bitmap_git);\n>       }\n>\n>       struct bitmap_show_data {\n>     @@ pack-bitmap.c: static void show_extended_objects(struct bitmap_index *bitmap_git\n>\n>      -          if (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))\n>      +          if (!bitmap_get(objects,\n>     -+                          st_add(bitmap_non_extended_bits(bitmap_git), i)))\n>     ++                          st_add(bitmap_num_objects_total(bitmap_git),\n>     ++                                 i)))\n>                         continue;\n>\n>                 obj = eindex->objects[i];\n>     @@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitma\n>          */\n>         for (i = 0; i < eindex->count; i++) {\n>      -          size_t pos = st_add(i, bitmap_num_objects(bitmap_git));\n>     -+          size_t pos = st_add(i, bitmap_non_extended_bits(bitmap_git));\n>     ++          size_t pos = st_add(i, bitmap_num_objects_total(bitmap_git));\n>                 if (eindex->objects[i]->type == type &&\n>                     bitmap_get(to_filter, pos) &&\n>                     !bitmap_get(tips, pos))\n>     @@ pack-bitmap.c: static unsigned long get_size_by_pos(struct bitmap_index *bitmap_\n>         oi.sizep = &size;\n>\n>      -  if (pos < bitmap_num_objects(bitmap_git)) {\n>     -+  if (pos < bitmap_non_extended_bits(bitmap_git)) {\n>     ++  if (pos < bitmap_num_objects_total(bitmap_git)) {\n>                 struct packed_git *pack;\n>                 off_t ofs;\n>\n>      @@ pack-bitmap.c: static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n>     +                   die(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n>                 }\n>         } else {\n>     ++          size_t eindex_pos = pos - bitmap_num_objects_total(bitmap_git);\n>                 struct eindex *eindex = &bitmap_git->ext_index;\n>      -          struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n>     -+          struct object *obj = eindex->objects[pos - bitmap_non_extended_bits(bitmap_git)];\n>     ++          struct object *obj = eindex->objects[eindex_pos];\n>                 if (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n>                                              &oi, 0) < 0)\n>                         die(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n>     @@ pack-bitmap.c: static void filter_packed_objects_from_bitmap(struct bitmap_index\n>         size_t i, pos;\n>\n>      -  objects_nr = bitmap_num_objects(bitmap_git);\n>     -+  objects_nr = bitmap_non_extended_bits(bitmap_git);\n>     ++  objects_nr = bitmap_num_objects_total(bitmap_git);\n>         pos = objects_nr / BITS_IN_EWORD;\n>\n>         if (pos > result->word_alloc)\n>     @@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git\n>                 if (eindex->objects[i]->type == type &&\n>                     bitmap_get(objects,\n>      -                         st_add(bitmap_num_objects(bitmap_git), i)))\n>     -+                         st_add(bitmap_non_extended_bits(bitmap_git), i)))\n>     ++                         st_add(bitmap_num_objects_total(bitmap_git), i)))\n>                         count++;\n>         }\n>\n>     @@ pack-bitmap.c: uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n>                     \"extension\");\n>\n>      -  num_objects = bitmap_num_objects(bitmap_git);\n>     -+  num_objects = bitmap_non_extended_bits(bitmap_git);\n>     ++  num_objects = bitmap_num_objects_total(bitmap_git);\n>         CALLOC_ARRAY(reposition, num_objects);\n>\n>         for (i = 0; i < num_objects; ++i) {\n>     @@ pack-bitmap.c: static off_t get_disk_usage_for_extended(struct bitmap_index *bit\n>\n>                 if (!bitmap_get(result,\n>      -                          st_add(bitmap_num_objects(bitmap_git), i)))\n>     -+                          st_add(bitmap_non_extended_bits(bitmap_git), i)))\n>     ++                          st_add(bitmap_num_objects_total(bitmap_git),\n>     ++                                 i)))\n>                         continue;\n>\n>                 if (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,\n>  3:  aca0318fb1 !  4:  922ea2f607 pack-bitmap.c: open and store incremental bitmap layers\n>     @@ Commit message\n>          with the previous MIDX layer.\n>\n>          The changes in this commit are mostly boilerplate to open the correct\n>     -    bitmap(s), add them to the chain bitmap layers along the \"base\" pointer,\n>     -    ensures that the correct packs and their reverse indexes are loaded\n>     -    across MIDX layers, etc.\n>     +    bitmap(s), add them to the chain of bitmap layers along the \"base\"\n>     +    pointer, ensure that the correct packs and their reverse indexes are\n>     +    loaded across MIDX layers, etc.\n>\n>          While we're at it, keep track of a base_nr field to indicate how many\n>          bitmap layers (including the current bitmap) exist. This will be used in\n>  4:  832fd0e8dc =  5:  8fedd96614 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs\n>  5:  c7c9f89956 =  6:  dccc1b2d2e pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs\n>  6:  14d3d80c3d !  7:  e31bddd240 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs\n>     @@ Commit message\n>\n>          Likewise, in reuse_partial_packfile_from_bitmap(), when reusing only a\n>          single pack from a MIDX, use the oldest layer's preferred pack as it is\n>     -    likely to contain the most amount of reusable sections.\n>     +    likely to contain the largest number of reusable sections.\n>\n>          Signed-off-by: Taylor Blau <me@ttaylorr.com>\n>\n>  7:  b45a9ccbc2 !  8:  d9dfcb5a1b pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs\n>     @@ pack-bitmap.c: static void test_show_commit(struct commit *commit, void *data)\n>      +  tdata->tags = ewah_to_bitmap(bitmap_git->tags);\n>      +\n>      +  if (bitmap_git->base) {\n>     -+          CALLOC_ARRAY(tdata->base_tdata, 1);\n>     ++          tdata->base_tdata = xmalloc(sizeof(struct bitmap_test_data));\n>      +          bitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);\n>      +  }\n>      +}\n>  8:  c1eefeae99 =  9:  b1bd60d25d pack-bitmap.c: compute disk-usage with incremental MIDXs\n>  9:  11c4b7b949 = 10:  7477a8ac03 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs\n> 10:  cb08ad6a62 ! 11:  0fbef17acc ewah: implement `struct ewah_or_iterator`\n>     @@ ewah/ewah_bitmap.c: void ewah_iterator_init(struct ewah_iterator *it, struct ewa\n>      +  return ret;\n>      +}\n>      +\n>     -+void ewah_or_iterator_free(struct ewah_or_iterator *it)\n>     ++void ewah_or_iterator_release(struct ewah_or_iterator *it)\n>      +{\n>      +  free(it->its);\n>      +}\n>     @@ ewah/ewok.h: void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitma\n>      +\n>      +int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n>      +\n>     -+void ewah_or_iterator_free(struct ewah_or_iterator *it);\n>     ++void ewah_or_iterator_release(struct ewah_or_iterator *it);\n>      +\n>       void ewah_xor(\n>         struct ewah_bitmap *ewah_i,\n> 11:  a29f4ee60d ! 12:  439e743fd5 pack-bitmap.c: keep track of each layer's type bitmaps\n>     @@ pack-bitmap.c: struct bitmap_index {\n>      +   * commits_all[n-2] is the commits field of this bitmap's\n>      +   * 'base', and so on.\n>      +   *\n>     -+   * When either associated either with a non-incremental MIDX, or\n>     -+   * a single packfile, these arrays each contain a single\n>     -+   * element.\n>     ++   * When associated either with a non-incremental MIDX or a\n>     ++   * single packfile, these arrays each contain a single element.\n>      +   */\n>      +  struct ewah_bitmap **commits_all;\n>      +  struct ewah_bitmap **trees_all;\n> 12:  a1cf65bedc ! 13:  dcb45e349e pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators\n>     @@ pack-bitmap.c: static void show_objects_for_type(\n>                 }\n>         }\n>      +\n>     -+  ewah_or_iterator_free(&it);\n>     ++  ewah_or_iterator_release(&it);\n>       }\n>\n>       static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n>     @@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitma\n>                         bitmap_unset(to_filter, pos);\n>         }\n>\n>     -+  ewah_or_iterator_free(&it);\n>     ++  ewah_or_iterator_release(&it);\n>         bitmap_free(tips);\n>       }\n>\n>     @@ pack-bitmap.c: static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_\n>                         bitmap_unset(to_filter, pos);\n>         }\n>\n>     -+  ewah_or_iterator_free(&it);\n>     ++  ewah_or_iterator_release(&it);\n>         bitmap_free(tips);\n>       }\n>\n>     @@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git\n>                         count++;\n>         }\n>\n>     -+  ewah_or_iterator_free(&it);\n>     ++  ewah_or_iterator_release(&it);\n>      +\n>         return count;\n>       }\n>     @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_\n>                 }\n>         }\n>\n>     -+  ewah_or_iterator_free(&it);\n>     ++  ewah_or_iterator_release(&it);\n>      +\n>         return total;\n>       }\n> 13:  d0d564685b ! 14:  13568cfa3b midx: implement writing incremental MIDX bitmaps\n>     @@ builtin/pack-objects.c: static void write_pack_file(void)\n>                                 bitmap_writer_build_type_index(&bitmap_writer,\n>                                                                written_list);\n>\n>     - ## ewah/ewah_bitmap.c ##\n>     -@@ ewah/ewah_bitmap.c: int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)\n>     -   return ret;\n>     - }\n>     -\n>     --void ewah_or_iterator_free(struct ewah_or_iterator *it)\n>     -+void ewah_or_iterator_release(struct ewah_or_iterator *it)\n>     - {\n>     -   free(it->its);\n>     - }\n>     -\n>     - ## ewah/ewok.h ##\n>     -@@ ewah/ewok.h: void ewah_or_iterator_init(struct ewah_or_iterator *it,\n>     -\n>     - int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);\n>     -\n>     --void ewah_or_iterator_free(struct ewah_or_iterator *it);\n>     -+void ewah_or_iterator_release(struct ewah_or_iterator *it);\n>     -\n>     - void ewah_xor(\n>     -   struct ewah_bitmap *ewah_i,\n>     -\n>       ## midx-write.c ##\n>      @@ midx-write.c: static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n>         return pack_order;\n>     @@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n>\n>         write_selected_commits_v1(writer, f, offsets);\n>\n>     - ## pack-bitmap.c ##\n>     -@@ pack-bitmap.c: static void show_objects_for_type(\n>     -           }\n>     -   }\n>     -\n>     --  ewah_or_iterator_free(&it);\n>     -+  ewah_or_iterator_release(&it);\n>     - }\n>     -\n>     - static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n>     -@@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n>     -                   bitmap_unset(to_filter, pos);\n>     -   }\n>     -\n>     --  ewah_or_iterator_free(&it);\n>     -+  ewah_or_iterator_release(&it);\n>     -   bitmap_free(tips);\n>     - }\n>     -\n>     -@@ pack-bitmap.c: static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n>     -                   bitmap_unset(to_filter, pos);\n>     -   }\n>     -\n>     --  ewah_or_iterator_free(&it);\n>     -+  ewah_or_iterator_release(&it);\n>     -   bitmap_free(tips);\n>     - }\n>     -\n>     -@@ pack-bitmap.c: static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n>     -                   count++;\n>     -   }\n>     -\n>     --  ewah_or_iterator_free(&it);\n>     -+  ewah_or_iterator_release(&it);\n>     -\n>     -   return count;\n>     - }\n>     -@@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n>     -           }\n>     -   }\n>     -\n>     --  ewah_or_iterator_free(&it);\n>     -+  ewah_or_iterator_release(&it);\n>     -\n>     -   return total;\n>     - }\n>     -\n>       ## pack-bitmap.h ##\n>      @@ pack-bitmap.h: struct bitmap_writer {\n>\n>     @@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'convert incrementa\n>      +'\n>      +\n>      +test_expect_success 'midx verify with multiple layers' '\n>     ++  test_path_is_file \"$midx_chain\" &&\n>     ++  test_line_count = 2 \"$midx_chain\" &&\n>     ++\n>      +  git multi-pack-index verify\n>      +'\n>      +\n>\n> base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e\n> --\n> 2.49.0.14.g88b49c1b34\n\nReading the range-diff plus the new first patch, all my feedback has\nbeen addressed with this round.\n"},{"id":"515108","messageId":"20250326180839.GD2508080@coredump.intra.peff.net","threadId":"61958","inReplyTo":"Z9oK2XBF9plTBbnl@nand.local","subject":"Re: [PATCH v4 02/13] pack-revindex: prepare for incremental MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-03-26T18:08:39Z","receivedAt":"2025-03-26T18:08:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 18, 2025 at 08:07:53PM -0400, Taylor Blau wrote:\n\n> > Right; some callers care about the number of objects in *their* layer,\n> > like computing the size of some bitmap extensions, bounds-checking\n> > pseudo-merge commit lookups, or generating positions for objects in the\n> > extended index.\n> >\n> > I'm happy to include that discussion somewhere in the commit message or\n> > as a comment nearby bitmap_non_extended_bits(), but I'm not sure which\n> > is better. If you have thoughts, LMK.\n> \n> I renamed this function to bitmap_num_objects_total(), which I think\n> more clearly distinguishes it from bitmap_num_objects(). If you have\n> other thoughts or things you think I should do in addition to that, LMK.\n\nYeah, that name is much more clear to me.\n\n-Peff\n"}]}