{"thread":{"id":"55464","subject":"[PATCH 00/22] multi-pack reachability bitmaps","startedAt":"2021-04-09T18:10:41Z","lastAt":"2021-09-02T09:45:03Z","messageCount":273,"participants":["Taylor Blau","Jonathan Tan","Junio C Hamano","Ævar Arnfjörð Bjarmason","Jeff King","Johannes Berg","brian m. carlson","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":22},"messages":[{"id":"421424","messageId":"cover.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":null,"subject":"[PATCH 00/22] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:10:33Z","receivedAt":"2021-04-09T18:10:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements multi-pack reachability bitmaps. It is based on\n'master' after merging 'tb/pack-preferred-tips-to-give-bitmap'.\n\nThis is an extension of the classic single-pack bitmaps. Instead of\nmapping between objects and bit positions according to each object's\npack-relative position, multi-pack bitmaps use each object's position in\na kind of \"pseudo pack\".\n\nThe pseudo pack doesn't refer to a physical packfile, but instead a\nconceptual ordering of objects in a multi-pack index. This ordering is\nreflected in the MIDX's .rev file, which is used extensively to power\nmulti-pack bitmaps.\n\nThis somewhat lengthy series is organized as follows:\n\n - The first eight patches are cleanup and preparation.\n\n - The next three patches factor out functions which have different\n implementations based on whether a bitmap is tied to a pack or MIDX.\n\n - The next two patches implement support for reading and writing\n multi-pack bitmaps.\n\n - The remaining tests prepare for a new\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP mode of running the test\n suite, and add new tests covering the new multi-pack bitmap\n behavior.\n\nYou can experiment with the new functionality by running \"git\nmulti-pack-index write --bitmap\", which updates the multi-pack index (if\nnecessary), and writes out a corresponding .bitmap file. Eventually,\nsupport for invoking the above during \"git repack\" will be introduced,\nbut this is done in a separate series.\n\nThese patches have been extracted from a version which has been running\non every repository on GitHub for the past few weeks.\n\nThanks in advance for your review (including on all of the many series leading\nup to this one).\n\nJeff King (1):\n t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n\nTaylor Blau (21):\n pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n pack-bitmap-write.c: free existing bitmaps\n Documentation: build 'technical/bitmap-format' by default\n Documentation: describe MIDX-based bitmaps\n midx: make a number of functions non-static\n midx: clear auxiliary .rev after replacing the MIDX\n midx: respect 'core.multiPackIndex' when writing\n pack-bitmap.c: introduce 'bitmap_num_objects()'\n pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n pack-bitmap: read multi-pack bitmaps\n pack-bitmap: write multi-pack bitmaps\n t5310: move some tests to lib-bitmap.sh\n t/helper/test-read-midx.c: add --checksum mode\n t5326: test multi-pack bitmap behavior\n t5319: don't write MIDX bitmaps in t5319\n t7700: update to work with MIDX bitmap test knob\n midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n p5310: extract full and partial bitmap tests\n p5326: perf tests for MIDX bitmaps\n\n Documentation/Makefile | 1 +\n Documentation/git-multi-pack-index.txt | 12 +-\n Documentation/technical/bitmap-format.txt | 72 ++-\n Documentation/technical/multi-pack-index.txt | 10 +-\n builtin/multi-pack-index.c | 2 +\n builtin/pack-objects.c | 15 +-\n builtin/repack.c | 13 +-\n ci/run-build-and-tests.sh | 1 +\n midx.c | 216 ++++++++-\n midx.h | 5 +\n pack-bitmap-write.c | 79 +++-\n pack-bitmap.c | 463 +++++++++++++++++--\n pack-bitmap.h | 8 +-\n packfile.c | 2 +-\n t/README | 4 +\n t/helper/test-read-midx.c | 16 +-\n t/lib-bitmap.sh | 216 +++++++++\n t/perf/lib-bitmap.sh | 69 +++\n t/perf/p5310-pack-bitmaps.sh | 65 +--\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++\n t/t5310-pack-bitmaps.sh | 208 +--------\n t/t5319-multi-pack-index.sh | 3 +-\n t/t5326-multi-pack-bitmaps.sh | 278 +++++++++++\n t/t7700-repack.sh | 18 +-\n 24 files changed, 1435 insertions(+), 384 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"421425","messageId":"2d1c6ccab5e2feaa4db1bcd08178a698b3e00241.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 01/22] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:10:39Z","receivedAt":"2021-04-09T18:10:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The special `--test-bitmap` mode of `git rev-list` is used to compare\nthe result of an object traversal with a bitmap to check its integrity.\nThis mode does not, however, assert that the types of reachable objects\nare stored correctly.\n\nHarden this mode by teaching it to also check that each time an object's\nbit is marked, the corresponding bit should be set in exactly one of the\ntype bitmaps (whose type matches the object's true type).\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3ed15431cd..d45e91db1e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1263,10 +1263,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,\n struct bitmap_test_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\n+\tstruct bitmap *commits;\n+\tstruct bitmap *trees;\n+\tstruct bitmap *blobs;\n+\tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n };\n \n+static void test_bitmap_type(struct bitmap_test_data *tdata,\n+\t\t\t struct object *obj, int pos)\n+{\n+\tenum object_type bitmap_type = OBJ_NONE;\n+\tint bitmaps_nr = 0;\n+\n+\tif (bitmap_get(tdata->commits, pos)) {\n+\t\tbitmap_type = OBJ_COMMIT;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->trees, pos)) {\n+\t\tbitmap_type = OBJ_TREE;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->blobs, pos)) {\n+\t\tbitmap_type = OBJ_BLOB;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->tags, pos)) {\n+\t\tbitmap_type = OBJ_TAG;\n+\t\tbitmaps_nr++;\n+\t}\n+\n+\tif (!bitmap_type)\n+\t\tdie(\"object %s not found in type bitmaps\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmaps_nr > 1)\n+\t\tdie(\"object %s does not have a unique type\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmap_type != obj->type)\n+\t\tdie(\"object %s: real type %s, expected: %s\",\n+\t\t oid_to_hex(&obj->oid),\n+\t\t type_name(obj->type),\n+\t\t type_name(bitmap_type));\n+}\n+\n static void test_show_object(struct object *object, const char *name,\n \t\t\t void *data)\n {\n@@ -1276,6 +1318,7 @@ static void test_show_object(struct object *object, const char *name,\n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&object->oid));\n+\ttest_bitmap_type(tdata, object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1290,6 +1333,7 @@ static void test_show_commit(struct commit *commit, void *data)\n \t\t\t\t &commit->object.oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&commit->object.oid));\n+\ttest_bitmap_type(tdata, &commit->object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1337,6 +1381,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \ttdata.bitmap_git = bitmap_git;\n \ttdata.base = bitmap_new();\n+\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n \ttdata.seen = 0;\n \n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421426","messageId":"d199954ef2246165f32a4f703d1e0ebf43847152.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 02/22] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:10:44Z","receivedAt":"2021-04-09T18:10:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The set of objects covered by a bitmap must be closed under\nreachability, since it must be the case that there is a valid bit\nposition assigned for every possible reachable object (otherwise the\nbitmaps would be incomplete).\n\nPack bitmaps are never written from 'git repack' unless repacking\nall-into-one, and so we never write non-closed bitmaps.\n\nBut multi-pack bitmaps change this, since it isn't known whether the\nset of objects in the MIDX is closed under reachability until walking\nthem. Plumb through a bit that is set when a reachable object isn't\nfound.\n\nAs soon as a reachable object isn't found in the set of objects to\ninclude in the bitmap, bitmap_writer_build() knows that the set is not\nclosed, and so it now fails gracefully.\n\n(The new conditional in builtin/pack-objects.c:bitmap_writer_build()\nguards against other failure modes, but is never triggered here, because\nof the all-into-one detail above. This return value will be important to\ncheck from the multi-pack index caller.)\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 +-\n pack-bitmap-write.c | 76 +++++++++++++++++++++++++++++-------------\n pack-bitmap.h | 2 +-\n 3 files changed, 56 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a1e33d7507..5205dde2e1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1116,7 +1116,8 @@ static void write_pack_file(void)\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n-\t\t\t\tbitmap_writer_build(&to_pack);\n+\t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n+\t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..e829c46649 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit)\n \twriter.selected_nr++;\n }\n \n-static uint32_t find_object_pos(const struct object_id *oid)\n+static uint32_t find_object_pos(const struct object_id *oid, int *found)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n \n \tif (!entry) {\n-\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\tif (found)\n+\t\t\t*found = 0;\n+\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\t\treturn 0;\n \t}\n \n+\tif (found)\n+\t\t*found = 1;\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n@@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n \n-static void fill_bitmap_tree(struct bitmap *bitmap,\n-\t\t\t struct tree *tree)\n+static int fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n {\n+\tint found;\n \tuint32_t pos;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t * If our bit is already set, then there is nothing to do. Both this\n \t * tree and all of its children will be set.\n \t */\n-\tpos = find_object_pos(&tree->object.oid);\n+\tpos = find_object_pos(&tree->object.oid, &found);\n+\tif (!found)\n+\t\treturn -1;\n \tif (bitmap_get(bitmap, pos))\n-\t\treturn;\n+\t\treturn 0;\n \tbitmap_set(bitmap, pos);\n \n \tif (parse_tree(tree) < 0)\n@@ -355,11 +363,15 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \twhile (tree_entry(&desc, &entry)) {\n \t\tswitch (object_type(entry.mode)) {\n \t\tcase OBJ_TREE:\n-\t\t\tfill_bitmap_tree(bitmap,\n-\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tif (fill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid)) < 0)\n+\t\t\t\treturn -1;\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n-\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tpos = find_object_pos(&entry.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(bitmap, pos);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\t/* Gitlink, etc; not reachable */\n@@ -368,15 +380,18 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t}\n \n \tfree_tree_buffer(tree);\n+\treturn 0;\n }\n \n-static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue,\n-\t\t\t struct prio_queue *tree_queue,\n-\t\t\t struct bitmap_index *old_bitmap,\n-\t\t\t const uint32_t *mapping)\n+static int fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n+\tint found;\n+\tuint32_t pos;\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n@@ -401,11 +416,16 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\tif (!found)\n+\t\t\treturn -1;\n+\t\tbitmap_set(ent->bitmap, pos);\n \t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n-\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n \t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n \t\t\t\tbitmap_set(ent->bitmap, pos);\n \t\t\t\tprio_queue_put(queue, p->item);\n@@ -413,8 +433,12 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t}\n \t}\n \n-\twhile (tree_queue->nr)\n-\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n+\twhile (tree_queue->nr) {\n+\t\tif (fill_bitmap_tree(ent->bitmap,\n+\t\t\t\t prio_queue_get(tree_queue)) < 0)\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -432,7 +456,7 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+int bitmap_writer_build(struct packing_data *to_pack)\n {\n \tstruct bitmap_builder bb;\n \tsize_t i;\n@@ -441,6 +465,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n \tuint32_t *mapping;\n+\tint closed = 1; /* until proven otherwise */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n-\t\t\t\t old_bitmap, mapping);\n+\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping) < 0) {\n+\t\t\tclosed = 0;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -499,7 +527,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \tstop_progress(&writer.progress);\n \n-\tcompute_xor_offsets();\n+\tif (closed)\n+\t\tcompute_xor_offsets();\n+\treturn closed;\n }\n \n /**\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 78f2b3ff79..988ed3a30d 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -86,7 +86,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n-void bitmap_writer_build(struct packing_data *to_pack);\n+int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421428","messageId":"014c18b8965fda2c5feb4ec5c4fe9d82f213a744.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 03/22] pack-bitmap-write.c: free existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:10:48Z","receivedAt":"2021-04-09T18:11:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new bitmap, the bitmap writer code attempts to read the\nexisting bitmap (if one is present). This is done in order to quickly\npermute the bits of any bitmaps for commits which appear in the existing\nbitmap, and were also selected for the new bitmap.\n\nBut since this code was added in 341fa34887 (pack-bitmap-write: use\nexisting bitmaps, 2020-12-08), the resources associated with opening an\nexisting bitmap were never released.\n\nIt's fine to ignore this, but it's bad hygiene. It will also cause a\nproblem for the multi-pack-index builtin, which will be responsible not\nonly for writing bitmaps, but also for expiring any old multi-pack\nbitmaps.\n\nIf an existing bitmap was reused here, it will also be expired. That\nwill cause a problem on platforms which require file resources to be\nclosed before unlinking them, like Windows. Avoid this by ensuring we\nclose reused bitmaps with free_bitmap_index() before removing them.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e829c46649..f90e100e3e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tclear_prio_queue(&queue);\n \tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree_bitmap_index(old_bitmap);\n \tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421429","messageId":"46de889cd2d5314cede0cb2bf6d570638015235b.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 04/22] Documentation: build 'technical/bitmap-format' by default","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:10:59Z","receivedAt":"2021-04-09T18:11:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Even though the 'TECH_DOCS' variable was introduced all the way back in\n5e00439f0a (Documentation: build html for all files in technical and\nhowto, 2012-10-23), the 'bitmap-format' document was never added to that\nlist when it was created.\n\nPrepare for changes to this file by including it in the list of\ntechnical documentation that 'make doc' will build by default.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/Makefile | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 874a01d7a8..6d60c8c165 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -83,6 +83,7 @@ SP_ARTICLES += $(API_DOCS)\n TECH_DOCS += MyFirstContribution\n TECH_DOCS += MyFirstObjectWalk\n TECH_DOCS += SubmittingPatches\n+TECH_DOCS += technical/bitmap-format\n TECH_DOCS += technical/hash-function-transition\n TECH_DOCS += technical/http-protocol\n TECH_DOCS += technical/index-format\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421430","messageId":"c76dfc198e2a458db241d1e53d91b07b9b2bc2ec.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 06/22] midx: make a number of functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:08Z","receivedAt":"2021-04-09T18:11:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These functions will be called from outside of midx.c in a subsequent\npatch.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 ++--\n midx.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 9e86583172..5249802326 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -48,12 +48,12 @@ static uint8_t oid_version(void)\n \t}\n }\n \n-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - the_hash_algo->rawsz;\n }\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex 8684cf0fef..1172df1a71 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -42,6 +42,8 @@ struct multi_pack_index {\n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n \n+const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+char *get_midx_filename(const char *object_dir);\n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421431","messageId":"0d4822a64e3b70cc315ff572145e5d5e95b958d9.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 05/22] Documentation: describe MIDX-based bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:03Z","receivedAt":"2021-04-09T18:11:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Update the technical documentation to describe the multi-pack bitmap\nformat. This patch merely introduces the new format, and describes its\nhigh-level ideas. Git does not yet know how to read nor write these\nmulti-pack variants, and so the subsequent patches will:\n\n - Introduce code to interpret multi-pack bitmaps, according to this\n document.\n\n - Then, introduce code to write multi-pack bitmaps from the 'git\n multi-pack-index write' sub-command.\n\nFinally, the implementation will gain tests in subsequent patches (as\nopposed to inline with the patch teaching Git how to write multi-pack\nbitmaps) to avoid a cyclic dependency.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 72 ++++++++++++++++----\n Documentation/technical/multi-pack-index.txt | 10 +--\n 2 files changed, 61 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f8c18a0f7a..25221c7ec8 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -1,6 +1,45 @@\n GIT bitmap v1 format\n ====================\n \n+== Pack and multi-pack bitmaps\n+\n+Bitmaps store reachability information about the set of objects in a packfile,\n+or a multi-pack index (MIDX). The former is defined obviously, and the latter is\n+defined as the union of objects in packs contained in the MIDX.\n+\n+A bitmap may belong to either one pack, or the repository's multi-pack index (if\n+it exists). A repository may have at most one bitmap.\n+\n+An object is uniquely described by its bit position within a bitmap:\n+\n+\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n+\tthe __n__th object in pack order. For a function `offset` which maps\n+\tobjects to their byte offset within a pack, pack order is defined as\n+\tfollows:\n+\n+\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n+\n+\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n+\t__n__th object in MIDX order. With an additional function `pack` which\n+\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n+\tis defined as follows:\n+\n+\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n+\n+\tThe ordering between packs is done lexicographically by the pack name,\n+\twith the exception of the preferred pack, which sorts ahead of all other\n+\tpacks.\n+\n+The on-disk representation (described below) of a bitmap is the same regardless\n+of whether or not that bitmap belongs to a packfile or a MIDX. The only\n+difference is the interpretation of the bits, which is described above.\n+\n+Certain bitmap extensions are supported (see: Appendix B). No extensions are\n+required for bitmaps corresponding to packfiles. For bitmaps that correspond to\n+MIDXs, both the bit-cache and rev-cache extensions are required.\n+\n+== On-disk format\n+\n \t- A header appears at the beginning:\n \n \t\t4-byte signature: {'B', 'I', 'T', 'M'}\n@@ -14,17 +53,19 @@ GIT bitmap v1 format\n \t\t\tThe following flags are supported:\n \n \t\t\t- BITMAP_OPT_FULL_DAG (0x1) REQUIRED\n-\t\t\tThis flag must always be present. It implies that the bitmap\n-\t\t\tindex has been generated for a packfile with full closure\n-\t\t\t(i.e. where every single object in the packfile can find\n-\t\t\t its parent links inside the same packfile). This is a\n-\t\t\trequirement for the bitmap index format, also present in JGit,\n-\t\t\tthat greatly reduces the complexity of the implementation.\n+\t\t\tThis flag must always be present. It implies that the\n+\t\t\tbitmap index has been generated for a packfile or\n+\t\t\tmulti-pack index (MIDX) with full closure (i.e. where\n+\t\t\tevery single object in the packfile/MIDX can find its\n+\t\t\tparent links inside the same packfile/MIDX). This is a\n+\t\t\trequirement for the bitmap index format, also present in\n+\t\t\tJGit, that greatly reduces the complexity of the\n+\t\t\timplementation.\n \n \t\t\t- BITMAP_OPT_HASH_CACHE (0x4)\n \t\t\tIf present, the end of the bitmap file contains\n \t\t\t`N` 32-bit name-hash values, one per object in the\n-\t\t\tpack. The format and meaning of the name-hash is\n+\t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n \t\t4-byte entry count (network byte order)\n@@ -33,7 +74,8 @@ GIT bitmap v1 format\n \n \t\t20-byte checksum\n \n-\t\t\tThe SHA1 checksum of the pack this bitmap index belongs to.\n+\t\t\tThe SHA1 checksum of the pack/MIDX this bitmap index\n+\t\t\tbelongs to.\n \n \t- 4 EWAH bitmaps that act as type indexes\n \n@@ -50,7 +92,7 @@ GIT bitmap v1 format\n \t\t\t- Tags\n \n \t\tIn each bitmap, the `n`th bit is set to true if the `n`th object\n-\t\tin the packfile is of that type.\n+\t\tin the packfile or multi-pack index is of that type.\n \n \t\tThe obvious consequence is that the OR of all 4 bitmaps will result\n \t\tin a full set (all bits set), and the AND of all 4 bitmaps will\n@@ -62,8 +104,9 @@ GIT bitmap v1 format\n \t\tEach entry contains the following:\n \n \t\t- 4-byte object position (network byte order)\n-\t\t\tThe position **in the index for the packfile** where the\n-\t\t\tbitmap for this commit is found.\n+\t\t\tThe position **in the index for the packfile or\n+\t\t\tmulti-pack index** where the bitmap for this commit is\n+\t\t\tfound.\n \n \t\t- 1-byte XOR-offset\n \t\t\tThe xor offset used to compress this bitmap. For an entry\n@@ -146,10 +189,11 @@ Name-hash cache\n ---------------\n \n If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains\n-a cache of 32-bit values, one per object in the pack. The value at\n+a cache of 32-bit values, one per object in the pack/MIDX. The value at\n position `i` is the hash of the pathname at which the `i`th object\n-(counting in index order) in the pack can be found. This can be fed\n-into the delta heuristics to compare objects with similar pathnames.\n+(counting in index or multi-pack index order) in the pack/MIDX can be found.\n+This can be fed into the delta heuristics to compare objects with similar\n+pathnames.\n \n The hash algorithm used is:\n \ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex fb688976c4..1a73c3ee20 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -71,14 +71,10 @@ Future Work\n still reducing the number of binary searches required for object\n lookups.\n \n-- The reachability bitmap is currently paired directly with a single\n- packfile, using the pack-order as the object order to hopefully\n- compress the bitmaps well using run-length encoding. This could be\n- extended to pair a reachability bitmap with a multi-pack-index. If\n- the multi-pack-index is extended to store a \"stable object order\"\n+- If the multi-pack-index is extended to store a \"stable object order\"\n (a function Order(hash) = integer that is constant for a given hash,\n- even as the multi-pack-index is updated) then a reachability bitmap\n- could point to a multi-pack-index and be updated independently.\n+ even as the multi-pack-index is updated) then MIDX bitmaps could be\n+ updated independently of the MIDX.\n \n - Packfiles can be marked as \"special\" using empty files that share\n the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421432","messageId":"26c3a312f9b2d073b0f50c44b78a7f3eba204eda.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 07/22] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:21Z","receivedAt":"2021-04-09T18:11:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nclean up any auxiliary files (currently just the MIDX's `.rev` file, but\nsoon to include a `.bitmap`, too) corresponding to the MIDX it's\nreplacing.\n\nThis step should happen after the new MIDX is written into place, since\ndoing so beforehand means that the old MIDX could be read without its\ncorresponding .rev file.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 5249802326..a24c36968d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1076,10 +1076,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n-\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421433","messageId":"8643174a67812f347856bef274d6b971b15dbffc.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 08/22] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:25Z","receivedAt":"2021-04-09T18:11:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nload any existing one to fill in some pieces of information. But it uses\nload_multi_pack_index(), which ignores the configuration\n\"core.multiPackIndex\", which indicates whether or not Git is allowed to\nread an existing multi-pack-index.\n\nReplace this with a routine that does respect that setting, to avoid\nreading multi-pack-index files when told not to.\n\nThis avoids a problem that would arise in subsequent patches due to the\ncombination of 'git repack' reopening the object store in-process and\nthe multi-pack index code not checking whether a pack already exists in\nthe object store when calling add_pack_to_midx().\n\nThis would ultimately lead to a cycle being created along the\n'packed_git' struct's '->next' pointer. That is obviously bad, but it\nhas hard-to-debug downstream effects like saying a bitmap can't be\nloaded for a pack because one already exists (for the same pack).\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex a24c36968d..567cdf0fcf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -908,8 +908,18 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (m)\n \t\tctx.m = m;\n-\telse\n-\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\telse {\n+\t\tstruct multi_pack_index *cur;\n+\n+\t\tprepare_multi_pack_index_one(the_repository, object_dir, 1);\n+\n+\t\tctx.m = NULL;\n+\t\tfor (cur = the_repository->objects->multi_pack_index; cur;\n+\t\t cur = cur->next) {\n+\t\t\tif (!strcmp(object_dir, cur->object_dir))\n+\t\t\t\tctx.m = cur;\n+\t\t}\n+\t}\n \n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421434","messageId":"af507f4b29b6fcb79556dd8e6daea2fcc0d4b9bf.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 09/22] pack-bitmap.c: introduce 'bitmap_num_objects()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:29Z","receivedAt":"2021-04-09T18:11:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to return how many objects are\ncontained in a bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 37 +++++++++++++++++++++----------------\n 1 file changed, 21 insertions(+), 16 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d45e91db1e..a6c616aa3e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn b;\n }\n \n+static uint32_t bitmap_num_objects(struct bitmap_index *index)\n+{\n+\treturn index->pack->num_objects;\n+}\n+\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n-\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tsize_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));\n \t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n@@ -399,7 +404,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -451,7 +456,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -647,7 +652,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\tif (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -808,7 +813,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == type &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos))\n@@ -835,7 +840,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < pack->num_objects) {\n+\tif (pos < bitmap_num_objects(bitmap_git)) {\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n@@ -845,7 +850,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - pack->num_objects];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -887,7 +892,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t}\n \n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == OBJ_BLOB &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos) &&\n@@ -1075,8 +1080,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_git->pack->num_objects)\n-\t\treturn; /* not actually in the pack */\n+\tif (pos >= bitmap_num_objects(bitmap_git))\n+\t\treturn; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n@@ -1142,6 +1147,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n+\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n \n \tassert(result);\n \n@@ -1149,8 +1155,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\ti++;\n \n \t/* Don't mark objects not in the packfile */\n-\tif (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)\n-\t\ti = bitmap_git->pack->num_objects / BITS_IN_EWORD;\n+\tif (i > objects_nr / BITS_IN_EWORD)\n+\t\ti = objects_nr / BITS_IN_EWORD;\n \n \treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n@@ -1234,7 +1240,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n-\t\t\tbitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\t\tbitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcount++;\n \t}\n \n@@ -1455,7 +1461,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n-\tnum_objects = bitmap_git->pack->num_objects;\n+\tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -1538,7 +1544,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \toff_t total = 0;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n@@ -1550,7 +1555,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \tfor (i = 0; i < eindex->count; i++) {\n \t\tstruct object *obj = eindex->objects[i];\n \n-\t\tif (!bitmap_get(result, pack->num_objects + i))\n+\t\tif (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421435","messageId":"a6fdf7234afda1b103342b13b17e026f15a7db9a.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 10/22] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:33Z","receivedAt":"2021-04-09T18:11:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to fetch the nth OID contained in\nthe bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex a6c616aa3e..97ee2d331d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)\n \n #define MAX_XOR_OFFSET 160\n \n+static void nth_bitmap_object_oid(struct bitmap_index *index,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t uint32_t n)\n+{\n+\tnth_packed_object_id(oid, index->pack, n);\n+}\n+\n static int load_bitmap_entries_v1(struct bitmap_index *index)\n {\n \tuint32_t i;\n@@ -242,9 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n-\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n-\t\t\t\t (unsigned)commit_idx_pos);\n+\t\tnth_bitmap_object_oid(index, &oid, commit_idx_pos);\n \n \t\tbitmap = read_bitmap_1(index);\n \t\tif (!bitmap)\n@@ -844,8 +849,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack,\n-\t\t\t\t\t pack_pos_to_index(pack, pos));\n+\t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n+\t\t\t\t\t pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421436","messageId":"a78f83a1279f51aa8f362fe471a078463cabf21a.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 11/22] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:37Z","receivedAt":"2021-04-09T18:11:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a recent commit, pack-objects learned support for the\n'pack.preferBitmapTips' configuration. This patch prepares the\nmulti-pack bitmap code to respect this configuration, too.\n\nSince the multi-pack bitmap code already does a traversal of all\nreferences (in order to discover the set of reachable commits in the\nmulti-pack index), it is more efficient to check whether or not each\nreference is a suffix of any value of 'pack.preferBitmapTips' rather\nthan do an additional traversal.\n\nImplement a function 'bitmap_is_preferred_refname()' which does just\nthat. The caller will be added in a subsequent patch.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 16 ++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 97ee2d331d..be52570b0f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1594,3 +1594,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\n }\n+\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname)\n+{\n+\tconst struct string_list *preferred_tips = bitmap_preferred_tips(r);\n+\tstruct string_list_item *item;\n+\n+\tif (!preferred_tips)\n+\t\treturn 0;\n+\n+\tfor_each_string_list_item(item, preferred_tips) {\n+\t\tif (starts_with(refname, item->string))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 988ed3a30d..0bf75ff2a7 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -93,5 +93,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint16_t options);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n #endif\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421437","messageId":"d5eeca4f112f70343b069fcb68fe61e26831843f.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 12/22] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:44Z","receivedAt":"2021-04-09T18:11:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This prepares the code in pack-bitmap to interpret the new multi-pack\nbitmaps described in Documentation/technical/bitmap-format.txt, which\nmostly involves converting bit positions to accommodate looking them up\nin a MIDX.\n\nNote that there are currently no writers who write multi-pack bitmaps,\nand that this will be implemented in the subsequent commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 12 +-\n pack-bitmap-write.c | 2 +-\n pack-bitmap.c | 349 +++++++++++++++++++++++++++++++++++++----\n pack-bitmap.h | 5 +\n packfile.c | 2 +-\n 5 files changed, 338 insertions(+), 32 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5205dde2e1..a4e4e4ebcc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -984,7 +984,17 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\toff_t pack_offs = bitmap_pack_offset(bitmap_git,\n+\t\t\t\t\t\t\t\t pos + offset);\n+\t\t\t\tuint32_t pos;\n+\n+\t\t\t\tif (offset_to_pack_pos(reuse_packfile, pack_offs, &pos) < 0)\n+\t\t\t\t\tdie(_(\"write_reused_pack: could not locate %\"PRIdMAX),\n+\t\t\t\t\t (intmax_t)pack_offs);\n+\t\t\t\twrite_reused_pack_one(pos, f, &w_curs);\n+\t\t\t} else\n+\t\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f90e100e3e..020c1774c8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)\n }\n \n /**\n- * Build the initial type index for the packfile\n+ * Build the initial type index for the packfile or multi-pack-index\n */\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex be52570b0f..e41fce9675 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -13,6 +13,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"midx.h\"\n #include \"config.h\"\n \n /*\n@@ -35,8 +36,15 @@ struct stored_bitmap {\n * the active bitmap index is the largest one.\n */\n struct bitmap_index {\n-\t/* Packfile to which this bitmap index belongs to */\n+\t/*\n+\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n+\t * to.\n+\t *\n+\t * Exactly one of these must be non-NULL; this specifies the object\n+\t * order used to interpret this bitmap.\n+\t */\n \tstruct packed_git *pack;\n+\tstruct multi_pack_index *midx;\n \n \t/*\n \t * Mark the first `reuse_objects` in the packfile as reused:\n@@ -71,6 +79,8 @@ struct bitmap_index {\n \t/* If not NULL, this is a name-hash cache pointing into map. */\n \tuint32_t *hashes;\n \n+\tconst unsigned char *checksum;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -138,6 +148,8 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n+\tif (index->midx)\n+\t\treturn index->midx->num_objects;\n \treturn index->pack->num_objects;\n }\n \n@@ -175,6 +187,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n+\tindex->checksum = header->checksum;\n \tindex->map_pos += header_size;\n \treturn 0;\n }\n@@ -227,7 +240,10 @@ static void nth_bitmap_object_oid(struct bitmap_index *index,\n \t\t\t\t struct object_id *oid,\n \t\t\t\t uint32_t n)\n {\n-\tnth_packed_object_id(oid, index->pack, n);\n+\tif (index->midx)\n+\t\tnth_midxed_object_oid(oid, index->midx, n);\n+\telse\n+\t\tnth_packed_object_id(oid, index->pack, n);\n }\n \n static int load_bitmap_entries_v1(struct bitmap_index *index)\n@@ -272,7 +288,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \treturn 0;\n }\n \n-static char *pack_bitmap_filename(struct packed_git *p)\n+char *midx_bitmap_filename(struct multi_pack_index *midx)\n+{\n+\treturn xstrfmt(\"%s-%s.bitmap\",\n+\t\t get_midx_filename(midx->object_dir),\n+\t\t hash_to_hex(get_midx_checksum(midx)));\n+}\n+\n+char *pack_bitmap_filename(struct packed_git *p)\n {\n \tsize_t len;\n \n@@ -281,6 +304,54 @@ static char *pack_bitmap_filename(struct packed_git *p)\n \treturn xstrfmt(\"%.*s.bitmap\", (int)len, p->pack_name);\n }\n \n+static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct stat st;\n+\tchar *idx_name = midx_bitmap_filename(midx);\n+\tint fd = git_open(idx_name);\n+\n+\tfree(idx_name);\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n+\t\treturn -1;\n+\t}\n+\n+\tbitmap_git->midx = midx;\n+\tbitmap_git->map_size = xsize_t(st.st_size);\n+\tbitmap_git->map_pos = 0;\n+\tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ,\n+\t\t\t\tMAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\tif (load_bitmap_header(bitmap_git) < 0)\n+\t\tgoto cleanup;\n+\n+\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum))\n+\t\tgoto cleanup;\n+\n+\tif (load_midx_revindex(bitmap_git->midx) < 0) {\n+\t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n+\t\tgoto cleanup;\n+\t}\n+\treturn 0;\n+\n+cleanup:\n+\tmunmap(bitmap_git->map, bitmap_git->map_size);\n+\tbitmap_git->map_size = 0;\n+\tbitmap_git->map = NULL;\n+\treturn -1;\n+}\n+\n static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git *packfile)\n {\n \tint fd;\n@@ -302,12 +373,18 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\treturn -1;\n \t}\n \n-\tif (bitmap_git->pack) {\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n \t\tclose(fd);\n \t\treturn -1;\n \t}\n \n+\tif (!is_pack_valid(packfile)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n \tbitmap_git->pack = packfile;\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n@@ -324,13 +401,36 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \treturn 0;\n }\n \n-static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n+static int load_reverse_index(struct bitmap_index *bitmap_git)\n+{\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tuint32_t i;\n+\t\tint ret;\n+\n+\t\tret = load_midx_revindex(bitmap_git->midx);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n+\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn load_pack_revindex(bitmap_git->pack);\n+}\n+\n+static int load_bitmap(struct bitmap_index *bitmap_git)\n {\n \tassert(bitmap_git->map);\n \n \tbitmap_git->bitmaps = kh_init_oid_map();\n \tbitmap_git->ext_index.positions = kh_init_oid_pos();\n-\tif (load_pack_revindex(bitmap_git->pack))\n+\n+\tif (load_reverse_index(bitmap_git))\n \t\tgoto failed;\n \n \tif (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) ||\n@@ -374,11 +474,35 @@ static int open_pack_bitmap(struct repository *r,\n \treturn ret;\n }\n \n+static int open_midx_bitmap(struct repository *r,\n+\t\t\t struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *midx;\n+\n+\tassert(!bitmap_git->map);\n+\n+\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n+\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\t\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n+static int open_bitmap(struct repository *r,\n+\t\t struct bitmap_index *bitmap_git)\n+{\n+\tassert(!bitmap_git->map);\n+\n+\tif (!open_midx_bitmap(r, bitmap_git))\n+\t\treturn 0;\n+\treturn open_pack_bitmap(r, bitmap_git);\n+}\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -428,10 +552,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \treturn pos;\n }\n \n+static int bitmap_position_midx(struct bitmap_index *bitmap_git,\n+\t\t\t\tconst struct object_id *oid)\n+{\n+\tuint32_t want, got;\n+\tif (!bsearch_midx(oid, bitmap_git->midx, &want))\n+\t\treturn -1;\n+\n+\tif (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0)\n+\t\treturn -1;\n+\treturn got;\n+}\n+\n static int bitmap_position(struct bitmap_index *bitmap_git,\n \t\t\t const struct object_id *oid)\n {\n-\tint pos = bitmap_position_packfile(bitmap_git, oid);\n+\tint pos;\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpos = bitmap_position_midx(bitmap_git, oid);\n+\telse\n+\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n }\n \n@@ -721,6 +861,7 @@ static void show_objects_for_type(\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n+\t\t\tstruct packed_git *pack;\n \t\t\tstruct object_id oid;\n \t\t\tuint32_t hash = 0, index_pos;\n \t\t\toff_t ofs;\n@@ -730,14 +871,28 @@ static void show_objects_for_type(\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n-\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\t\t\tuint32_t pack_id;\n+\n+\t\t\t\tindex_pos = pack_pos_to_midx(m, pos + offset);\n+\t\t\t\tofs = nth_midxed_offset(m, index_pos);\n+\t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n+\n+\t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t} else {\n+\t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\t\tnth_bitmap_object_oid(bitmap_git, &oid, index_pos);\n+\n+\t\t\t\tpack = bitmap_git->pack;\n+\t\t\t}\n \n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n }\n@@ -749,8 +904,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n-\t\t\treturn 1;\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n+\t\t\t\treturn 1;\n+\t\t} else {\n+\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -839,14 +999,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git,\n static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\t\t uint32_t pos)\n {\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tunsigned long size;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = &size;\n \n \tif (pos < bitmap_num_objects(bitmap_git)) {\n-\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tstruct packed_git *pack;\n+\t\toff_t ofs;\n+\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\n+\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\t\t} else {\n+\t\t\tpack = bitmap_git->pack;\n+\t\t\tofs = pack_pos_to_offset(pack, pos);\n+\t\t}\n+\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n \t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n@@ -990,7 +1162,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t/* try to open a bitmapped pack, but don't parse it yet\n \t * because we may not need to use it */\n \tCALLOC_ARRAY(bitmap_git, 1);\n-\tif (open_pack_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (open_bitmap(revs->repo, bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tfor (i = 0; i < revs->pending.nr; ++i) {\n@@ -1034,7 +1206,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_pack_bitmap(bitmap_git) < 0)\n+\tif (load_bitmap(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tobject_array_clear(&revs->pending);\n@@ -1081,15 +1253,29 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n-\toff_t offset, header;\n+\tstruct packed_git *pack;\n+\toff_t offset, delta_obj_offset;\n \tenum object_type type;\n \tunsigned long size;\n \n \tif (pos >= bitmap_num_objects(bitmap_git))\n \t\treturn; /* not actually in the pack or MIDX */\n \n-\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n-\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tuint32_t pack_id, midx_pos;\n+\n+\t\tmidx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\tpack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\n+\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\toffset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\t} else {\n+\t\tpack = bitmap_git->pack;\n+\t\toffset = pack_pos_to_offset(bitmap_git->pack, pos);\n+\t}\n+\n+\tdelta_obj_offset = offset;\n+\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn; /* broken packfile, punt */\n \n@@ -1105,11 +1291,11 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t &offset, type, header);\n+\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn;\n-\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n \t\t\treturn;\n \n \t\t/*\n@@ -1120,6 +1306,16 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * packs we write fresh, and OFS_DELTA is the default). But\n \t\t * let's double check to make sure the pack wasn't written with\n \t\t * odd parameters.\n+\t\t *\n+\t\t * Note that the base does not need to be repositioned, i.e.,\n+\t\t * the MIDX is guaranteed to have selected the copy of \"base\"\n+\t\t * from the same pack, since this function is only ever called\n+\t\t * on the preferred pack (and all duplicate objects are resolved\n+\t\t * in favor of the preferred pack).\n+\t\t *\n+\t\t * This means that we can reuse base_pos when looking up the bit\n+\t\t * in the reuse bitmap, too, since bits corresponding to the\n+\t\t * preferred pack precede all bits from other packs.\n \t\t */\n \t\tif (base_pos >= pos)\n \t\t\treturn;\n@@ -1142,6 +1338,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tbitmap_set(reuse, pos);\n }\n \n+static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *m = bitmap_git->midx;\n+\tif (!m)\n+\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n+\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n+}\n+\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t struct packed_git **packfile_out,\n \t\t\t\t uint32_t *entries,\n@@ -1153,13 +1357,29 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tsize_t i = 0;\n \tuint32_t offset;\n \tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n+\tuint32_t preferred_pack = 0;\n \n \tassert(result);\n \n+\tload_reverse_index(bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tpreferred_pack = midx_preferred_pack(bitmap_git);\n+\t\tobjects_nr = bitmap_git->midx->packs[preferred_pack]->num_objects;\n+\t} else\n+\t\tobjects_nr = bitmap_git->pack->num_objects;\n+\n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n \n-\t/* Don't mark objects not in the packfile */\n+\t/*\n+\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n+\t * marks objects eligible for reuse, but the pack-reuse code only\n+\t * understands how to reuse a single pack. Since the preferred pack is\n+\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n+\t * we use it instead of another pack. In single-pack bitmaps, the choice\n+\t * is made for us.\n+\t */\n \tif (i > objects_nr / BITS_IN_EWORD)\n \t\ti = objects_nr / BITS_IN_EWORD;\n \n@@ -1175,6 +1395,14 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\t/*\n+\t\t\t\t * Can't reuse from a non-preferred pack (see\n+\t\t\t\t * above).\n+\t\t\t\t */\n+\t\t\t\tif (pos + offset >= objects_nr)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n \t\t}\n \t}\n@@ -1192,7 +1420,9 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = bitmap_git->pack;\n+\t*packfile_out = bitmap_git->pack ?\n+\t\tbitmap_git->pack :\n+\t\tbitmap_git->midx->packs[preferred_pack];\n \t*reuse_out = reuse;\n \treturn 0;\n }\n@@ -1466,6 +1696,12 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n+\tif (!bitmap_is_midx(bitmap_git))\n+\t\tload_reverse_index(bitmap_git);\n+\telse if (load_midx_revindex(bitmap_git->midx) < 0)\n+\t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n+\t\t \"extension\");\n+\n \tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n@@ -1473,8 +1709,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tstruct object_id oid;\n \t\tstruct object_entry *oe;\n \n-\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n-\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n+\t\tif (bitmap_is_midx(bitmap_git))\n+\t\t\tnth_midxed_object_oid(&oid,\n+\t\t\t\t\t bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n+\t\telse\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n@@ -1500,6 +1741,19 @@ void free_bitmap_index(struct bitmap_index *b)\n \tfree(b->ext_index.hashes);\n \tbitmap_free(b->result);\n \tbitmap_free(b->haves);\n+\tif (bitmap_is_midx(b)) {\n+\t\t/*\n+\t\t * Multi-pack bitmaps need to have resources associated with\n+\t\t * their on-disk reverse indexes unmapped so that stale .rev and\n+\t\t * .bitmap files can be removed.\n+\t\t *\n+\t\t * Unlike pack-based bitmaps, multi-pack bitmaps can be read and\n+\t\t * written in the same 'git multi-pack-index write --bitmap'\n+\t\t * process. Close resources so they can be removed safely on\n+\t\t * platforms like Windows.\n+\t\t */\n+\t\tclose_midx_revindex(b->midx);\n+\t}\n \tfree(b);\n }\n \n@@ -1514,7 +1768,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\t enum object_type object_type)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n+\tstruct packed_git *pack;\n \toff_t total = 0;\n \tstruct ewah_iterator it;\n \teword_t filter;\n@@ -1538,6 +1792,29 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \t\t\tpos = base + offset;\n+\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tuint32_t pack_pos;\n+\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\n+\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n+\t\t\t\t\tstruct object_id oid;\n+\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n+\n+\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n+\t\t\t\t\t oid_to_hex(&oid),\n+\t\t\t\t\t pack_id,\n+\t\t\t\t\t (uintmax_t)offset);\n+\t\t\t\t}\n+\n+\t\t\t\tpos = pack_pos;\n+\t\t\t} else\n+\t\t\t\tpack = bitmap_git->pack;\n+\n \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n \t\t\t\t pack_pos_to_offset(pack, pos);\n \t\t}\n@@ -1590,6 +1867,20 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n \treturn total;\n }\n \n+int bitmap_is_midx(struct bitmap_index *bitmap_git)\n+{\n+\treturn !!bitmap_git->midx;\n+}\n+\n+off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos)\n+{\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\treturn nth_midxed_offset(bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, pos));\n+\treturn nth_packed_object_offset(bitmap_git->pack,\n+\t\t\t\t\tpack_pos_to_index(bitmap_git->pack, pos));\n+}\n+\n const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 0bf75ff2a7..0dc6f7a7e4 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -91,6 +91,11 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n \t\t\t uint16_t options);\n+char *midx_bitmap_filename(struct multi_pack_index *midx);\n+char *pack_bitmap_filename(struct packed_git *p);\n+\n+int bitmap_is_midx(struct bitmap_index *bitmap_git);\n+off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n int bitmap_is_preferred_refname(struct repository *r, const char *refname);\ndiff --git a/packfile.c b/packfile.c\nindex 8668345d93..c444e365a3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -863,7 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n-\t ends_with(file_name, \".rev\"))\n+\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421438","messageId":"fd320c5ed48c7431b64b898f49101b0f53655a95.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 13/22] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:48Z","receivedAt":"2021-04-09T18:12:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Write multi-pack bitmaps in the format described by\nDocumentation/technical/bitmap-format.txt, inferring their presence with\nthe absence of '--bitmap'.\n\nTo write a multi-pack bitmap, this patch attempts to reuse as much of\nthe existing machinery from pack-objects as possible. Specifically, the\nMIDX code prepares a packing_data struct that pretends as if a single\npackfile has been generated containing all of the objects contained\nwithin the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 12 +-\n builtin/multi-pack-index.c | 2 +\n midx.c | 195 ++++++++++++++++++++++++-\n midx.h | 1 +\n 4 files changed, 202 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex ffd601bc17..ada14deb2c 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n-\t[--preferred-pack=] \n+\t[--preferred-pack=] [--[no-]bitmap] \n \n DESCRIPTION\n -----------\n@@ -40,6 +40,9 @@ write::\n \t\tmultiple packs contain the same object. If not given,\n \t\tties are broken in favor of the pack with the lowest\n \t\tmtime.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\n@@ -81,6 +84,13 @@ EXAMPLES\n $ git multi-pack-index write\n -----------------------------------------------\n \n+* Write a MIDX file for the packfiles in the current .git folder with a\n+corresponding bitmap.\n++\n+-------------------------------------------------------------\n+$ git multi-pack-index write --preferred-pack --bitmap\n+-------------------------------------------------------------\n+\n * Write a MIDX file for the packfiles in an alternate object store.\n +\n -----------------------------------------------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5d3ea445fd..bf6fa982e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n \t\t\t N_(\"preferred-pack\"),\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/midx.c b/midx.c\nindex 567cdf0fcf..32d7d184c0 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,10 @@\n #include \"repository.h\"\n #include \"chunk-format.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -885,6 +889,145 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n static void clear_midx_files_ext(struct repository *r, const char *ext,\n \t\t\t\t unsigned char *keep_hash);\n \n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tif (oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t data->ctx->entries_nr,\n+\t\t bitmap_oid_access) > -1) {\n+\t\tALLOC_GROW(data->commits, data->commits_nr + 1,\n+\t\t\t data->commits_alloc);\n+\t\tdata->commits[data->commits_nr++] = commit;\n+\t}\n+}\n+\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb;\n+\n+\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tfor_each_ref(add_ref_to_pending, &revs);\n+\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t unsigned flags)\n+{\n+\tstruct packing_data pdata;\n+\tstruct pack_idx_entry **index;\n+\tstruct commit **commits = NULL;\n+\tuint32_t i, commits_nr;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n+\tint ret;\n+\n+\tprepare_midx_packing_data(&pdata, ctx);\n+\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata.nr_objects);\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_select_commits expects objects in lex order, but\n+\t * pack_order gives us exactly that. use it directly instead of\n+\t * re-sorting the array\n+\t */\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(&pdata);\n+\tif (!ret)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\treturn ret;\n+}\n+\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -930,9 +1073,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n \n+\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\terror(_(\"could not load pack %s\"),\n+\t\t\t\t ctx.m->pack_names[i]);\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\tctx.info[ctx.nr].expired = 0;\n \t\t\tctx.nr++;\n \t\t}\n@@ -947,8 +1097,26 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n-\t\tgoto cleanup;\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_bitmap_git(the_repository);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n+\t\t\t\t\t\t NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n \n \tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n@@ -1048,9 +1216,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n-\tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n@@ -1081,14 +1246,17 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tif (flags & MIDX_WRITE_BITMAP)\n+\t\twrite_midx_bitmap(midx_name, midx_hash, &ctx, flags);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n \tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n cleanup:\n@@ -1096,6 +1264,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tif (ctx.info[i].p) {\n \t\t\tclose_pack(ctx.info[i].p);\n \t\t\tfree(ctx.info[i].p);\n+\t\t\tif (ctx.m) {\n+\t\t\t\t/*\n+\t\t\t\t * Destroy a stale reference to the pack in\n+\t\t\t\t * 'ctx.m'.\n+\t\t\t\t */\n+\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n+\t\t\t\tif (orig < ctx.m->num_packs)\n+\t\t\t\t\tctx.m->packs[orig] = NULL;\n+\t\t\t}\n \t\t}\n \t\tfree(ctx.info[i].pack_name);\n \t}\n@@ -1105,6 +1282,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n+\tif (ctx.m)\n+\t\tclose_midx(ctx.m);\n+\n \treturn result;\n }\n \n@@ -1166,6 +1346,7 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".bitmap\", NULL);\n \tclear_midx_files_ext(r, \".rev\", NULL);\n \n \tfree(midx);\ndiff --git a/midx.h b/midx.h\nindex 1172df1a71..350f4d0a7b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -41,6 +41,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n+#define MIDX_WRITE_BITMAP (1 << 2)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n char *get_midx_filename(const char *object_dir);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421439","messageId":"570e3de9ed9d597cb2b9dcc38490836e45998e58.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 14/22] t5310: move some tests to lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:52Z","receivedAt":"2021-04-09T18:12:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We'll soon be adding a test script that will cover many of the same\nbitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many\nof the applicable tests as we can so we don't have to rewrite them.\n\nThere should be no functional change to t5310; we still run the same\noperations in the same order.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 212 ++++++++++++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 204 +-------------------------------------\n 2 files changed, 216 insertions(+), 200 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex fe3f98be24..c655a9bf35 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,3 +1,6 @@\n+# Helpers for scripts testing bitamp functionality; see t5310 for\n+# example usage.\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -24,3 +27,212 @@ test_bitmap_traversal () {\n \ttest_cmp \"$1.normalized\" \"$2.normalized\" &&\n \trm -f \"$1.normalized\" \"$2.normalized\"\n }\n+\n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other second\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-second * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for second (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n+#\n+# second: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n+setup_bitmap_history() {\n+\ttest_expect_success 'setup repo with moderate-sized history' '\n+\t\ttest_commit_bulk --id=file 10 &&\n+\t\tgit branch -M second &&\n+\t\tgit checkout -b other HEAD~5 &&\n+\t\ttest_commit_bulk --id=side 10 &&\n+\n+\t\t# add complicated history setup, including merges and\n+\t\t# ambiguous merge-bases\n+\n+\t\tgit checkout -b merge-left other~2 &&\n+\t\tgit merge second~2 -m \"merge-left\" &&\n+\n+\t\tgit checkout -b merge-right second~1 &&\n+\t\tgit merge other~1 -m \"merge-right\" &&\n+\n+\t\tgit checkout -b octo-second second &&\n+\t\tgit merge merge-left merge-right -m \"octopus-second\" &&\n+\n+\t\tgit checkout -b octo-other other &&\n+\t\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\t\tgit checkout other &&\n+\t\tgit merge octo-other -m \"pull octopus\" &&\n+\n+\t\tgit checkout second &&\n+\t\tgit merge octo-second -m \"pull octopus\" &&\n+\n+\t\t# Remove these branches so they are not selected\n+\t\t# as bitmap tips\n+\t\tgit branch -D merge-left &&\n+\t\tgit branch -D merge-right &&\n+\t\tgit branch -D octo-other &&\n+\t\tgit branch -D octo-second &&\n+\n+\t\t# add padding to make these merges less interesting\n+\t\t# and avoid having them selected for bitmaps\n+\t\ttest_commit_bulk --id=file 100 &&\n+\t\tgit checkout other &&\n+\t\ttest_commit_bulk --id=side 100 &&\n+\t\tgit checkout second &&\n+\n+\t\tbitmaptip=$(git rev-parse second) &&\n+\t\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n+\t\tgit tag tagged-blob $blob\n+\t'\n+}\n+\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n+\t\tgit rev-list --count other...second >expect &&\n+\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n+\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n+\t\ttest_bitmap_traversal expect actual\n+\t'\n+\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n+\t\tgrep $blob actual\n+\t'\n+}\n+\n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"second\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n+basic_bitmap_tests () {\n+\ttip=\"$1\"\n+\ttest_expect_success 'rev-list --test-bitmap verifies bitmaps' \"\n+\t\tgit rev-list --test-bitmap \"${tip:-HEAD}\"\n+\t\"\n+\n+\trev_list_tests 'full bitmap'\n+\n+\ttest_expect_success 'clone from bitmapped repository' '\n+\t\trm -fr clone.git &&\n+\t\tgit clone --no-local --bare . clone.git &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'partial clone from bitmapped repository' '\n+\t\ttest_config uploadpack.allowfilter true &&\n+\t\trm -fr partial-clone.git &&\n+\t\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n+\t\t(\n+\t\t\tcd partial-clone.git &&\n+\t\t\tpack=$(echo objects/pack/*.pack) &&\n+\t\t\tgit verify-pack -v \"$pack\" >have &&\n+\t\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n+\t\t\t# we expect this single blob because of the direct ref\n+\t\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n+\t\t\ttest_cmp expect blobs\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'setup further non-bitmapped commits' '\n+\t\ttest_commit_bulk --id=further 10\n+\t'\n+\n+\trev_list_tests 'partial bitmap'\n+\n+\ttest_expect_success 'fetch (partial bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+}\n+\n+# have_delta \n+#\n+# Note that because this relies on cat-file, it might find _any_ copy of an\n+# object in the repository. The caller is responsible for making sure\n+# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n+have_delta () {\n+\techo $2 >expect &&\n+\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n+\ttest_cmp expect actual\n+}\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f53efc8229..4318f84d53 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -25,93 +25,10 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-# To ensure the logic for \"maximal commits\" is exercised, make\n-# the repository a bit more complicated.\n-#\n-# other second\n-# * *\n-# (99 commits) (99 commits)\n-# * *\n-# |\\ /|\n-# | * octo-other octo-second * |\n-# |/|\\_________ ____________/|\\|\n-# | \\ \\/ __________/ |\n-# | | ________/\\ / |\n-# * |/ * merge-right *\n-# | _|__________/ \\____________ |\n-# |/ | \\|\n-# (l1) * * merge-left * (r1)\n-# | / \\________________________ |\n-# |/ \\|\n-# (l2) * * (r2)\n-# \\___________________________ |\n-# \\|\n-# * (base)\n-#\n-# We only push bits down the first-parent history, which\n-# makes some of these commits unimportant!\n-#\n-# The important part for the maximal commit algorithm is how\n-# the bitmasks are extended. Assuming starting bit positions\n-# for second (bit 0) and other (bit 1), the bitmasks at the\n-# end should be:\n-#\n-# second: 1 (maximal, selected)\n-# other: 01 (maximal, selected)\n-# (base): 11 (maximal)\n-#\n-# This complicated history was important for a previous\n-# version of the walk that guarantees never walking a\n-# commit multiple times. That goal might be important\n-# again, so preserve this complicated case. For now, this\n-# test will guarantee that the bitmaps are computed\n-# correctly, even with the repeat calculations.\n+setup_bitmap_history\n \n-test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 10 &&\n-\tgit branch -M second &&\n-\tgit checkout -b other HEAD~5 &&\n-\ttest_commit_bulk --id=side 10 &&\n-\n-\t# add complicated history setup, including merges and\n-\t# ambiguous merge-bases\n-\n-\tgit checkout -b merge-left other~2 &&\n-\tgit merge second~2 -m \"merge-left\" &&\n-\n-\tgit checkout -b merge-right second~1 &&\n-\tgit merge other~1 -m \"merge-right\" &&\n-\n-\tgit checkout -b octo-second second &&\n-\tgit merge merge-left merge-right -m \"octopus-second\" &&\n-\n-\tgit checkout -b octo-other other &&\n-\tgit merge merge-left merge-right -m \"octopus-other\" &&\n-\n-\tgit checkout other &&\n-\tgit merge octo-other -m \"pull octopus\" &&\n-\n-\tgit checkout second &&\n-\tgit merge octo-second -m \"pull octopus\" &&\n-\n-\t# Remove these branches so they are not selected\n-\t# as bitmap tips\n-\tgit branch -D merge-left &&\n-\tgit branch -D merge-right &&\n-\tgit branch -D octo-other &&\n-\tgit branch -D octo-second &&\n-\n-\t# add padding to make these merges less interesting\n-\t# and avoid having them selected for bitmaps\n-\ttest_commit_bulk --id=file 100 &&\n-\tgit checkout other &&\n-\ttest_commit_bulk --id=side 100 &&\n-\tgit checkout second &&\n-\n-\tbitmaptip=$(git rev-parse second) &&\n-\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n-\tgit tag tagged-blob $blob &&\n-\tgit config repack.writebitmaps true\n+test_expect_success 'setup writing bitmaps during repack' '\n+\tgit config repack.writeBitmaps true\n '\n \n test_expect_success 'full repack creates bitmaps' '\n@@ -123,109 +40,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n-test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-\tgit rev-list --test-bitmap HEAD\n-'\n-\n-rev_list_tests_head () {\n-\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch~5..$branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n-\t\tgit rev-list --count -n 1 $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n-\t\tgit rev-list --count other...second >expect &&\n-\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n-\t\tgit rev-list --count $branch -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count --objects $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n-\t\tgit rev-list --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list $branch >expect &&\n-\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list --objects $branch >expect &&\n-\t\ttest_bitmap_traversal expect actual\n-\t'\n-\n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n-\t\tgrep $blob actual\n-\t'\n-}\n-\n-rev_list_tests () {\n-\tstate=$1\n-\n-\tfor branch in \"second\" \"other\"\n-\tdo\n-\t\trev_list_tests_head\n-\tdone\n-}\n-\n-rev_list_tests 'full bitmap'\n-\n-test_expect_success 'clone from bitmapped repository' '\n-\tgit clone --no-local --bare . clone.git &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'partial clone from bitmapped repository' '\n-\ttest_config uploadpack.allowfilter true &&\n-\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n-\t(\n-\t\tcd partial-clone.git &&\n-\t\tpack=$(echo objects/pack/*.pack) &&\n-\t\tgit verify-pack -v \"$pack\" >have &&\n-\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n-\t\t# we expect this single blob because of the direct ref\n-\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n-\t\ttest_cmp expect blobs\n-\t)\n-'\n-\n-test_expect_success 'setup further non-bitmapped commits' '\n-\ttest_commit_bulk --id=further 10\n-'\n-\n-rev_list_tests 'partial bitmap'\n-\n-test_expect_success 'fetch (partial bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+basic_bitmap_tests\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -461,17 +276,6 @@ test_expect_success 'truncated bitmap fails gracefully (cache)' '\n \ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n-# have_delta \n-#\n-# Note that because this relies on cat-file, it might find _any_ copy of an\n-# object in the repository. The caller is responsible for making sure\n-# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n-have_delta () {\n-\techo $2 >expect &&\n-\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n-\ttest_cmp expect actual\n-}\n-\n # Create a state of history with these properties:\n #\n # - refs that allow a client to fetch some new history, while sharing some old\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421440","messageId":"ff74181e85975690b4fccfb6b72fb80045f4adc7.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 16/22] t5326: test multi-pack bitmap behavior","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:00Z","receivedAt":"2021-04-09T18:12:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch introduces a new test, t5326, which tests the basic\nfunctionality of multi-pack bitmaps.\n\nSome trivial behavior is tested, such as:\n\n - Whether bitmaps can be generated with more than one pack.\n - Whether clones can be served with all objects in the bitmap.\n - Whether follow-up fetches can be served with some objects outside of\n the server's bitmap\n\nThese use lib-bitmap's tests (which in turn were pulled from t5310), and\nwe cover cases where the MIDX represents both a single pack and multiple\npacks.\n\nIn addition, some non-trivial and MIDX-specific behavior is tested, too,\nincluding:\n\n - Whether multi-pack bitmaps behave correctly with respect to the\n pack-reuse machinery when the base for some object is selected from\n a different pack than the delta.\n - Whether multi-pack bitmaps correctly respect the\n pack.preferBitmapTips configuration.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 278 ++++++++++++++++++++++++++++++++++\n 1 file changed, 278 insertions(+)\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..51c4f9ad78\n--- /dev/null\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,278 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality'\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+setup_bitmap_history\n+\n+test_expect_success 'enable core.multiPackIndex' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_expect_success 'create single-pack midx with bitmaps' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success 'create new additional packs' '\n+\tfor i in $(test_seq 1 16)\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\t\tgit repack -d\n+\tdone &&\n+\n+\tgit checkout -b other2 HEAD~8 &&\n+\tfor i in $(test_seq 1 8)\n+\tdo\n+\t\ttest_commit \"side-$i\" &&\n+\t\tgit repack -d\n+\tdone &&\n+\tgit checkout second\n+'\n+\n+test_expect_success 'create multi-pack midx with bitmaps' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\tls $objdir/pack/pack-*.pack >packs &&\n+\ttest_line_count = 26 packs &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\ttest_commit respect--no-bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\tgit multi-pack-index write --no-bitmap &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+test_expect_success 'setup midx with base from later pack' '\n+\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t# prefers to delete contents out of a base rather than add to a shorter\n+\t# object.\n+\ttest_seq 1 128 >a &&\n+\ttest_seq 1 130 >b &&\n+\n+\tgit add a b &&\n+\tgit commit -m \"initial commit\" &&\n+\n+\ta=$(git rev-parse HEAD:a) &&\n+\tb=$(git rev-parse HEAD:b) &&\n+\n+\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$a\n+\t$b\n+\tEOF\n+\t) &&\n+\n+\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t# any other object.\n+\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$b\n+\t$(git rev-parse HEAD)\n+\t$(git rev-parse HEAD^{tree})\n+\tEOF\n+\t) &&\n+\n+\tgit prune-packed &&\n+\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t# reuse.\n+\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\thave_delta $a $b &&\n+\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+'\n+\n+rev_list_tests 'full bitmap with backwards delta'\n+\n+test_expect_success 'clone with bitmaps enabled' '\n+\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\tgit rev-parse HEAD >expect &&\n+\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+bitmap_reuse_tests() {\n+\tfrom=$1\n+\tto=$2\n+\n+\ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk 16 &&\n+\t\t\tgit tag old-tip &&\n+\n+\t\t\tgit config core.multiPackIndex true &&\n+\t\t\tif test \"MIDX\" = \"$from\"\n+\t\t\tthen\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk --id=further 16 &&\n+\t\t\tgit tag new-tip &&\n+\n+\t\t\tif test \"MIDX\" = \"$to\"\n+\t\t\tthen\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit for-each-ref &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n+\t\t)\n+\t'\n+}\n+\n+bitmap_reuse_tests 'pack' 'MIDX'\n+bitmap_reuse_tests 'MIDX' 'pack'\n+bitmap_reuse_tests 'MIDX' 'MIDX'\n+\n+test_expect_success 'missing object closure fails gracefully' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit loose &&\n+\t\ttest_commit packed &&\n+\n+\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t# commit.\n+\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t$(git rev-parse packed)\n+\t\tEOF\n+\n+\t\tgit multi-pack-index write --bitmap 2>err &&\n+\t\tgrep \"doesn.t have full closure\" err &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+\t)\n+'\n+\n+test_expect_success 'setup partial bitmaps' '\n+\ttest_commit packed &&\n+\tgit repack &&\n+\ttest_commit loose &&\n+\tgit multi-pack-index write --bitmap 2>err &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests HEAD~\n+\n+test_expect_success 'removing a MIDX clears stale bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm $midx &&\n+\n+\t\t# Then write a new MIDX.\n+\t\ttest_commit new &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_missing $stale_bitmap\n+\t)\n+'\n+\n+test_expect_success 'pack.preferBitmapTips' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\tbitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421441","messageId":"060ee427bebdb6f45766c173878a9c056f312baa.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 15/22] t/helper/test-read-midx.c: add --checksum mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:11:56Z","receivedAt":"2021-04-09T18:12:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subsequent tests will want to check for the existence of a multi-pack\nbitmap which matches the multi-pack-index stored in the pack directory.\n\nThe multi-pack bitmap includes the hex checksum of the MIDX it\ncorresponds to in its filename (for example,\n'$packdir/multi-pack-index-.bitmap'). As a result, some tests\nwant a way to learn what '' is.\n\nThis helper addresses that need by printing the checksum of the\nrepository's multi-pack-index.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 16 +++++++++++++++-\n t/lib-bitmap.sh | 4 ++++\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 7c2eb11a8e..cb0d27049a 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \treturn 0;\n }\n \n+static int read_midx_checksum(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m;\n+\n+\tsetup_git_directory();\n+\tm = load_multi_pack_index(object_dir, 1);\n+\tif (!m)\n+\t\treturn 1;\n+\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n+\telse if (!strcmp(argv[1], \"--checksum\"))\n+\t\treturn read_midx_checksum(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex c655a9bf35..a5ac8b41a7 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -236,3 +236,7 @@ have_delta () {\n \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n \ttest_cmp expect actual\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"${1:-.git/objects}\"\n+}\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421442","messageId":"8f328bb5bc46149bd05860c29be032fe219951a4.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 17/22] t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:04Z","receivedAt":"2021-04-09T18:12:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap confuses many of the tests in t5310, which\nexpect to control whether and how bitmaps are written. Since the\nrelevant MIDX-bitmap tests here are covered already in t5326, let's just\ndisable the flag for the whole t5310 script.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 4318f84d53..673baa5c3c 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n . \"$TEST_DIRECTORY\"/lib-bundle.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n+# their place.\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421443","messageId":"ee952e43007b9fd7481ad0e51aafe241e27bd8b1.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 19/22] t7700: update to work with MIDX bitmap test knob","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:13Z","receivedAt":"2021-04-09T18:12:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A number of these tests are focused only on pack-based bitmaps and need\nto be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where\nnecessary.\n\nSigned-off-by: Taylor Blau \n---\n t/t7700-repack.sh | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 25b235c063..98eda3bfeb 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit repack -Adbl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '\n \n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n-\tgit -C bare.git repack -ad &&\n+\trm -f bare.git/objects/pack/*.bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421444","messageId":"dbd953815b640dfafa9044a70dc6dacbbf9302a4.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 18/22] t5319: don't write MIDX bitmaps in t5319","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:09Z","receivedAt":"2021-04-09T18:12:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This test is specifically about generating a midx still respecting a\npack-based bitmap file. Generating a MIDX bitmap would confuse the test.\nLet's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to\nmake sure we don't do so.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 5641d158df..69f1c815aa 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -474,7 +474,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421445","messageId":"2f3836bf2e2e7dafc6ad3c1cb20e00de49f638ef.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 21/22] p5310: extract full and partial bitmap tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:21Z","receivedAt":"2021-04-09T18:12:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A new p5326 introduced by the next patch will want these same tests,\ninterjecting its own setup in between. Move them out so that both perf\ntests can reuse them.\n\nSigned-off-by: Taylor Blau \n---\n t/perf/lib-bitmap.sh | 69 ++++++++++++++++++++++++++++++++++++\n t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------\n 2 files changed, 72 insertions(+), 62 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n\ndiff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh\nnew file mode 100644\nindex 0000000000..63d3bc7cec\n--- /dev/null\n+++ b/t/perf/lib-bitmap.sh\n@@ -0,0 +1,69 @@\n+# Helper functions for testing bitmap performance; see p5310.\n+\n+test_full_bitmap () {\n+\ttest_perf 'simulated clone' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'simulated fetch' '\n+\t\thave=$(git rev-list HEAD~100 -1) &&\n+\t\t{\n+\t\t\techo HEAD &&\n+\t\t\techo ^$have\n+\t\t} | git pack-objects --revs --stdout >/dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (commits)' '\n+\t\tgit rev-list --all --use-bitmap-index >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (objects)' '\n+\t\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tag negated via --not --all (objects)' '\n+\t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with negative tag (objects)' '\n+\t\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:none' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:none >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:limit=1k' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:limit=1k >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with tree:0' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+\n+\ttest_perf 'simulated partial clone' '\n+\t\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n+\t'\n+}\n+\n+test_partial_bitmap () {\n+\ttest_perf 'clone (partial bitmap)' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (partial bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tree filter (partial bitmap)' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+}\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 452be01056..7ad4f237bc 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -2,6 +2,7 @@\n \n test_description='Tests pack performance using bitmaps'\n . ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n test_perf_large_repo\n \n@@ -25,56 +26,7 @@ test_perf 'repack to disk' '\n \tgit repack -ad\n '\n \n-test_perf 'simulated clone' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'simulated fetch' '\n-\thave=$(git rev-list HEAD~100 -1) &&\n-\t{\n-\t\techo HEAD &&\n-\t\techo ^$have\n-\t} | git pack-objects --revs --stdout >/dev/null\n-'\n-\n-test_perf 'pack to file (bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n-'\n-\n-test_perf 'rev-list (commits)' '\n-\tgit rev-list --all --use-bitmap-index >/dev/null\n-'\n-\n-test_perf 'rev-list (objects)' '\n-\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with tag negated via --not --all (objects)' '\n-\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with negative tag (objects)' '\n-\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:none' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:none >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:limit=1k' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:limit=1k >/dev/null\n-'\n-\n-test_perf 'rev-list count with tree:0' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n-\n-test_perf 'simulated partial clone' '\n-\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n-'\n+test_full_bitmap\n \n test_expect_success 'create partial bitmap state' '\n \t# pick a commit to represent the repo tip in the past\n@@ -97,17 +49,6 @@ test_expect_success 'create partial bitmap state' '\n \tgit update-ref HEAD $orig_tip\n '\n \n-test_perf 'clone (partial bitmap)' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'pack to file (partial bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n-'\n-\n-test_perf 'rev-list with tree filter (partial bitmap)' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n+test_partial_bitmap\n \n test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421446","messageId":"83614f928486577892f9345f6a91a6240b47e173.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 20/22] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:17Z","receivedAt":"2021-04-09T18:12:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment\nvariable to also write a multi-pack bitmap when\n'GIT_TEST_MULTI_PACK_INDEX' is set.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 13 ++++++++++---\n ci/run-build-and-tests.sh | 1 +\n midx.h | 2 ++\n t/README | 4 ++++\n 4 files changed, 17 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 2847fdfbab..3cb843fb59 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -515,7 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n \t\t !is_bare_repository())\n \t\t\twrite_bitmaps = 0;\n-\t}\n+\t} else if (write_bitmaps &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\twrite_bitmaps = 0;\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n \n@@ -720,8 +723,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tupdate_server_info(0);\n \tremove_temporary_files();\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n+\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t}\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex a66b5e8c75..7c55a5033e 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -22,6 +22,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_WRITE_REV_INDEX=1\ndiff --git a/midx.h b/midx.h\nindex 350f4d0a7b..aa3da557bb 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,8 @@ struct pack_entry;\n struct repository;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex fd9375b146..956731da44 100644\n--- a/t/README\n+++ b/t/README\n@@ -420,6 +420,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n+'--bitmap' option on all invocations of 'git multi-pack-index write',\n+and ignores pack-objects' '--write-bitmap-index'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"421447","messageId":"b75b534446088b20e77cbaa7f09705510ddec908.1617991824.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH 22/22] p5326: perf tests for MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-09T18:12:25Z","receivedAt":"2021-04-09T18:12:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These new performance tests demonstrate effectively the same behavior as\np5310, but use a multi-pack bitmap instead of a single-pack one.\n\nNotably, p5326 does not create a MIDX bitmap with multiple packs. This\nis so we can measure a direct comparison between it and p5310. Any\ndifference between the two is measuring just the overhead of using MIDX\nbitmaps.\n\nHere are the results of p5310 and p5326 together, measured at the same\ntime and on the same machine (using a Xenon W-2255 CPU):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5310.2: repack to disk 96.78(93.39+11.33)\n 5310.3: simulated clone 9.98(9.79+0.19)\n 5310.4: simulated fetch 1.75(4.26+0.19)\n 5310.5: pack to file (bitmap) 28.20(27.87+8.70)\n 5310.6: rev-list (commits) 0.41(0.36+0.05)\n 5310.7: rev-list (objects) 1.61(1.54+0.07)\n 5310.8: rev-list count with blob:none 0.25(0.21+0.04)\n 5310.9: rev-list count with blob:limit=1k 2.65(2.54+0.10)\n 5310.10: rev-list count with tree:0 0.23(0.19+0.04)\n 5310.11: simulated partial clone 4.34(4.21+0.12)\n 5310.13: clone (partial bitmap) 11.05(12.21+0.48)\n 5310.14: pack to file (partial bitmap) 31.25(34.22+3.70)\n 5310.15: rev-list with tree filter (partial bitmap) 0.26(0.22+0.04)\n\nversus the same tests (this time using a multi-pack index):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5326.2: setup multi-pack index 78.99(75.29+11.58)\n 5326.3: simulated clone 11.78(11.56+0.22)\n 5326.4: simulated fetch 1.70(4.49+0.13)\n 5326.5: pack to file (bitmap) 28.02(27.72+8.76)\n 5326.6: rev-list (commits) 0.42(0.36+0.06)\n 5326.7: rev-list (objects) 1.65(1.58+0.06)\n 5326.8: rev-list count with blob:none 0.26(0.21+0.05)\n 5326.9: rev-list count with blob:limit=1k 2.97(2.86+0.10)\n 5326.10: rev-list count with tree:0 0.25(0.20+0.04)\n 5326.11: simulated partial clone 5.65(5.49+0.16)\n 5326.13: clone (partial bitmap) 12.22(13.43+0.38)\n 5326.14: pack to file (partial bitmap) 30.05(31.57+7.25)\n 5326.15: rev-list with tree filter (partial bitmap) 0.24(0.20+0.04)\n\nThere is slight overhead in \"simulated clone\", \"simulated partial\nclone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\ndue to using the MIDX's reverse index to map between bit positions and\nMIDX positions.\n\nThis can be reproduced by running \"git repack -adb\" along with \"git\nmulti-pack-index write --bitmap\" in a large-ish repository. Then run:\n\n $ perf record -o pack.perf git -c core.multiPackIndex=false \\\n pack-objects --all --stdout >/dev/null /dev/null \n---\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..5845109ac7\n--- /dev/null\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,43 @@\n+#!/bin/sh\n+\n+test_description='Tests performance using midx bitmaps'\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n+\n+test_perf_large_repo\n+\n+test_expect_success 'enable multi-pack index' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_perf 'setup multi-pack index' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap\n+'\n+\n+test_full_bitmap\n+\n+test_expect_success 'create partial bitmap state' '\n+\t# pick a commit to represent the repo tip in the past\n+\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t# now pretend we have just one tip\n+\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\tgit update-ref HEAD $cutoff &&\n+\n+\t# and then repack, which will leave us with a nice\n+\t# big bitmap pack of the \"old\" history, and all of\n+\t# the new history will be loose, as if it had been pushed\n+\t# up incrementally and exploded via unpack-objects\n+\tgit repack -Ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\n+\t# and now restore our original tip, as if the pushes\n+\t# had happened\n+\tgit update-ref HEAD $orig_tip\n+'\n+\n+test_partial_bitmap\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"422077","messageId":"20210416023925.16736-1-jonathantanmy@google.com","threadId":"55464","inReplyTo":"d5eeca4f112f70343b069fcb68fe61e26831843f.1617991824.git.me@ttaylorr.com","subject":"Re: [PATCH 12/22] pack-bitmap: read multi-pack bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-04-16T02:39:25Z","receivedAt":"2021-04-16T02:40:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I'll review until this patch for now. Hopefully I'll get to the rest\nsoon.\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5205dde2e1..a4e4e4ebcc 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -984,7 +984,17 @@ static void write_reused_pack(struct hashfile *f)\n> \t\t\t\tbreak;\n> \n> \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> -\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n> +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\t\toff_t pack_offs = bitmap_pack_offset(bitmap_git,\n> +\t\t\t\t\t\t\t\t pos + offset);\n> +\t\t\t\tuint32_t pos;\n> +\n> +\t\t\t\tif (offset_to_pack_pos(reuse_packfile, pack_offs, &pos) < 0)\n> +\t\t\t\t\tdie(_(\"write_reused_pack: could not locate %\"PRIdMAX),\n> +\t\t\t\t\t (intmax_t)pack_offs);\n> +\t\t\t\twrite_reused_pack_one(pos, f, &w_curs);\n> +\t\t\t} else\n> +\t\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n> \t\t\tdisplay_progress(progress_state, ++written);\n> \t\t}\n> \t}\n\nWhen bitmaps are used, pos + offset is the pseudo-pack (a virtual\nconcatenation of all packfiles in the MIDX) position (as in, first\nobject is 0, second object is 1, and so on), not a position in\na single packfile. From it, we obtain a pack offset, and from it, we\nobtain a position in the reused packfile (reuse_packfile). In this way,\nthe code is equivalent to the non-MIDX case. Looks good.\n\n(There is no need to select a packfile here in the case of MIDX because,\nas the code later shows, we always reuse only one packfile - assigned to\nreuse_packfile.)\n\n> @@ -35,8 +36,15 @@ struct stored_bitmap {\n> * the active bitmap index is the largest one.\n> */\n> struct bitmap_index {\n> -\t/* Packfile to which this bitmap index belongs to */\n> +\t/*\n> +\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n> +\t * to.\n> +\t *\n> +\t * Exactly one of these must be non-NULL; this specifies the object\n> +\t * order used to interpret this bitmap.\n> +\t */\n> \tstruct packed_git *pack;\n> +\tstruct multi_pack_index *midx;\n\nMakes sense.\n\n> @@ -71,6 +79,8 @@ struct bitmap_index {\n> \t/* If not NULL, this is a name-hash cache pointing into map. */\n> \tuint32_t *hashes;\n> \n> +\tconst unsigned char *checksum;\n> +\n> \t/*\n> \t * Extended index.\n> \t *\n\nI see later that this checksum is used, OK. Maybe comment that this\npoints into map (just like \"hashes\", as quoted above).\n\n> @@ -281,6 +304,54 @@ static char *pack_bitmap_filename(struct packed_git *p)\n> \treturn xstrfmt(\"%.*s.bitmap\", (int)len, p->pack_name);\n> }\n> \n> +static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> +\t\t\t struct multi_pack_index *midx)\n> +{\n> +\tstruct stat st;\n> +\tchar *idx_name = midx_bitmap_filename(midx);\n> +\tint fd = git_open(idx_name);\n> +\n> +\tfree(idx_name);\n> +\n> +\tif (fd < 0)\n> +\t\treturn -1;\n> +\n> +\tif (fstat(fd, &st)) {\n> +\t\tclose(fd);\n> +\t\treturn -1;\n> +\t}\n> +\n> +\tif (bitmap_git->pack || bitmap_git->midx) {\n> +\t\t/* ignore extra bitmap file; we can only handle one */\n> +\t\treturn -1;\n\nHere, fd is not closed? Maybe better to have multiple cleanup stages\n(one when the mmap has been built, and one when not).\n\n> @@ -302,12 +373,18 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n> \t\treturn -1;\n> \t}\n> \n> -\tif (bitmap_git->pack) {\n> +\tif (bitmap_git->pack || bitmap_git->midx) {\n> +\t\t/* ignore extra bitmap file; we can only handle one */\n> \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n> \t\tclose(fd);\n> \t\treturn -1;\n> \t}\n> \n> +\tif (!is_pack_valid(packfile)) {\n> +\t\tclose(fd);\n> +\t\treturn -1;\n> +\t}\n\nWhy is this needed now (and presumably, not before)?\n\n> -static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n> +static int load_reverse_index(struct bitmap_index *bitmap_git)\n> +{\n> +\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\tuint32_t i;\n> +\t\tint ret;\n> +\n> +\t\tret = load_midx_revindex(bitmap_git->midx);\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\n> +\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> +\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> +\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n> +\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n\nI was thinking about why we still need per-pack revindex, but I think\nthe answer is that we still need to convert pack offsets (roughly\nspeaking, 0 to size of packfile in bytes) to pack positions (0 to number\nof objects) (and one such conversion is in the quoted section of\nbuiltin/pack-objects.c above), and MIDX does not provide this. OK, makes\nsense.\n\n> +\t\t\tif (ret)\n> +\t\t\t\treturn ret;\n> +\t\t}\n> +\t\treturn 0;\n> +\t}\n> +\treturn load_pack_revindex(bitmap_git->pack);\n> +}\n\n[snip]\n\n> @@ -428,10 +552,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n> \treturn pos;\n> }\n> \n> +static int bitmap_position_midx(struct bitmap_index *bitmap_git,\n> +\t\t\t\tconst struct object_id *oid)\n> +{\n> +\tuint32_t want, got;\n> +\tif (!bsearch_midx(oid, bitmap_git->midx, &want))\n> +\t\treturn -1;\n> +\n> +\tif (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0)\n> +\t\treturn -1;\n> +\treturn got;\n> +}\n\nbsearch_midx() gives us the position in the MIDX (e.g. if we had an\nobject with the name 00...00, \"want\" will be 0, and if we had an\nobject with the name ff...ff, \"want\" will be the number of objects\nminus 1). midx_to_pack_pos() converts that into the position in the\npseudo-pack, which is what we want. OK.\n\n> @@ -730,14 +871,28 @@ static void show_objects_for_type(\n> \n> \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> \n> -\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n> -\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n> -\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n> +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n> +\t\t\t\tuint32_t pack_id;\n> +\n> +\t\t\t\tindex_pos = pack_pos_to_midx(m, pos + offset);\n> +\t\t\t\tofs = nth_midxed_offset(m, index_pos);\n> +\t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n> +\n> +\t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n> +\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n\nThis is similar to the builtin/pack-objects.c case right at the start of\nthis patch. (bitmap_pack_offset(), used in builtin/pack-objects.c, is\npack_pos_to_midx() and nth_midx_offset() chained.) OK.\n\n> +\t\t\t} else {\n> +\t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n> +\t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n> +\t\t\t\tnth_bitmap_object_oid(bitmap_git, &oid, index_pos);\n> +\n> +\t\t\t\tpack = bitmap_git->pack;\n> +\t\t\t}\n> \n> \t\t\tif (bitmap_git->hashes)\n> \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n> \n> -\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n> +\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n> \t\t}\n> \t}\n> }\n> @@ -749,8 +904,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n> \t\tstruct object *object = roots->item;\n> \t\troots = roots->next;\n> \n> -\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n> -\t\t\treturn 1;\n> +\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n> +\t\t\t\treturn 1;\n> +\t\t} else {\n> +\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n> +\t\t\t\treturn 1;\n> +\t\t}\n> \t}\n> \n> \treturn 0;\n\nOK - we don't actually care about the position, just that it exists,\nwhich is why we can pass NULL as the last argument to bsearch_midx().\n\n> @@ -839,14 +999,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git,\n> static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n> \t\t\t\t uint32_t pos)\n> {\n> -\tstruct packed_git *pack = bitmap_git->pack;\n> \tunsigned long size;\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> \n> \toi.sizep = &size;\n> \n> \tif (pos < bitmap_num_objects(bitmap_git)) {\n> -\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n> +\t\tstruct packed_git *pack;\n> +\t\toff_t ofs;\n> +\n> +\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> +\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> +\n> +\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> +\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> +\t\t} else {\n> +\t\t\tpack = bitmap_git->pack;\n> +\t\t\tofs = pack_pos_to_offset(pack, pos);\n> +\t\t}\n> +\n> \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n> \t\t\tstruct object_id oid;\n> \t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n\nMakes sense - \"pos\" is the position in the pseudo-pack. From it we get\nthe MIDX position, and then we can get the pack ID and pack offset as\nusual.\n\n> @@ -1081,15 +1253,29 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n> \t\t\t struct bitmap *reuse,\n> \t\t\t struct pack_window **w_curs)\n> {\n> -\toff_t offset, header;\n> +\tstruct packed_git *pack;\n> +\toff_t offset, delta_obj_offset;\n> \tenum object_type type;\n> \tunsigned long size;\n> \n> \tif (pos >= bitmap_num_objects(bitmap_git))\n> \t\treturn; /* not actually in the pack or MIDX */\n> \n> -\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n> -\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n> +\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\tuint32_t pack_id, midx_pos;\n> +\n> +\t\tmidx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> +\t\tpack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> +\n> +\t\tpack = bitmap_git->midx->packs[pack_id];\n> +\t\toffset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n\nWould it be useful to assert somewhere here that \"pack\" is the preferred\npack?\n\nGoing further, is it reasonable to say that positions 0..n in the\npreferred pack (where n is the number of objects in the preferred pack)\nmatch positions 0..n in the pseudo-pack exactly? If yes, maybe we can\nsimplify things by explaining that we can operate in the MIDX case\nexactly (or as similarly as possible) like we operate on a single\npackfile because of this, instead of always needing to consider if a\ndelta base could appear in the MIDX as belonging to another packfile.\n\n> @@ -1538,6 +1792,29 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n> \n> \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> \t\t\tpos = base + offset;\n> +\n> +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\t\tuint32_t pack_pos;\n> +\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> +\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> +\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> +\n> +\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> +\n> +\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n> +\t\t\t\t\tstruct object_id oid;\n> +\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n> +\n> +\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n> +\t\t\t\t\t oid_to_hex(&oid),\n> +\t\t\t\t\t pack_id,\n> +\t\t\t\t\t (uintmax_t)offset);\n> +\t\t\t\t}\n> +\n> +\t\t\t\tpos = pack_pos;\n> +\t\t\t} else\n> +\t\t\t\tpack = bitmap_git->pack;\n> +\n> \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n> \t\t\t\t pack_pos_to_offset(pack, pos);\n> \t\t}\n\n\"pos\" is assigned to twice in the MIDX case (with different semantics).\nI think it's better to do it like in the rest of the patch - use \"base +\noffset\" as the argument to pack_pos_to_midx, and then you wouldn't need\nto assign to \"pos\" twice.\n\n> diff --git a/packfile.c b/packfile.c\n> index 8668345d93..c444e365a3 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -863,7 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n> \tif (!strcmp(file_name, \"multi-pack-index\"))\n> \t\treturn;\n> \tif (starts_with(file_name, \"multi-pack-index\") &&\n> -\t ends_with(file_name, \".rev\"))\n> +\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n> \t\treturn;\n> \tif (ends_with(file_name, \".idx\") ||\n> \t ends_with(file_name, \".rev\") ||\n\nI guess this will come into play when we start writing MIDX bitmaps?\n"},{"id":"422078","messageId":"20210416024657.17563-1-jonathantanmy@google.com","threadId":"55464","inReplyTo":"d199954ef2246165f32a4f703d1e0ebf43847152.1617991824.git.me@ttaylorr.com","subject":"Re: [PATCH 02/22] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-04-16T02:46:57Z","receivedAt":"2021-04-16T02:47:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit)\n> \twriter.selected_nr++;\n> }\n> \n> -static uint32_t find_object_pos(const struct object_id *oid)\n> +static uint32_t find_object_pos(const struct object_id *oid, int *found)\n\nfind_object_pos() is only called by fill_bitmap_tree() and\nfill_bitmap_commit(). fill_bitmap_tree() is only called by itself and\nfill_bitmap_commit(). fill_bitmap_commit() is only called by\nbitmap_writer_build(). And bitmap_writer_build() is only called by\nwrite_pack_file(), which has been changed to die when\nbitmap_writer_build() fails. So looks like everything is accounted for.\n"},{"id":"422081","messageId":"YHkA0m8yZJ5lc/yo@nand.local","threadId":"55464","inReplyTo":"20210416023925.16736-1-jonathantanmy@google.com","subject":"Re: [PATCH 12/22] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-16T03:13:22Z","receivedAt":"2021-04-16T03:13:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Apr 15, 2021 at 07:39:25PM -0700, Jonathan Tan wrote:\n> I'll review until this patch for now. Hopefully I'll get to the rest\n> soon.\n\nThanks in advance. I always find that you leave insightful comments, so\nI appreciate you taking the time to review my patches.\n\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index 5205dde2e1..a4e4e4ebcc 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -984,7 +984,17 @@ static void write_reused_pack(struct hashfile *f)\n> > \t\t\t\tbreak;\n> >\n> > \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> > -\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n> > +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\t\t\toff_t pack_offs = bitmap_pack_offset(bitmap_git,\n> > +\t\t\t\t\t\t\t\t pos + offset);\n> > +\t\t\t\tuint32_t pos;\n> > +\n> > +\t\t\t\tif (offset_to_pack_pos(reuse_packfile, pack_offs, &pos) < 0)\n> > +\t\t\t\t\tdie(_(\"write_reused_pack: could not locate %\"PRIdMAX),\n> > +\t\t\t\t\t (intmax_t)pack_offs);\n> > +\t\t\t\twrite_reused_pack_one(pos, f, &w_curs);\n> > +\t\t\t} else\n> > +\t\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n> > \t\t\tdisplay_progress(progress_state, ++written);\n> > \t\t}\n> > \t}\n>\n> When bitmaps are used, pos + offset is the pseudo-pack (a virtual\n> concatenation of all packfiles in the MIDX) position (as in, first\n> object is 0, second object is 1, and so on), not a position in\n> a single packfile. From it, we obtain a pack offset, and from it, we\n> obtain a position in the reused packfile (reuse_packfile). In this way,\n> the code is equivalent to the non-MIDX case. Looks good.\n>\n> (There is no need to select a packfile here in the case of MIDX because,\n> as the code later shows, we always reuse only one packfile - assigned to\n> reuse_packfile.)\n\nYou're exactly right here on both points.\n\nIt's worth noting that the \"reuse\" you're describing here is only about\nreusing sections of the original packfile byte-for-byte (with the\nexception of fixing the offsets in any OFS_DELTAs). That's not to be\nconfused with delta reuse, which is entirely different.\n\nI think that both Peff and I are dubious that the pack-reuse stuff is\nkicking in all that much, since there are some heuristics in place about\nwhen it is allowed to take over and when it isn't, but that's a topic\nfor another thread.\n\n> > @@ -35,8 +36,15 @@ struct stored_bitmap {\n> > * the active bitmap index is the largest one.\n> > */\n> > struct bitmap_index {\n> > -\t/* Packfile to which this bitmap index belongs to */\n> > +\t/*\n> > +\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n> > +\t * to.\n> > +\t *\n> > +\t * Exactly one of these must be non-NULL; this specifies the object\n> > +\t * order used to interpret this bitmap.\n> > +\t */\n> > \tstruct packed_git *pack;\n> > +\tstruct multi_pack_index *midx;\n>\n> Makes sense.\n>\n> > @@ -71,6 +79,8 @@ struct bitmap_index {\n> > \t/* If not NULL, this is a name-hash cache pointing into map. */\n> > \tuint32_t *hashes;\n> >\n> > +\tconst unsigned char *checksum;\n> > +\n> > \t/*\n> > \t * Extended index.\n> > \t *\n>\n> I see later that this checksum is used, OK. Maybe comment that this\n> points into map (just like \"hashes\", as quoted above).\n\nYep, quite fair.\n\n> > +\tif (bitmap_git->pack || bitmap_git->midx) {\n> > +\t\t/* ignore extra bitmap file; we can only handle one */\n> > +\t\treturn -1;\n>\n> Here, fd is not closed? Maybe better to have multiple cleanup stages\n> (one when the mmap has been built, and one when not).\n\nGood eyes. That's an oversight, and we should be closing fd there, too.\nIt looks like we're also missing a warning(), although I am skeptical\nthat the warning would ever kick in. The pack-based version of this\nfunction is run in a loop over all packs, but the loop doesn't terminate\nonce a pack bitmap is opened, since we make sure that no *other* packs\nhave bitmaps, too.\n\nBut we don't do the same for multi-pack bitmaps, i.e., once we find a\nMIDX that has a bitmap, we terminate immediately. It may be worth\nscanning through the list of all MIDXs to make sure that only one has a\nbitmap, but to be honest I could go either way on that point, too, since\nany MIDX bitmap is worth loading. But the warning doesn't hurt, so I'll\nadd that, too.\n\n> > +\tif (!is_pack_valid(packfile)) {\n> > +\t\tclose(fd);\n> > +\t\treturn -1;\n> > +\t}\n>\n> Why is this needed now (and presumably, not before)?\n\nIt does appear as a stray hunk, and I'm sure that it probably could be\nextracted into its own patch. I can't recall anything about this\nparticular patch that makes it necessary, but maybe Peff remembers\nsomething I don't.\n\n> > @@ -1081,15 +1253,29 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n> > \t\t\t struct bitmap *reuse,\n> > \t\t\t struct pack_window **w_curs)\n> > {\n> > -\toff_t offset, header;\n> > +\tstruct packed_git *pack;\n> > +\toff_t offset, delta_obj_offset;\n> > \tenum object_type type;\n> > \tunsigned long size;\n> >\n> > \tif (pos >= bitmap_num_objects(bitmap_git))\n> > \t\treturn; /* not actually in the pack or MIDX */\n> >\n> > -\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n> > -\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n> > +\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\tuint32_t pack_id, midx_pos;\n> > +\n> > +\t\tmidx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> > +\t\tpack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> > +\n> > +\t\tpack = bitmap_git->midx->packs[pack_id];\n> > +\t\toffset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n>\n> Would it be useful to assert somewhere here that \"pack\" is the preferred\n> pack?\n\nAn assertion like that may hurt this function's cache performance, since\nthe way we determine the preferred pack is by looking at which pack is\nthe donor for the 0th object in the MIDX's .rev file. And this function\nis rather hot, since it is invoked once per-bit. So it may cause us to\nhit more page faults than we currently do.\n\nThat all said, the assertion may not be helping much since we only call\nthis method on objects from a single pack (the bitmapped pack in the\nsingle-pack case, or the preferred pack in the MIDX case). There's a\ncomment in reuse_partial_packfile_from_bitmap() to this effect, which\nmay or may not be good enough ;).\n\n> Going further, is it reasonable to say that positions 0..n in the\n> preferred pack (where n is the number of objects in the preferred pack)\n> match positions 0..n in the pseudo-pack exactly? If yes, maybe we can\n> simplify things by explaining that we can operate in the MIDX case\n> exactly (or as similarly as possible) like we operate on a single\n> packfile because of this, instead of always needing to consider if a\n> delta base could appear in the MIDX as belonging to another packfile.\n\nYou're right, and there are two things going on here which allow us to\nmake that assumption:\n\n - The preferred pack sorts ahead of all other packs in the MIDX when\n assembling the pseudo-pack order, so bits 0..n (where 'n' is the\n number of objects in the preferred pack) of the pseudo pack are\n designated to the preferred pack.\n\n - When duplicates of objects exist, the MIDX *always* breaks ties in\n favor of the preferred pack, so it's never the case that a delta'd\n object from the preferred pack will find its base in another pack\n (if it asked the MIDX to locate a copy of the base object).\n\nSo we can safely remove the conditional on bitmap_is_midx() in the first\npart of this function for exactly the reasons above, which is good. That\nprobably merits moving the comment beginning with \"Note that the base\ndoes not need to be repositioned ...\" earlier in this function, to make\nclear that we really can treat bits from the preferred pack as if they\ndon't have anything to do with the MIDX at all.\n\nSo long as we determine the preferred pack ahead of time (and not once\nper-call), I think that it would be a win.\n\n> > @@ -1538,6 +1792,29 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n> >\n> > \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> > \t\t\tpos = base + offset;\n> > +\n> > +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\t\t\tuint32_t pack_pos;\n> > +\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> > +\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> > +\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> > +\n> > +\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> > +\n> > +\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n> > +\t\t\t\t\tstruct object_id oid;\n> > +\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n> > +\n> > +\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n> > +\t\t\t\t\t oid_to_hex(&oid),\n> > +\t\t\t\t\t pack_id,\n> > +\t\t\t\t\t (uintmax_t)offset);\n> > +\t\t\t\t}\n> > +\n> > +\t\t\t\tpos = pack_pos;\n> > +\t\t\t} else\n> > +\t\t\t\tpack = bitmap_git->pack;\n> > +\n> > \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n> > \t\t\t\t pack_pos_to_offset(pack, pos);\n> > \t\t}\n>\n> \"pos\" is assigned to twice in the MIDX case (with different semantics).\n> I think it's better to do it like in the rest of the patch - use \"base +\n> offset\" as the argument to pack_pos_to_midx, and then you wouldn't need\n> to assign to \"pos\" twice.\n\nGood idea, thanks. Skimming again over the patch, this is the only place\nthat I could find where I double-assign pos like this.\n\n> > diff --git a/packfile.c b/packfile.c\n> > index 8668345d93..c444e365a3 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -863,7 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n> > \tif (!strcmp(file_name, \"multi-pack-index\"))\n> > \t\treturn;\n> > \tif (starts_with(file_name, \"multi-pack-index\") &&\n> > -\t ends_with(file_name, \".rev\"))\n> > +\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n> > \t\treturn;\n> > \tif (ends_with(file_name, \".idx\") ||\n> > \t ends_with(file_name, \".rev\") ||\n>\n> I guess this will come into play when we start writing MIDX bitmaps?\n\nYep, that's right. Since this patch is about making sure we can handle\nthe MIDX bitmap as described in\nDocumentation/technical/bitmap-format.txt, this is part of that.\n\nThanks,\nTaylor\n"},{"id":"423562","messageId":"20210504050230.2915390-1-jonathantanmy@google.com","threadId":"55464","inReplyTo":"fd320c5ed48c7431b64b898f49101b0f53655a95.1617991824.git.me@ttaylorr.com","subject":"Re: [PATCH 13/22] pack-bitmap: write multi-pack bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-04T05:02:30Z","receivedAt":"2021-05-04T05:02:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Write multi-pack bitmaps in the format described by\n> Documentation/technical/bitmap-format.txt, inferring their presence with\n> the absence of '--bitmap'.\n> \n> To write a multi-pack bitmap, this patch attempts to reuse as much of\n> the existing machinery from pack-objects as possible. Specifically, the\n> MIDX code prepares a packing_data struct that pretends as if a single\n> packfile has been generated containing all of the objects contained\n> within the MIDX.\n\nSounds good, and makes sense. Conceptually, the MIDX bitmap is the same\nas a regular packfile bitmap, just that the order of objects in the\nbitmap is defined differently.\n\n> +static void prepare_midx_packing_data(struct packing_data *pdata,\n> +\t\t\t\t struct write_midx_context *ctx)\n> +{\n> +\tuint32_t i;\n> +\n> +\tmemset(pdata, 0, sizeof(struct packing_data));\n> +\tprepare_packing_data(the_repository, pdata);\n> +\n> +\tfor (i = 0; i < ctx->entries_nr; i++) {\n> +\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n> +\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n> +\n> +\t\toe_set_in_pack(pdata, to,\n> +\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n> +\t}\n> +}\n\nIt is surprising to see this right at the top. Scrolling down, I guess\nthat there is more information needed than just the packing_data struct.\n\n> +static int add_ref_to_pending(const char *refname,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t int flag, void *cb_data)\n> +{\n> +\tstruct rev_info *revs = (struct rev_info*)cb_data;\n> +\tstruct object *object;\n> +\n> +\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n> +\t\twarning(\"symbolic ref is dangling: %s\", refname);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tobject = parse_object_or_die(oid, refname);\n> +\tif (object->type != OBJ_COMMIT)\n> +\t\treturn 0;\n> +\n> +\tadd_pending_object(revs, object, \"\");\n> +\tif (bitmap_is_preferred_refname(revs->repo, refname))\n> +\t\tobject->flags |= NEEDS_BITMAP;\n> +\treturn 0;\n> +}\n\nMakes sense. We need to flag certain commits as NEEDS_BITMAP because\nbitmaps are not made for all commits but only certain ones.\n\n> +struct bitmap_commit_cb {\n> +\tstruct commit **commits;\n> +\tsize_t commits_nr, commits_alloc;\n> +\n> +\tstruct write_midx_context *ctx;\n> +};\n> +\n> +static const struct object_id *bitmap_oid_access(size_t index,\n> +\t\t\t\t\t\t const void *_entries)\n> +{\n> +\tconst struct pack_midx_entry *entries = _entries;\n> +\treturn &entries[index].oid;\n> +}\n> +\n> +static void bitmap_show_commit(struct commit *commit, void *_data)\n> +{\n> +\tstruct bitmap_commit_cb *data = _data;\n> +\tif (oid_pos(&commit->object.oid, data->ctx->entries,\n> +\t\t data->ctx->entries_nr,\n> +\t\t bitmap_oid_access) > -1) {\n> +\t\tALLOC_GROW(data->commits, data->commits_nr + 1,\n> +\t\t\t data->commits_alloc);\n> +\t\tdata->commits[data->commits_nr++] = commit;\n> +\t}\n> +}\n> +\n> +static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n> +\t\t\t\t\t\t struct write_midx_context *ctx)\n> +{\n> +\tstruct rev_info revs;\n> +\tstruct bitmap_commit_cb cb;\n> +\n> +\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n> +\tcb.ctx = ctx;\n> +\n> +\trepo_init_revisions(the_repository, &revs, NULL);\n> +\tfor_each_ref(add_ref_to_pending, &revs);\n> +\n> +\tfetch_if_missing = 0;\n> +\trevs.exclude_promisor_objects = 1;\n\nI think that the MIDX bitmap requires all objects be present? If yes, we\nshould omit these 2 lines.\n\n> +\n> +\tif (prepare_revision_walk(&revs))\n> +\t\tdie(_(\"revision walk setup failed\"));\n> +\n> +\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n> +\tif (indexed_commits_nr_p)\n> +\t\t*indexed_commits_nr_p = cb.commits_nr;\n> +\n> +\treturn cb.commits;\n> +}\n\nHmm...I might be missing something obvious, but this function and its\ncallbacks seem to be written like this in order to put the returned\ncommits in a certain order. But later on in write_midx_bitmap(), the\nreturn value of this function is passed to\nbitmap_writer_select_commits(), which resorts the list anyway?\n\n> +static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n> +\t\t\t struct write_midx_context *ctx,\n> +\t\t\t unsigned flags)\n> +{\n> +\tstruct packing_data pdata;\n> +\tstruct pack_idx_entry **index;\n> +\tstruct commit **commits = NULL;\n> +\tuint32_t i, commits_nr;\n> +\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n> +\tint ret;\n> +\n> +\tprepare_midx_packing_data(&pdata, ctx);\n> +\n> +\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n> +\n> +\t/*\n> +\t * Build the MIDX-order index based on pdata.objects (which is already\n> +\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n> +\t * this order).\n> +\t */\n> +\tALLOC_ARRAY(index, pdata.nr_objects);\n> +\tfor (i = 0; i < pdata.nr_objects; i++)\n> +\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n> +\n> +\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n> +\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n> +\n> +\t/*\n> +\t * bitmap_writer_select_commits expects objects in lex order, but\n> +\t * pack_order gives us exactly that. use it directly instead of\n> +\t * re-sorting the array\n> +\t */\n> +\tfor (i = 0; i < pdata.nr_objects; i++)\n> +\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n> +\n> +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n\nThe comment above says bitmap_writer_select_commits() expects objects in\nlex order, but (1) you're putting \"index\" in lex order, not \"commits\",\nand (2) the first thing in bitmap_writer_select_commits() is a QSORT.\nDid you mean another function?\n\n> +\tret = bitmap_writer_build(&pdata);\n> +\tif (!ret)\n> +\t\tgoto cleanup;\n> +\n> +\tbitmap_writer_set_checksum(midx_hash);\n> +\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n\nSo bitmap_writer_build_type_index() and bitmap_writer_finish() are\ncalled with 2 different orders of commits. Is this expected? If yes,\nmaybe this is worth a comment.\n\n> +\n> +cleanup:\n> +\tfree(index);\n> +\tfree(bitmap_name);\n> +\treturn ret;\n> +}\n\n[snip]\n\n> @@ -930,9 +1073,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> \n> +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> +\t\t\t\terror(_(\"could not load pack %s\"),\n> +\t\t\t\t ctx.m->pack_names[i]);\n> +\t\t\t\tresult = 1;\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n> +\n> \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n> \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n> -\t\t\tctx.info[ctx.nr].p = NULL;\n> +\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n> \t\t\tctx.info[ctx.nr].expired = 0;\n> \t\t\tctx.nr++;\n> \t\t}\n\nWhy is this needed now and not before? From what I see in this function,\nnothing seems to happen to this .p pack except that they are closed\nlater.\n\n> @@ -1096,6 +1264,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\tif (ctx.info[i].p) {\n> \t\t\tclose_pack(ctx.info[i].p);\n> \t\t\tfree(ctx.info[i].p);\n> +\t\t\tif (ctx.m) {\n> +\t\t\t\t/*\n> +\t\t\t\t * Destroy a stale reference to the pack in\n> +\t\t\t\t * 'ctx.m'.\n> +\t\t\t\t */\n> +\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n> +\t\t\t\tif (orig < ctx.m->num_packs)\n> +\t\t\t\t\tctx.m->packs[orig] = NULL;\n> +\t\t\t}\n> \t\t}\n> \t\tfree(ctx.info[i].pack_name);\n> \t}\n\nIs this hunk needed? \"ctx\" is a local variable and will not outlast this\nfunction.\n\nI'll review the rest tomorrow. It seems like I've gotten over the most\ndifficult patches.\n"},{"id":"423601","messageId":"20210504175125.2987651-1-jonathantanmy@google.com","threadId":"55464","inReplyTo":"ff74181e85975690b4fccfb6b72fb80045f4adc7.1617991824.git.me@ttaylorr.com","subject":"Re: [PATCH 16/22] t5326: test multi-pack bitmap behavior","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-04T17:51:25Z","receivedAt":"2021-05-04T17:51:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +test_expect_success 'clone with bitmaps enabled' '\n> +\tgit clone --no-local --bare . clone-reverse-delta.git &&\n> +\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n> +\n> +\tgit rev-parse HEAD >expect &&\n> +\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n> +\ttest_cmp expect actual\n> +'\n\nWhat is this test testing? That bitmaps are used? (I'm not sure how to\nverify that though - we seem to have tracing for bitmap writing but not\nfor reading, for example.)\n\n> +bitmap_reuse_tests() {\n> +\tfrom=$1\n> +\tto=$2\n> +\n> +\ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\ttest_commit_bulk 16 &&\n> +\t\t\tgit tag old-tip &&\n> +\n> +\t\t\tgit config core.multiPackIndex true &&\n> +\t\t\tif test \"MIDX\" = \"$from\"\n> +\t\t\tthen\n> +\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&\n> +\t\t\t\tgit multi-pack-index write --bitmap\n> +\t\t\telse\n> +\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n> +\t\t\tfi\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\ttest_commit_bulk --id=further 16 &&\n> +\t\t\tgit tag new-tip &&\n> +\n> +\t\t\tif test \"MIDX\" = \"$to\"\n> +\t\t\tthen\n> +\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n> +\t\t\t\tgit multi-pack-index write --bitmap\n> +\t\t\telse\n> +\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n> +\t\t\tfi\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit for-each-ref &&\n> +\t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n> +\t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n> +\t\t)\n> +\t'\n> +}\n> +\n> +bitmap_reuse_tests 'pack' 'MIDX'\n> +bitmap_reuse_tests 'MIDX' 'pack'\n> +bitmap_reuse_tests 'MIDX' 'MIDX'\n\nIs it possible to verify that the bitmaps have truly been reused (and\nnot, say, created from scratch)? (E.g. is there any nature of the\nbitmap created - for example, the order of commits?)\n\n> +test_expect_success 'pack.preferBitmapTips' '\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit_bulk --message=\"%s\" 103 &&\n> +\n> +\t\tgit log --format=\"%H\" >commits.raw &&\n> +\t\tsort commits &&\n> +\n> +\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> +\t\tgit update-ref --stdin +\n> +\t\tgit multi-pack-index write --bitmap &&\n> +\t\ttest_path_is_file $midx &&\n> +\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\n> +\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\tcomm -13 bitmaps commits >before &&\n> +\t\ttest_line_count = 1 before &&\n> +\n> +\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n> +\t\t\t +\n> +\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n> +\t\trm -fr $midx &&\n> +\n> +\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n> +\t\t\tmulti-pack-index write --bitmap &&\n> +\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\tcomm -13 bitmaps commits >after &&\n> +\n> +\t\t! test_cmp before after\n> +\t)\n> +'\n\nCould we have a more precise comparison of \"before\" and \"after\" (besides\nthe fact that they're different)?\n\nBesides that, all the patches up to this one look good (including patch\n14, verified with \"--color-moved\n--color-moved-ws=allow-indentation-change\").\n"},{"id":"423602","messageId":"20210504180058.2988580-1-jonathantanmy@google.com","threadId":"55464","inReplyTo":"b75b534446088b20e77cbaa7f09705510ddec908.1617991824.git.me@ttaylorr.com","subject":"Re: [PATCH 22/22] p5326: perf tests for MIDX bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-04T18:00:58Z","receivedAt":"2021-05-04T18:01:05Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> There is slight overhead in \"simulated clone\", \"simulated partial\n> clone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\n> due to using the MIDX's reverse index to map between bit positions and\n> MIDX positions.\n\nThanks - it's great to see that accessing a MIDX bitmap doesn't add much\noverhead (as compared to accessing a single-pack bitmap of the same\nsize).\n\nAll the remaining patches up to and including this one look good.\nOverall, I did have some comments here and there, but I am happy with\nthe overall design.\n"},{"id":"423629","messageId":"xmqqh7ji578n.fsf@gitster.g","threadId":"55464","inReplyTo":"20210504180058.2988580-1-jonathantanmy@google.com","subject":"Re: [PATCH 22/22] p5326: perf tests for MIDX bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-05T00:55:52Z","receivedAt":"2021-05-05T00:56:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n>> There is slight overhead in \"simulated clone\", \"simulated partial\n>> clone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\n>> due to using the MIDX's reverse index to map between bit positions and\n>> MIDX positions.\n>\n> Thanks - it's great to see that accessing a MIDX bitmap doesn't add much\n> overhead (as compared to accessing a single-pack bitmap of the same\n> size).\n>\n> All the remaining patches up to and including this one look good.\n> Overall, I did have some comments here and there, but I am happy with\n> the overall design.\n\nThanks for a review.\n"},{"id":"423814","messageId":"YJRPMgBgg65ohRg0@nand.local","threadId":"55464","inReplyTo":"20210504050230.2915390-1-jonathantanmy@google.com","subject":"Re: [PATCH 13/22] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-05-06T20:18:58Z","receivedAt":"2021-05-06T20:19:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 03, 2021 at 10:02:30PM -0700, Jonathan Tan wrote:\n> > +static void prepare_midx_packing_data(struct packing_data *pdata,\n> > +\t\t\t\t struct write_midx_context *ctx)\n> > +{\n> > +\tuint32_t i;\n> > +\n> > +\tmemset(pdata, 0, sizeof(struct packing_data));\n> > +\tprepare_packing_data(the_repository, pdata);\n> > +\n> > +\tfor (i = 0; i < ctx->entries_nr; i++) {\n> > +\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n> > +\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n> > +\n> > +\t\toe_set_in_pack(pdata, to,\n> > +\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n> > +\t}\n> > +}\n>\n> It is surprising to see this right at the top. Scrolling down, I guess\n> that there is more information needed than just the packing_data struct.\n\nHmm, which part is surprising to you? This function is setting up the\npacking_data structure that I mentioned in the commit message, which\nhappens in two steps. First, we allocate and call\nprepare_packing_data(). And then we call packlist_alloc() for each\nobject in the MIDX, setting up some information about each object\n(like its OID and which physical pack it came from).\n\nBut if any of this is unclear, let me know which part and I'd be happy\nto add a clarifying comment.\n\n> > +static int add_ref_to_pending(const char *refname,\n> > +\t\t\t const struct object_id *oid,\n> > +\t\t\t int flag, void *cb_data)\n> > +{\n> > +\tstruct rev_info *revs = (struct rev_info*)cb_data;\n> > +\tstruct object *object;\n> > +\n> > +\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n> > +\t\twarning(\"symbolic ref is dangling: %s\", refname);\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > +\tobject = parse_object_or_die(oid, refname);\n> > +\tif (object->type != OBJ_COMMIT)\n> > +\t\treturn 0;\n> > +\n> > +\tadd_pending_object(revs, object, \"\");\n> > +\tif (bitmap_is_preferred_refname(revs->repo, refname))\n> > +\t\tobject->flags |= NEEDS_BITMAP;\n> > +\treturn 0;\n> > +}\n>\n> Makes sense. We need to flag certain commits as NEEDS_BITMAP because\n> bitmaps are not made for all commits but only certain ones.\n\nRight, and the NEEDS_BITMAP is a bit of a misnomer. It's true meaning is\nmore like BITMAPPING_THIS_WOULD_BE_A_GOOD_IDEA, since it roughly\ntranslates to \"bitmap this commit before any others in its window\". More\ndetails are in bitmap_writer_select_commits(), but in all honesty I find\nthe implementation there somewhat confusing.\n\n> > +static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n> > +\t\t\t\t\t\t struct write_midx_context *ctx)\n> > +{\n> > +\tstruct rev_info revs;\n> > +\tstruct bitmap_commit_cb cb;\n> > +\n> > +\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n> > +\tcb.ctx = ctx;\n> > +\n> > +\trepo_init_revisions(the_repository, &revs, NULL);\n> > +\tfor_each_ref(add_ref_to_pending, &revs);\n> > +\n> > +\tfetch_if_missing = 0;\n> > +\trevs.exclude_promisor_objects = 1;\n>\n> I think that the MIDX bitmap requires all objects be present? If yes, we\n> should omit these 2 lines.\n\nIt does require that all objects are present, but if we fetched any\npromisor objects at this stage it would be too late. That's because by\nthe time we're in this function, all of the packs that are to be\nincluded in the MIDX should already exist on disk.\n\nSkipping promisor objects here is intentional, since it only excludes\nthem from the list of reachable commits that we want to select from when\ncomputing the selection of MIDX'd commits to receive bitmaps.\n\nBut, if one of those promisor objects is reachable from another object\nthat is included in the bitmap, then we will complain later on that we\ncouldn't find a reachability closure (and fail appropriately).\n\nThat said, I'm not sure any of that is obvious from reading this code,\nso I'll add a comment to that effect around these lines.\n\n> > +\n> > +\tif (prepare_revision_walk(&revs))\n> > +\t\tdie(_(\"revision walk setup failed\"));\n> > +\n> > +\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n> > +\tif (indexed_commits_nr_p)\n> > +\t\t*indexed_commits_nr_p = cb.commits_nr;\n> > +\n> > +\treturn cb.commits;\n> > +}\n>\n> Hmm...I might be missing something obvious, but this function and its\n> callbacks seem to be written like this in order to put the returned\n> commits in a certain order. But later on in write_midx_bitmap(), the\n> return value of this function is passed to\n> bitmap_writer_select_commits(), which resorts the list anyway?\n\nIt isn't intentional, but rather just to build up the list in topo\norder. In fact, the order we build it up in isn't quite the same as how\nthe pack bitmap code generates it (it is in true topo order, at least on\nGitHub's servers, as a side effect of using delta islands).\n\nThe fact that we resort according to date_compare makes me wonder why\nchanging that seemed to make such a difference for us. The whole\nselection code is a mystery to me.\n\nBut no, the order shouldn't matter since we QSORT it later. Any code\nhere that looks like it's putting it in a certain order has much more to\ndo with convenience than anything else.\n\n>\n> > +static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n> > +\t\t\t struct write_midx_context *ctx,\n> > +\t\t\t unsigned flags)\n> > +{\n> > +\tstruct packing_data pdata;\n> > +\tstruct pack_idx_entry **index;\n> > +\tstruct commit **commits = NULL;\n> > +\tuint32_t i, commits_nr;\n> > +\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n> > +\tint ret;\n> > +\n> > +\tprepare_midx_packing_data(&pdata, ctx);\n> > +\n> > +\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n> > +\n> > +\t/*\n> > +\t * Build the MIDX-order index based on pdata.objects (which is already\n> > +\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n> > +\t * this order).\n> > +\t */\n> > +\tALLOC_ARRAY(index, pdata.nr_objects);\n> > +\tfor (i = 0; i < pdata.nr_objects; i++)\n> > +\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n> > +\n> > +\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n> > +\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n> > +\n> > +\t/*\n> > +\t * bitmap_writer_select_commits expects objects in lex order, but\n> > +\t * pack_order gives us exactly that. use it directly instead of\n> > +\t * re-sorting the array\n> > +\t */\n> > +\tfor (i = 0; i < pdata.nr_objects; i++)\n> > +\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n> > +\n> > +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n>\n> The comment above says bitmap_writer_select_commits() expects objects in\n> lex order, but (1) you're putting \"index\" in lex order, not \"commits\",\n> and (2) the first thing in bitmap_writer_select_commits() is a QSORT.\n> Did you mean another function?\n\nAck, I definitely meant bitmap_writer_build(). Thanks for catching.\n\n> > +\tret = bitmap_writer_build(&pdata);\n> > +\tif (!ret)\n> > +\t\tgoto cleanup;\n> > +\n> > +\tbitmap_writer_set_checksum(midx_hash);\n> > +\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n>\n> So bitmap_writer_build_type_index() and bitmap_writer_finish() are\n> called with 2 different orders of commits. Is this expected? If yes,\n> maybe this is worth a comment.\n\nConfusingly so, but yes, these two do expect different orders. You can\nsee the same re-sorting going on much more subtly in\npack-write.c:write_idx_file(), which is called by\nbuiltin/pack-objects.c:finish_tmp_packfile(), which happens between\nbitmap_writer_build_type_index() and bitmap_writer_finish().\n\nDefinitely worth adding a comment.\n\n> > @@ -930,9 +1073,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> > \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> >\n> > +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> > +\t\t\t\terror(_(\"could not load pack %s\"),\n> > +\t\t\t\t ctx.m->pack_names[i]);\n> > +\t\t\t\tresult = 1;\n> > +\t\t\t\tgoto cleanup;\n> > +\t\t\t}\n> > +\n> > \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n> > \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n> > -\t\t\tctx.info[ctx.nr].p = NULL;\n> > +\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n> > \t\t\tctx.info[ctx.nr].expired = 0;\n> > \t\t\tctx.nr++;\n> > \t\t}\n>\n> Why is this needed now and not before? From what I see in this function,\n> nothing seems to happen to this .p pack except that they are closed\n> later.\n\nThese are used by prepare_midx_packing_data().\n\n> > @@ -1096,6 +1264,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \t\tif (ctx.info[i].p) {\n> > \t\t\tclose_pack(ctx.info[i].p);\n> > \t\t\tfree(ctx.info[i].p);\n> > +\t\t\tif (ctx.m) {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * Destroy a stale reference to the pack in\n> > +\t\t\t\t * 'ctx.m'.\n> > +\t\t\t\t */\n> > +\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n> > +\t\t\t\tif (orig < ctx.m->num_packs)\n> > +\t\t\t\t\tctx.m->packs[orig] = NULL;\n> > +\t\t\t}\n> > \t\t}\n> > \t\tfree(ctx.info[i].pack_name);\n> > \t}\n>\n> Is this hunk needed? \"ctx\" is a local variable and will not outlast this\n> function.\n\nI can't remember exactly why I added this. I'll play around with it and\neither remove it or add a comment why it's necessary before the next\nreroll.\n\n> I'll review the rest tomorrow. It seems like I've gotten over the most\n> difficult patches.\n\nThanks, and sorry that this took me a few days to get back to. I\nappreciate your review immensely.\n\nThanks,\nTaylor\n"},{"id":"423826","messageId":"20210506220035.366165-1-jonathantanmy@google.com","threadId":"55464","inReplyTo":"YJRPMgBgg65ohRg0@nand.local","subject":"Re: [PATCH 13/22] pack-bitmap: write multi-pack bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-06T22:00:35Z","receivedAt":"2021-05-06T22:00:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Mon, May 03, 2021 at 10:02:30PM -0700, Jonathan Tan wrote:\n> > > +static void prepare_midx_packing_data(struct packing_data *pdata,\n> > > +\t\t\t\t struct write_midx_context *ctx)\n> > > +{\n> > > +\tuint32_t i;\n> > > +\n> > > +\tmemset(pdata, 0, sizeof(struct packing_data));\n> > > +\tprepare_packing_data(the_repository, pdata);\n> > > +\n> > > +\tfor (i = 0; i < ctx->entries_nr; i++) {\n> > > +\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n> > > +\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n> > > +\n> > > +\t\toe_set_in_pack(pdata, to,\n> > > +\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n> > > +\t}\n> > > +}\n> >\n> > It is surprising to see this right at the top. Scrolling down, I guess\n> > that there is more information needed than just the packing_data struct.\n> \n> Hmm, which part is surprising to you? This function is setting up the\n> packing_data structure that I mentioned in the commit message, which\n> happens in two steps. First, we allocate and call\n> prepare_packing_data(). And then we call packlist_alloc() for each\n> object in the MIDX, setting up some information about each object\n> (like its OID and which physical pack it came from).\n> \n> But if any of this is unclear, let me know which part and I'd be happy\n> to add a clarifying comment.\n\nAh, I think I was unclear. I was thinking that the commit message led me\nto believe that all information needed for creating a bitmap lies in the\npacking_data struct, so I would have expected several helper functions\nfollowed by a function that actually writes the packing_data struct.\nMaybe the commit message could be rewritten to avoid that confusion, but\nit's probably not a big deal.\n\n> > > +static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n> > > +\t\t\t\t\t\t struct write_midx_context *ctx)\n> > > +{\n> > > +\tstruct rev_info revs;\n> > > +\tstruct bitmap_commit_cb cb;\n> > > +\n> > > +\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n> > > +\tcb.ctx = ctx;\n> > > +\n> > > +\trepo_init_revisions(the_repository, &revs, NULL);\n> > > +\tfor_each_ref(add_ref_to_pending, &revs);\n> > > +\n> > > +\tfetch_if_missing = 0;\n> > > +\trevs.exclude_promisor_objects = 1;\n> >\n> > I think that the MIDX bitmap requires all objects be present? If yes, we\n> > should omit these 2 lines.\n> \n> It does require that all objects are present, but if we fetched any\n> promisor objects at this stage it would be too late. That's because by\n> the time we're in this function, all of the packs that are to be\n> included in the MIDX should already exist on disk.\n> \n> Skipping promisor objects here is intentional, since it only excludes\n> them from the list of reachable commits that we want to select from when\n> computing the selection of MIDX'd commits to receive bitmaps.\n> \n> But, if one of those promisor objects is reachable from another object\n> that is included in the bitmap, then we will complain later on that we\n> couldn't find a reachability closure (and fail appropriately).\n> \n> That said, I'm not sure any of that is obvious from reading this code,\n> so I'll add a comment to that effect around these lines.\n\nSo you're saying that if we have missing promisor commits as in the\nfollowing graph:\n\n A\n / \\\n B C\n | |\n . .\n . .\n . .\n\nwhere B is missing but promised, and only C is NEEDS_BITMAP, then the\nMIDX bitmap write will still work? (So the rev walk is intended to walk\nthrough A and C but not B, and because we are only building bitmaps for\nC and potentially its ancestors, we only need the objects in C's\ntransitive closure.) Even if this is true, \"exclude_promisor_objects\" is\nthe wrong option here, because it will exclude all commits that came\nfrom a promisor remote (regardless of whether it is present locally).\n(That's how \"promisor object\" is defined in partial-clone.txt.) What we\nneed would be an option that permits missing links.\n\nAnd even if we go with that option that permits missing links, it still\nremains that we have very little support for missing promisor commits in\nGit right now.\n\nIt might be better to just assume that MIDX will only be used for full\nclones. If you want, you can add a NEEDSWORK explaining the above case.\n\n> > > +\n> > > +\tif (prepare_revision_walk(&revs))\n> > > +\t\tdie(_(\"revision walk setup failed\"));\n> > > +\n> > > +\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n> > > +\tif (indexed_commits_nr_p)\n> > > +\t\t*indexed_commits_nr_p = cb.commits_nr;\n> > > +\n> > > +\treturn cb.commits;\n> > > +}\n> >\n> > Hmm...I might be missing something obvious, but this function and its\n> > callbacks seem to be written like this in order to put the returned\n> > commits in a certain order. But later on in write_midx_bitmap(), the\n> > return value of this function is passed to\n> > bitmap_writer_select_commits(), which resorts the list anyway?\n> \n> It isn't intentional, but rather just to build up the list in topo\n> order. In fact, the order we build it up in isn't quite the same as how\n> the pack bitmap code generates it (it is in true topo order, at least on\n> GitHub's servers, as a side effect of using delta islands).\n> \n> The fact that we resort according to date_compare makes me wonder why\n> changing that seemed to make such a difference for us. The whole\n> selection code is a mystery to me.\n> \n> But no, the order shouldn't matter since we QSORT it later. Any code\n> here that looks like it's putting it in a certain order has much more to\n> do with convenience than anything else.\n\nIf the order doesn't matter, why don't you just copy one-by-one from\ndata->ctx->entries into data->commits? (Unless data->ctx->entries has\nextra commits?)\n\n> > > +\tret = bitmap_writer_build(&pdata);\n> > > +\tif (!ret)\n> > > +\t\tgoto cleanup;\n> > > +\n> > > +\tbitmap_writer_set_checksum(midx_hash);\n> > > +\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n> >\n> > So bitmap_writer_build_type_index() and bitmap_writer_finish() are\n> > called with 2 different orders of commits. Is this expected? If yes,\n> > maybe this is worth a comment.\n> \n> Confusingly so, but yes, these two do expect different orders. You can\n> see the same re-sorting going on much more subtly in\n> pack-write.c:write_idx_file(), which is called by\n> builtin/pack-objects.c:finish_tmp_packfile(), which happens between\n> bitmap_writer_build_type_index() and bitmap_writer_finish().\n> \n> Definitely worth adding a comment.\n\nAh, I see. Thanks for your explanation.\n\n> > > @@ -930,9 +1073,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > > \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> > > \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> > >\n> > > +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> > > +\t\t\t\terror(_(\"could not load pack %s\"),\n> > > +\t\t\t\t ctx.m->pack_names[i]);\n> > > +\t\t\t\tresult = 1;\n> > > +\t\t\t\tgoto cleanup;\n> > > +\t\t\t}\n> > > +\n> > > \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n> > > \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n> > > -\t\t\tctx.info[ctx.nr].p = NULL;\n> > > +\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n> > > \t\t\tctx.info[ctx.nr].expired = 0;\n> > > \t\t\tctx.nr++;\n> > > \t\t}\n> >\n> > Why is this needed now and not before? From what I see in this function,\n> > nothing seems to happen to this .p pack except that they are closed\n> > later.\n> \n> These are used by prepare_midx_packing_data().\n\nAh, thanks.\n\n> > > @@ -1096,6 +1264,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > > \t\tif (ctx.info[i].p) {\n> > > \t\t\tclose_pack(ctx.info[i].p);\n> > > \t\t\tfree(ctx.info[i].p);\n> > > +\t\t\tif (ctx.m) {\n> > > +\t\t\t\t/*\n> > > +\t\t\t\t * Destroy a stale reference to the pack in\n> > > +\t\t\t\t * 'ctx.m'.\n> > > +\t\t\t\t */\n> > > +\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n> > > +\t\t\t\tif (orig < ctx.m->num_packs)\n> > > +\t\t\t\t\tctx.m->packs[orig] = NULL;\n> > > +\t\t\t}\n> > > \t\t}\n> > > \t\tfree(ctx.info[i].pack_name);\n> > > \t}\n> >\n> > Is this hunk needed? \"ctx\" is a local variable and will not outlast this\n> > function.\n> \n> I can't remember exactly why I added this. I'll play around with it and\n> either remove it or add a comment why it's necessary before the next\n> reroll.\n\nOK.\n\n> \n> > I'll review the rest tomorrow. It seems like I've gotten over the most\n> > difficult patches.\n> \n> Thanks, and sorry that this took me a few days to get back to. I\n> appreciate your review immensely.\n\nNo worries, and thanks for these patches.\n"},{"id":"428118","messageId":"cover.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH v2 00/24] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:24:56Z","receivedAt":"2021-06-21T22:25:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll of my series to implement multi-pack reachability bitmaps. It\nis based on 'master', and incorporates a handful of changes from an earlier\nround of review from Jonathan Tan, as well as a handful of tweaks useful to us\nat GitHub that we've picked up over a few months of running these patches in\nproduction.\n\nI have been quite behind in sending this to the list because a number of\nnon-work things that have kept me busy. But those seem to have settled down, so\nhere is a second reroll.\n\nNotable changes since last time are summarized here (though a complete\nrange-diff is below):\n\n - A preferred pack is inferred when not otherwise specified. This fixes a\n nasty bug dependent on readdir() ordering which can cause bitmap corruption.\n See the new ninth patch for the gory details.\n - A bug which broke CI on 'seen' is fixed where t0410.27 would fail when\n writing a bitmap (as is the case when\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 is set).\n - Comments in unclear portions of the code having to do with promisor objects,\n and object order when fed to bitmap writing routines are added.\n - A number of spots dealing with pack reuse were simplified to avoid using the\n MIDX's .rev file where unnecessary (along with a comment explaining why the\n optimization is possible in the first place).\n\nThanks in advance for your review, and sorry for the wait.\n\nJeff King (2):\n t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n\nTaylor Blau (22):\n pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n pack-bitmap-write.c: free existing bitmaps\n Documentation: build 'technical/bitmap-format' by default\n Documentation: describe MIDX-based bitmaps\n midx: make a number of functions non-static\n midx: clear auxiliary .rev after replacing the MIDX\n midx: respect 'core.multiPackIndex' when writing\n midx: infer preferred pack when not given one\n pack-bitmap.c: introduce 'bitmap_num_objects()'\n pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n pack-bitmap: read multi-pack bitmaps\n pack-bitmap: write multi-pack bitmaps\n t5310: move some tests to lib-bitmap.sh\n t/helper/test-read-midx.c: add --checksum mode\n t5326: test multi-pack bitmap behavior\n t5319: don't write MIDX bitmaps in t5319\n t7700: update to work with MIDX bitmap test knob\n midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n p5310: extract full and partial bitmap tests\n p5326: perf tests for MIDX bitmaps\n\n Documentation/Makefile | 1 +\n Documentation/git-multi-pack-index.txt | 12 +-\n Documentation/technical/bitmap-format.txt | 72 ++-\n Documentation/technical/multi-pack-index.txt | 10 +-\n builtin/multi-pack-index.c | 2 +\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 13 +-\n ci/run-build-and-tests.sh | 1 +\n midx.c | 288 +++++++++++-\n midx.h | 5 +\n pack-bitmap-write.c | 79 +++-\n pack-bitmap.c | 470 +++++++++++++++++--\n pack-bitmap.h | 8 +-\n packfile.c | 2 +-\n t/README | 4 +\n t/helper/test-read-midx.c | 16 +-\n t/lib-bitmap.sh | 240 ++++++++++\n t/perf/lib-bitmap.sh | 69 +++\n t/perf/p5310-pack-bitmaps.sh | 65 +--\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++\n t/t0410-partial-clone.sh | 12 +-\n t/t5310-pack-bitmaps.sh | 231 +--------\n t/t5319-multi-pack-index.sh | 3 +-\n t/t5326-multi-pack-bitmaps.sh | 277 +++++++++++\n t/t7700-repack.sh | 18 +-\n 25 files changed, 1534 insertions(+), 415 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\nRange-diff against v1:\n 1: 2d1c6ccab5 = 1: a18baeb0b4 pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n 2: d199954ef2 ! 2: 3e637d9ec8 pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n @@ Commit message\n bitmaps would be incomplete).\n \n Pack bitmaps are never written from 'git repack' unless repacking\n - all-into-one, and so we never write non-closed bitmaps.\n + all-into-one, and so we never write non-closed bitmaps (except in the\n + case of partial clones where we aren't guaranteed to have all objects).\n \n But multi-pack bitmaps change this, since it isn't known whether the\n set of objects in the MIDX is closed under reachability until walking\n @@ Commit message\n include in the bitmap, bitmap_writer_build() knows that the set is not\n closed, and so it now fails gracefully.\n \n - (The new conditional in builtin/pack-objects.c:bitmap_writer_build()\n - guards against other failure modes, but is never triggered here, because\n - of the all-into-one detail above. This return value will be important to\n - check from the multi-pack index caller.)\n + A test is added in t0410 to trigger a bitmap write without full\n + reachability closure by removing local copies of some reachable objects\n + from a promisor remote.\n \n Signed-off-by: Taylor Blau \n \n @@ pack-bitmap-write.c: void bitmap_writer_build(struct packing_data *to_pack)\n -\tcompute_xor_offsets();\n +\tif (closed)\n +\t\tcompute_xor_offsets();\n -+\treturn closed;\n ++\treturn closed ? 0 : -1;\n }\n \n /**\n @@ pack-bitmap.h: struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n +\n + ## t/t0410-partial-clone.sh ##\n +@@ t/t0410-partial-clone.sh: test_expect_success 'gc does not repack promisor objects if there are none' '\n + repack_and_check () {\n + \trm -rf repo2 &&\n + \tcp -r repo repo2 &&\n +-\tgit -C repo2 repack $1 -d &&\n ++\tif test x\"$1\" = \"x--must-fail\"\n ++\tthen\n ++\t\tshift\n ++\t\ttest_must_fail git -C repo2 repack $1 -d\n ++\telse\n ++\t\tgit -C repo2 repack $1 -d\n ++\tfi &&\n + \tgit -C repo2 fsck &&\n + \n + \tgit -C repo2 cat-file -e $2 &&\n +@@ t/t0410-partial-clone.sh: test_expect_success 'repack -d does not irreversibly delete promisor objects' '\n + \tprintf \"$THREE\\n\" | pack_as_from_promisor &&\n + \tdelete_object repo \"$ONE\" &&\n + \n ++\trepack_and_check --must-fail -ab \"$TWO\" \"$THREE\" &&\n + \trepack_and_check -a \"$TWO\" \"$THREE\" &&\n + \trepack_and_check -A \"$TWO\" \"$THREE\" &&\n + \trepack_and_check -l \"$TWO\" \"$THREE\"\n 3: 014c18b896 = 3: 490d733d12 pack-bitmap-write.c: free existing bitmaps\n 4: 46de889cd2 = 4: b0bb2e8051 Documentation: build 'technical/bitmap-format' by default\n 5: 0d4822a64e = 5: 64a260e0c6 Documentation: describe MIDX-based bitmaps\n 6: c76dfc198e = 6: b3a12424d7 midx: make a number of functions non-static\n 7: 26c3a312f9 = 7: 1448ca0d2b midx: clear auxiliary .rev after replacing the MIDX\n 8: 8643174a67 = 8: dfd1daacc5 midx: respect 'core.multiPackIndex' when writing\n -: ---------- > 9: 9495f6869d midx: infer preferred pack when not given one\n 9: af507f4b29 = 10: 373aa47528 pack-bitmap.c: introduce 'bitmap_num_objects()'\n10: a6fdf7234a = 11: ac1f46aa1f pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n11: a78f83a127 = 12: c474d2eda5 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n12: d5eeca4f11 ! 13: 7d44ba6299 pack-bitmap: read multi-pack bitmaps\n @@ builtin/pack-objects.c: static void write_reused_pack(struct hashfile *f)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n --\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n -+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n -+\t\t\t\toff_t pack_offs = bitmap_pack_offset(bitmap_git,\n -+\t\t\t\t\t\t\t\t pos + offset);\n -+\t\t\t\tuint32_t pos;\n -+\n -+\t\t\t\tif (offset_to_pack_pos(reuse_packfile, pack_offs, &pos) < 0)\n -+\t\t\t\t\tdie(_(\"write_reused_pack: could not locate %\"PRIdMAX),\n -+\t\t\t\t\t (intmax_t)pack_offs);\n -+\t\t\t\twrite_reused_pack_one(pos, f, &w_curs);\n -+\t\t\t} else\n -+\t\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n ++\t\t\t/*\n ++\t\t\t * Can use bit positions directly, even for MIDX\n ++\t\t\t * bitmaps. See comment in try_partial_reuse()\n ++\t\t\t * for why.\n ++\t\t\t */\n + \t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n - \t}\n \n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: void bitmap_writer_show_progress(int show)\n @@ pack-bitmap.c: struct bitmap_index {\n \t/* If not NULL, this is a name-hash cache pointing into map. */\n \tuint32_t *hashes;\n \n ++\t/* The checksum of the packfile or MIDX; points into map. */\n +\tconst unsigned char *checksum;\n +\n \t/*\n @@ pack-bitmap.c: static char *pack_bitmap_filename(struct packed_git *p)\n +\n +\tif (bitmap_git->pack || bitmap_git->midx) {\n +\t\t/* ignore extra bitmap file; we can only handle one */\n ++\t\twarning(\"ignoring extra bitmap file: %s\",\n ++\t\t\tget_midx_filename(midx->object_dir));\n ++\t\tclose(fd);\n +\t\treturn -1;\n +\t}\n +\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\tgoto cleanup;\n \n \tobject_array_clear(&revs->pending);\n -@@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n +@@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n + }\n + \n + static void try_partial_reuse(struct bitmap_index *bitmap_git,\n ++\t\t\t struct packed_git *pack,\n + \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n -\toff_t offset, header;\n -+\tstruct packed_git *pack;\n +\toff_t offset, delta_obj_offset;\n \tenum object_type type;\n \tunsigned long size;\n \n - \tif (pos >= bitmap_num_objects(bitmap_git))\n - \t\treturn; /* not actually in the pack or MIDX */\n +-\tif (pos >= bitmap_num_objects(bitmap_git))\n +-\t\treturn; /* not actually in the pack or MIDX */\n ++\t/*\n ++\t * try_partial_reuse() is called either on (a) objects in the\n ++\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n ++\t * objects in the preferred pack of a multi-pack bitmap.\n ++\t * Importantly, the latter can pretend as if only a single pack\n ++\t * exists because:\n ++\t *\n ++\t * - The first pack->num_objects bits of a MIDX bitmap are\n ++\t * reserved for the preferred pack, and\n ++\t *\n ++\t * - Ties due to duplicate objects are always resolved in\n ++\t * favor of the preferred pack.\n ++\t *\n ++\t * Therefore we do not need to ever ask the MIDX for its copy of\n ++\t * an object by OID, since it will always select it from the\n ++\t * preferred pack. Likewise, the selected copy of the base\n ++\t * object for any deltas will reside in the same pack.\n ++\t *\n ++\t * This means that we can reuse pos when looking up the bit in\n ++\t * the reuse bitmap, too, since bits corresponding to the\n ++\t * preferred pack precede all bits from other packs.\n ++\t */\n \n -\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n -\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n -+\tif (bitmap_is_midx(bitmap_git)) {\n -+\t\tuint32_t pack_id, midx_pos;\n ++\tif (pos >= pack->num_objects)\n ++\t\treturn; /* not actually in the pack or MIDX preferred pack */\n +\n -+\t\tmidx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n -+\t\tpack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n -+\n -+\t\tpack = bitmap_git->midx->packs[pack_id];\n -+\t\toffset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n -+\t} else {\n -+\t\tpack = bitmap_git->pack;\n -+\t\toffset = pack_pos_to_offset(bitmap_git->pack, pos);\n -+\t}\n -+\n -+\tdelta_obj_offset = offset;\n ++\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n +\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn; /* broken packfile, punt */\n @@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t\treturn;\n \n \t\t/*\n -@@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n - \t\t * packs we write fresh, and OFS_DELTA is the default). But\n - \t\t * let's double check to make sure the pack wasn't written with\n - \t\t * odd parameters.\n -+\t\t *\n -+\t\t * Note that the base does not need to be repositioned, i.e.,\n -+\t\t * the MIDX is guaranteed to have selected the copy of \"base\"\n -+\t\t * from the same pack, since this function is only ever called\n -+\t\t * on the preferred pack (and all duplicate objects are resolved\n -+\t\t * in favor of the preferred pack).\n -+\t\t *\n -+\t\t * This means that we can reuse base_pos when looking up the bit\n -+\t\t * in the reuse bitmap, too, since bits corresponding to the\n -+\t\t * preferred pack precede all bits from other packs.\n - \t\t */\n - \t\tif (base_pos >= pos)\n - \t\t\treturn;\n @@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tbitmap_set(reuse, pos);\n }\n @@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t struct packed_git **packfile_out,\n \t\t\t\t uint32_t *entries,\n -@@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n + \t\t\t\t struct bitmap **reuse_out)\n + {\n ++\tstruct packed_git *pack;\n + \tstruct bitmap *result = bitmap_git->result;\n + \tstruct bitmap *reuse;\n + \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n - \tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n -+\tuint32_t preferred_pack = 0;\n +-\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n ++\tuint32_t objects_nr;\n \n \tassert(result);\n \n +\tload_reverse_index(bitmap_git);\n +\n -+\tif (bitmap_is_midx(bitmap_git)) {\n -+\t\tpreferred_pack = midx_preferred_pack(bitmap_git);\n -+\t\tobjects_nr = bitmap_git->midx->packs[preferred_pack]->num_objects;\n -+\t} else\n -+\t\tobjects_nr = bitmap_git->pack->num_objects;\n ++\tif (bitmap_is_midx(bitmap_git))\n ++\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n ++\telse\n ++\t\tpack = bitmap_git->pack;\n ++\tobjects_nr = pack->num_objects;\n +\n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitma\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n +-\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n +\t\t\t\t/*\n +\t\t\t\t * Can't reuse from a non-preferred pack (see\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitma\n +\t\t\t\tif (pos + offset >= objects_nr)\n +\t\t\t\t\tcontinue;\n +\t\t\t}\n - \t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n ++\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n \t\t}\n \t}\n + \n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n -\t*packfile_out = bitmap_git->pack;\n -+\t*packfile_out = bitmap_git->pack ?\n -+\t\tbitmap_git->pack :\n -+\t\tbitmap_git->midx->packs[preferred_pack];\n ++\t*packfile_out = pack;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_\n \tstruct ewah_iterator it;\n \teword_t filter;\n @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n + \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n - \t\t\tpos = base + offset;\n +-\t\t\tpos = base + offset;\n +\n +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n +\t\t\t\tuint32_t pack_pos;\n -+\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n ++\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n +\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n +\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n +\n @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_\n +\t\t\t\t}\n +\n +\t\t\t\tpos = pack_pos;\n -+\t\t\t} else\n ++\t\t\t} else {\n +\t\t\t\tpack = bitmap_git->pack;\n ++\t\t\t\tpos = base + offset;\n ++\t\t\t}\n +\n \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n \t\t\t\t pack_pos_to_offset(pack, pos);\n13: fd320c5ed4 ! 14: a8cec2463d pack-bitmap: write multi-pack bitmaps\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +\trepo_init_revisions(the_repository, &revs, NULL);\n +\tfor_each_ref(add_ref_to_pending, &revs);\n +\n ++\t/*\n ++\t * Skipping promisor objects here is intentional, since it only excludes\n ++\t * them from the list of reachable commits that we want to select from\n ++\t * when computing the selection of MIDX'd commits to receive bitmaps.\n ++\t *\n ++\t * Reachability bitmaps do require that their objects be closed under\n ++\t * reachability, but fetching any objects missing from promisors at this\n ++\t * point is too late. But, if one of those objects can be reached from\n ++\t * an another object that is included in the bitmap, then we will\n ++\t * complain later that we don't have reachability closure (and fail\n ++\t * appropriately).\n ++\t */\n +\tfetch_if_missing = 0;\n +\trevs.exclude_promisor_objects = 1;\n +\n ++\t/*\n ++\t * Pass selected commits in topo order to match the behavior of\n ++\t * pack-bitmaps when configured with delta islands.\n ++\t */\n ++\trevs.topo_order = 1;\n ++\trevs.sort_order = REV_SORT_IN_GRAPH_ORDER;\n ++\n +\tif (prepare_revision_walk(&revs))\n +\t\tdie(_(\"revision walk setup failed\"));\n +\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n +\n +\t/*\n -+\t * bitmap_writer_select_commits expects objects in lex order, but\n -+\t * pack_order gives us exactly that. use it directly instead of\n -+\t * re-sorting the array\n ++\t * bitmap_writer_finish expects objects in lex order, but pack_order\n ++\t * gives us exactly that. use it directly instead of re-sorting the\n ++\t * array.\n ++\t *\n ++\t * This changes the order of objects in 'index' between\n ++\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n ++\t *\n ++\t * The same re-ordering takes place in the single-pack bitmap code via\n ++\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n ++\t * happens between bitmap_writer_build_type_index() and\n ++\t * bitmap_writer_finish().\n +\t */\n +\tfor (i = 0; i < pdata.nr_objects; i++)\n +\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n +\n +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n +\tret = bitmap_writer_build(&pdata);\n -+\tif (!ret)\n ++\tif (ret < 0)\n +\t\tgoto cleanup;\n +\n +\tbitmap_writer_set_checksum(midx_hash);\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\t}\n +\t}\n \n - \tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n + \t\tint found = 0;\n +@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n + \t\tif (!found)\n + \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n + \t\t\t\tpreferred_pack_name);\n +-\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n ++\t} else if (ctx.nr &&\n ++\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n + \t\ttime_t oldest = ctx.info[0].p->mtime;\n + \t\tctx.preferred_pack_idx = 0;\n + \n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n -+\tif (flags & MIDX_WRITE_BITMAP)\n -+\t\twrite_midx_bitmap(midx_name, midx_hash, &ctx, flags);\n ++\tif (flags & MIDX_WRITE_BITMAP) {\n ++\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n ++\t\t\terror(_(\"could not write multi-pack bitmap\"));\n ++\t\t\tresult = 1;\n ++\t\t\tgoto cleanup;\n ++\t\t}\n ++\t}\n \n \tcommit_lock_file(&lk);\n \n14: 570e3de9ed ! 15: c63eb637c8 t5310: move some tests to lib-bitmap.sh\n @@ t/lib-bitmap.sh: test_bitmap_traversal () {\n +\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n +\t\ttest_cmp expect actual\n +\t'\n ++\n ++\ttest_expect_success 'enumerating progress counts pack-reused objects' '\n ++\t\tcount=$(git rev-list --objects --all --count) &&\n ++\t\tgit repack -adb &&\n ++\n ++\t\t# check first with only reused objects; confirm that our\n ++\t\t# progress showed the right number, and also that we did\n ++\t\t# pack-reuse as expected. Check only the final \"done\"\n ++\t\t# line of the meter (there may be an arbitrary number of\n ++\t\t# intermediate lines ending with CR).\n ++\t\tGIT_PROGRESS_DELAY=0 \\\n ++\t\t\tgit pack-objects --all --stdout --progress \\\n ++\t\t\t/dev/null 2>stderr &&\n ++\t\tgrep \"Enumerating objects: $count, done\" stderr &&\n ++\t\tgrep \"pack-reused $count\" stderr &&\n ++\n ++\t\t# now the same but with one non-reused object\n ++\t\tgit commit --allow-empty -m \"an extra commit object\" &&\n ++\t\tGIT_PROGRESS_DELAY=0 \\\n ++\t\t\tgit pack-objects --all --stdout --progress \\\n ++\t\t\t/dev/null 2>stderr &&\n ++\t\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n ++\t\tgrep \"pack-reused $count\" stderr\n ++\t'\n +}\n +\n +# have_delta \n @@ t/t5310-pack-bitmaps.sh: test_expect_success 'truncated bitmap fails gracefully\n \ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n +-test_expect_success 'enumerating progress counts pack-reused objects' '\n +-\tcount=$(git rev-list --objects --all --count) &&\n +-\tgit repack -adb &&\n +-\n +-\t# check first with only reused objects; confirm that our progress\n +-\t# showed the right number, and also that we did pack-reuse as expected.\n +-\t# Check only the final \"done\" line of the meter (there may be an\n +-\t# arbitrary number of intermediate lines ending with CR).\n +-\tGIT_PROGRESS_DELAY=0 \\\n +-\t\tgit pack-objects --all --stdout --progress \\\n +-\t\t/dev/null 2>stderr &&\n +-\tgrep \"Enumerating objects: $count, done\" stderr &&\n +-\tgrep \"pack-reused $count\" stderr &&\n +-\n +-\t# now the same but with one non-reused object\n +-\tgit commit --allow-empty -m \"an extra commit object\" &&\n +-\tGIT_PROGRESS_DELAY=0 \\\n +-\t\tgit pack-objects --all --stdout --progress \\\n +-\t\t/dev/null 2>stderr &&\n +-\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n +-\tgrep \"pack-reused $count\" stderr\n +-'\n +-\n -# have_delta \n -#\n -# Note that because this relies on cat-file, it might find _any_ copy of an\n15: 060ee427be = 16: bedb7afb37 t/helper/test-read-midx.c: add --checksum mode\n16: ff74181e85 ! 17: fbfac4ae8e t5326: test multi-pack bitmap behavior\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\tgit multi-pack-index write --bitmap &&\n +\n +\tls $objdir/pack/pack-*.pack >packs &&\n -+\ttest_line_count = 26 packs &&\n ++\ttest_line_count = 25 packs &&\n +\n +\ttest_path_is_file $midx &&\n +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\t\t$(git rev-parse packed)\n +\t\tEOF\n +\n -+\t\tgit multi-pack-index write --bitmap 2>err &&\n ++\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n +\t\tgrep \"doesn.t have full closure\" err &&\n -+\t\ttest_path_is_file $midx &&\n -+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n ++\t\ttest_path_is_missing $midx\n +\t)\n +'\n +\n -: ---------- > 18: 2a5df1832a t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n17: 8f328bb5bc = 19: 2d24c5b7ad t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n18: dbd953815b = 20: 4cbfaa0e97 t5319: don't write MIDX bitmaps in t5319\n19: ee952e4300 = 21: 839a7a79eb t7700: update to work with MIDX bitmap test knob\n20: 83614f9284 = 22: 00418d5b09 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n21: 2f3836bf2e = 23: 98fa73a76a p5310: extract full and partial bitmap tests\n22: b75b534446 = 24: ec0f53b424 p5326: perf tests for MIDX bitmaps\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"428119","messageId":"a18baeb0b42994ebcb216df5fe69459ba9a33795.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 01/24] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:24:59Z","receivedAt":"2021-06-21T22:25:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The special `--test-bitmap` mode of `git rev-list` is used to compare\nthe result of an object traversal with a bitmap to check its integrity.\nThis mode does not, however, assert that the types of reachable objects\nare stored correctly.\n\nHarden this mode by teaching it to also check that each time an object's\nbit is marked, the corresponding bit should be set in exactly one of the\ntype bitmaps (whose type matches the object's true type).\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d90e1d9d8c..368fa59a42 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1301,10 +1301,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,\n struct bitmap_test_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\n+\tstruct bitmap *commits;\n+\tstruct bitmap *trees;\n+\tstruct bitmap *blobs;\n+\tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n };\n \n+static void test_bitmap_type(struct bitmap_test_data *tdata,\n+\t\t\t struct object *obj, int pos)\n+{\n+\tenum object_type bitmap_type = OBJ_NONE;\n+\tint bitmaps_nr = 0;\n+\n+\tif (bitmap_get(tdata->commits, pos)) {\n+\t\tbitmap_type = OBJ_COMMIT;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->trees, pos)) {\n+\t\tbitmap_type = OBJ_TREE;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->blobs, pos)) {\n+\t\tbitmap_type = OBJ_BLOB;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->tags, pos)) {\n+\t\tbitmap_type = OBJ_TAG;\n+\t\tbitmaps_nr++;\n+\t}\n+\n+\tif (!bitmap_type)\n+\t\tdie(\"object %s not found in type bitmaps\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmaps_nr > 1)\n+\t\tdie(\"object %s does not have a unique type\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmap_type != obj->type)\n+\t\tdie(\"object %s: real type %s, expected: %s\",\n+\t\t oid_to_hex(&obj->oid),\n+\t\t type_name(obj->type),\n+\t\t type_name(bitmap_type));\n+}\n+\n static void test_show_object(struct object *object, const char *name,\n \t\t\t void *data)\n {\n@@ -1314,6 +1356,7 @@ static void test_show_object(struct object *object, const char *name,\n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&object->oid));\n+\ttest_bitmap_type(tdata, object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1328,6 +1371,7 @@ static void test_show_commit(struct commit *commit, void *data)\n \t\t\t\t &commit->object.oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&commit->object.oid));\n+\ttest_bitmap_type(tdata, &commit->object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1375,6 +1419,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \ttdata.bitmap_git = bitmap_git;\n \ttdata.base = bitmap_new();\n+\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n \ttdata.seen = 0;\n \n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428120","messageId":"3e637d9ec83435540ad32b8325b0dce87f61bae0.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:01Z","receivedAt":"2021-06-21T22:25:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The set of objects covered by a bitmap must be closed under\nreachability, since it must be the case that there is a valid bit\nposition assigned for every possible reachable object (otherwise the\nbitmaps would be incomplete).\n\nPack bitmaps are never written from 'git repack' unless repacking\nall-into-one, and so we never write non-closed bitmaps (except in the\ncase of partial clones where we aren't guaranteed to have all objects).\n\nBut multi-pack bitmaps change this, since it isn't known whether the\nset of objects in the MIDX is closed under reachability until walking\nthem. Plumb through a bit that is set when a reachable object isn't\nfound.\n\nAs soon as a reachable object isn't found in the set of objects to\ninclude in the bitmap, bitmap_writer_build() knows that the set is not\nclosed, and so it now fails gracefully.\n\nA test is added in t0410 to trigger a bitmap write without full\nreachability closure by removing local copies of some reachable objects\nfrom a promisor remote.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 +-\n pack-bitmap-write.c | 76 ++++++++++++++++++++++++++++------------\n pack-bitmap.h | 2 +-\n t/t0410-partial-clone.sh | 9 ++++-\n 4 files changed, 64 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex de00adbb9e..8a523624a1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1256,7 +1256,8 @@ static void write_pack_file(void)\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n-\t\t\t\tbitmap_writer_build(&to_pack);\n+\t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n+\t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..d374f7884b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit)\n \twriter.selected_nr++;\n }\n \n-static uint32_t find_object_pos(const struct object_id *oid)\n+static uint32_t find_object_pos(const struct object_id *oid, int *found)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n \n \tif (!entry) {\n-\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\tif (found)\n+\t\t\t*found = 0;\n+\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\t\treturn 0;\n \t}\n \n+\tif (found)\n+\t\t*found = 1;\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n@@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n \n-static void fill_bitmap_tree(struct bitmap *bitmap,\n-\t\t\t struct tree *tree)\n+static int fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n {\n+\tint found;\n \tuint32_t pos;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t * If our bit is already set, then there is nothing to do. Both this\n \t * tree and all of its children will be set.\n \t */\n-\tpos = find_object_pos(&tree->object.oid);\n+\tpos = find_object_pos(&tree->object.oid, &found);\n+\tif (!found)\n+\t\treturn -1;\n \tif (bitmap_get(bitmap, pos))\n-\t\treturn;\n+\t\treturn 0;\n \tbitmap_set(bitmap, pos);\n \n \tif (parse_tree(tree) < 0)\n@@ -355,11 +363,15 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \twhile (tree_entry(&desc, &entry)) {\n \t\tswitch (object_type(entry.mode)) {\n \t\tcase OBJ_TREE:\n-\t\t\tfill_bitmap_tree(bitmap,\n-\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tif (fill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid)) < 0)\n+\t\t\t\treturn -1;\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n-\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tpos = find_object_pos(&entry.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(bitmap, pos);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\t/* Gitlink, etc; not reachable */\n@@ -368,15 +380,18 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t}\n \n \tfree_tree_buffer(tree);\n+\treturn 0;\n }\n \n-static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue,\n-\t\t\t struct prio_queue *tree_queue,\n-\t\t\t struct bitmap_index *old_bitmap,\n-\t\t\t const uint32_t *mapping)\n+static int fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n+\tint found;\n+\tuint32_t pos;\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n@@ -401,11 +416,16 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\tif (!found)\n+\t\t\treturn -1;\n+\t\tbitmap_set(ent->bitmap, pos);\n \t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n-\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n \t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n \t\t\t\tbitmap_set(ent->bitmap, pos);\n \t\t\t\tprio_queue_put(queue, p->item);\n@@ -413,8 +433,12 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t}\n \t}\n \n-\twhile (tree_queue->nr)\n-\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n+\twhile (tree_queue->nr) {\n+\t\tif (fill_bitmap_tree(ent->bitmap,\n+\t\t\t\t prio_queue_get(tree_queue)) < 0)\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -432,7 +456,7 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+int bitmap_writer_build(struct packing_data *to_pack)\n {\n \tstruct bitmap_builder bb;\n \tsize_t i;\n@@ -441,6 +465,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n \tuint32_t *mapping;\n+\tint closed = 1; /* until proven otherwise */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n-\t\t\t\t old_bitmap, mapping);\n+\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping) < 0) {\n+\t\t\tclosed = 0;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -499,7 +527,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \tstop_progress(&writer.progress);\n \n-\tcompute_xor_offsets();\n+\tif (closed)\n+\t\tcompute_xor_offsets();\n+\treturn closed ? 0 : -1;\n }\n \n /**\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 99d733eb26..020cd8d868 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -87,7 +87,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n-void bitmap_writer_build(struct packing_data *to_pack);\n+int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 584a039b85..1667450917 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '\n repack_and_check () {\n \trm -rf repo2 &&\n \tcp -r repo repo2 &&\n-\tgit -C repo2 repack $1 -d &&\n+\tif test x\"$1\" = \"x--must-fail\"\n+\tthen\n+\t\tshift\n+\t\ttest_must_fail git -C repo2 repack $1 -d\n+\telse\n+\t\tgit -C repo2 repack $1 -d\n+\tfi &&\n \tgit -C repo2 fsck &&\n \n \tgit -C repo2 cat-file -e $2 &&\n@@ -561,6 +567,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' '\n \tprintf \"$THREE\\n\" | pack_as_from_promisor &&\n \tdelete_object repo \"$ONE\" &&\n \n+\trepack_and_check --must-fail -ab \"$TWO\" \"$THREE\" &&\n \trepack_and_check -a \"$TWO\" \"$THREE\" &&\n \trepack_and_check -A \"$TWO\" \"$THREE\" &&\n \trepack_and_check -l \"$TWO\" \"$THREE\"\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428121","messageId":"490d733d121e206ccdc335812c03f31c380bcd86.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 03/24] pack-bitmap-write.c: free existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:04Z","receivedAt":"2021-06-21T22:25:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new bitmap, the bitmap writer code attempts to read the\nexisting bitmap (if one is present). This is done in order to quickly\npermute the bits of any bitmaps for commits which appear in the existing\nbitmap, and were also selected for the new bitmap.\n\nBut since this code was added in 341fa34887 (pack-bitmap-write: use\nexisting bitmaps, 2020-12-08), the resources associated with opening an\nexisting bitmap were never released.\n\nIt's fine to ignore this, but it's bad hygiene. It will also cause a\nproblem for the multi-pack-index builtin, which will be responsible not\nonly for writing bitmaps, but also for expiring any old multi-pack\nbitmaps.\n\nIf an existing bitmap was reused here, it will also be expired. That\nwill cause a problem on platforms which require file resources to be\nclosed before unlinking them, like Windows. Avoid this by ensuring we\nclose reused bitmaps with free_bitmap_index() before removing them.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d374f7884b..142fd0adb8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tclear_prio_queue(&queue);\n \tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree_bitmap_index(old_bitmap);\n \tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428122","messageId":"b0bb2e8051f19ec47140fda6500e092e37c6bea8.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:07Z","receivedAt":"2021-06-21T22:25:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Even though the 'TECH_DOCS' variable was introduced all the way back in\n5e00439f0a (Documentation: build html for all files in technical and\nhowto, 2012-10-23), the 'bitmap-format' document was never added to that\nlist when it was created.\n\nPrepare for changes to this file by including it in the list of\ntechnical documentation that 'make doc' will build by default.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/Makefile | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex f5605b7767..7d7b778b28 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -90,6 +90,7 @@ SP_ARTICLES += $(API_DOCS)\n TECH_DOCS += MyFirstContribution\n TECH_DOCS += MyFirstObjectWalk\n TECH_DOCS += SubmittingPatches\n+TECH_DOCS += technical/bitmap-format\n TECH_DOCS += technical/hash-function-transition\n TECH_DOCS += technical/http-protocol\n TECH_DOCS += technical/index-format\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428123","messageId":"64a260e0c6a116b7c6fa6fea2b9fd96bf416cb18.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 05/24] Documentation: describe MIDX-based bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:10Z","receivedAt":"2021-06-21T22:25:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Update the technical documentation to describe the multi-pack bitmap\nformat. This patch merely introduces the new format, and describes its\nhigh-level ideas. Git does not yet know how to read nor write these\nmulti-pack variants, and so the subsequent patches will:\n\n - Introduce code to interpret multi-pack bitmaps, according to this\n document.\n\n - Then, introduce code to write multi-pack bitmaps from the 'git\n multi-pack-index write' sub-command.\n\nFinally, the implementation will gain tests in subsequent patches (as\nopposed to inline with the patch teaching Git how to write multi-pack\nbitmaps) to avoid a cyclic dependency.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 72 ++++++++++++++++----\n Documentation/technical/multi-pack-index.txt | 10 +--\n 2 files changed, 61 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f8c18a0f7a..25221c7ec8 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -1,6 +1,45 @@\n GIT bitmap v1 format\n ====================\n \n+== Pack and multi-pack bitmaps\n+\n+Bitmaps store reachability information about the set of objects in a packfile,\n+or a multi-pack index (MIDX). The former is defined obviously, and the latter is\n+defined as the union of objects in packs contained in the MIDX.\n+\n+A bitmap may belong to either one pack, or the repository's multi-pack index (if\n+it exists). A repository may have at most one bitmap.\n+\n+An object is uniquely described by its bit position within a bitmap:\n+\n+\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n+\tthe __n__th object in pack order. For a function `offset` which maps\n+\tobjects to their byte offset within a pack, pack order is defined as\n+\tfollows:\n+\n+\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n+\n+\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n+\t__n__th object in MIDX order. With an additional function `pack` which\n+\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n+\tis defined as follows:\n+\n+\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n+\n+\tThe ordering between packs is done lexicographically by the pack name,\n+\twith the exception of the preferred pack, which sorts ahead of all other\n+\tpacks.\n+\n+The on-disk representation (described below) of a bitmap is the same regardless\n+of whether or not that bitmap belongs to a packfile or a MIDX. The only\n+difference is the interpretation of the bits, which is described above.\n+\n+Certain bitmap extensions are supported (see: Appendix B). No extensions are\n+required for bitmaps corresponding to packfiles. For bitmaps that correspond to\n+MIDXs, both the bit-cache and rev-cache extensions are required.\n+\n+== On-disk format\n+\n \t- A header appears at the beginning:\n \n \t\t4-byte signature: {'B', 'I', 'T', 'M'}\n@@ -14,17 +53,19 @@ GIT bitmap v1 format\n \t\t\tThe following flags are supported:\n \n \t\t\t- BITMAP_OPT_FULL_DAG (0x1) REQUIRED\n-\t\t\tThis flag must always be present. It implies that the bitmap\n-\t\t\tindex has been generated for a packfile with full closure\n-\t\t\t(i.e. where every single object in the packfile can find\n-\t\t\t its parent links inside the same packfile). This is a\n-\t\t\trequirement for the bitmap index format, also present in JGit,\n-\t\t\tthat greatly reduces the complexity of the implementation.\n+\t\t\tThis flag must always be present. It implies that the\n+\t\t\tbitmap index has been generated for a packfile or\n+\t\t\tmulti-pack index (MIDX) with full closure (i.e. where\n+\t\t\tevery single object in the packfile/MIDX can find its\n+\t\t\tparent links inside the same packfile/MIDX). This is a\n+\t\t\trequirement for the bitmap index format, also present in\n+\t\t\tJGit, that greatly reduces the complexity of the\n+\t\t\timplementation.\n \n \t\t\t- BITMAP_OPT_HASH_CACHE (0x4)\n \t\t\tIf present, the end of the bitmap file contains\n \t\t\t`N` 32-bit name-hash values, one per object in the\n-\t\t\tpack. The format and meaning of the name-hash is\n+\t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n \t\t4-byte entry count (network byte order)\n@@ -33,7 +74,8 @@ GIT bitmap v1 format\n \n \t\t20-byte checksum\n \n-\t\t\tThe SHA1 checksum of the pack this bitmap index belongs to.\n+\t\t\tThe SHA1 checksum of the pack/MIDX this bitmap index\n+\t\t\tbelongs to.\n \n \t- 4 EWAH bitmaps that act as type indexes\n \n@@ -50,7 +92,7 @@ GIT bitmap v1 format\n \t\t\t- Tags\n \n \t\tIn each bitmap, the `n`th bit is set to true if the `n`th object\n-\t\tin the packfile is of that type.\n+\t\tin the packfile or multi-pack index is of that type.\n \n \t\tThe obvious consequence is that the OR of all 4 bitmaps will result\n \t\tin a full set (all bits set), and the AND of all 4 bitmaps will\n@@ -62,8 +104,9 @@ GIT bitmap v1 format\n \t\tEach entry contains the following:\n \n \t\t- 4-byte object position (network byte order)\n-\t\t\tThe position **in the index for the packfile** where the\n-\t\t\tbitmap for this commit is found.\n+\t\t\tThe position **in the index for the packfile or\n+\t\t\tmulti-pack index** where the bitmap for this commit is\n+\t\t\tfound.\n \n \t\t- 1-byte XOR-offset\n \t\t\tThe xor offset used to compress this bitmap. For an entry\n@@ -146,10 +189,11 @@ Name-hash cache\n ---------------\n \n If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains\n-a cache of 32-bit values, one per object in the pack. The value at\n+a cache of 32-bit values, one per object in the pack/MIDX. The value at\n position `i` is the hash of the pathname at which the `i`th object\n-(counting in index order) in the pack can be found. This can be fed\n-into the delta heuristics to compare objects with similar pathnames.\n+(counting in index or multi-pack index order) in the pack/MIDX can be found.\n+This can be fed into the delta heuristics to compare objects with similar\n+pathnames.\n \n The hash algorithm used is:\n \ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex fb688976c4..1a73c3ee20 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -71,14 +71,10 @@ Future Work\n still reducing the number of binary searches required for object\n lookups.\n \n-- The reachability bitmap is currently paired directly with a single\n- packfile, using the pack-order as the object order to hopefully\n- compress the bitmaps well using run-length encoding. This could be\n- extended to pair a reachability bitmap with a multi-pack-index. If\n- the multi-pack-index is extended to store a \"stable object order\"\n+- If the multi-pack-index is extended to store a \"stable object order\"\n (a function Order(hash) = integer that is constant for a given hash,\n- even as the multi-pack-index is updated) then a reachability bitmap\n- could point to a multi-pack-index and be updated independently.\n+ even as the multi-pack-index is updated) then MIDX bitmaps could be\n+ updated independently of the MIDX.\n \n - Packfiles can be marked as \"special\" using empty files that share\n the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428124","messageId":"b3a12424d78e80553741f5c7a0672490a59b6f7d.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 06/24] midx: make a number of functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:12Z","receivedAt":"2021-06-21T22:25:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These functions will be called from outside of midx.c in a subsequent\npatch.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 ++--\n midx.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 21d6a05e88..fa23d57a24 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -48,12 +48,12 @@ static uint8_t oid_version(void)\n \t}\n }\n \n-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - the_hash_algo->rawsz;\n }\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex 8684cf0fef..1172df1a71 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -42,6 +42,8 @@ struct multi_pack_index {\n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n \n+const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+char *get_midx_filename(const char *object_dir);\n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428125","messageId":"1448ca0d2ba265db2dce414a7f7d6b1f4bcb5a08.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 07/24] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:15Z","receivedAt":"2021-06-21T22:25:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nclean up any auxiliary files (currently just the MIDX's `.rev` file, but\nsoon to include a `.bitmap`, too) corresponding to the MIDX it's\nreplacing.\n\nThis step should happen after the new MIDX is written into place, since\ndoing so beforehand means that the old MIDX could be read without its\ncorresponding .rev file.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex fa23d57a24..40eb7974ba 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1076,10 +1076,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n-\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428126","messageId":"dfd1daacc5b12d470bb6deec3448cf7dbde2bf0f.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:18Z","receivedAt":"2021-06-21T22:25:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nload any existing one to fill in some pieces of information. But it uses\nload_multi_pack_index(), which ignores the configuration\n\"core.multiPackIndex\", which indicates whether or not Git is allowed to\nread an existing multi-pack-index.\n\nReplace this with a routine that does respect that setting, to avoid\nreading multi-pack-index files when told not to.\n\nThis avoids a problem that would arise in subsequent patches due to the\ncombination of 'git repack' reopening the object store in-process and\nthe multi-pack index code not checking whether a pack already exists in\nthe object store when calling add_pack_to_midx().\n\nThis would ultimately lead to a cycle being created along the\n'packed_git' struct's '->next' pointer. That is obviously bad, but it\nhas hard-to-debug downstream effects like saying a bitmap can't be\nloaded for a pack because one already exists (for the same pack).\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 40eb7974ba..759007d5a8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -908,8 +908,18 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (m)\n \t\tctx.m = m;\n-\telse\n-\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\telse {\n+\t\tstruct multi_pack_index *cur;\n+\n+\t\tprepare_multi_pack_index_one(the_repository, object_dir, 1);\n+\n+\t\tctx.m = NULL;\n+\t\tfor (cur = the_repository->objects->multi_pack_index; cur;\n+\t\t cur = cur->next) {\n+\t\t\tif (!strcmp(object_dir, cur->object_dir))\n+\t\t\t\tctx.m = cur;\n+\t\t}\n+\t}\n \n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428127","messageId":"9495f6869d792264c4366c9914fcf93d544caa6a.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 09/24] midx: infer preferred pack when not given one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:21Z","receivedAt":"2021-06-21T22:25:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the\nmulti-pack index code learned how to select a pack which all duplicate\nobjects are selected from. That is, if an object appears in multiple\npacks, select the copy in the preferred pack before breaking ties\naccording to the other rules like pack mtime and readdir() order.\n\nNot specifying a preferred pack can cause serious problems with\nmulti-pack reachability bitmaps, because these bitmaps rely on having at\nleast one pack from which all duplicates are selected. Not having such a\npack causes problems with the pack reuse code (e.g., like assuming that\na base object was sent from that pack via reuse when in fact the base\nwas selected from a different pack).\n\nSo why does not marking a pack preferred cause problems here? The reason\nis roughly as follows:\n\n - Ties are broken (when handling duplicate objects) by sorting\n according to midx_oid_compare(), which sorts objects by OID,\n preferred-ness, pack mtime, and finally pack ID (more on that\n later).\n\n - The psuedo pack-order (described in\n Documentation/technical/bitmap-format.txt) is computed by\n midx_pack_order(), and sorts by pack ID and pack offset, with\n preferred packs sorting first.\n\n - But! Pack IDs come from incrementing the pack count in\n add_pack_to_midx(), which is a callback to\n for_each_file_in_pack_dir(), meaning that pack IDs are assigned in\n readdir() order.\n\nWhen specifying a preferred pack, all of that works fine, because\nduplicate objects are correctly resolved in favor of the copy in the\npreferred pack, and the preferred pack sorts first in the object order.\n\n\"Sorting first\" is critical, because the bitmap code relies on finding\nout which pack holds the first object in the MIDX's pseudo pack-order to\ndetermine which pack is preferred.\n\nBut if we didn't specify a preferred pack, and the pack which comes\nfirst in readdir() order does not also have the lowest timestamp, then\nit's possible that that pack (the one that sorts first in pseudo-pack\norder, which the bitmap code will treat as the preferred one) did *not*\nhave all duplicate objects resolved in its favor, resulting in breakage.\n\nThe fix is simple: pick a (semi-arbitrary) preferred pack when none was\nspecified. This forces that pack to have duplicates resolved in its\nfavor, and (critically) to sort first in pseudo-pack order.\nUnfortunately, testing this behavior portably isn't possible, since it\ndepends on readdir() order which isn't guaranteed by POSIX.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 39 +++++++++++++++++++++++++++++++++------\n 1 file changed, 33 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 759007d5a8..752d36c57f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -950,15 +950,46 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n+\t\tint found = 0;\n \t\tfor (i = 0; i < ctx.nr; i++) {\n \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tfound = 1;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n+\n+\t\tif (!found)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t\ttime_t oldest = ctx.info[0].p->mtime;\n+\t\tctx.preferred_pack_idx = 0;\n+\n+\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n+\n+\t\t/*\n+\t\t * set a preferred pack when writing a bitmap to ensure that\n+\t\t * the pack from which the first object is selected in pseudo\n+\t\t * pack-order has all of its objects selected from that pack\n+\t\t * (and not another pack containing a duplicate)\n+\t\t */\n+\t\tfor (i = 1; i < ctx.nr; i++) {\n+\t\t\ttime_t mtime = ctx.info[i].p->mtime;\n+\t\t\tif (mtime < oldest) {\n+\t\t\t\toldest = mtime;\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t}\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * otherwise don't mark any pack as preferred to avoid\n+\t\t * interfering with expiration logic below\n+\t\t */\n+\t\tctx.preferred_pack_idx = -1;\n \t}\n \n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n@@ -1029,11 +1060,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n-\n-\t\tif (!preferred)\n-\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n-\t\telse {\n+\t\tif (preferred) {\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428128","messageId":"373aa47528ce8bd4bb044a82a7e80312476f1b5f.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 10/24] pack-bitmap.c: introduce 'bitmap_num_objects()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:23Z","receivedAt":"2021-06-21T22:25:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to return how many objects are\ncontained in a bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 37 +++++++++++++++++++++----------------\n 1 file changed, 21 insertions(+), 16 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 368fa59a42..2dc135d34a 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn b;\n }\n \n+static uint32_t bitmap_num_objects(struct bitmap_index *index)\n+{\n+\treturn index->pack->num_objects;\n+}\n+\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n-\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tsize_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));\n \t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n@@ -399,7 +404,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -451,7 +456,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -650,7 +655,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\tif (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -808,7 +813,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == type &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos))\n@@ -835,7 +840,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < pack->num_objects) {\n+\tif (pos < bitmap_num_objects(bitmap_git)) {\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n@@ -845,7 +850,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - pack->num_objects];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -887,7 +892,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t}\n \n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == OBJ_BLOB &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos) &&\n@@ -1113,8 +1118,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_git->pack->num_objects)\n-\t\treturn; /* not actually in the pack */\n+\tif (pos >= bitmap_num_objects(bitmap_git))\n+\t\treturn; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n@@ -1180,6 +1185,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n+\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n \n \tassert(result);\n \n@@ -1187,8 +1193,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\ti++;\n \n \t/* Don't mark objects not in the packfile */\n-\tif (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)\n-\t\ti = bitmap_git->pack->num_objects / BITS_IN_EWORD;\n+\tif (i > objects_nr / BITS_IN_EWORD)\n+\t\ti = objects_nr / BITS_IN_EWORD;\n \n \treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n@@ -1272,7 +1278,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n-\t\t\tbitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\t\tbitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcount++;\n \t}\n \n@@ -1493,7 +1499,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n-\tnum_objects = bitmap_git->pack->num_objects;\n+\tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -1576,7 +1582,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \toff_t total = 0;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n@@ -1588,7 +1593,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \tfor (i = 0; i < eindex->count; i++) {\n \t\tstruct object *obj = eindex->objects[i];\n \n-\t\tif (!bitmap_get(result, pack->num_objects + i))\n+\t\tif (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428129","messageId":"ac1f46aa1f0dbc7fba45229555b11b390fe104a0.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 11/24] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:26Z","receivedAt":"2021-06-21T22:25:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to fetch the nth OID contained in\nthe bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2dc135d34a..9757cd0fbb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)\n \n #define MAX_XOR_OFFSET 160\n \n+static void nth_bitmap_object_oid(struct bitmap_index *index,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t uint32_t n)\n+{\n+\tnth_packed_object_id(oid, index->pack, n);\n+}\n+\n static int load_bitmap_entries_v1(struct bitmap_index *index)\n {\n \tuint32_t i;\n@@ -242,9 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n-\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n-\t\t\t\t (unsigned)commit_idx_pos);\n+\t\tnth_bitmap_object_oid(index, &oid, commit_idx_pos);\n \n \t\tbitmap = read_bitmap_1(index);\n \t\tif (!bitmap)\n@@ -844,8 +849,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack,\n-\t\t\t\t\t pack_pos_to_index(pack, pos));\n+\t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n+\t\t\t\t\t pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428130","messageId":"c474d2eda5b0b4393616109915626134429085d5.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 12/24] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:29Z","receivedAt":"2021-06-21T22:25:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a recent commit, pack-objects learned support for the\n'pack.preferBitmapTips' configuration. This patch prepares the\nmulti-pack bitmap code to respect this configuration, too.\n\nSince the multi-pack bitmap code already does a traversal of all\nreferences (in order to discover the set of reachable commits in the\nmulti-pack index), it is more efficient to check whether or not each\nreference is a suffix of any value of 'pack.preferBitmapTips' rather\nthan do an additional traversal.\n\nImplement a function 'bitmap_is_preferred_refname()' which does just\nthat. The caller will be added in a subsequent patch.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 16 ++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9757cd0fbb..d882bf7ce1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1632,3 +1632,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\n }\n+\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname)\n+{\n+\tconst struct string_list *preferred_tips = bitmap_preferred_tips(r);\n+\tstruct string_list_item *item;\n+\n+\tif (!preferred_tips)\n+\t\treturn 0;\n+\n+\tfor_each_string_list_item(item, preferred_tips) {\n+\t\tif (starts_with(refname, item->string))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 020cd8d868..52ea10de51 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint16_t options);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n #endif\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428131","messageId":"7d44ba6299c06c956d5ac8ba01a0288d109c3cae.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 13/24] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:31Z","receivedAt":"2021-06-21T22:25:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This prepares the code in pack-bitmap to interpret the new multi-pack\nbitmaps described in Documentation/technical/bitmap-format.txt, which\nmostly involves converting bit positions to accommodate looking them up\nin a MIDX.\n\nNote that there are currently no writers who write multi-pack bitmaps,\nand that this will be implemented in the subsequent commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 5 +\n pack-bitmap-write.c | 2 +-\n pack-bitmap.c | 362 +++++++++++++++++++++++++++++++++++++----\n pack-bitmap.h | 5 +\n packfile.c | 2 +-\n 5 files changed, 340 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8a523624a1..e11d3ac2e5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\t/*\n+\t\t\t * Can use bit positions directly, even for MIDX\n+\t\t\t * bitmaps. See comment in try_partial_reuse()\n+\t\t\t * for why.\n+\t\t\t */\n \t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 142fd0adb8..9c55c1531e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)\n }\n \n /**\n- * Build the initial type index for the packfile\n+ * Build the initial type index for the packfile or multi-pack-index\n */\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d882bf7ce1..4110d23ca1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -13,6 +13,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"midx.h\"\n #include \"config.h\"\n \n /*\n@@ -35,8 +36,15 @@ struct stored_bitmap {\n * the active bitmap index is the largest one.\n */\n struct bitmap_index {\n-\t/* Packfile to which this bitmap index belongs to */\n+\t/*\n+\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n+\t * to.\n+\t *\n+\t * Exactly one of these must be non-NULL; this specifies the object\n+\t * order used to interpret this bitmap.\n+\t */\n \tstruct packed_git *pack;\n+\tstruct multi_pack_index *midx;\n \n \t/*\n \t * Mark the first `reuse_objects` in the packfile as reused:\n@@ -71,6 +79,9 @@ struct bitmap_index {\n \t/* If not NULL, this is a name-hash cache pointing into map. */\n \tuint32_t *hashes;\n \n+\t/* The checksum of the packfile or MIDX; points into map. */\n+\tconst unsigned char *checksum;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -138,6 +149,8 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n+\tif (index->midx)\n+\t\treturn index->midx->num_objects;\n \treturn index->pack->num_objects;\n }\n \n@@ -175,6 +188,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n+\tindex->checksum = header->checksum;\n \tindex->map_pos += header_size;\n \treturn 0;\n }\n@@ -227,7 +241,10 @@ static void nth_bitmap_object_oid(struct bitmap_index *index,\n \t\t\t\t struct object_id *oid,\n \t\t\t\t uint32_t n)\n {\n-\tnth_packed_object_id(oid, index->pack, n);\n+\tif (index->midx)\n+\t\tnth_midxed_object_oid(oid, index->midx, n);\n+\telse\n+\t\tnth_packed_object_id(oid, index->pack, n);\n }\n \n static int load_bitmap_entries_v1(struct bitmap_index *index)\n@@ -272,7 +289,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \treturn 0;\n }\n \n-static char *pack_bitmap_filename(struct packed_git *p)\n+char *midx_bitmap_filename(struct multi_pack_index *midx)\n+{\n+\treturn xstrfmt(\"%s-%s.bitmap\",\n+\t\t get_midx_filename(midx->object_dir),\n+\t\t hash_to_hex(get_midx_checksum(midx)));\n+}\n+\n+char *pack_bitmap_filename(struct packed_git *p)\n {\n \tsize_t len;\n \n@@ -281,6 +305,57 @@ static char *pack_bitmap_filename(struct packed_git *p)\n \treturn xstrfmt(\"%.*s.bitmap\", (int)len, p->pack_name);\n }\n \n+static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct stat st;\n+\tchar *idx_name = midx_bitmap_filename(midx);\n+\tint fd = git_open(idx_name);\n+\n+\tfree(idx_name);\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n+\t\twarning(\"ignoring extra bitmap file: %s\",\n+\t\t\tget_midx_filename(midx->object_dir));\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tbitmap_git->midx = midx;\n+\tbitmap_git->map_size = xsize_t(st.st_size);\n+\tbitmap_git->map_pos = 0;\n+\tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ,\n+\t\t\t\tMAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\tif (load_bitmap_header(bitmap_git) < 0)\n+\t\tgoto cleanup;\n+\n+\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum))\n+\t\tgoto cleanup;\n+\n+\tif (load_midx_revindex(bitmap_git->midx) < 0) {\n+\t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n+\t\tgoto cleanup;\n+\t}\n+\treturn 0;\n+\n+cleanup:\n+\tmunmap(bitmap_git->map, bitmap_git->map_size);\n+\tbitmap_git->map_size = 0;\n+\tbitmap_git->map = NULL;\n+\treturn -1;\n+}\n+\n static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git *packfile)\n {\n \tint fd;\n@@ -302,12 +377,18 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\treturn -1;\n \t}\n \n-\tif (bitmap_git->pack) {\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n \t\tclose(fd);\n \t\treturn -1;\n \t}\n \n+\tif (!is_pack_valid(packfile)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n \tbitmap_git->pack = packfile;\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n@@ -324,13 +405,36 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \treturn 0;\n }\n \n-static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n+static int load_reverse_index(struct bitmap_index *bitmap_git)\n+{\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tuint32_t i;\n+\t\tint ret;\n+\n+\t\tret = load_midx_revindex(bitmap_git->midx);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n+\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn load_pack_revindex(bitmap_git->pack);\n+}\n+\n+static int load_bitmap(struct bitmap_index *bitmap_git)\n {\n \tassert(bitmap_git->map);\n \n \tbitmap_git->bitmaps = kh_init_oid_map();\n \tbitmap_git->ext_index.positions = kh_init_oid_pos();\n-\tif (load_pack_revindex(bitmap_git->pack))\n+\n+\tif (load_reverse_index(bitmap_git))\n \t\tgoto failed;\n \n \tif (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) ||\n@@ -374,11 +478,35 @@ static int open_pack_bitmap(struct repository *r,\n \treturn ret;\n }\n \n+static int open_midx_bitmap(struct repository *r,\n+\t\t\t struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *midx;\n+\n+\tassert(!bitmap_git->map);\n+\n+\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n+\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\t\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n+static int open_bitmap(struct repository *r,\n+\t\t struct bitmap_index *bitmap_git)\n+{\n+\tassert(!bitmap_git->map);\n+\n+\tif (!open_midx_bitmap(r, bitmap_git))\n+\t\treturn 0;\n+\treturn open_pack_bitmap(r, bitmap_git);\n+}\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -428,10 +556,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \treturn pos;\n }\n \n+static int bitmap_position_midx(struct bitmap_index *bitmap_git,\n+\t\t\t\tconst struct object_id *oid)\n+{\n+\tuint32_t want, got;\n+\tif (!bsearch_midx(oid, bitmap_git->midx, &want))\n+\t\treturn -1;\n+\n+\tif (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0)\n+\t\treturn -1;\n+\treturn got;\n+}\n+\n static int bitmap_position(struct bitmap_index *bitmap_git,\n \t\t\t const struct object_id *oid)\n {\n-\tint pos = bitmap_position_packfile(bitmap_git, oid);\n+\tint pos;\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpos = bitmap_position_midx(bitmap_git, oid);\n+\telse\n+\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n }\n \n@@ -724,6 +868,7 @@ static void show_objects_for_type(\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n+\t\t\tstruct packed_git *pack;\n \t\t\tstruct object_id oid;\n \t\t\tuint32_t hash = 0, index_pos;\n \t\t\toff_t ofs;\n@@ -733,14 +878,28 @@ static void show_objects_for_type(\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n-\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\t\t\tuint32_t pack_id;\n+\n+\t\t\t\tindex_pos = pack_pos_to_midx(m, pos + offset);\n+\t\t\t\tofs = nth_midxed_offset(m, index_pos);\n+\t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n+\n+\t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t} else {\n+\t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\t\tnth_bitmap_object_oid(bitmap_git, &oid, index_pos);\n+\n+\t\t\t\tpack = bitmap_git->pack;\n+\t\t\t}\n \n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n }\n@@ -752,8 +911,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n-\t\t\treturn 1;\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n+\t\t\t\treturn 1;\n+\t\t} else {\n+\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -839,14 +1003,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git,\n static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\t\t uint32_t pos)\n {\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tunsigned long size;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = &size;\n \n \tif (pos < bitmap_num_objects(bitmap_git)) {\n-\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tstruct packed_git *pack;\n+\t\toff_t ofs;\n+\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\n+\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\t\t} else {\n+\t\t\tpack = bitmap_git->pack;\n+\t\t\tofs = pack_pos_to_offset(pack, pos);\n+\t\t}\n+\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n \t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n@@ -1027,7 +1203,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t/* try to open a bitmapped pack, but don't parse it yet\n \t * because we may not need to use it */\n \tCALLOC_ARRAY(bitmap_git, 1);\n-\tif (open_pack_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (open_bitmap(revs->repo, bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tfor (i = 0; i < revs->pending.nr; ++i) {\n@@ -1071,7 +1247,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_pack_bitmap(bitmap_git) < 0)\n+\tif (load_bitmap(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tobject_array_clear(&revs->pending);\n@@ -1115,19 +1291,43 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n }\n \n static void try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t struct packed_git *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n-\toff_t offset, header;\n+\toff_t offset, delta_obj_offset;\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn; /* not actually in the pack or MIDX */\n+\t/*\n+\t * try_partial_reuse() is called either on (a) objects in the\n+\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n+\t * objects in the preferred pack of a multi-pack bitmap.\n+\t * Importantly, the latter can pretend as if only a single pack\n+\t * exists because:\n+\t *\n+\t * - The first pack->num_objects bits of a MIDX bitmap are\n+\t * reserved for the preferred pack, and\n+\t *\n+\t * - Ties due to duplicate objects are always resolved in\n+\t * favor of the preferred pack.\n+\t *\n+\t * Therefore we do not need to ever ask the MIDX for its copy of\n+\t * an object by OID, since it will always select it from the\n+\t * preferred pack. Likewise, the selected copy of the base\n+\t * object for any deltas will reside in the same pack.\n+\t *\n+\t * This means that we can reuse pos when looking up the bit in\n+\t * the reuse bitmap, too, since bits corresponding to the\n+\t * preferred pack precede all bits from other packs.\n+\t */\n \n-\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n-\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n+\tif (pos >= pack->num_objects)\n+\t\treturn; /* not actually in the pack or MIDX preferred pack */\n+\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n+\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn; /* broken packfile, punt */\n \n@@ -1143,11 +1343,11 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t &offset, type, header);\n+\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn;\n-\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n \t\t\treturn;\n \n \t\t/*\n@@ -1180,24 +1380,48 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tbitmap_set(reuse, pos);\n }\n \n+static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *m = bitmap_git->midx;\n+\tif (!m)\n+\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n+\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n+}\n+\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t struct packed_git **packfile_out,\n \t\t\t\t uint32_t *entries,\n \t\t\t\t struct bitmap **reuse_out)\n {\n+\tstruct packed_git *pack;\n \tstruct bitmap *result = bitmap_git->result;\n \tstruct bitmap *reuse;\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n-\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n+\tuint32_t objects_nr;\n \n \tassert(result);\n \n+\tload_reverse_index(bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\telse\n+\t\tpack = bitmap_git->pack;\n+\tobjects_nr = pack->num_objects;\n+\n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n \n-\t/* Don't mark objects not in the packfile */\n+\t/*\n+\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n+\t * marks objects eligible for reuse, but the pack-reuse code only\n+\t * understands how to reuse a single pack. Since the preferred pack is\n+\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n+\t * we use it instead of another pack. In single-pack bitmaps, the choice\n+\t * is made for us.\n+\t */\n \tif (i > objects_nr / BITS_IN_EWORD)\n \t\ti = objects_nr / BITS_IN_EWORD;\n \n@@ -1213,7 +1437,15 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\t/*\n+\t\t\t\t * Can't reuse from a non-preferred pack (see\n+\t\t\t\t * above).\n+\t\t\t\t */\n+\t\t\t\tif (pos + offset >= objects_nr)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n \t\t}\n \t}\n \n@@ -1230,7 +1462,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = bitmap_git->pack;\n+\t*packfile_out = pack;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n@@ -1504,6 +1736,12 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n+\tif (!bitmap_is_midx(bitmap_git))\n+\t\tload_reverse_index(bitmap_git);\n+\telse if (load_midx_revindex(bitmap_git->midx) < 0)\n+\t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n+\t\t \"extension\");\n+\n \tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n@@ -1511,8 +1749,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tstruct object_id oid;\n \t\tstruct object_entry *oe;\n \n-\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n-\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n+\t\tif (bitmap_is_midx(bitmap_git))\n+\t\t\tnth_midxed_object_oid(&oid,\n+\t\t\t\t\t bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n+\t\telse\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n@@ -1538,6 +1781,19 @@ void free_bitmap_index(struct bitmap_index *b)\n \tfree(b->ext_index.hashes);\n \tbitmap_free(b->result);\n \tbitmap_free(b->haves);\n+\tif (bitmap_is_midx(b)) {\n+\t\t/*\n+\t\t * Multi-pack bitmaps need to have resources associated with\n+\t\t * their on-disk reverse indexes unmapped so that stale .rev and\n+\t\t * .bitmap files can be removed.\n+\t\t *\n+\t\t * Unlike pack-based bitmaps, multi-pack bitmaps can be read and\n+\t\t * written in the same 'git multi-pack-index write --bitmap'\n+\t\t * process. Close resources so they can be removed safely on\n+\t\t * platforms like Windows.\n+\t\t */\n+\t\tclose_midx_revindex(b->midx);\n+\t}\n \tfree(b);\n }\n \n@@ -1552,7 +1808,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\t enum object_type object_type)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n+\tstruct packed_git *pack;\n \toff_t total = 0;\n \tstruct ewah_iterator it;\n \teword_t filter;\n@@ -1575,7 +1831,31 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tpos = base + offset;\n+\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tuint32_t pack_pos;\n+\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n+\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\n+\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n+\t\t\t\t\tstruct object_id oid;\n+\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n+\n+\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n+\t\t\t\t\t oid_to_hex(&oid),\n+\t\t\t\t\t pack_id,\n+\t\t\t\t\t (uintmax_t)offset);\n+\t\t\t\t}\n+\n+\t\t\t\tpos = pack_pos;\n+\t\t\t} else {\n+\t\t\t\tpack = bitmap_git->pack;\n+\t\t\t\tpos = base + offset;\n+\t\t\t}\n+\n \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n \t\t\t\t pack_pos_to_offset(pack, pos);\n \t\t}\n@@ -1628,6 +1908,20 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n \treturn total;\n }\n \n+int bitmap_is_midx(struct bitmap_index *bitmap_git)\n+{\n+\treturn !!bitmap_git->midx;\n+}\n+\n+off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos)\n+{\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\treturn nth_midxed_offset(bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, pos));\n+\treturn nth_packed_object_offset(bitmap_git->pack,\n+\t\t\t\t\tpack_pos_to_index(bitmap_git->pack, pos));\n+}\n+\n const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 52ea10de51..30396a7a4a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -92,6 +92,11 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n \t\t\t uint16_t options);\n+char *midx_bitmap_filename(struct multi_pack_index *midx);\n+char *pack_bitmap_filename(struct packed_git *p);\n+\n+int bitmap_is_midx(struct bitmap_index *bitmap_git);\n+off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n int bitmap_is_preferred_refname(struct repository *r, const char *refname);\ndiff --git a/packfile.c b/packfile.c\nindex 755aa7aec5..e855b93208 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n-\t ends_with(file_name, \".rev\"))\n+\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428132","messageId":"a8cec2463d0993b1118abdd31cb6c9e88a32e0c4.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:34Z","receivedAt":"2021-06-21T22:25:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Write multi-pack bitmaps in the format described by\nDocumentation/technical/bitmap-format.txt, inferring their presence with\nthe absence of '--bitmap'.\n\nTo write a multi-pack bitmap, this patch attempts to reuse as much of\nthe existing machinery from pack-objects as possible. Specifically, the\nMIDX code prepares a packing_data struct that pretends as if a single\npackfile has been generated containing all of the objects contained\nwithin the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 12 +-\n builtin/multi-pack-index.c | 2 +\n midx.c | 230 ++++++++++++++++++++++++-\n midx.h | 1 +\n 4 files changed, 236 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex ffd601bc17..ada14deb2c 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n-\t[--preferred-pack=] \n+\t[--preferred-pack=] [--[no-]bitmap] \n \n DESCRIPTION\n -----------\n@@ -40,6 +40,9 @@ write::\n \t\tmultiple packs contain the same object. If not given,\n \t\tties are broken in favor of the pack with the lowest\n \t\tmtime.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\n@@ -81,6 +84,13 @@ EXAMPLES\n $ git multi-pack-index write\n -----------------------------------------------\n \n+* Write a MIDX file for the packfiles in the current .git folder with a\n+corresponding bitmap.\n++\n+-------------------------------------------------------------\n+$ git multi-pack-index write --preferred-pack --bitmap\n+-------------------------------------------------------------\n+\n * Write a MIDX file for the packfiles in an alternate object store.\n +\n -----------------------------------------------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5d3ea445fd..bf6fa982e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n \t\t\t N_(\"preferred-pack\"),\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/midx.c b/midx.c\nindex 752d36c57f..a58cca707b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,10 @@\n #include \"repository.h\"\n #include \"chunk-format.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -885,6 +889,172 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n static void clear_midx_files_ext(struct repository *r, const char *ext,\n \t\t\t\t unsigned char *keep_hash);\n \n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tif (oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t data->ctx->entries_nr,\n+\t\t bitmap_oid_access) > -1) {\n+\t\tALLOC_GROW(data->commits, data->commits_nr + 1,\n+\t\t\t data->commits_alloc);\n+\t\tdata->commits[data->commits_nr++] = commit;\n+\t}\n+}\n+\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb;\n+\n+\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tfor_each_ref(add_ref_to_pending, &revs);\n+\n+\t/*\n+\t * Skipping promisor objects here is intentional, since it only excludes\n+\t * them from the list of reachable commits that we want to select from\n+\t * when computing the selection of MIDX'd commits to receive bitmaps.\n+\t *\n+\t * Reachability bitmaps do require that their objects be closed under\n+\t * reachability, but fetching any objects missing from promisors at this\n+\t * point is too late. But, if one of those objects can be reached from\n+\t * an another object that is included in the bitmap, then we will\n+\t * complain later that we don't have reachability closure (and fail\n+\t * appropriately).\n+\t */\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\t/*\n+\t * Pass selected commits in topo order to match the behavior of\n+\t * pack-bitmaps when configured with delta islands.\n+\t */\n+\trevs.topo_order = 1;\n+\trevs.sort_order = REV_SORT_IN_GRAPH_ORDER;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t unsigned flags)\n+{\n+\tstruct packing_data pdata;\n+\tstruct pack_idx_entry **index;\n+\tstruct commit **commits = NULL;\n+\tuint32_t i, commits_nr;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n+\tint ret;\n+\n+\tprepare_midx_packing_data(&pdata, ctx);\n+\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata.nr_objects);\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_finish expects objects in lex order, but pack_order\n+\t * gives us exactly that. use it directly instead of re-sorting the\n+\t * array.\n+\t *\n+\t * This changes the order of objects in 'index' between\n+\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n+\t *\n+\t * The same re-ordering takes place in the single-pack bitmap code via\n+\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n+\t * happens between bitmap_writer_build_type_index() and\n+\t * bitmap_writer_finish().\n+\t */\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(&pdata);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\treturn ret;\n+}\n+\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -930,9 +1100,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n \n+\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\terror(_(\"could not load pack %s\"),\n+\t\t\t\t ctx.m->pack_names[i]);\n+\t\t\t\tresult = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\tctx.info[ctx.nr].expired = 0;\n \t\t\tctx.nr++;\n \t\t}\n@@ -947,8 +1124,26 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n-\t\tgoto cleanup;\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_bitmap_git(the_repository);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n+\t\t\t\t\t\t NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n \n \tif (preferred_pack_name) {\n \t\tint found = 0;\n@@ -964,7 +1159,8 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tif (!found)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n \t\t\t\tpreferred_pack_name);\n-\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t} else if (ctx.nr &&\n+\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\ttime_t oldest = ctx.info[0].p->mtime;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1075,9 +1271,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n-\tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n@@ -1108,14 +1301,22 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tif (flags & MIDX_WRITE_BITMAP) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\t\terror(_(\"could not write multi-pack bitmap\"));\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n \tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n cleanup:\n@@ -1123,6 +1324,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tif (ctx.info[i].p) {\n \t\t\tclose_pack(ctx.info[i].p);\n \t\t\tfree(ctx.info[i].p);\n+\t\t\tif (ctx.m) {\n+\t\t\t\t/*\n+\t\t\t\t * Destroy a stale reference to the pack in\n+\t\t\t\t * 'ctx.m'.\n+\t\t\t\t */\n+\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n+\t\t\t\tif (orig < ctx.m->num_packs)\n+\t\t\t\t\tctx.m->packs[orig] = NULL;\n+\t\t\t}\n \t\t}\n \t\tfree(ctx.info[i].pack_name);\n \t}\n@@ -1132,6 +1342,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n+\tif (ctx.m)\n+\t\tclose_midx(ctx.m);\n+\n \treturn result;\n }\n \n@@ -1193,6 +1406,7 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".bitmap\", NULL);\n \tclear_midx_files_ext(r, \".rev\", NULL);\n \n \tfree(midx);\ndiff --git a/midx.h b/midx.h\nindex 1172df1a71..350f4d0a7b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -41,6 +41,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n+#define MIDX_WRITE_BITMAP (1 << 2)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n char *get_midx_filename(const char *object_dir);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428133","messageId":"c63eb637c8d082e0ae13ef1d1f4b7f530a22881a.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 15/24] t5310: move some tests to lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:37Z","receivedAt":"2021-06-21T22:25:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We'll soon be adding a test script that will cover many of the same\nbitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many\nof the applicable tests as we can so we don't have to rewrite them.\n\nThere should be no functional change to t5310; we still run the same\noperations in the same order.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 236 ++++++++++++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 227 +-------------------------------------\n 2 files changed, 240 insertions(+), 223 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex fe3f98be24..ecb5d0e05d 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,3 +1,6 @@\n+# Helpers for scripts testing bitamp functionality; see t5310 for\n+# example usage.\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -24,3 +27,236 @@ test_bitmap_traversal () {\n \ttest_cmp \"$1.normalized\" \"$2.normalized\" &&\n \trm -f \"$1.normalized\" \"$2.normalized\"\n }\n+\n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other second\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-second * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for second (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n+#\n+# second: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n+setup_bitmap_history() {\n+\ttest_expect_success 'setup repo with moderate-sized history' '\n+\t\ttest_commit_bulk --id=file 10 &&\n+\t\tgit branch -M second &&\n+\t\tgit checkout -b other HEAD~5 &&\n+\t\ttest_commit_bulk --id=side 10 &&\n+\n+\t\t# add complicated history setup, including merges and\n+\t\t# ambiguous merge-bases\n+\n+\t\tgit checkout -b merge-left other~2 &&\n+\t\tgit merge second~2 -m \"merge-left\" &&\n+\n+\t\tgit checkout -b merge-right second~1 &&\n+\t\tgit merge other~1 -m \"merge-right\" &&\n+\n+\t\tgit checkout -b octo-second second &&\n+\t\tgit merge merge-left merge-right -m \"octopus-second\" &&\n+\n+\t\tgit checkout -b octo-other other &&\n+\t\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\t\tgit checkout other &&\n+\t\tgit merge octo-other -m \"pull octopus\" &&\n+\n+\t\tgit checkout second &&\n+\t\tgit merge octo-second -m \"pull octopus\" &&\n+\n+\t\t# Remove these branches so they are not selected\n+\t\t# as bitmap tips\n+\t\tgit branch -D merge-left &&\n+\t\tgit branch -D merge-right &&\n+\t\tgit branch -D octo-other &&\n+\t\tgit branch -D octo-second &&\n+\n+\t\t# add padding to make these merges less interesting\n+\t\t# and avoid having them selected for bitmaps\n+\t\ttest_commit_bulk --id=file 100 &&\n+\t\tgit checkout other &&\n+\t\ttest_commit_bulk --id=side 100 &&\n+\t\tgit checkout second &&\n+\n+\t\tbitmaptip=$(git rev-parse second) &&\n+\t\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n+\t\tgit tag tagged-blob $blob\n+\t'\n+}\n+\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n+\t\tgit rev-list --count other...second >expect &&\n+\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n+\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n+\t\ttest_bitmap_traversal expect actual\n+\t'\n+\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n+\t\tgrep $blob actual\n+\t'\n+}\n+\n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"second\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n+basic_bitmap_tests () {\n+\ttip=\"$1\"\n+\ttest_expect_success 'rev-list --test-bitmap verifies bitmaps' \"\n+\t\tgit rev-list --test-bitmap \"${tip:-HEAD}\"\n+\t\"\n+\n+\trev_list_tests 'full bitmap'\n+\n+\ttest_expect_success 'clone from bitmapped repository' '\n+\t\trm -fr clone.git &&\n+\t\tgit clone --no-local --bare . clone.git &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'partial clone from bitmapped repository' '\n+\t\ttest_config uploadpack.allowfilter true &&\n+\t\trm -fr partial-clone.git &&\n+\t\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n+\t\t(\n+\t\t\tcd partial-clone.git &&\n+\t\t\tpack=$(echo objects/pack/*.pack) &&\n+\t\t\tgit verify-pack -v \"$pack\" >have &&\n+\t\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n+\t\t\t# we expect this single blob because of the direct ref\n+\t\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n+\t\t\ttest_cmp expect blobs\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'setup further non-bitmapped commits' '\n+\t\ttest_commit_bulk --id=further 10\n+\t'\n+\n+\trev_list_tests 'partial bitmap'\n+\n+\ttest_expect_success 'fetch (partial bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'enumerating progress counts pack-reused objects' '\n+\t\tcount=$(git rev-list --objects --all --count) &&\n+\t\tgit repack -adb &&\n+\n+\t\t# check first with only reused objects; confirm that our\n+\t\t# progress showed the right number, and also that we did\n+\t\t# pack-reuse as expected. Check only the final \"done\"\n+\t\t# line of the meter (there may be an arbitrary number of\n+\t\t# intermediate lines ending with CR).\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $count, done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr &&\n+\n+\t\t# now the same but with one non-reused object\n+\t\tgit commit --allow-empty -m \"an extra commit object\" &&\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr\n+\t'\n+}\n+\n+# have_delta \n+#\n+# Note that because this relies on cat-file, it might find _any_ copy of an\n+# object in the repository. The caller is responsible for making sure\n+# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n+have_delta () {\n+\techo $2 >expect &&\n+\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n+\ttest_cmp expect actual\n+}\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex b02838750e..4318f84d53 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -25,93 +25,10 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-# To ensure the logic for \"maximal commits\" is exercised, make\n-# the repository a bit more complicated.\n-#\n-# other second\n-# * *\n-# (99 commits) (99 commits)\n-# * *\n-# |\\ /|\n-# | * octo-other octo-second * |\n-# |/|\\_________ ____________/|\\|\n-# | \\ \\/ __________/ |\n-# | | ________/\\ / |\n-# * |/ * merge-right *\n-# | _|__________/ \\____________ |\n-# |/ | \\|\n-# (l1) * * merge-left * (r1)\n-# | / \\________________________ |\n-# |/ \\|\n-# (l2) * * (r2)\n-# \\___________________________ |\n-# \\|\n-# * (base)\n-#\n-# We only push bits down the first-parent history, which\n-# makes some of these commits unimportant!\n-#\n-# The important part for the maximal commit algorithm is how\n-# the bitmasks are extended. Assuming starting bit positions\n-# for second (bit 0) and other (bit 1), the bitmasks at the\n-# end should be:\n-#\n-# second: 1 (maximal, selected)\n-# other: 01 (maximal, selected)\n-# (base): 11 (maximal)\n-#\n-# This complicated history was important for a previous\n-# version of the walk that guarantees never walking a\n-# commit multiple times. That goal might be important\n-# again, so preserve this complicated case. For now, this\n-# test will guarantee that the bitmaps are computed\n-# correctly, even with the repeat calculations.\n+setup_bitmap_history\n \n-test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 10 &&\n-\tgit branch -M second &&\n-\tgit checkout -b other HEAD~5 &&\n-\ttest_commit_bulk --id=side 10 &&\n-\n-\t# add complicated history setup, including merges and\n-\t# ambiguous merge-bases\n-\n-\tgit checkout -b merge-left other~2 &&\n-\tgit merge second~2 -m \"merge-left\" &&\n-\n-\tgit checkout -b merge-right second~1 &&\n-\tgit merge other~1 -m \"merge-right\" &&\n-\n-\tgit checkout -b octo-second second &&\n-\tgit merge merge-left merge-right -m \"octopus-second\" &&\n-\n-\tgit checkout -b octo-other other &&\n-\tgit merge merge-left merge-right -m \"octopus-other\" &&\n-\n-\tgit checkout other &&\n-\tgit merge octo-other -m \"pull octopus\" &&\n-\n-\tgit checkout second &&\n-\tgit merge octo-second -m \"pull octopus\" &&\n-\n-\t# Remove these branches so they are not selected\n-\t# as bitmap tips\n-\tgit branch -D merge-left &&\n-\tgit branch -D merge-right &&\n-\tgit branch -D octo-other &&\n-\tgit branch -D octo-second &&\n-\n-\t# add padding to make these merges less interesting\n-\t# and avoid having them selected for bitmaps\n-\ttest_commit_bulk --id=file 100 &&\n-\tgit checkout other &&\n-\ttest_commit_bulk --id=side 100 &&\n-\tgit checkout second &&\n-\n-\tbitmaptip=$(git rev-parse second) &&\n-\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n-\tgit tag tagged-blob $blob &&\n-\tgit config repack.writebitmaps true\n+test_expect_success 'setup writing bitmaps during repack' '\n+\tgit config repack.writeBitmaps true\n '\n \n test_expect_success 'full repack creates bitmaps' '\n@@ -123,109 +40,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n-test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-\tgit rev-list --test-bitmap HEAD\n-'\n-\n-rev_list_tests_head () {\n-\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch~5..$branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n-\t\tgit rev-list --count -n 1 $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n-\t\tgit rev-list --count other...second >expect &&\n-\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n-\t\tgit rev-list --count $branch -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count --objects $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n-\t\tgit rev-list --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list $branch >expect &&\n-\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list --objects $branch >expect &&\n-\t\ttest_bitmap_traversal expect actual\n-\t'\n-\n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n-\t\tgrep $blob actual\n-\t'\n-}\n-\n-rev_list_tests () {\n-\tstate=$1\n-\n-\tfor branch in \"second\" \"other\"\n-\tdo\n-\t\trev_list_tests_head\n-\tdone\n-}\n-\n-rev_list_tests 'full bitmap'\n-\n-test_expect_success 'clone from bitmapped repository' '\n-\tgit clone --no-local --bare . clone.git &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'partial clone from bitmapped repository' '\n-\ttest_config uploadpack.allowfilter true &&\n-\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n-\t(\n-\t\tcd partial-clone.git &&\n-\t\tpack=$(echo objects/pack/*.pack) &&\n-\t\tgit verify-pack -v \"$pack\" >have &&\n-\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n-\t\t# we expect this single blob because of the direct ref\n-\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n-\t\ttest_cmp expect blobs\n-\t)\n-'\n-\n-test_expect_success 'setup further non-bitmapped commits' '\n-\ttest_commit_bulk --id=further 10\n-'\n-\n-rev_list_tests 'partial bitmap'\n-\n-test_expect_success 'fetch (partial bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+basic_bitmap_tests\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -461,40 +276,6 @@ test_expect_success 'truncated bitmap fails gracefully (cache)' '\n \ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n-test_expect_success 'enumerating progress counts pack-reused objects' '\n-\tcount=$(git rev-list --objects --all --count) &&\n-\tgit repack -adb &&\n-\n-\t# check first with only reused objects; confirm that our progress\n-\t# showed the right number, and also that we did pack-reuse as expected.\n-\t# Check only the final \"done\" line of the meter (there may be an\n-\t# arbitrary number of intermediate lines ending with CR).\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $count, done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr &&\n-\n-\t# now the same but with one non-reused object\n-\tgit commit --allow-empty -m \"an extra commit object\" &&\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr\n-'\n-\n-# have_delta \n-#\n-# Note that because this relies on cat-file, it might find _any_ copy of an\n-# object in the repository. The caller is responsible for making sure\n-# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n-have_delta () {\n-\techo $2 >expect &&\n-\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n-\ttest_cmp expect actual\n-}\n-\n # Create a state of history with these properties:\n #\n # - refs that allow a client to fetch some new history, while sharing some old\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428134","messageId":"bedb7afb37664385ca286527afa55c09cec61895.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 16/24] t/helper/test-read-midx.c: add --checksum mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:40Z","receivedAt":"2021-06-21T22:25:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subsequent tests will want to check for the existence of a multi-pack\nbitmap which matches the multi-pack-index stored in the pack directory.\n\nThe multi-pack bitmap includes the hex checksum of the MIDX it\ncorresponds to in its filename (for example,\n'$packdir/multi-pack-index-.bitmap'). As a result, some tests\nwant a way to learn what '' is.\n\nThis helper addresses that need by printing the checksum of the\nrepository's multi-pack-index.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 16 +++++++++++++++-\n t/lib-bitmap.sh | 4 ++++\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 7c2eb11a8e..cb0d27049a 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \treturn 0;\n }\n \n+static int read_midx_checksum(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m;\n+\n+\tsetup_git_directory();\n+\tm = load_multi_pack_index(object_dir, 1);\n+\tif (!m)\n+\t\treturn 1;\n+\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n+\telse if (!strcmp(argv[1], \"--checksum\"))\n+\t\treturn read_midx_checksum(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex ecb5d0e05d..09cd036f4d 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -260,3 +260,7 @@ have_delta () {\n \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n \ttest_cmp expect actual\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"${1:-.git/objects}\"\n+}\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428135","messageId":"2a5df1832a340bfedba80bbb1b223b82d14ce3f9.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 18/24] t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:45Z","receivedAt":"2021-06-21T22:25:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap causes tests which repack in a partial clone to\nfail because they are missing objects. Missing objects is an expected\ncomponent of tests in t0410, so disable this knob altogether. Graceful\ndegradation when writing a bitmap with missing objects is tested in\nt5326.\n\nSigned-off-by: Taylor Blau \n---\n t/t0410-partial-clone.sh | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 1667450917..4fd8e83da1 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -4,6 +4,9 @@ test_description='partial clone'\n \n . ./test-lib.sh\n \n+# missing promisor objects cause repacks which write bitmaps to fail\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n delete_object () {\n \trm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428136","messageId":"fbfac4ae8e14eee6b825b0f2b71a81a6b448d437.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 17/24] t5326: test multi-pack bitmap behavior","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:42Z","receivedAt":"2021-06-21T22:25:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch introduces a new test, t5326, which tests the basic\nfunctionality of multi-pack bitmaps.\n\nSome trivial behavior is tested, such as:\n\n - Whether bitmaps can be generated with more than one pack.\n - Whether clones can be served with all objects in the bitmap.\n - Whether follow-up fetches can be served with some objects outside of\n the server's bitmap\n\nThese use lib-bitmap's tests (which in turn were pulled from t5310), and\nwe cover cases where the MIDX represents both a single pack and multiple\npacks.\n\nIn addition, some non-trivial and MIDX-specific behavior is tested, too,\nincluding:\n\n - Whether multi-pack bitmaps behave correctly with respect to the\n pack-reuse machinery when the base for some object is selected from\n a different pack than the delta.\n - Whether multi-pack bitmaps correctly respect the\n pack.preferBitmapTips configuration.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 277 ++++++++++++++++++++++++++++++++++\n 1 file changed, 277 insertions(+)\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..c1b7d633e2\n--- /dev/null\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,277 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality'\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+setup_bitmap_history\n+\n+test_expect_success 'enable core.multiPackIndex' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_expect_success 'create single-pack midx with bitmaps' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success 'create new additional packs' '\n+\tfor i in $(test_seq 1 16)\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\t\tgit repack -d\n+\tdone &&\n+\n+\tgit checkout -b other2 HEAD~8 &&\n+\tfor i in $(test_seq 1 8)\n+\tdo\n+\t\ttest_commit \"side-$i\" &&\n+\t\tgit repack -d\n+\tdone &&\n+\tgit checkout second\n+'\n+\n+test_expect_success 'create multi-pack midx with bitmaps' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\tls $objdir/pack/pack-*.pack >packs &&\n+\ttest_line_count = 25 packs &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\ttest_commit respect--no-bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\tgit multi-pack-index write --no-bitmap &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+test_expect_success 'setup midx with base from later pack' '\n+\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t# prefers to delete contents out of a base rather than add to a shorter\n+\t# object.\n+\ttest_seq 1 128 >a &&\n+\ttest_seq 1 130 >b &&\n+\n+\tgit add a b &&\n+\tgit commit -m \"initial commit\" &&\n+\n+\ta=$(git rev-parse HEAD:a) &&\n+\tb=$(git rev-parse HEAD:b) &&\n+\n+\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$a\n+\t$b\n+\tEOF\n+\t) &&\n+\n+\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t# any other object.\n+\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$b\n+\t$(git rev-parse HEAD)\n+\t$(git rev-parse HEAD^{tree})\n+\tEOF\n+\t) &&\n+\n+\tgit prune-packed &&\n+\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t# reuse.\n+\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\thave_delta $a $b &&\n+\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+'\n+\n+rev_list_tests 'full bitmap with backwards delta'\n+\n+test_expect_success 'clone with bitmaps enabled' '\n+\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\tgit rev-parse HEAD >expect &&\n+\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+bitmap_reuse_tests() {\n+\tfrom=$1\n+\tto=$2\n+\n+\ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk 16 &&\n+\t\t\tgit tag old-tip &&\n+\n+\t\t\tgit config core.multiPackIndex true &&\n+\t\t\tif test \"MIDX\" = \"$from\"\n+\t\t\tthen\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk --id=further 16 &&\n+\t\t\tgit tag new-tip &&\n+\n+\t\t\tif test \"MIDX\" = \"$to\"\n+\t\t\tthen\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit for-each-ref &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n+\t\t)\n+\t'\n+}\n+\n+bitmap_reuse_tests 'pack' 'MIDX'\n+bitmap_reuse_tests 'MIDX' 'pack'\n+bitmap_reuse_tests 'MIDX' 'MIDX'\n+\n+test_expect_success 'missing object closure fails gracefully' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit loose &&\n+\t\ttest_commit packed &&\n+\n+\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t# commit.\n+\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t$(git rev-parse packed)\n+\t\tEOF\n+\n+\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\tgrep \"doesn.t have full closure\" err &&\n+\t\ttest_path_is_missing $midx\n+\t)\n+'\n+\n+test_expect_success 'setup partial bitmaps' '\n+\ttest_commit packed &&\n+\tgit repack &&\n+\ttest_commit loose &&\n+\tgit multi-pack-index write --bitmap 2>err &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests HEAD~\n+\n+test_expect_success 'removing a MIDX clears stale bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm $midx &&\n+\n+\t\t# Then write a new MIDX.\n+\t\ttest_commit new &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_missing $stale_bitmap\n+\t)\n+'\n+\n+test_expect_success 'pack.preferBitmapTips' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\tbitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428137","messageId":"2d24c5b7ad0835f433428c16dfd137449688d93b.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 19/24] t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:48Z","receivedAt":"2021-06-21T22:25:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap confuses many of the tests in t5310, which\nexpect to control whether and how bitmaps are written. Since the\nrelevant MIDX-bitmap tests here are covered already in t5326, let's just\ndisable the flag for the whole t5310 script.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 4318f84d53..673baa5c3c 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n . \"$TEST_DIRECTORY\"/lib-bundle.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n+# their place.\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428138","messageId":"4cbfaa0e978fd68bd5acd6ce5f96b34e9cd43656.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 20/24] t5319: don't write MIDX bitmaps in t5319","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:50Z","receivedAt":"2021-06-21T22:25:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This test is specifically about generating a midx still respecting a\npack-based bitmap file. Generating a MIDX bitmap would confuse the test.\nLet's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to\nmake sure we don't do so.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 5641d158df..69f1c815aa 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -474,7 +474,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428139","messageId":"839a7a79eb1fedd25948213a5d64c97e34d38a31.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 21/24] t7700: update to work with MIDX bitmap test knob","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:53Z","receivedAt":"2021-06-21T22:25:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A number of these tests are focused only on pack-based bitmaps and need\nto be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where\nnecessary.\n\nSigned-off-by: Taylor Blau \n---\n t/t7700-repack.sh | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 25b235c063..98eda3bfeb 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit repack -Adbl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '\n \n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n-\tgit -C bare.git repack -ad &&\n+\trm -f bare.git/objects/pack/*.bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428140","messageId":"00418d5b096c049ddfc738b5d51ef65eae019607.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 22/24] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:56Z","receivedAt":"2021-06-21T22:26:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment\nvariable to also write a multi-pack bitmap when\n'GIT_TEST_MULTI_PACK_INDEX' is set.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 13 ++++++++++---\n ci/run-build-and-tests.sh | 1 +\n midx.h | 2 ++\n t/README | 4 ++++\n 4 files changed, 17 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..77f6f03057 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -515,7 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n \t\t !is_bare_repository())\n \t\t\twrite_bitmaps = 0;\n-\t}\n+\t} else if (write_bitmaps &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\twrite_bitmaps = 0;\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n \n@@ -725,8 +728,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tupdate_server_info(0);\n \tremove_temporary_files();\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n+\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t}\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..7ee9ba9325 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -23,6 +23,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_WRITE_REV_INDEX=1\ndiff --git a/midx.h b/midx.h\nindex 350f4d0a7b..aa3da557bb 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,8 @@ struct pack_entry;\n struct repository;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex 1a2072b2c8..1311b8e17a 100644\n--- a/t/README\n+++ b/t/README\n@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n+'--bitmap' option on all invocations of 'git multi-pack-index write',\n+and ignores pack-objects' '--write-bitmap-index'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428141","messageId":"98fa73a76a981ce328dc9ad1dc816d6a62bc0659.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 23/24] p5310: extract full and partial bitmap tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:25:58Z","receivedAt":"2021-06-21T22:26:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A new p5326 introduced by the next patch will want these same tests,\ninterjecting its own setup in between. Move them out so that both perf\ntests can reuse them.\n\nSigned-off-by: Taylor Blau \n---\n t/perf/lib-bitmap.sh | 69 ++++++++++++++++++++++++++++++++++++\n t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------\n 2 files changed, 72 insertions(+), 62 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n\ndiff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh\nnew file mode 100644\nindex 0000000000..63d3bc7cec\n--- /dev/null\n+++ b/t/perf/lib-bitmap.sh\n@@ -0,0 +1,69 @@\n+# Helper functions for testing bitmap performance; see p5310.\n+\n+test_full_bitmap () {\n+\ttest_perf 'simulated clone' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'simulated fetch' '\n+\t\thave=$(git rev-list HEAD~100 -1) &&\n+\t\t{\n+\t\t\techo HEAD &&\n+\t\t\techo ^$have\n+\t\t} | git pack-objects --revs --stdout >/dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (commits)' '\n+\t\tgit rev-list --all --use-bitmap-index >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (objects)' '\n+\t\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tag negated via --not --all (objects)' '\n+\t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with negative tag (objects)' '\n+\t\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:none' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:none >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:limit=1k' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:limit=1k >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with tree:0' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+\n+\ttest_perf 'simulated partial clone' '\n+\t\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n+\t'\n+}\n+\n+test_partial_bitmap () {\n+\ttest_perf 'clone (partial bitmap)' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (partial bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tree filter (partial bitmap)' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+}\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 452be01056..7ad4f237bc 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -2,6 +2,7 @@\n \n test_description='Tests pack performance using bitmaps'\n . ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n test_perf_large_repo\n \n@@ -25,56 +26,7 @@ test_perf 'repack to disk' '\n \tgit repack -ad\n '\n \n-test_perf 'simulated clone' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'simulated fetch' '\n-\thave=$(git rev-list HEAD~100 -1) &&\n-\t{\n-\t\techo HEAD &&\n-\t\techo ^$have\n-\t} | git pack-objects --revs --stdout >/dev/null\n-'\n-\n-test_perf 'pack to file (bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n-'\n-\n-test_perf 'rev-list (commits)' '\n-\tgit rev-list --all --use-bitmap-index >/dev/null\n-'\n-\n-test_perf 'rev-list (objects)' '\n-\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with tag negated via --not --all (objects)' '\n-\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with negative tag (objects)' '\n-\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:none' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:none >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:limit=1k' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:limit=1k >/dev/null\n-'\n-\n-test_perf 'rev-list count with tree:0' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n-\n-test_perf 'simulated partial clone' '\n-\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n-'\n+test_full_bitmap\n \n test_expect_success 'create partial bitmap state' '\n \t# pick a commit to represent the repo tip in the past\n@@ -97,17 +49,6 @@ test_expect_success 'create partial bitmap state' '\n \tgit update-ref HEAD $orig_tip\n '\n \n-test_perf 'clone (partial bitmap)' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'pack to file (partial bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n-'\n-\n-test_perf 'rev-list with tree filter (partial bitmap)' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n+test_partial_bitmap\n \n test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"428142","messageId":"ec0f53b4249ff668106319473f49260ab8478314.1624314293.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"[PATCH v2 24/24] p5326: perf tests for MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-21T22:26:01Z","receivedAt":"2021-06-21T22:26:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These new performance tests demonstrate effectively the same behavior as\np5310, but use a multi-pack bitmap instead of a single-pack one.\n\nNotably, p5326 does not create a MIDX bitmap with multiple packs. This\nis so we can measure a direct comparison between it and p5310. Any\ndifference between the two is measuring just the overhead of using MIDX\nbitmaps.\n\nHere are the results of p5310 and p5326 together, measured at the same\ntime and on the same machine (using a Xenon W-2255 CPU):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5310.2: repack to disk 96.78(93.39+11.33)\n 5310.3: simulated clone 9.98(9.79+0.19)\n 5310.4: simulated fetch 1.75(4.26+0.19)\n 5310.5: pack to file (bitmap) 28.20(27.87+8.70)\n 5310.6: rev-list (commits) 0.41(0.36+0.05)\n 5310.7: rev-list (objects) 1.61(1.54+0.07)\n 5310.8: rev-list count with blob:none 0.25(0.21+0.04)\n 5310.9: rev-list count with blob:limit=1k 2.65(2.54+0.10)\n 5310.10: rev-list count with tree:0 0.23(0.19+0.04)\n 5310.11: simulated partial clone 4.34(4.21+0.12)\n 5310.13: clone (partial bitmap) 11.05(12.21+0.48)\n 5310.14: pack to file (partial bitmap) 31.25(34.22+3.70)\n 5310.15: rev-list with tree filter (partial bitmap) 0.26(0.22+0.04)\n\nversus the same tests (this time using a multi-pack index):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5326.2: setup multi-pack index 78.99(75.29+11.58)\n 5326.3: simulated clone 11.78(11.56+0.22)\n 5326.4: simulated fetch 1.70(4.49+0.13)\n 5326.5: pack to file (bitmap) 28.02(27.72+8.76)\n 5326.6: rev-list (commits) 0.42(0.36+0.06)\n 5326.7: rev-list (objects) 1.65(1.58+0.06)\n 5326.8: rev-list count with blob:none 0.26(0.21+0.05)\n 5326.9: rev-list count with blob:limit=1k 2.97(2.86+0.10)\n 5326.10: rev-list count with tree:0 0.25(0.20+0.04)\n 5326.11: simulated partial clone 5.65(5.49+0.16)\n 5326.13: clone (partial bitmap) 12.22(13.43+0.38)\n 5326.14: pack to file (partial bitmap) 30.05(31.57+7.25)\n 5326.15: rev-list with tree filter (partial bitmap) 0.24(0.20+0.04)\n\nThere is slight overhead in \"simulated clone\", \"simulated partial\nclone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\ndue to using the MIDX's reverse index to map between bit positions and\nMIDX positions.\n\nThis can be reproduced by running \"git repack -adb\" along with \"git\nmulti-pack-index write --bitmap\" in a large-ish repository. Then run:\n\n $ perf record -o pack.perf git -c core.multiPackIndex=false \\\n pack-objects --all --stdout >/dev/null /dev/null \n---\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..5845109ac7\n--- /dev/null\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,43 @@\n+#!/bin/sh\n+\n+test_description='Tests performance using midx bitmaps'\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n+\n+test_perf_large_repo\n+\n+test_expect_success 'enable multi-pack index' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_perf 'setup multi-pack index' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap\n+'\n+\n+test_full_bitmap\n+\n+test_expect_success 'create partial bitmap state' '\n+\t# pick a commit to represent the repo tip in the past\n+\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t# now pretend we have just one tip\n+\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\tgit update-ref HEAD $cutoff &&\n+\n+\t# and then repack, which will leave us with a nice\n+\t# big bitmap pack of the \"old\" history, and all of\n+\t# the new history will be loose, as if it had been pushed\n+\t# up incrementally and exploded via unpack-objects\n+\tgit repack -Ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\n+\t# and now restore our original tip, as if the pushes\n+\t# had happened\n+\tgit update-ref HEAD $orig_tip\n+'\n+\n+test_partial_bitmap\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"428379","messageId":"YNSdzWSb8pZv1xWg@nand.local","threadId":"55464","inReplyTo":"ac1f46aa1f0dbc7fba45229555b11b390fe104a0.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 11/24] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-24T14:59:25Z","receivedAt":"2021-06-24T14:59:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:26PM -0400, Taylor Blau wrote:\n> static int load_bitmap_entries_v1(struct bitmap_index *index)\n> {\n> \tuint32_t i;\n> @@ -242,9 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n> \t\txor_offset = read_u8(index->map, &index->map_pos);\n> \t\tflags = read_u8(index->map, &index->map_pos);\n>\n> -\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n> -\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n> -\t\t\t\t (unsigned)commit_idx_pos);\n> +\t\tnth_bitmap_object_oid(index, &oid, commit_idx_pos);\n\nOops. I was reading code in this area and noticed that this patch drops\nthe check introduced by c6b0c3910c (pack-bitmap.c: check reads more\naggressively when loading, 2020-12-08).\n\nI fixed it up locally by restoring the check (but on the new function\nnth_bitmap_object_oid() instead), and will send it in a reroll.\n\nThanks,\nTaylor\n"},{"id":"428437","messageId":"87k0mizws5.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"a18baeb0b42994ebcb216df5fe69459ba9a33795.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 01/24] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T23:02:56Z","receivedAt":"2021-06-24T23:07:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> +\tenum object_type bitmap_type = OBJ_NONE;\n> +\tint bitmaps_nr = 0;\n> +\n> +\tif (bitmap_get(tdata->commits, pos)) {\n> +\t\tbitmap_type = OBJ_COMMIT;\n> +\t\tbitmaps_nr++;\n> +\t}\n> +\tif (bitmap_get(tdata->trees, pos)) {\n> +\t\tbitmap_type = OBJ_TREE;\n> +\t\tbitmaps_nr++;\n> +\t}\n> +\tif (bitmap_get(tdata->blobs, pos)) {\n> +\t\tbitmap_type = OBJ_BLOB;\n> +\t\tbitmaps_nr++;\n> +\t}\n> +\tif (bitmap_get(tdata->tags, pos)) {\n> +\t\tbitmap_type = OBJ_TAG;\n> +\t\tbitmaps_nr++;\n> +\t}\n\nThis made me wonder if this could be better with something like the\nHAS_MULTI_BITS() macro, but that trick probably can't be applied here in\nany shape or form :)\n\n> +\n> +\tif (!bitmap_type)\n> +\t\tdie(\"object %s not found in type bitmaps\",\n> +\t\t oid_to_hex(&obj->oid));\n\nIt feels a bit magical to use an enum and then assume to know the enum's\nvalues, I know we do \"type < 0\" all over the place, but I'd think \"if\n(bitmap_type == OBJ_NONE)\" would be better here....\n\n> +\n> +\tif (bitmaps_nr > 1)\n> +\t\tdie(\"object %s does not have a unique type\",\n> +\t\t oid_to_hex(&obj->oid));\n\nOr just check the bitmaps_nr instead:\n\n if (!bitmaps_nr)\n die(\"found none\");\n else if (bitmaps_nr > 1)\n ...;\n\nJust bikeshedding...\n\n> +\n> +\tif (bitmap_type != obj->type)\n> +\t\tdie(\"object %s: real type %s, expected: %s\",\n> +\t\t oid_to_hex(&obj->oid),\n> +\t\t type_name(obj->type),\n> +\t\t type_name(bitmap_type));\n\nTo argue against myself (sort of) about that \"== OBJ_NONE\" above, if\nwe're not assuming that then it's sort of weird not to also assume that\ntype_name(type) won't return a NULL in the case of OBJ_NONE, which it\ndoes (but this code guards against).\n"},{"id":"428440","messageId":"87eecqzvld.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"3e637d9ec83435540ad32b8325b0dce87f61bae0.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T23:23:40Z","receivedAt":"2021-06-24T23:33:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> -static uint32_t find_object_pos(const struct object_id *oid)\n> +static uint32_t find_object_pos(const struct object_id *oid, int *found)\n> {\n> \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n> \n> \tif (!entry) {\n> -\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> +\t\tif (found)\n> +\t\t\t*found = 0;\n> +\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n> +\t\treturn 0;\n> \t}\n> \n> +\tif (found)\n> +\t\t*found = 1;\n> \treturn oe_in_pack_pos(writer.to_pack, entry);\n> }\n> \n> @@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n> \tbb->commits_nr = bb->commits_alloc = 0;\n> }\n> \n> -static void fill_bitmap_tree(struct bitmap *bitmap,\n> -\t\t\t struct tree *tree)\n> +static int fill_bitmap_tree(struct bitmap *bitmap,\n> +\t\t\t struct tree *tree)\n> {\n> +\tint found;\n> \tuint32_t pos;\n> \tstruct tree_desc desc;\n> \tstruct name_entry entry;\n> @@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n> \t * If our bit is already set, then there is nothing to do. Both this\n> \t * tree and all of its children will be set.\n> \t */\n> -\tpos = find_object_pos(&tree->object.oid);\n> +\tpos = find_object_pos(&tree->object.oid, &found);\n> +\tif (!found)\n> +\t\treturn -1;\n\nSo, a function that returns an unsigned 32 bit int won't (presumably)\nhave enough space for an \"is bad\", but before it died so it didn't\nmatter.\n\nNow it warns, so it needs a \"is bad\", so we add another \"int\" to pass\nthat information around.\n\nSo if we're already paying for that extra space (which, on some\nplatforms would already be a 64 bit int, and on some so would the\nuint32_t, it's just \"at least 32 bits\").\n\nWouldn't it be more idiomatic to just have find_object_pos() return\nint64_t now, if it's -1 it's an error, otherwise the \"pos\" is cast to\nuint32_t:\n\t\n\tdiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n\tindex 88d9e696a54..d71fa6f607a 100644\n\t--- a/pack-bitmap-write.c\n\t+++ b/pack-bitmap-write.c\n\t@@ -125,14 +125,12 @@ static inline void push_bitmapped_commit(struct commit *commit)\n\t \twriter.selected_nr++;\n\t }\n\t \n\t-static uint32_t find_object_pos(const struct object_id *oid)\n\t+static int64_t find_object_pos(const struct object_id *oid)\n\t {\n\t \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n\t \n\t-\tif (!entry) {\n\t-\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n\t-\t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n\t-\t}\n\t+\tif (!entry)\n\t+\t\treturn -1;\n\t \n\t \treturn oe_in_pack_pos(writer.to_pack, entry);\n\t }\n\t@@ -334,7 +332,7 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n\t static void fill_bitmap_tree(struct bitmap *bitmap,\n\t \t\t\t struct tree *tree)\n\t {\n\t-\tuint32_t pos;\n\t+\tint64_t pos;\n\t \tstruct tree_desc desc;\n\t \tstruct name_entry entry;\n\t \n\t@@ -343,6 +341,9 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n\t \t * tree and all of its children will be set.\n\t \t */\n\t \tpos = find_object_pos(&tree->object.oid);\n\t+\tif (pos < 0)\n\t+\t\tdie(\"unhappy: %s\", oid_to_hex(&tree->object.oid));\n\t+\n\t \tif (bitmap_get(bitmap, pos))\n\t \t\treturn;\n\t \tbitmap_set(bitmap, pos);\n\nI mean, you don't want the die() part of that, but to me the rest looks\nbetter.\n\n> [...]\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 584a039b85..1667450917 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '\n> repack_and_check () {\n> \trm -rf repo2 &&\n> \tcp -r repo repo2 &&\n> -\tgit -C repo2 repack $1 -d &&\n> +\tif test x\"$1\" = \"x--must-fail\"\n> +\tthen\n> +\t\tshift\n> +\t\ttest_must_fail git -C repo2 repack $1 -d\n> +\telse\n> +\t\tgit -C repo2 repack $1 -d\n> +\tfi &&\n> \tgit -C repo2 fsck &&\n\nThis sent me down the rabbit hole of\nhttps://lore.kernel.org/git/60c67bdf5b77e_f569220858@natae.notmuch/\nagain.\n"},{"id":"428443","messageId":"87bl7uzvaf.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"b0bb2e8051f19ec47140fda6500e092e37c6bea8.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T23:35:48Z","receivedAt":"2021-06-24T23:40:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> Even though the 'TECH_DOCS' variable was introduced all the way back in\n> 5e00439f0a (Documentation: build html for all files in technical and\n> howto, 2012-10-23), the 'bitmap-format' document was never added to that\n> list when it was created.\n>\n> Prepare for changes to this file by including it in the list of\n> technical documentation that 'make doc' will build by default.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/Makefile | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/Documentation/Makefile b/Documentation/Makefile\n> index f5605b7767..7d7b778b28 100644\n> --- a/Documentation/Makefile\n> +++ b/Documentation/Makefile\n> @@ -90,6 +90,7 @@ SP_ARTICLES += $(API_DOCS)\n> TECH_DOCS += MyFirstContribution\n> TECH_DOCS += MyFirstObjectWalk\n> TECH_DOCS += SubmittingPatches\n> +TECH_DOCS += technical/bitmap-format\n> TECH_DOCS += technical/hash-function-transition\n> TECH_DOCS += technical/http-protocol\n> TECH_DOCS += technical/index-format\n\nAs a mostly aside I've got a local series queued up to move all of these\n\"format\" docs to e.g. gitformat-bitmap(5), i.e. to make them first-class\nmanpages, so other pages can link to them. Right now we mostly don't,\nbut when our manpages do they link to the generated HTML, which e.g. I\ndon't have installed by default.\n\nSo since you're linking to it: Does anyone prefer this state of a\naffairs, and isn't it mainly useful for built docs such as\nhttps://git-scm.com/docs/git-multi-pack-index.\n\nBut there's still (but maybe later in this series) a link to\nbitmap-format anywhere from another manual page (but there is for\ne.g. technical/pack-format.html).\n"},{"id":"428445","messageId":"878s2yzv65.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"b3a12424d78e80553741f5c7a0672490a59b6f7d.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 06/24] midx: make a number of functions non-static","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T23:42:06Z","receivedAt":"2021-06-24T23:42:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> These functions will be called from outside of midx.c in a subsequent\n> patch.\n\nSo \"a number\" is \"two\" and \"a subsequent patch\" appears to be 13/24. I\nthink this would be clearer just squashed into whatever needs it, or at\nleast if it comes right before the new use in the series.\n"},{"id":"428446","messageId":"875yy2zv4e.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"dfd1daacc5b12d470bb6deec3448cf7dbde2bf0f.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T23:43:05Z","receivedAt":"2021-06-24T23:43:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> When writing a new multi-pack index, write_midx_internal() attempts to\n> load any existing one to fill in some pieces of information. But it uses\n> load_multi_pack_index(), which ignores the configuration\n> \"core.multiPackIndex\", which indicates whether or not Git is allowed to\n> read an existing multi-pack-index.\n>\n> Replace this with a routine that does respect that setting, to avoid\n> reading multi-pack-index files when told not to.\n>\n> This avoids a problem that would arise in subsequent patches due to the\n> combination of 'git repack' reopening the object store in-process and\n> the multi-pack index code not checking whether a pack already exists in\n> the object store when calling add_pack_to_midx().\n>\n> This would ultimately lead to a cycle being created along the\n> 'packed_git' struct's '->next' pointer. That is obviously bad, but it\n> has hard-to-debug downstream effects like saying a bitmap can't be\n> loaded for a pack because one already exists (for the same pack).\n>\n> Signed-off-by: Taylor Blau \n> ---\n> midx.c | 14 ++++++++++++--\n> 1 file changed, 12 insertions(+), 2 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 40eb7974ba..759007d5a8 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -908,8 +908,18 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \n> \tif (m)\n> \t\tctx.m = m;\n> -\telse\n> -\t\tctx.m = load_multi_pack_index(object_dir, 1);\n> +\telse {\n\nStyle nit: leaves the initial \"if\" braceless now that the \"else\" gained\nbraces.\n"},{"id":"428447","messageId":"8735t6zucn.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"a8cec2463d0993b1118abdd31cb6c9e88a32e0c4.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T23:45:46Z","receivedAt":"2021-06-25T00:02:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> Write multi-pack bitmaps in the format described by\n> Documentation/technical/bitmap-format.txt, inferring their presence with\n> the absence of '--bitmap'.\n>\n> To write a multi-pack bitmap, this patch attempts to reuse as much of\n> the existing machinery from pack-objects as possible. Specifically, the\n> MIDX code prepares a packing_data struct that pretends as if a single\n> packfile has been generated containing all of the objects contained\n> within the MIDX.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-multi-pack-index.txt | 12 +-\n> builtin/multi-pack-index.c | 2 +\n> midx.c | 230 ++++++++++++++++++++++++-\n> midx.h | 1 +\n> 4 files changed, 236 insertions(+), 9 deletions(-)\n>\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index ffd601bc17..ada14deb2c 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -10,7 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n> -\t[--preferred-pack=] \n> +\t[--preferred-pack=] [--[no-]bitmap] \n> \n> DESCRIPTION\n> -----------\n> @@ -40,6 +40,9 @@ write::\n> \t\tmultiple packs contain the same object. If not given,\n> \t\tties are broken in favor of the pack with the lowest\n> \t\tmtime.\n> +\n> +\t--[no-]bitmap::\n> +\t\tControl whether or not a multi-pack bitmap is written.\n> --\n> \n> verify::\n> @@ -81,6 +84,13 @@ EXAMPLES\n> $ git multi-pack-index write\n> -----------------------------------------------\n> \n> +* Write a MIDX file for the packfiles in the current .git folder with a\n> +corresponding bitmap.\n> ++\n> +-------------------------------------------------------------\n> +$ git multi-pack-index write --preferred-pack --bitmap\n> +-------------------------------------------------------------\n> +\n\nI wondered if this was a positional argument, but it's just the\nargument for --preferred-pack, even though the synopsis uses the \"=\"\nstyle for it. Even if parse-options.c is loose about it, let's use one\nor the other in examples consistently.\n\n> * Write a MIDX file for the packfiles in an alternate object store.\n> +\n> -----------------------------------------------\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 5d3ea445fd..bf6fa982e3 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n> \t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n> \t\t\t N_(\"preferred-pack\"),\n> \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n> +\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n> +\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n> \t\tOPT_END(),\n> \t};\n> \n> diff --git a/midx.c b/midx.c\n> index 752d36c57f..a58cca707b 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -13,6 +13,10 @@\n> #include \"repository.h\"\n> #include \"chunk-format.h\"\n> #include \"pack.h\"\n> +#include \"pack-bitmap.h\"\n> +#include \"refs.h\"\n> +#include \"revision.h\"\n> +#include \"list-objects.h\"\n> \n> #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n> #define MIDX_VERSION 1\n> @@ -885,6 +889,172 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n> static void clear_midx_files_ext(struct repository *r, const char *ext,\n> \t\t\t\t unsigned char *keep_hash);\n> \n> +static void prepare_midx_packing_data(struct packing_data *pdata,\n> +\t\t\t\t struct write_midx_context *ctx)\n> +{\n> +\tuint32_t i;\n> +\n> +\tmemset(pdata, 0, sizeof(struct packing_data));\n\nWe initialize this on the stack in write_midx_bitmap(), shouldn't we\njust there do:\n\n struct packing_data pdata = {0}\n\nInstead of:\n\n struct packing_data pdata;\n\nAnd then doing this memset() here?\n\n> +\tprepare_packing_data(the_repository, pdata);\n> +\n> +\tfor (i = 0; i < ctx->entries_nr; i++) {\n> +\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n> +\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n> +\n> +\t\toe_set_in_pack(pdata, to,\n> +\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n> +\t}\n> +}\n> +\n> +static int add_ref_to_pending(const char *refname,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t int flag, void *cb_data)\n> +{\n> +\tstruct rev_info *revs = (struct rev_info*)cb_data;\n> +\tstruct object *object;\n> +\n> +\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n\nJust since I'd mentioned HAS_MULTI_BITS() offhand on another patch of\nyours, it's for cases like this, so:\n\n- if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+ if (HAS_MULTI_BITS(flag & (REF_ISSYMREF|REF_ISBROKEN)) {\n\nSaves you 3 bytes of code:) Anyway, you don't need to use it, just an\nintresting function... :)\n\n> +{\n> +\tstruct rev_info revs;\n> +\tstruct bitmap_commit_cb cb;\n> +\n> +\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n\nAnother case of s/memset/\"= {0}\"/g ?\n\n> +\tcb.ctx = ctx;\n> +\n> +\trepo_init_revisions(the_repository, &revs, NULL);\n> +\tfor_each_ref(add_ref_to_pending, &revs);\n> +\n> +\t/*\n> +\t * Skipping promisor objects here is intentional, since it only excludes\n> +\t * them from the list of reachable commits that we want to select from\n> +\t * when computing the selection of MIDX'd commits to receive bitmaps.\n> +\t *\n> +\t * Reachability bitmaps do require that their objects be closed under\n> +\t * reachability, but fetching any objects missing from promisors at this\n> +\t * point is too late. But, if one of those objects can be reached from\n> +\t * an another object that is included in the bitmap, then we will\n> +\t * complain later that we don't have reachability closure (and fail\n> +\t * appropriately).\n> +\t */\n> +\tfetch_if_missing = 0;\n> +\trevs.exclude_promisor_objects = 1;\n> +\n> +\t/*\n> +\t * Pass selected commits in topo order to match the behavior of\n> +\t * pack-bitmaps when configured with delta islands.\n> +\t */\n> +\trevs.topo_order = 1;\n> +\trevs.sort_order = REV_SORT_IN_GRAPH_ORDER;\n> +\n> +\tif (prepare_revision_walk(&revs))\n> +\t\tdie(_(\"revision walk setup failed\"));\n> +\n> +\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n> +\tif (indexed_commits_nr_p)\n> +\t\t*indexed_commits_nr_p = cb.commits_nr;\n> +\n> +\treturn cb.commits;\n> +}\n> +\n> +static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n> +\t\t\t struct write_midx_context *ctx,\n> +\t\t\t unsigned flags)\n> +{\n> +\tstruct packing_data pdata;\n> +\tstruct pack_idx_entry **index;\n> +\tstruct commit **commits = NULL;\n> +\tuint32_t i, commits_nr;\n> +\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n> +\tint ret;\n> +\n> +\tprepare_midx_packing_data(&pdata, ctx);\n> +\n> +\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n> +\n> +\t/*\n> +\t * Build the MIDX-order index based on pdata.objects (which is already\n> +\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n> +\t * this order).\n> +\t */\n> +\tALLOC_ARRAY(index, pdata.nr_objects);\n> +\tfor (i = 0; i < pdata.nr_objects; i++)\n> +\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n> +\n> +\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n> +\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n> +\n> +\t/*\n> +\t * bitmap_writer_finish expects objects in lex order, but pack_order\n> +\t * gives us exactly that. use it directly instead of re-sorting the\n> +\t * array.\n> +\t *\n> +\t * This changes the order of objects in 'index' between\n> +\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n> +\t *\n> +\t * The same re-ordering takes place in the single-pack bitmap code via\n> +\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n> +\t * happens between bitmap_writer_build_type_index() and\n> +\t * bitmap_writer_finish().\n> +\t */\n> +\tfor (i = 0; i < pdata.nr_objects; i++)\n> +\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n> +\n> +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n> +\tret = bitmap_writer_build(&pdata);\n> +\tif (ret < 0)\n> +\t\tgoto cleanup;\n> +\n> +\tbitmap_writer_set_checksum(midx_hash);\n> +\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n> +\n> +cleanup:\n> +\tfree(index);\n> +\tfree(bitmap_name);\n> +\treturn ret;\n> +}\n> +\n> static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n> \t\t\t struct string_list *packs_to_drop,\n> \t\t\t const char *preferred_pack_name,\n> @@ -930,9 +1100,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> \n> +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> +\t\t\t\terror(_(\"could not load pack %s\"),\n> +\t\t\t\t ctx.m->pack_names[i]);\n\nIsn't the prepare_midx_pack() tasked with populating that pack_names[i]\nthat you can't load (the strbuf_addf() it does), but it can also exit\nbefore that, do we get an empty string here then? Maybe I'm misreading\nit (I haven't run this, just skimmed the code).\n\n> @@ -1132,6 +1342,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \tfree(ctx.pack_perm);\n> \tfree(ctx.pack_order);\n> \tfree(midx_name);\n> +\tif (ctx.m)\n> +\t\tclose_midx(ctx.m);\n> +\n\nI see Stolee made close_midx() just return silently if !ctx.m in\n1dcd9f2043a (midx: close multi-pack-index on repack, 2018-10-12), but\ngrepping the uses of it it seems calls to it are similarly guarded by\n\"if\"'s.\n\nJust a nit, weird to have a free-like function not invoked like\nfree. Perhaps (and maybe better for an unrelated cleanup) to either drop\nthe conditionals, or make it BUG() if it's called with NULL, but at\nleast we should pick one :)\n"},{"id":"428448","messageId":"87zgveyfk2.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"00418d5b096c049ddfc738b5d51ef65eae019607.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 22/24] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-25T00:03:24Z","receivedAt":"2021-06-25T00:05:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment\n> variable to also write a multi-pack bitmap when\n> 'GIT_TEST_MULTI_PACK_INDEX' is set.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/repack.c | 13 ++++++++++---\n> ci/run-build-and-tests.sh | 1 +\n> midx.h | 2 ++\n> t/README | 4 ++++\n> 4 files changed, 17 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 5f9bc74adc..77f6f03057 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -515,7 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n> \t\t !is_bare_repository())\n> \t\t\twrite_bitmaps = 0;\n> -\t}\n> +\t} else if (write_bitmaps &&\n> +\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n> +\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n> +\t\twrite_bitmaps = 0;\n\nStyle nit: more if/else/elseif some with braces, some not...\n"},{"id":"428459","messageId":"87lf6yxqeh.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"cover.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/24] multi-pack reachability bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-25T09:06:21Z","receivedAt":"2021-06-25T09:08:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, Taylor Blau wrote:\n\n> Thanks in advance for your review, and sorry for the wait.\n\nThanks for working on this, exciting feature!\n\nJust a note on my comments on this. I left them after some light reading\nand would describe them as some combination of \"musings\", \"shallow\",\n\"nit-y\" and \"bikesheddy\".\n\nI.e. I did not have time (or I feel, the familiarity) to give this\nseries the sort of review it actually deserves as far as the actual\nimportant bits go, i.e. nits aside whether this feature works and\nbehaves as desired. Sorry, but hopefully at least some of comments were\nsomewhat useful anyway.\n"},{"id":"430087","messageId":"YO8dwitqG/pw2Mqn@nand.local","threadId":"55464","inReplyTo":"87k0mizws5.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 01/24] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-14T17:24:18Z","receivedAt":"2021-07-14T17:24:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 25, 2021 at 01:02:56AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Jun 21 2021, Taylor Blau wrote:\n>\n> > +\tenum object_type bitmap_type = OBJ_NONE;\n> > +\tint bitmaps_nr = 0;\n> > +\n> > +\tif (bitmap_get(tdata->commits, pos)) {\n> > +\t\tbitmap_type = OBJ_COMMIT;\n> > +\t\tbitmaps_nr++;\n> > +\t}\n> > +\tif (bitmap_get(tdata->trees, pos)) {\n> > +\t\tbitmap_type = OBJ_TREE;\n> > +\t\tbitmaps_nr++;\n> > +\t}\n> > +\tif (bitmap_get(tdata->blobs, pos)) {\n> > +\t\tbitmap_type = OBJ_BLOB;\n> > +\t\tbitmaps_nr++;\n> > +\t}\n> > +\tif (bitmap_get(tdata->tags, pos)) {\n> > +\t\tbitmap_type = OBJ_TAG;\n> > +\t\tbitmaps_nr++;\n> > +\t}\n>\n> This made me wonder if this could be better with something like the\n> HAS_MULTI_BITS() macro, but that trick probably can't be applied here in\n> any shape or form :)\n\nRight; since we're looking at the same bit position in each of the\ntype-level bitmaps, we can't just OR them together, since all of the\nbits are in the same place.\n\nAnd really, the object_type enum doesn't have values that tell us the\ntype of an object by looking at just a single bit. So\nHAS_MULTI_BITS(OBJ_BLOB) would return \"true\", since OBJ_BLOB is 3.\n\n> > +\n> > +\tif (bitmap_type != obj->type)\n> > +\t\tdie(\"object %s: real type %s, expected: %s\",\n> > +\t\t oid_to_hex(&obj->oid),\n> > +\t\t type_name(obj->type),\n> > +\t\t type_name(bitmap_type));\n>\n> To argue against myself (sort of) about that \"== OBJ_NONE\" above, if\n> we're not assuming that then it's sort of weird not to also assume that\n> type_name(type) won't return a NULL in the case of OBJ_NONE, which it\n> does (but this code guards against).\n\nI tend to agree. To restate what you're saying: by the time we get to\nthe type_name(bitmap_type) we know that bitmap_type is non-zero, so we\nassume it's OK to call type_name() on it.\n\nOf course, the object_type_strings does handle the zero argument, so\nthis is probably a little academic, but good to think through\nnonetheless.\n\nThanks,\nTaylor\n"},{"id":"430091","messageId":"YO8fpVZhoCiEiurR@nand.local","threadId":"55464","inReplyTo":"87eecqzvld.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-14T17:32:21Z","receivedAt":"2021-07-14T17:32:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 25, 2021 at 01:23:40AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Jun 21 2021, Taylor Blau wrote:\n>\n> > -static uint32_t find_object_pos(const struct object_id *oid)\n> > +static uint32_t find_object_pos(const struct object_id *oid, int *found)\n> > {\n> > \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n> >\n> > \tif (!entry) {\n> > -\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> > +\t\tif (found)\n> > +\t\t\t*found = 0;\n> > +\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n> > \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n> > +\t\treturn 0;\n> > \t}\n> >\n> > +\tif (found)\n> > +\t\t*found = 1;\n> > \treturn oe_in_pack_pos(writer.to_pack, entry);\n> > }\n>\n> So, a function that returns an unsigned 32 bit int won't (presumably)\n> have enough space for an \"is bad\", but before it died so it didn't\n> matter.\n>\n> Now it warns, so it needs a \"is bad\", so we add another \"int\" to pass\n> that information around.\n\nRight. You could imagine using the most-significant bit to indicate\n\"bad\" (which in this case is \"I couldn't find this object that I'm\nsupposed to be able to reach\"), but of course it cuts our maximum number\nof objects in a bitmap in half.\n\n> So if we're already paying for that extra space (which, on some\n> platforms would already be a 64 bit int, and on some so would the\n> uint32_t, it's just \"at least 32 bits\").\n>\n> Wouldn't it be more idiomatic to just have find_object_pos() return\n> int64_t now, if it's -1 it's an error, otherwise the \"pos\" is cast to\n> uint32_t:\n\nI'm not sure. It does save the extra argument, which is arguably more\nconvenient for callers, but the cost for doing so is a cast from a\nsigned integer type to an unsigned one (and a narrower destination type,\nat that).\n\nThat seems easier to get wrong to me than passing a pointer to a pure\n\"int\" and keeping the return type a uint32_t. So, I'm probably more\ncontent to leave it as-is rather than change it.\n\nI don't feel too strongly about it, though, so if you do I'd be happy to\nhear more.\n\nThanks,\nTaylor\n"},{"id":"430094","messageId":"YO8h0smhBPuIjFAa@nand.local","threadId":"55464","inReplyTo":"87bl7uzvaf.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-14T17:41:38Z","receivedAt":"2021-07-14T17:41:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 25, 2021 at 01:35:48AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> As a mostly aside I've got a local series queued up to move all of these\n> \"format\" docs to e.g. gitformat-bitmap(5), i.e. to make them first-class\n> manpages, so other pages can link to them. Right now we mostly don't,\n> but when our manpages do they link to the generated HTML, which e.g. I\n> don't have installed by default.\n\nIt would be nice to be able to look it up with \"man 5 gitformat-bitmap\".\nI actually don't have strong feelings about this particular patch\ngetting picked up or not, since it doesn't add the actual format changes\nto the file itself.\n\nThis does pick up the bitmap-format document in \"make -C Documentation\nhtml\", which is nice(r than \"make -C Documentation\ntechnical/bitmap-format.html\" IMHO).\n\n> But there's still (but maybe later in this series) a link to\n> bitmap-format anywhere from another manual page (but there is for\n> e.g. technical/pack-format.html).\n\nNo, I didn't add any links pointed at bitmap-format.\n\nThanks,\nTaylor\n"},{"id":"430101","messageId":"874kcw20nq.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"YO8fpVZhoCiEiurR@nand.local","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T18:44:27Z","receivedAt":"2021-07-14T18:47:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 14 2021, Taylor Blau wrote:\n\n> On Fri, Jun 25, 2021 at 01:23:40AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Mon, Jun 21 2021, Taylor Blau wrote:\n>>\n>> > -static uint32_t find_object_pos(const struct object_id *oid)\n>> > +static uint32_t find_object_pos(const struct object_id *oid, int *found)\n>> > {\n>> > \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n>> >\n>> > \tif (!entry) {\n>> > -\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n>> > +\t\tif (found)\n>> > +\t\t\t*found = 0;\n>> > +\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n>> > \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n>> > +\t\treturn 0;\n>> > \t}\n>> >\n>> > +\tif (found)\n>> > +\t\t*found = 1;\n>> > \treturn oe_in_pack_pos(writer.to_pack, entry);\n>> > }\n>>\n>> So, a function that returns an unsigned 32 bit int won't (presumably)\n>> have enough space for an \"is bad\", but before it died so it didn't\n>> matter.\n>>\n>> Now it warns, so it needs a \"is bad\", so we add another \"int\" to pass\n>> that information around.\n>\n> Right. You could imagine using the most-significant bit to indicate\n> \"bad\" (which in this case is \"I couldn't find this object that I'm\n> supposed to be able to reach\"), but of course it cuts our maximum number\n> of objects in a bitmap in half.\n>\n>> So if we're already paying for that extra space (which, on some\n>> platforms would already be a 64 bit int, and on some so would the\n>> uint32_t, it's just \"at least 32 bits\").\n>>\n>> Wouldn't it be more idiomatic to just have find_object_pos() return\n>> int64_t now, if it's -1 it's an error, otherwise the \"pos\" is cast to\n>> uint32_t:\n>\n> I'm not sure. It does save the extra argument, which is arguably more\n> convenient for callers, but the cost for doing so is a cast from a\n> signed integer type to an unsigned one (and a narrower destination type,\n> at that).\n>\n> That seems easier to get wrong to me than passing a pointer to a pure\n> \"int\" and keeping the return type a uint32_t. So, I'm probably more\n> content to leave it as-is rather than change it.\n>\n> I don't feel too strongly about it, though, so if you do I'd be happy to\n> hear more.\n\nI don't really care, it just looked a bit weird at first, and I wondered\nwhy it couldn't return -1.\n\nAside from this case do you mean that such a cast would be too expensive\nin general, or fears abou going past the 32 bits? I assumed that there\nwould be checks here for that already (and if not, we'd have wrap-around\nnow...).\n"},{"id":"430155","messageId":"87eec0zej2.fsf@evledraar.gmail.com","threadId":"55464","inReplyTo":"YO8h0smhBPuIjFAa@nand.local","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-14T22:58:00Z","receivedAt":"2021-07-14T23:01:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 14 2021, Taylor Blau wrote:\n\n> On Fri, Jun 25, 2021 at 01:35:48AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> As a mostly aside I've got a local series queued up to move all of these\n>> \"format\" docs to e.g. gitformat-bitmap(5), i.e. to make them first-class\n>> manpages, so other pages can link to them. Right now we mostly don't,\n>> but when our manpages do they link to the generated HTML, which e.g. I\n>> don't have installed by default.\n>\n> It would be nice to be able to look it up with \"man 5 gitformat-bitmap\".\n> I actually don't have strong feelings about this particular patch\n> getting picked up or not, since it doesn't add the actual format changes\n> to the file itself.\n>\n> This does pick up the bitmap-format document in \"make -C Documentation\n> html\", which is nice(r than \"make -C Documentation\n> technical/bitmap-format.html\" IMHO).\n\nOh yes, I'm not saying don't add the target. Just a musing on how we\nended up with such a large set of things in \"Documentation/technical/*\"\nas opposed to just man pages.\n\nI guess if there's good reasons for it they'll come out if/when I submit\nthat series...\n\n>> But there's still (but maybe later in this series) a link to\n>> bitmap-format anywhere from another manual page (but there is for\n>> e.g. technical/pack-format.html).\n>\n> No, I didn't add any links pointed at bitmap-format.\n\nI see https://git-scm.com/docs/bitmap-format has somehow managed to get\nindexed by Google, perhaps through some magic :)\n"},{"id":"430156","messageId":"YO9s3NM8CYtbbo82@nand.local","threadId":"55464","inReplyTo":"878s2yzv65.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 06/24] midx: make a number of functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-14T23:01:48Z","receivedAt":"2021-07-14T23:01:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 25, 2021 at 01:42:06AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Jun 21 2021, Taylor Blau wrote:\n>\n> > These functions will be called from outside of midx.c in a subsequent\n> > patch.\n>\n> So \"a number\" is \"two\" and \"a subsequent patch\" appears to be 13/24. I\n> think this would be clearer just squashed into whatever needs it, or at\n> least if it comes right before the new use in the series.\n\nGood suggestion, thanks. This was probably written at a time when the\nnumber of functions I needed was larger (or perhaps when this and\npatches 10-12 were all jumbled together).\n\nIn any case, I dropped this patch and squashed its contents into 13/24\nwhere it is used.\n\nThanks,\nTaylor\n"},{"id":"430203","messageId":"YPBHSWaACJcQRy2K@nand.local","threadId":"55464","inReplyTo":"8735t6zucn.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-15T14:33:45Z","receivedAt":"2021-07-15T14:33:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 25, 2021 at 01:45:46AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Jun 21 2021, Taylor Blau wrote:\n> > +* Write a MIDX file for the packfiles in the current .git folder with a\n> > +corresponding bitmap.\n> > ++\n> > +-------------------------------------------------------------\n> > +$ git multi-pack-index write --preferred-pack --bitmap\n> > +-------------------------------------------------------------\n> > +\n>\n> I wondered if this was a positional argument, but it's just the\n> argument for --preferred-pack, even though the synopsis uses the \"=\"\n> style for it. Even if parse-options.c is loose about it, let's use one\n> or the other in examples consistently.\n\nThe example below (for writing a MIDX in an alternate object store)\ndoesn't include the '=', but probably would be clearer if it did. I\nthink it's a good suggestion, though, so I'll fix up my addition here.\n\n> > +\tmemset(pdata, 0, sizeof(struct packing_data));\n>\n> We initialize this on the stack in write_midx_bitmap(), shouldn't we\n> just there do:\n>\n> struct packing_data pdata = {0}\n>\n> Instead of:\n>\n> struct packing_data pdata;\n>\n> And then doing this memset() here?\n\nI could go either way. Part of me prefers the memset() since it lets\ncallers of prepare_midx_packing_data() pass in anything they want,\nincluding a pointer to uninitialized memory. Of course, there is only\none such caller, so it probably doesn't really matter.\n\nAnd the other caller of prepare_packing_data() which is in\nbuiltin/pack-objects.c operates on a pointer to a statically allocated\nvariable, so its bytes are already zero'd.\n\nI don't feel strongly about it, though, so I'd just as soon err on the\nside of flexibility than changing the declaration.\n\n> > +{\n> > +\tstruct rev_info revs;\n> > +\tstruct bitmap_commit_cb cb;\n> > +\n> > +\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n>\n> Another case of s/memset/\"= {0}\"/g ?\n\nAh, in this case I'd prefer the aggregate-style initialization, since\nwe're zero-ing it out in the same function.\n\n> > static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n> > \t\t\t struct string_list *packs_to_drop,\n> > \t\t\t const char *preferred_pack_name,\n> > @@ -930,9 +1100,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> > \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> >\n> > +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> > +\t\t\t\terror(_(\"could not load pack %s\"),\n> > +\t\t\t\t ctx.m->pack_names[i]);\n>\n> Isn't the prepare_midx_pack() tasked with populating that pack_names[i]\n> that you can't load (the strbuf_addf() it does), but it can also exit\n> before that, do we get an empty string here then? Maybe I'm misreading\n> it (I haven't run this, just skimmed the code).\n\nNice catch, we can't rely on ctx->m.pack_names[i] being safe to read\n(and at the same time know that we're going to get a non-empty string).\n\nSince prepare_midx_pack() can fail because the pack itself couldn't be\nloaded, I think the easiest thing to do here is to just opaquely say\n\"could not load pack\" without adding any pack name that we may or may\nnot have.\n\n> > @@ -1132,6 +1342,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \tfree(ctx.pack_perm);\n> > \tfree(ctx.pack_order);\n> > \tfree(midx_name);\n> > +\tif (ctx.m)\n> > +\t\tclose_midx(ctx.m);\n> > +\n>\n> I see Stolee made close_midx() just return silently if !ctx.m in\n> 1dcd9f2043a (midx: close multi-pack-index on repack, 2018-10-12), but\n> grepping the uses of it it seems calls to it are similarly guarded by\n> \"if\"'s.\n>\n> Just a nit, weird to have a free-like function not invoked like\n> free. Perhaps (and maybe better for an unrelated cleanup) to either drop\n> the conditionals, or make it BUG() if it's called with NULL, but at\n> least we should pick one :)\n\nI agree with the direction of 1dcd9f2043a, so I'm happy to just drop the\nconditional and call close_midx() with an argument that may or may not\nbe NULL.\n\nThanks,\nTaylor\n"},{"id":"430204","messageId":"YPBIBXUB53b0Hixx@nand.local","threadId":"55464","inReplyTo":"87lf6yxqeh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 00/24] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-15T14:36:53Z","receivedAt":"2021-07-15T14:36:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 25, 2021 at 11:06:21AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Jun 21 2021, Taylor Blau wrote:\n>\n> > Thanks in advance for your review, and sorry for the wait.\n>\n> Thanks for working on this, exciting feature!\n>\n> Just a note on my comments on this. I left them after some light reading\n> and would describe them as some combination of \"musings\", \"shallow\",\n> \"nit-y\" and \"bikesheddy\".\n\nThanks for your review. It did help me catch some important issues, like\nreferring to ctx->m.pack_names[i] when a read at \"i\" may have been\ninvalid.\n\n> I.e. I did not have time (or I feel, the familiarity) to give this\n> series the sort of review it actually deserves as far as the actual\n> important bits go, i.e. nits aside whether this feature works and\n> behaves as desired. Sorry, but hopefully at least some of comments were\n> somewhat useful anyway.\n\nThey were useful indeed. I'll sit on the changes locally since most of\nthem are pretty benign and I think subsequent review could be done on\ntop of v2 without seeing my local changes.\n\nI know that reviewing this is on Peff's list of things to do, but there\nare competing priorities (and we have all-company meetings this week, so\nI would not be surprised to see a lack of movement until at least next\nweek).\n\nThanks,\nTaylor\n"},{"id":"430747","messageId":"YPft87yCjR9e+93E@coredump.intra.peff.net","threadId":"55464","inReplyTo":"3e637d9ec83435540ad32b8325b0dce87f61bae0.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T09:50:43Z","receivedAt":"2021-07-21T10:04:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:01PM -0400, Taylor Blau wrote:\n\n> The set of objects covered by a bitmap must be closed under\n> reachability, since it must be the case that there is a valid bit\n> position assigned for every possible reachable object (otherwise the\n> bitmaps would be incomplete).\n> \n> Pack bitmaps are never written from 'git repack' unless repacking\n> all-into-one, and so we never write non-closed bitmaps (except in the\n> case of partial clones where we aren't guaranteed to have all objects).\n> \n> But multi-pack bitmaps change this, since it isn't known whether the\n> set of objects in the MIDX is closed under reachability until walking\n> them. Plumb through a bit that is set when a reachable object isn't\n> found.\n> \n> As soon as a reachable object isn't found in the set of objects to\n> include in the bitmap, bitmap_writer_build() knows that the set is not\n> closed, and so it now fails gracefully.\n\nLeaving aside your intended use here, I think it's nice to get rid of a\ndeep-buried die() like this in general.\n\nThe amount of error-plumbing you had to do is a little unpleasant, but I\nthink is unavoidable. The only non-obvious part was this hunk:\n\n> @@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \t\tstruct commit *child;\n> \t\tint reused = 0;\n> \n> -\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n> -\t\t\t\t old_bitmap, mapping);\n> +\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n> +\t\t\t\t old_bitmap, mapping) < 0) {\n> +\t\t\tclosed = 0;\n> +\t\t\tbreak;\n> +\t\t}\n> \n> \t\tif (ent->selected) {\n> \t\t\tstore_selected(ent, commit);\n\nThis is the right thing to do because we still want to free memory, stop\nprogress, etc. I gave a look over what will run after breaking out of\nthe loop, and compute_xor_offsets(), which you already handled, is the\nonly thing we'd want to avoid running. Good.\n\n-Peff\n"},{"id":"430748","messageId":"YPfsqJrCTFQXhltJ@coredump.intra.peff.net","threadId":"55464","inReplyTo":"a18baeb0b42994ebcb216df5fe69459ba9a33795.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 01/24] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T09:45:12Z","receivedAt":"2021-07-21T10:04:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:24:59PM -0400, Taylor Blau wrote:\n\n> The special `--test-bitmap` mode of `git rev-list` is used to compare\n> the result of an object traversal with a bitmap to check its integrity.\n> This mode does not, however, assert that the types of reachable objects\n> are stored correctly.\n> \n> Harden this mode by teaching it to also check that each time an object's\n> bit is marked, the corresponding bit should be set in exactly one of the\n> type bitmaps (whose type matches the object's true type).\n\nYep, makes sense, and the patch looks good.\n\n> +{\n> +\tenum object_type bitmap_type = OBJ_NONE;\n> [...]\n> +\n> +\tif (!bitmap_type)\n> +\t\tdie(\"object %s not found in type bitmaps\",\n> +\t\t oid_to_hex(&obj->oid));\n\nI think the suggestion to do:\n\n if (bitmap_type == OBJ_NONE)\n\nis reasonable here, as it assumes less about the enum. I do think\nOBJ_BAD and OBJ_NONE were chosen with these kind of numeric comparisons\nin mind, but there is no reason to rely on them in places we don't need\nto.\n\n-Peff\n"},{"id":"430749","messageId":"YPfuo1GY2WSqHn8j@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YO8fpVZhoCiEiurR@nand.local","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T09:53:39Z","receivedAt":"2021-07-21T10:04:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 14, 2021 at 01:32:21PM -0400, Taylor Blau wrote:\n\n> > So if we're already paying for that extra space (which, on some\n> > platforms would already be a 64 bit int, and on some so would the\n> > uint32_t, it's just \"at least 32 bits\").\n> >\n> > Wouldn't it be more idiomatic to just have find_object_pos() return\n> > int64_t now, if it's -1 it's an error, otherwise the \"pos\" is cast to\n> > uint32_t:\n> \n> I'm not sure. It does save the extra argument, which is arguably more\n> convenient for callers, but the cost for doing so is a cast from a\n> signed integer type to an unsigned one (and a narrower destination type,\n> at that).\n> \n> That seems easier to get wrong to me than passing a pointer to a pure\n> \"int\" and keeping the return type a uint32_t. So, I'm probably more\n> content to leave it as-is rather than change it.\n\nI agree that the separate \"found\" value makes things more obvious.\nCasting to a smaller size means explaining why that is OK, whereas it is\nquite clear that things are correct with two separate variables. And\nhaving an int64_t makes one wonder whether a value like 2^35 is possible\n(it isn't; this is a uint32_t because that is the limit of objects in\nthe pack format).\n\n-Peff\n"},{"id":"430750","messageId":"YPfu6rdpaisTEcRh@coredump.intra.peff.net","threadId":"55464","inReplyTo":"490d733d121e206ccdc335812c03f31c380bcd86.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 03/24] pack-bitmap-write.c: free existing bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T09:54:50Z","receivedAt":"2021-07-21T10:04:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:04PM -0400, Taylor Blau wrote:\n\n> When writing a new bitmap, the bitmap writer code attempts to read the\n> existing bitmap (if one is present). This is done in order to quickly\n> permute the bits of any bitmaps for commits which appear in the existing\n> bitmap, and were also selected for the new bitmap.\n> \n> But since this code was added in 341fa34887 (pack-bitmap-write: use\n> existing bitmaps, 2020-12-08), the resources associated with opening an\n> existing bitmap were never released.\n> \n> It's fine to ignore this, but it's bad hygiene. It will also cause a\n> problem for the multi-pack-index builtin, which will be responsible not\n> only for writing bitmaps, but also for expiring any old multi-pack\n> bitmaps.\n> \n> If an existing bitmap was reused here, it will also be expired. That\n> will cause a problem on platforms which require file resources to be\n> closed before unlinking them, like Windows. Avoid this by ensuring we\n> close reused bitmaps with free_bitmap_index() before removing them.\n\nI agree with all of that. But just \"it's a memory leak\" would have\ncontented me, too. :)\n\n-Peff\n"},{"id":"430751","messageId":"YPfv0YoLtpYp9866@coredump.intra.peff.net","threadId":"55464","inReplyTo":"b0bb2e8051f19ec47140fda6500e092e37c6bea8.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T09:58:41Z","receivedAt":"2021-07-21T10:10:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:07PM -0400, Taylor Blau wrote:\n\n> Even though the 'TECH_DOCS' variable was introduced all the way back in\n> 5e00439f0a (Documentation: build html for all files in technical and\n> howto, 2012-10-23), the 'bitmap-format' document was never added to that\n> list when it was created.\n> \n> Prepare for changes to this file by including it in the list of\n> technical documentation that 'make doc' will build by default.\n\nOK. I don't care that much about being able to format this as html, but\nI agree it's good to be consistent with the other stuff in technical/.\n\nThe big question is whether it looks OK rendered by asciidoc, and the\nanswer seems to be \"yes\" (from a cursory look I gave it).\n\n-Peff\n"},{"id":"430752","messageId":"YPfyEiXw7szt5mjl@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPfv0YoLtpYp9866@coredump.intra.peff.net","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:08:18Z","receivedAt":"2021-07-21T11:24:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 05:58:41AM -0400, Jeff King wrote:\n\n> On Mon, Jun 21, 2021 at 06:25:07PM -0400, Taylor Blau wrote:\n> \n> > Even though the 'TECH_DOCS' variable was introduced all the way back in\n> > 5e00439f0a (Documentation: build html for all files in technical and\n> > howto, 2012-10-23), the 'bitmap-format' document was never added to that\n> > list when it was created.\n> > \n> > Prepare for changes to this file by including it in the list of\n> > technical documentation that 'make doc' will build by default.\n> \n> OK. I don't care that much about being able to format this as html, but\n> I agree it's good to be consistent with the other stuff in technical/.\n> \n> The big question is whether it looks OK rendered by asciidoc, and the\n> answer seems to be \"yes\" (from a cursory look I gave it).\n\nActually, I take it back. After looking more carefully, it renders quite\npoorly. There's a lot of structural indentation that ends up being\nconfused as code blocks.\n\nI don't know if it's better to have a poorly-formatted HTML file, or\nnone at all. :)\n\nPersonally, I would just read the source. And I have a slight concern\nthat if we start \"cleaning it up\" to render as asciidoc, the source\nmight end up a lot less readable (though I'd reserve judgement until\nactually seeing it).\n\n-Peff\n"},{"id":"430753","messageId":"YPfyqKdzB6HV5onf@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPfxH9RW6QGjZlQv@coredump.intra.peff.net","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:10:48Z","receivedAt":"2021-07-21T11:24:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 06:04:15AM -0400, Jeff King wrote:\n\n> The git-scm.com site doesn't use our Documentation/Makefile, so it has\n> been building the bitmap-format page for a while. It doesn't look\n> correct, though (it is missing everything before the \"on-disk format\"\n> section).\n\nDoh, nevermind. That nice header section does not exist before Taylor's\nseries. ;)\n\nThe git-scm.com page does seem to throw away the \"GIT bitmap v1 format\"\ntitle entirely. And it suffers the same rendering problems I saw when\nbuilding with asciidoc locally.\n\n-Peff\n"},{"id":"430754","messageId":"YPfxH9RW6QGjZlQv@coredump.intra.peff.net","threadId":"55464","inReplyTo":"87eec0zej2.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:04:15Z","receivedAt":"2021-07-21T11:27:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 15, 2021 at 12:58:00AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> >> But there's still (but maybe later in this series) a link to\n> >> bitmap-format anywhere from another manual page (but there is for\n> >> e.g. technical/pack-format.html).\n> >\n> > No, I didn't add any links pointed at bitmap-format.\n> \n> I see https://git-scm.com/docs/bitmap-format has somehow managed to get\n> indexed by Google, perhaps through some magic :)\n\nPresumably somebody somewhere mentioned it (if not, the list archive now\nlinks to it ;) ).\n\nThe git-scm.com site doesn't use our Documentation/Makefile, so it has\nbeen building the bitmap-format page for a while. It doesn't look\ncorrect, though (it is missing everything before the \"on-disk format\"\nsection).\n\n-Peff\n"},{"id":"430755","messageId":"YPf0hivipY6o5Y3B@coredump.intra.peff.net","threadId":"55464","inReplyTo":"64a260e0c6a116b7c6fa6fea2b9fd96bf416cb18.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 05/24] Documentation: describe MIDX-based bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:18:46Z","receivedAt":"2021-07-21T11:27:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:10PM -0400, Taylor Blau wrote:\n\n> +An object is uniquely described by its bit position within a bitmap:\n> +\n> +\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n> +\tthe __n__th object in pack order. For a function `offset` which maps\n> +\tobjects to their byte offset within a pack, pack order is defined as\n> +\tfollows:\n> +\n> +\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n> +\n> +\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n> +\t__n__th object in MIDX order. With an additional function `pack` which\n> +\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n> +\tis defined as follows:\n> +\n> +\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n> +\n> +\tThe ordering between packs is done lexicographically by the pack name,\n> +\twith the exception of the preferred pack, which sorts ahead of all other\n> +\tpacks.\n\nThis doesn't render well as asciidoc (the final paragraph is taken as\nmore of the code block). But that is a problem through the whole file. I\nthink we should ignore it for now, and worry about asciidoc-ifying the\nwhole thing later, if we choose to.\n\n> +\tThe ordering between packs is done lexicographically by the pack name,\n> +\twith the exception of the preferred pack, which sorts ahead of all other\n> +\tpacks.\n\nHmm, I'm not sure if this \"lexicographically\" part is true. Really we're\nbuilding on the midx .rev format here. And that says \"defined by the\nMIDX's pack list\" (though I can't offhand remember if that is\nlexicographic, or if it is in the reverse-mtime order).\n\nAt any rate, should we just be referencing the rev documentation?\n\n> [...]\n\nThe rest of the changes to the document seemed quite sensible to me.\n\n-Peff\n"},{"id":"430756","messageId":"YPf0yduzM9ReOYfX@coredump.intra.peff.net","threadId":"55464","inReplyTo":"1448ca0d2ba265db2dce414a7f7d6b1f4bcb5a08.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 07/24] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:19:53Z","receivedAt":"2021-07-21T11:27:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:15PM -0400, Taylor Blau wrote:\n\n> When writing a new multi-pack index, write_midx_internal() attempts to\n> clean up any auxiliary files (currently just the MIDX's `.rev` file, but\n> soon to include a `.bitmap`, too) corresponding to the MIDX it's\n> replacing.\n> \n> This step should happen after the new MIDX is written into place, since\n> doing so beforehand means that the old MIDX could be read without its\n> corresponding .rev file.\n\nGood catch. The patch looks obviously correct.\n\n-Peff\n"},{"id":"430757","messageId":"YPf1m01mcdJ3HNBt@coredump.intra.peff.net","threadId":"55464","inReplyTo":"dfd1daacc5b12d470bb6deec3448cf7dbde2bf0f.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:23:23Z","receivedAt":"2021-07-21T11:27:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:18PM -0400, Taylor Blau wrote:\n\n> When writing a new multi-pack index, write_midx_internal() attempts to\n> load any existing one to fill in some pieces of information. But it uses\n> load_multi_pack_index(), which ignores the configuration\n> \"core.multiPackIndex\", which indicates whether or not Git is allowed to\n> read an existing multi-pack-index.\n> \n> Replace this with a routine that does respect that setting, to avoid\n> reading multi-pack-index files when told not to.\n> \n> This avoids a problem that would arise in subsequent patches due to the\n> combination of 'git repack' reopening the object store in-process and\n> the multi-pack index code not checking whether a pack already exists in\n> the object store when calling add_pack_to_midx().\n> \n> This would ultimately lead to a cycle being created along the\n> 'packed_git' struct's '->next' pointer. That is obviously bad, but it\n> has hard-to-debug downstream effects like saying a bitmap can't be\n> loaded for a pack because one already exists (for the same pack).\n\nI'm not sure I completely understand the bug that this causes.\n\nBut another question: does this impact how\n\n git -c core.multipackindex=false multi-pack-index write\n\nbehaves? I.e., do we still write, but just avoid reading the existing\nmidx? That itself seems like a more sensible behavior (e.g., trying to\nrecover from a broken midx state).\n\n-Peff\n"},{"id":"430758","messageId":"YPf49fd7EDncfdl/@coredump.intra.peff.net","threadId":"55464","inReplyTo":"ac1f46aa1f0dbc7fba45229555b11b390fe104a0.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 11/24] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:37:41Z","receivedAt":"2021-07-21T11:27:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:26PM -0400, Taylor Blau wrote:\n\n> A subsequent patch to support reading MIDX bitmaps will be less noisy\n> after extracting a generic function to fetch the nth OID contained in\n> the bitmap.\n\nMakes sense, but...\n\n> +static void nth_bitmap_object_oid(struct bitmap_index *index,\n> +\t\t\t\t struct object_id *oid,\n> +\t\t\t\t uint32_t n)\n> +{\n> +\tnth_packed_object_id(oid, index->pack, n);\n> +}\n> +\n> static int load_bitmap_entries_v1(struct bitmap_index *index)\n> {\n> \tuint32_t i;\n> @@ -242,9 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n> \t\txor_offset = read_u8(index->map, &index->map_pos);\n> \t\tflags = read_u8(index->map, &index->map_pos);\n> \n> -\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n> -\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n> -\t\t\t\t (unsigned)commit_idx_pos);\n> +\t\tnth_bitmap_object_oid(index, &oid, commit_idx_pos);\n\nWhat happened to our error check here?\n\nShould nth_bitmap_object_oid() be returning the value from\nnth_packed_object_id()?\n\n-Peff\n"},{"id":"430761","messageId":"YPf4MTDpbvinoIia@coredump.intra.peff.net","threadId":"55464","inReplyTo":"9495f6869d792264c4366c9914fcf93d544caa6a.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 09/24] midx: infer preferred pack when not given one","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:34:25Z","receivedAt":"2021-07-21T11:27:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:21PM -0400, Taylor Blau wrote:\n\n> In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the\n> multi-pack index code learned how to select a pack which all duplicate\n> objects are selected from. That is, if an object appears in multiple\n> packs, select the copy in the preferred pack before breaking ties\n> according to the other rules like pack mtime and readdir() order.\n> \n> Not specifying a preferred pack can cause serious problems with\n> multi-pack reachability bitmaps, because these bitmaps rely on having at\n> least one pack from which all duplicates are selected. Not having such a\n> pack causes problems with the pack reuse code (e.g., like assuming that\n> a base object was sent from that pack via reuse when in fact the base\n> was selected from a different pack).\n\nIt might be helpful to use a more descriptive name for \"pack reuse code\"\nhere, since it's kind of vague for people who have not been actively\nworking on bitmaps.\n\nI don't have a short name for that chunk of code, but maybe:\n\n ...causes problems with the code in pack-objects to reuse packs\n verbatim (e.g., that code assumes that a delta object in a chunk of\n pack sent verbatim will have its base object sent from the same pack).\n\n> So why does not marking a pack preferred cause problems here? The reason\n> is roughly as follows:\n> \n> - Ties are broken (when handling duplicate objects) by sorting\n> according to midx_oid_compare(), which sorts objects by OID,\n> preferred-ness, pack mtime, and finally pack ID (more on that\n> later).\n> \n> - The psuedo pack-order (described in\n> Documentation/technical/bitmap-format.txt) is computed by\n> midx_pack_order(), and sorts by pack ID and pack offset, with\n> preferred packs sorting first.\n\nI think the .rev description in pack-format.txt may be a better\nreference here.\n\n> - But! Pack IDs come from incrementing the pack count in\n> add_pack_to_midx(), which is a callback to\n> for_each_file_in_pack_dir(), meaning that pack IDs are assigned in\n> readdir() order.\n> \n> When specifying a preferred pack, all of that works fine, because\n> duplicate objects are correctly resolved in favor of the copy in the\n> preferred pack, and the preferred pack sorts first in the object order.\n> \n> \"Sorting first\" is critical, because the bitmap code relies on finding\n> out which pack holds the first object in the MIDX's pseudo pack-order to\n> determine which pack is preferred.\n> \n> But if we didn't specify a preferred pack, and the pack which comes\n> first in readdir() order does not also have the lowest timestamp, then\n> it's possible that that pack (the one that sorts first in pseudo-pack\n> order, which the bitmap code will treat as the preferred one) did *not*\n> have all duplicate objects resolved in its favor, resulting in breakage.\n> \n> The fix is simple: pick a (semi-arbitrary) preferred pack when none was\n> specified. This forces that pack to have duplicates resolved in its\n> favor, and (critically) to sort first in pseudo-pack order.\n> Unfortunately, testing this behavior portably isn't possible, since it\n> depends on readdir() order which isn't guaranteed by POSIX.\n\nThis explanation is rather confusing, but I'm not sure if we can do much\nbetter. I followed all of it, because I was there when we found the bug\nthat this is fixing. And of course that happened _after_ we implemented\nmidx bitmaps and in particular adapted the verbatim reuse stuff in\npack-objects to make use of it.\n\nI see why you'd want to float the fix up before then, so we don't ever\nhave the broken state. But it's hard to understand what bug this is\nfixing, because the bug does not even exist yet at this point in\nthe series!\n\nI dunno. Like I said, I was able to follow it, so maybe it is\nsufficient. I'm just not sure others would be able to.\n\n> +\n> +\t\tif (!found)\n> +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n> +\t\t\t\tpreferred_pack_name);\n> +\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n> +\t\ttime_t oldest = ctx.info[0].p->mtime;\n> +\t\tctx.preferred_pack_idx = 0;\n> +\n> +\t\tif (packs_to_drop && packs_to_drop->nr)\n> +\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n\nLikewise, this BUG() feels somewhat out-of-place. At this point in the\nseries, we don't have bitmaps yet. :)\n\nI can live with that, though. And I don't want to make a lot of work by\ntrying to re-order this patch within the series. Mostly I want to make\nsure that if somebody stumbles on this commit via git-log or in a\nbisection, that they can make some sense of what it's trying to do.\n\n-Peff\n"},{"id":"430759","messageId":"YPf5a4LzBwNweI3W@coredump.intra.peff.net","threadId":"55464","inReplyTo":"c474d2eda5b0b4393616109915626134429085d5.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/24] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:39:39Z","receivedAt":"2021-07-21T11:27:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:29PM -0400, Taylor Blau wrote:\n\n> In a recent commit, pack-objects learned support for the\n> 'pack.preferBitmapTips' configuration. This patch prepares the\n> multi-pack bitmap code to respect this configuration, too.\n> \n> Since the multi-pack bitmap code already does a traversal of all\n> references (in order to discover the set of reachable commits in the\n> multi-pack index), it is more efficient to check whether or not each\n> reference is a suffix of any value of 'pack.preferBitmapTips' rather\n> than do an additional traversal.\n> \n> Implement a function 'bitmap_is_preferred_refname()' which does just\n> that. The caller will be added in a subsequent patch.\n\nI suspect there was some patch reordering here. We don't have any\nmulti-pack bitmap code yet. :)\n\nProbably this needs to say something like \"in preparation for adding\nmulti-pack bitmap code...\" or similar?\n\n-Peff\n"},{"id":"430760","messageId":"YPf4YPbKccZuAVm0@coredump.intra.peff.net","threadId":"55464","inReplyTo":"373aa47528ce8bd4bb044a82a7e80312476f1b5f.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/24] pack-bitmap.c: introduce 'bitmap_num_objects()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:35:12Z","receivedAt":"2021-07-21T11:27:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:23PM -0400, Taylor Blau wrote:\n\n> A subsequent patch to support reading MIDX bitmaps will be less noisy\n> after extracting a generic function to return how many objects are\n> contained in a bitmap.\n\nThanks for this. These kinds of preparatory patches make reading the\nactual tricky changes so much easier.\n\n-Peff\n"},{"id":"430763","messageId":"YPf5HqH8PbMtRnly@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPf49fd7EDncfdl/@coredump.intra.peff.net","subject":"Re: [PATCH v2 11/24] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T10:38:22Z","receivedAt":"2021-07-21T11:27:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 06:37:41AM -0400, Jeff King wrote:\n\n> > @@ -242,9 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n> > \t\txor_offset = read_u8(index->map, &index->map_pos);\n> > \t\tflags = read_u8(index->map, &index->map_pos);\n> > \n> > -\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n> > -\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n> > -\t\t\t\t (unsigned)commit_idx_pos);\n> > +\t\tnth_bitmap_object_oid(index, &oid, commit_idx_pos);\n> \n> What happened to our error check here?\n> \n> Should nth_bitmap_object_oid() be returning the value from\n> nth_packed_object_id()?\n\nAh, sorry, I just saw your followup message. I'll look for the fix in\nthe re-roll.\n\n-Peff\n"},{"id":"430764","messageId":"YPgF4X2PeFvBuJXm@coredump.intra.peff.net","threadId":"55464","inReplyTo":"7d44ba6299c06c956d5ac8ba01a0288d109c3cae.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 13/24] pack-bitmap: read multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T11:32:49Z","receivedAt":"2021-07-21T11:56:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:31PM -0400, Taylor Blau wrote:\n\n> This prepares the code in pack-bitmap to interpret the new multi-pack\n> bitmaps described in Documentation/technical/bitmap-format.txt, which\n> mostly involves converting bit positions to accommodate looking them up\n> in a MIDX.\n> \n> Note that there are currently no writers who write multi-pack bitmaps,\n> and that this will be implemented in the subsequent commit.\n\nThere's quite a lot going on in this one, of course, but most of it\nlooks right. A few hunks did puzzle me:\n\n> @@ -302,12 +377,18 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n> \t\treturn -1;\n> \t}\n> \n> -\tif (bitmap_git->pack) {\n> +\tif (bitmap_git->pack || bitmap_git->midx) {\n> +\t\t/* ignore extra bitmap file; we can only handle one */\n> \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n> \t\tclose(fd);\n> \t\treturn -1;\n> \t}\n> \n> +\tif (!is_pack_valid(packfile)) {\n> +\t\tclose(fd);\n> +\t\treturn -1;\n> +\t}\n> +\n\nWhat's this extra is_pack_valid() doing? I wouldn't expect many changes\nat all to this non-midx code path (aside from the \"did we already load a\nmidx bitmap\" in the earlier part of the hunk, which makes sense).\n\n> -static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n> +static int load_reverse_index(struct bitmap_index *bitmap_git)\n> +{\n> +\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\tuint32_t i;\n> +\t\tint ret;\n> +\n> +\t\tret = load_midx_revindex(bitmap_git->midx);\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\n> +\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> +\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> +\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n> +\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n> +\t\t\tif (ret)\n> +\t\t\t\treturn ret;\n> +\t\t}\n> +\t\treturn 0;\n> +\t}\n> +\treturn load_pack_revindex(bitmap_git->pack);\n> +}\n\nOK, this new function is used in load_bitmap(), which is used for both\npack and midx bitmaps. So if we have a midx bitmap, we'll\nunconditionally load the revindex here. But:\n\n - why do we then load individual pack revindexes? I can believe it may\n be necessary to meet the assumptions of some other part of the code,\n but it would be nice to have a comment giving us some clue.\n\n - in open_midx_bitmap_1(), we also unconditionally load the midx\n reverse index. I think that will always happen before us here (we\n cannot load_bitmap() a bitmap that has not been opened). So is this\n load_midx_revindex() call always a noop?\n\n> +static int open_bitmap(struct repository *r,\n> +\t\t struct bitmap_index *bitmap_git)\n> +{\n> +\tassert(!bitmap_git->map);\n> +\n> +\tif (!open_midx_bitmap(r, bitmap_git))\n> +\t\treturn 0;\n> +\treturn open_pack_bitmap(r, bitmap_git);\n> +}\n\nWe always prefer a midx bitmap over a pack one. That makes sense, since\nthat means we can leave old pack bitmaps in place when generating midx\nbitmaps, if we choose to.\n\n> static int bitmap_position(struct bitmap_index *bitmap_git,\n> \t\t\t const struct object_id *oid)\n> {\n> -\tint pos = bitmap_position_packfile(bitmap_git, oid);\n> +\tint pos;\n> +\tif (bitmap_is_midx(bitmap_git))\n> +\t\tpos = bitmap_position_midx(bitmap_git, oid);\n> +\telse\n> +\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n> \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n> }\n\nMakes sense. Not new in your patch, but this \"int\" return is fudging the\nsame 32-bit space we were talking about elsewhere (i.e., \"pos\" really\ncould be 2^32, or even more due to extended objects).\n\nIn practice I think even 2^31 objects is pretty out-of-reach, but it may\nbe worth changing the return type (and the callers), or even just\ncatching the overflow with an assertion.\n\n> @@ -752,8 +911,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n> \t\tstruct object *object = roots->item;\n> \t\troots = roots->next;\n> \n> -\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n> -\t\t\treturn 1;\n> +\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n> +\t\t\t\treturn 1;\n> +\t\t} else {\n> +\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n> +\t\t\t\treturn 1;\n> +\t\t}\n> \t}\n\nMakes sense. TBH, I am not sure this in_bitmapped_pack() function is all\nthat useful. It is used only as part of a heuristic to avoid bitmaps\nwhen we don't have coverage of any \"have\" commits. But I'm not sure that\nheuristic is actually useful.\n\nAnyway, we should definitely not get into ripping it out here. This\nseries is complicated enough. :) Just a note for possible future work.\n\n> \tif (pos < bitmap_num_objects(bitmap_git)) {\n> -\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n> +\t\tstruct packed_git *pack;\n> +\t\toff_t ofs;\n> +\n> +\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> +\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> +\n> +\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> +\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> +\t\t} else {\n> +\t\t\tpack = bitmap_git->pack;\n> +\t\t\tofs = pack_pos_to_offset(pack, pos);\n> +\t\t}\n> +\n\nAll of the hunks like this make perfect sense. The big problem would be\nif we _missed_ a place that needed conversion to handle midx. But the\nnice thing is that it would segfault quickly in such an instance. So\nthere I'm mostly relying on test coverage, plus our experience running\nwith this code at scale.\n\n> static void try_partial_reuse(struct bitmap_index *bitmap_git,\n> +\t\t\t struct packed_git *pack,\n> \t\t\t size_t pos,\n> \t\t\t struct bitmap *reuse,\n> \t\t\t struct pack_window **w_curs)\n> {\n> -\toff_t offset, header;\n> +\toff_t offset, delta_obj_offset;\n\nI'm OK with all of this in one big patch. But I suspect you _could_\njust put:\n\n if (bitmap_git->midx)\n\treturn; /* partial reuse not implemented for midx yet */\n\nto start with, and then actually implement it later. I call out this\ncode in particular just because it's got a lot of subtleties (the\n\"reuse\" bits are much more intimate with the assumptions of packs and\nbitmaps than most other code).\n\nI'm not sure if it's worth the trouble at this point or not.\n\n> \tenum object_type type;\n> \tunsigned long size;\n> \n> -\tif (pos >= bitmap_num_objects(bitmap_git))\n> -\t\treturn; /* not actually in the pack or MIDX */\n> +\t/*\n> +\t * try_partial_reuse() is called either on (a) objects in the\n> +\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n> +\t * objects in the preferred pack of a multi-pack bitmap.\n> +\t * Importantly, the latter can pretend as if only a single pack\n> +\t * exists because:\n> +\t *\n> +\t * - The first pack->num_objects bits of a MIDX bitmap are\n> +\t * reserved for the preferred pack, and\n> +\t *\n> +\t * - Ties due to duplicate objects are always resolved in\n> +\t * favor of the preferred pack.\n> +\t *\n> +\t * Therefore we do not need to ever ask the MIDX for its copy of\n> +\t * an object by OID, since it will always select it from the\n> +\t * preferred pack. Likewise, the selected copy of the base\n> +\t * object for any deltas will reside in the same pack.\n> +\t *\n> +\t * This means that we can reuse pos when looking up the bit in\n> +\t * the reuse bitmap, too, since bits corresponding to the\n> +\t * preferred pack precede all bits from other packs.\n> +\t */\n> \n> -\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n> -\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n> +\tif (pos >= pack->num_objects)\n> +\t\treturn; /* not actually in the pack or MIDX preferred pack */\n\nIt feels weird to go from bitmap_num_objects() back to\npack->num_objects. But I agree it's the right thing for the \"pretend as\nif only a single pack exists\" reasons given above.\n\n> +static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n> +{\n> +\tstruct multi_pack_index *m = bitmap_git->midx;\n> +\tif (!m)\n> +\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n> +\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n> +}\n\nThis part is really subtle. We infer the preferred pack by looking at\nthe pack of the 0th bit position. In general that works, since that's\npart of the definition of the preferred pack.\n\nCould this ever be fooled if we had a preferred pack with 0 objects in\nit? I don't know why we would have such a thing, but just trying to\nthink of cases where our assumptions might not hold (and what bad things\ncould happen).\n\n> +\tif (bitmap_is_midx(bitmap_git))\n> +\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n> +\telse\n> +\t\tpack = bitmap_git->pack;\n> +\tobjects_nr = pack->num_objects;\n> +\n> \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n> \t\ti++;\n> \n> -\t/* Don't mark objects not in the packfile */\n> +\t/*\n> +\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n> +\t * marks objects eligible for reuse, but the pack-reuse code only\n> +\t * understands how to reuse a single pack. Since the preferred pack is\n> +\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n> +\t * we use it instead of another pack. In single-pack bitmaps, the choice\n> +\t * is made for us.\n> +\t */\n> \tif (i > objects_nr / BITS_IN_EWORD)\n> \t\ti = objects_nr / BITS_IN_EWORD;\n\nOK, so this clamps our \"quick\" contiguous set of bits to the number of\nobjects in the preferred pack. Makes sense. And then we hit the\nobject-by-object loop below...\n\n> @@ -1213,7 +1437,15 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n> \t\t\t\tbreak;\n> \n> \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> -\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n> +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\t\t/*\n> +\t\t\t\t * Can't reuse from a non-preferred pack (see\n> +\t\t\t\t * above).\n> +\t\t\t\t */\n> +\t\t\t\tif (pos + offset >= objects_nr)\n> +\t\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n\n...and this likewise makes sure we never go past that first pack. Good.\n\nI think this \"continue\" could actually be a \"break\", as the loop is\niterating over \"offset\" (and \"pos + offset\" always gets larger). In\nfact, it could break out of the outer loop as well (which is\nincrementing \"pos\"). It's probably a pretty small efficiency in\npractice, though.\n\n> @@ -1511,8 +1749,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n> \t\tstruct object_id oid;\n> \t\tstruct object_entry *oe;\n> \n> -\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n> -\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n> +\t\tif (bitmap_is_midx(bitmap_git))\n> +\t\t\tnth_midxed_object_oid(&oid,\n> +\t\t\t\t\t bitmap_git->midx,\n> +\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n> +\t\telse\n> +\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n> +\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n> \t\toe = packlist_find(mapping, &oid);\n\nCould this be using nth_bitmap_object_oid()? I guess not, because we are\nfeeding from pack_pos_to_*. I'm not sure if another helper function is\nworth it (pack_pos_to_bitmap_index() or something?).\n\n> @@ -1575,7 +1831,31 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n> \t\t\t\tbreak;\n> \n> \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> -\t\t\tpos = base + offset;\n> +\n> +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\t\t\tuint32_t pack_pos;\n> +\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n> +\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> +\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> +\n> +\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> +\n> +\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n> +\t\t\t\t\tstruct object_id oid;\n> +\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n> +\n> +\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n> +\t\t\t\t\t oid_to_hex(&oid),\n> +\t\t\t\t\t pack_id,\n> +\t\t\t\t\t (uintmax_t)offset);\n> +\t\t\t\t}\n> +\n> +\t\t\t\tpos = pack_pos;\n> +\t\t\t} else {\n> +\t\t\t\tpack = bitmap_git->pack;\n> +\t\t\t\tpos = base + offset;\n> +\t\t\t}\n> +\n> \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n> \t\t\t\t pack_pos_to_offset(pack, pos);\n> \t\t}\n\nIn the midx case, we have to go from midx-bitmap-pos to midx-index-pos,\nto then get the pack/ofs combo, which then gives us a real \"pos\" in the\npack. I don't think there's a faster way to do it (and this is still\nmuch faster than looking up objects in the pack only to check their\nrevindex).\n\nBut then with the result, we compare the offset of \"pos\" and \"pos + 1\".\nWe need to know \"pos\" to find \"pos + 1\". But in the midx case, don't we\nalready have the offset of \"pos\" (it is \"offset\" in the bitmap_is_midx()\nconditional, which is shadowing the completely unrelated \"offset\" in the\nouter loop).\n\nWe could reuse it, saving ourselves an extra round-trip of pack_pos to\nindex_pos to offset. It would just mean stuffing the \"total +=\" line\ninto the two sides of the conditional.\n\n> +off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos)\n> +{\n> +\tif (bitmap_is_midx(bitmap_git))\n> +\t\treturn nth_midxed_offset(bitmap_git->midx,\n> +\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, pos));\n> +\treturn nth_packed_object_offset(bitmap_git->pack,\n> +\t\t\t\t\tpack_pos_to_index(bitmap_git->pack, pos));\n> +}\n\nDoes anybody call this function? I don't see any users by the end of the\nseries.\n\n-Peff\n"},{"id":"430765","messageId":"YPgObwXjt/tzAJvV@coredump.intra.peff.net","threadId":"55464","inReplyTo":"a8cec2463d0993b1118abdd31cb6c9e88a32e0c4.1624314293.git.me@ttaylorr.com","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T12:09:19Z","receivedAt":"2021-07-21T12:09:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 21, 2021 at 06:25:34PM -0400, Taylor Blau wrote:\n\n> +static int add_ref_to_pending(const char *refname,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t int flag, void *cb_data)\n> +{\n> +\tstruct rev_info *revs = (struct rev_info*)cb_data;\n> +\tstruct object *object;\n> +\n> +\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n> +\t\twarning(\"symbolic ref is dangling: %s\", refname);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tobject = parse_object_or_die(oid, refname);\n> +\tif (object->type != OBJ_COMMIT)\n> +\t\treturn 0;\n> +\n> +\tadd_pending_object(revs, object, \"\");\n> +\tif (bitmap_is_preferred_refname(revs->repo, refname))\n> +\t\tobject->flags |= NEEDS_BITMAP;\n> +\treturn 0;\n> +}\n\nOK, so we'll look at each ref to get the set of commits that we want to\ntraverse to put into the bitmap. Which is roughly the same as what the\npack bitmap does. We only generate bitmaps for all-into-one repacks, so\nit is traversing all of the reachable objects. It is a little different\nin that the pack version is probably hitting reflogs, but IMHO we are\nbetter off to ignore reflogs for the purposes of bitmaps (I would\nsuggest to do so in the pack-bitmap case, too, except that it is\ncombined with the \"what to pack\" traversal there, and by the time we see\neach commit we don't know how we got there).\n\n> +struct bitmap_commit_cb {\n> +\tstruct commit **commits;\n> +\tsize_t commits_nr, commits_alloc;\n> +\n> +\tstruct write_midx_context *ctx;\n> +};\n> +\n> +static const struct object_id *bitmap_oid_access(size_t index,\n> +\t\t\t\t\t\t const void *_entries)\n> +{\n> +\tconst struct pack_midx_entry *entries = _entries;\n> +\treturn &entries[index].oid;\n> +}\n> +\n> +static void bitmap_show_commit(struct commit *commit, void *_data)\n> +{\n> +\tstruct bitmap_commit_cb *data = _data;\n> +\tif (oid_pos(&commit->object.oid, data->ctx->entries,\n> +\t\t data->ctx->entries_nr,\n> +\t\t bitmap_oid_access) > -1) {\n\nThis \"> -1\" struck me as a little bit funny. Perhaps \">= 0\" would be a\nmore obvious way of saying \"we found it\"?\n\n> +\t/*\n> +\t * Skipping promisor objects here is intentional, since it only excludes\n> +\t * them from the list of reachable commits that we want to select from\n> +\t * when computing the selection of MIDX'd commits to receive bitmaps.\n> +\t *\n> +\t * Reachability bitmaps do require that their objects be closed under\n> +\t * reachability, but fetching any objects missing from promisors at this\n> +\t * point is too late. But, if one of those objects can be reached from\n> +\t * an another object that is included in the bitmap, then we will\n> +\t * complain later that we don't have reachability closure (and fail\n> +\t * appropriately).\n> +\t */\n> +\tfetch_if_missing = 0;\n> +\trevs.exclude_promisor_objects = 1;\n\nMakes sense.\n\n> +\t/*\n> +\t * Pass selected commits in topo order to match the behavior of\n> +\t * pack-bitmaps when configured with delta islands.\n> +\t */\n> +\trevs.topo_order = 1;\n> +\trevs.sort_order = REV_SORT_IN_GRAPH_ORDER;\n\nHmm. Why do we want to match this side effect of delta islands here?\n\nThe only impact this has is on the order of commits we feed for bitmap\nselection (and during the actual generation phase, it may impact\nvisitation order).\n\nNow I'm of the opinion that topo order is probably the best thing for\nbitmap generation (since the bitmaps themselves are connected to the\ngraph structure). But if it is the best thing, shouldn't we perhaps be\nturning on topo-order for single-pack bitmaps, too?\n\nAnd if it isn't the best thing, then why would we want it here?\n\n> +\tif (prepare_revision_walk(&revs))\n> +\t\tdie(_(\"revision walk setup failed\"));\n\nWe call init_revisions(), and then go straight to\nprepare_revision_walk() with no call to setup_revisions() between. It\ndoesn't seem to be clearly documented, but I think you're supposed to,\nas it finalizes some bits like diff_setup_done().\n\nI suspect it works OK in practice, and I did find a few other spots that\ndo not call it (e.g., builtin/am.c:write_commit_patch). But most spots\ndo at least an empty setup_revisions(0, NULL, &rev, NULL).\n\n> +\t/*\n> +\t * Build the MIDX-order index based on pdata.objects (which is already\n> +\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n> +\t * this order).\n> +\t */\n> +\tALLOC_ARRAY(index, pdata.nr_objects);\n> +\tfor (i = 0; i < pdata.nr_objects; i++)\n> +\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n\nThis cast is correct because the pack_idx_entry is at the start of each\nobject_entry. But maybe:\n\n index[i] = &pdata.objects[i].idx;\n\nwould be less scary looking?\n\n> +\t/*\n> +\t * bitmap_writer_finish expects objects in lex order, but pack_order\n> +\t * gives us exactly that. use it directly instead of re-sorting the\n> +\t * array.\n> +\t *\n> +\t * This changes the order of objects in 'index' between\n> +\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n> +\t *\n> +\t * The same re-ordering takes place in the single-pack bitmap code via\n> +\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n> +\t * happens between bitmap_writer_build_type_index() and\n> +\t * bitmap_writer_finish().\n> +\t */\n> +\tfor (i = 0; i < pdata.nr_objects; i++)\n> +\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n\nDitto here.\n\n> +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n\nNot related to your patch, but I had to refresh my memory on what this\n\"-1\" was for. It's \"max_bitmaps\", and is ignored if it's negative. But\nthe only callers pass \"-1\"! So we could get rid of it entirely.\n\nIt probably makes sense to leave that cleanup out of this\nalready-complicated series. But maybe worth doing later on top.\n\n> @@ -930,9 +1100,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> \n> +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> +\t\t\t\terror(_(\"could not load pack %s\"),\n> +\t\t\t\t ctx.m->pack_names[i]);\n> +\t\t\t\tresult = 1;\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n\nIt might be worth a comment here. I can easily believe that there is\nsome later part of the bitmap generation code that assumes the packs are\nloaded. But somebody reading this is not likely to understand why it's\nhere.\n\nShould this be done conditionally only if we're writing a bitmap? (That\nmight also make it obvious why we are doing it).\n\n> @@ -947,8 +1124,26 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n> \tstop_progress(&ctx.progress);\n> \n> -\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n> -\t\tgoto cleanup;\n> +\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n> +\t\tstruct bitmap_index *bitmap_git;\n> +\t\tint bitmap_exists;\n> +\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n> +\n> +\t\tbitmap_git = prepare_bitmap_git(the_repository);\n> +\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n> +\t\tfree_bitmap_index(bitmap_git);\n> +\n> +\t\tif (bitmap_exists || !want_bitmap) {\n> +\t\t\t/*\n> +\t\t\t * The correct MIDX already exists, and so does a\n> +\t\t\t * corresponding bitmap (or one wasn't requested).\n> +\t\t\t */\n> +\t\t\tif (!want_bitmap)\n> +\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n> +\t\t\t\t\t\t NULL);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\t}\n\nSo this makes \"git multi-pack-index write --write-bitmap\" actually write\na bitmap, even if the midx itself didn't need updating? Sounds good.\nLikewise, we'll delete a bitmap if one exists but we were not requested\nto write one. Makes sense.\n\nI do think nice-to-have bits like this could have come in a separate\npatch with their own explanation and tests. It may not be worth trying\nto extract it at this point, though.\n\n> @@ -1075,9 +1271,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n> \n> -\tif (ctx.m)\n> -\t\tclose_midx(ctx.m);\n> -\n> \tif (ctx.nr - dropped_packs == 0) {\n> \t\terror(_(\"no pack files to index.\"));\n> \t\tresult = 1;\n\nI'm not sure what this hunk is doing. We do pick up the close_midx()\ncall at the end of the function, amidst the other cleanup.\n\nI expect the answer is something like \"we need it open when we generate\nthe bitmaps\". But it makes me wonder if we could hit any cases where we\ntry to overwrite it while it's still open, which would cause problems on\nWindows.\n\n-Peff\n"},{"id":"430766","messageId":"YPgPNVfwedWFGqcd@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPBIBXUB53b0Hixx@nand.local","subject":"Re: [PATCH v2 00/24] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-21T12:12:37Z","receivedAt":"2021-07-21T12:12:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 15, 2021 at 10:36:53AM -0400, Taylor Blau wrote:\n\n> I know that reviewing this is on Peff's list of things to do, but there\n> are competing priorities (and we have all-company meetings this week, so\n> I would not be surprised to see a lack of movement until at least next\n> week).\n\nI made it through a very careful read of all of the actual code changes,\nthrough patch 14. After that, it looks like it's all tests, but my brain\nis now effectively fried. I had some comments, but nothing\nearth-shattering.\n\nI'll pick up on 15-24 later, though I think between my comments and\nÆvar's there's some light changes to be made. So don't hesitate to post\na new version in the meantime, which can save us a round-trip.\n\n-Peff\n"},{"id":"430779","messageId":"YPhWPVpiGufLVysG@nand.local","threadId":"55464","inReplyTo":"YPfsqJrCTFQXhltJ@coredump.intra.peff.net","subject":"Re: [PATCH v2 01/24] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T17:15:41Z","receivedAt":"2021-07-21T17:15:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 05:45:12AM -0400, Jeff King wrote:\n> > +{\n> > +\tenum object_type bitmap_type = OBJ_NONE;\n> > [...]\n> > +\n> > +\tif (!bitmap_type)\n> > +\t\tdie(\"object %s not found in type bitmaps\",\n> > +\t\t oid_to_hex(&obj->oid));\n>\n> I think the suggestion to do:\n>\n> if (bitmap_type == OBJ_NONE)\n>\n> is reasonable here, as it assumes less about the enum. I do think\n> OBJ_BAD and OBJ_NONE were chosen with these kind of numeric comparisons\n> in mind, but there is no reason to rely on them in places we don't need\n> to.\n\nI had to double check your suggestion, because my first question was\n\"what if bitmap_type is OBJ_BAD?\" We can call type_name() on OBJ_BAD, but\nit will return NULL, and we use the return value in a format string\nunconditionally.\n\nSo that would be a problem, but it's impossible for this to ever be\nOBJ_BAD, because we only set it based on the type bitmaps; so it's\neither a commit/tree/blob/tag, or none (but not bad).\n\nI took your suggestion, thanks.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"430780","messageId":"YPhXb9Zns8S6aIod@nand.local","threadId":"55464","inReplyTo":"YPft87yCjR9e+93E@coredump.intra.peff.net","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T17:20:47Z","receivedAt":"2021-07-21T17:20:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 05:50:43AM -0400, Jeff King wrote:\n> The amount of error-plumbing you had to do is a little unpleasant, but I\n> think is unavoidable. The only non-obvious part was this hunk:\n\nAgreed, at least on the amount of plumbing required to get this to work\n;).\n\n> > @@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> > \t\tstruct commit *child;\n> > \t\tint reused = 0;\n> >\n> > -\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n> > -\t\t\t\t old_bitmap, mapping);\n> > +\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n> > +\t\t\t\t old_bitmap, mapping) < 0) {\n> > +\t\t\tclosed = 0;\n> > +\t\t\tbreak;\n> > +\t\t}\n> >\n> > \t\tif (ent->selected) {\n> > \t\t\tstore_selected(ent, commit);\n>\n> This is the right thing to do because we still want to free memory, stop\n> progress, etc. I gave a look over what will run after breaking out of\n> the loop, and compute_xor_offsets(), which you already handled, is the\n> only thing we'd want to avoid running. Good.\n\nRight. The key is that we return \"closed ? 0 : -1\" (of course, being\ncareful to invert \"closed\" where \"1\" OK into a suitable return value for\nbitmap_writer_build, where \"0\" means OK, and a negative number means\n\"error\").\n\nWhile I'm thinking about that inversion, we *could* call this variable\n\"open\" and set it to \"0\" until proven otherwise. Then the conditional\nbecomes \"if (!open)\", but the return value is still \"return open ? -1 :\n0\" (since I assume we'd want to use 0/1 values for \"open\" instead of -1,\nmeaning we'd have to do some translation).\n\nAnyway, this is definitely an annoying detail that doesn't really\nmatter (and just rambling on my part) ;).\n\nThanks,\nTaylor\n"},{"id":"430781","messageId":"YPhYFnudmHJ9lQek@nand.local","threadId":"55464","inReplyTo":"YPfyEiXw7szt5mjl@coredump.intra.peff.net","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T17:23:34Z","receivedAt":"2021-07-21T17:23:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 06:08:18AM -0400, Jeff King wrote:\n> On Wed, Jul 21, 2021 at 05:58:41AM -0400, Jeff King wrote:\n>\n> > On Mon, Jun 21, 2021 at 06:25:07PM -0400, Taylor Blau wrote:\n> >\n> > > Even though the 'TECH_DOCS' variable was introduced all the way back in\n> > > 5e00439f0a (Documentation: build html for all files in technical and\n> > > howto, 2012-10-23), the 'bitmap-format' document was never added to that\n> > > list when it was created.\n> > >\n> > > Prepare for changes to this file by including it in the list of\n> > > technical documentation that 'make doc' will build by default.\n> >\n> > OK. I don't care that much about being able to format this as html, but\n> > I agree it's good to be consistent with the other stuff in technical/.\n> >\n> > The big question is whether it looks OK rendered by asciidoc, and the\n> > answer seems to be \"yes\" (from a cursory look I gave it).\n>\n> Actually, I take it back. After looking more carefully, it renders quite\n> poorly. There's a lot of structural indentation that ends up being\n> confused as code blocks.\n>\n> I don't know if it's better to have a poorly-formatted HTML file, or\n> none at all. :)\n>\n> Personally, I would just read the source. And I have a slight concern\n> that if we start \"cleaning it up\" to render as asciidoc, the source\n> might end up a lot less readable (though I'd reserve judgement until\n> actually seeing it).\n\nYeah, the actual source is pretty readable (and it's what I had been\nlooking at, although it is sometimes convenient to have a version I can\nread in my web browser). But it's definitely not good Asciidoc.\n\nI briefly considered cleaning it up, but decided against it. Usually I\nwould opt to clean it up, but this series is already so large that I\nfigured it would make a negative impact on the reviewer experience to\nread a clean-up patch here.\n\nI wouldn't be opposed to coming back to it in the future, once the dust\nsettles. I guess we can consider this #leftoverbits until then.\n\nThanks,\nTaylor\n"},{"id":"430784","messageId":"YPhfJNubkJpOn4Sm@nand.local","threadId":"55464","inReplyTo":"YPf0hivipY6o5Y3B@coredump.intra.peff.net","subject":"Re: [PATCH v2 05/24] Documentation: describe MIDX-based bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T17:53:40Z","receivedAt":"2021-07-21T17:53:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 06:18:46AM -0400, Jeff King wrote:\n> On Mon, Jun 21, 2021 at 06:25:10PM -0400, Taylor Blau wrote:\n>\n> > +An object is uniquely described by its bit position within a bitmap:\n> > +\n> > +\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n> > +\tthe __n__th object in pack order. For a function `offset` which maps\n> > +\tobjects to their byte offset within a pack, pack order is defined as\n> > +\tfollows:\n> > +\n> > +\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n> > +\n> > +\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n> > +\t__n__th object in MIDX order. With an additional function `pack` which\n> > +\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n> > +\tis defined as follows:\n> > +\n> > +\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n> > +\n> > +\tThe ordering between packs is done lexicographically by the pack name,\n> > +\twith the exception of the preferred pack, which sorts ahead of all other\n> > +\tpacks.\n>\n> This doesn't render well as asciidoc (the final paragraph is taken as\n> more of the code block). But that is a problem through the whole file. I\n> think we should ignore it for now, and worry about asciidoc-ifying the\n> whole thing later, if we choose to.\n\nAgreed; let's ignore it for now.\n\n> > +\tThe ordering between packs is done lexicographically by the pack name,\n> > +\twith the exception of the preferred pack, which sorts ahead of all other\n> > +\tpacks.\n>\n> Hmm, I'm not sure if this \"lexicographically\" part is true. Really we're\n> building on the midx .rev format here. And that says \"defined by the\n> MIDX's pack list\" (though I can't offhand remember if that is\n> lexicographic, or if it is in the reverse-mtime order).\n>\n> At any rate, should we just be referencing the rev documentation?\n\nThe packs are listed in lex order in the MIDX, but that is so we can\nbinary search that list to determine whether a pack is included in the\nMIDX or not.\n\nI had to check, but we do use the lex order to resolve duplicate\nobjects, too. See (at the tip of this branch):\n\n QSORT(ctx.info, ctx.nr, pack_info_compare);\n\nfrom within midx.c:write_midx_internal(). Here, ctx.info contains the\nlist of packs, and pack_info_compare is a thin wrapper around\nstrcmp()-ing the pack_name values of two packed_git structures.\n\nArguably, you'd get better EWAH compression of the bits between packs\nif we sorted packs in reverse order according to their mtime. But I\nsuspect that it doesn't matter much in practice, since the number of\nobjects vastly outpaces the number of packs (but I haven't measured to\nbe certain, so take that with a grain of salt).\n\nIn any case, I think that you're right that adding too much detail hurts\nus here, so we should really be mentioning the MIDX's .rev-file\ndocumentation (unfortunately, we can't linkgit it, so mentioning it by\nname will have to suffice). I plan to reroll with something like this on\ntop:\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 25221c7ec8..04b3ec2178 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -26,9 +26,8 @@ An object is uniquely described by its bit position within a bitmap:\n\n \t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n\n-\tThe ordering between packs is done lexicographically by the pack name,\n-\twith the exception of the preferred pack, which sorts ahead of all other\n-\tpacks.\n+\tThe ordering between packs is done according to the MIDX's .rev file.\n+\tNotably, the preferred pack sorts ahead of all other packs.\n\n The on-disk representation (described below) of a bitmap is the same regardless\n of whether or not that bitmap belongs to a packfile or a MIDX. The only\n\nThanks,\nTaylor\n"},{"id":"430787","messageId":"YPhz+iOMu4Q7zjY4@nand.local","threadId":"55464","inReplyTo":"YPf1m01mcdJ3HNBt@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T19:22:34Z","receivedAt":"2021-07-21T19:22:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 06:23:23AM -0400, Jeff King wrote:\n> On Mon, Jun 21, 2021 at 06:25:18PM -0400, Taylor Blau wrote:\n>\n> > When writing a new multi-pack index, write_midx_internal() attempts to\n> > load any existing one to fill in some pieces of information. But it uses\n> > load_multi_pack_index(), which ignores the configuration\n> > \"core.multiPackIndex\", which indicates whether or not Git is allowed to\n> > read an existing multi-pack-index.\n> >\n> > Replace this with a routine that does respect that setting, to avoid\n> > reading multi-pack-index files when told not to.\n> >\n> > This avoids a problem that would arise in subsequent patches due to the\n> > combination of 'git repack' reopening the object store in-process and\n> > the multi-pack index code not checking whether a pack already exists in\n> > the object store when calling add_pack_to_midx().\n> >\n> > This would ultimately lead to a cycle being created along the\n> > 'packed_git' struct's '->next' pointer. That is obviously bad, but it\n> > has hard-to-debug downstream effects like saying a bitmap can't be\n> > loaded for a pack because one already exists (for the same pack).\n>\n> I'm not sure I completely understand the bug that this causes.\n\nOff-hand, I can't quite remember either. But it is important; I do have\na distinct memory of dropping this patch and then watching a 'git repack\n--write-midx' (that option will be introduced in a later series) fail\nhorribly.\n\nIf I remember correctly, the bug has to do with loading a MIDX twice in\nthe same process. When we call add_packed_git() from within\nprepare_midx_pack(), we load the pack without caring whether or not it's\nalready loaded. So loading a MIDX twice in the same process will fail.\n\nSo really I think that this is papering over that bug: we're just\nremoving one of the times that we happened to load a MIDX from during\nthe writing phase.\n\nWhat I do remember is that this bug was a huge pain to figure out ;).\nI'm happy to look further if you aren't satisfied with my vague\nexplanation here (and I wouldn't blame you).\n\n> But another question: does this impact how\n>\n> git -c core.multipackindex=false multi-pack-index write\n>\n> behaves? I.e., do we still write, but just avoid reading the existing\n> midx? That itself seems like a more sensible behavior (e.g., trying to\n> recover from a broken midx state).\n\nYes. Before this patch, that invocation would still load and use any\nexisting MIDX to write a new one. Now we don't, because (unlike\nload_multi_pack_index()) prepare_multi_pack_index_one() does check\ncore.multiPackIndex before returning anything.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"430788","messageId":"YPiAhw2eP2MOksUF@nand.local","threadId":"55464","inReplyTo":"YPf4MTDpbvinoIia@coredump.intra.peff.net","subject":"Re: [PATCH v2 09/24] midx: infer preferred pack when not given one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T20:16:07Z","receivedAt":"2021-07-21T20:16:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 06:34:25AM -0400, Jeff King wrote:\n> On Mon, Jun 21, 2021 at 06:25:21PM -0400, Taylor Blau wrote:\n>\n> > In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the\n> > multi-pack index code learned how to select a pack which all duplicate\n> > objects are selected from. That is, if an object appears in multiple\n> > packs, select the copy in the preferred pack before breaking ties\n> > according to the other rules like pack mtime and readdir() order.\n> >\n> > Not specifying a preferred pack can cause serious problems with\n> > multi-pack reachability bitmaps, because these bitmaps rely on having at\n> > least one pack from which all duplicates are selected. Not having such a\n> > pack causes problems with the pack reuse code (e.g., like assuming that\n> > a base object was sent from that pack via reuse when in fact the base\n> > was selected from a different pack).\n>\n> It might be helpful to use a more descriptive name for \"pack reuse code\"\n> here, since it's kind of vague for people who have not been actively\n> working on bitmaps.\n>\n> I don't have a short name for that chunk of code, but maybe:\n>\n> ...causes problems with the code in pack-objects to reuse packs\n> verbatim (e.g., that code assumes that a delta object in a chunk of\n> pack sent verbatim will have its base object sent from the same pack).\n\nThanks; I like what you wrote here.\n\n> > - The psuedo pack-order (described in\n> > Documentation/technical/bitmap-format.txt) is computed by\n> > midx_pack_order(), and sorts by pack ID and pack offset, with\n> > preferred packs sorting first.\n>\n> I think the .rev description in pack-format.txt may be a better\n> reference here.\n\nDitto, I changed that, too.\n\n> > - But! Pack IDs come from incrementing the pack count in\n> > add_pack_to_midx(), which is a callback to\n> > for_each_file_in_pack_dir(), meaning that pack IDs are assigned in\n> > readdir() order.\n> >\n> > [ ... ]\n>\n> This explanation is rather confusing, but I'm not sure if we can do much\n> better. I followed all of it, because I was there when we found the bug\n> that this is fixing. And of course that happened _after_ we implemented\n> midx bitmaps and in particular adapted the verbatim reuse stuff in\n> pack-objects to make use of it.\n>\n> I see why you'd want to float the fix up before then, so we don't ever\n> have the broken state. But it's hard to understand what bug this is\n> fixing, because the bug does not even exist yet at this point in\n> the series!\n>\n> I dunno. Like I said, I was able to follow it, so maybe it is\n> sufficient. I'm just not sure others would be able to.\n\nI think that others will follow it, too. But I agree that it is\nconfusing, since we're fixing a bug that doesn't yet exist. In reality,\nI wrote this patch after sending v1, and then reordered its position to\ncome before the implementation of MIDX bitmaps for that reason.\n\nSo in one sense, I prefer it this way because we don't ever introduce\nthe bug. But in another sense, it is very jarring to read about an\ninteraction that has no basis in the code (yet).\n\nI think that the best thing we could do without adding any significant\nreordering would be to just call out the situation we're in. I added\nthis onto the end of the commit message which I think makes things a\nlittle clearer:\n\n (Note that multi-pack reachability bitmaps have yet to be\n implemented; so in that sense this patch is fixing a bug which does\n not yet exist. But by having this patch beforehand, we can prevent\n the bug from ever materializing.)\n\nThanks,\nTaylor\n"},{"id":"430790","messageId":"YPiBJKWUV+BQJyif@nand.local","threadId":"55464","inReplyTo":"YPf5a4LzBwNweI3W@coredump.intra.peff.net","subject":"Re: [PATCH v2 12/24] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T20:18:44Z","receivedAt":"2021-07-21T20:18:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 06:39:39AM -0400, Jeff King wrote:\n> On Mon, Jun 21, 2021 at 06:25:29PM -0400, Taylor Blau wrote:\n> > [...]\n> >\n> > Implement a function 'bitmap_is_preferred_refname()' which does just\n> > that. The caller will be added in a subsequent patch.\n>\n> I suspect there was some patch reordering here. We don't have any\n> multi-pack bitmap code yet. :)\n>\n> Probably this needs to say something like \"in preparation for adding\n> multi-pack bitmap code...\" or similar?\n\nOops. Good catch!\n\nThanks,\nTaylor\n"},{"id":"430821","messageId":"YPinPFW50Mj8cVkP@nand.local","threadId":"55464","inReplyTo":"YPgF4X2PeFvBuJXm@coredump.intra.peff.net","subject":"Re: [PATCH v2 13/24] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-21T23:01:16Z","receivedAt":"2021-07-21T23:01:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 07:32:49AM -0400, Jeff King wrote:\n> On Mon, Jun 21, 2021 at 06:25:31PM -0400, Taylor Blau wrote:\n> > +\tif (!is_pack_valid(packfile)) {\n> > +\t\tclose(fd);\n> > +\t\treturn -1;\n> > +\t}\n> > +\n>\n> What's this extra is_pack_valid() doing? I wouldn't expect many changes\n> at all to this non-midx code path (aside from the \"did we already load a\n> midx bitmap\" in the earlier part of the hunk, which makes sense).\n\nThat looks like a mistake to me. I did a little digging and tried to\nremember if it could have ever been useful, but I think that it's just a\nstray change that has no value. Removed.\n\n> > -static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n> > +static int load_reverse_index(struct bitmap_index *bitmap_git)\n> > +{\n> > +\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\tuint32_t i;\n> > +\t\tint ret;\n> > +\n> > +\t\tret = load_midx_revindex(bitmap_git->midx);\n> > +\t\tif (ret)\n> > +\t\t\treturn ret;\n> > +\n> > +\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> > +\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> > +\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n> > +\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n> > +\t\t\tif (ret)\n> > +\t\t\t\treturn ret;\n> > +\t\t}\n> > +\t\treturn 0;\n> > +\t}\n> > +\treturn load_pack_revindex(bitmap_git->pack);\n> > +}\n>\n> OK, this new function is used in load_bitmap(), which is used for both\n> pack and midx bitmaps. So if we have a midx bitmap, we'll\n> unconditionally load the revindex here. But:\n>\n> - why do we then load individual pack revindexes? I can believe it may\n> be necessary to meet the assumptions of some other part of the code,\n> but it would be nice to have a comment giving us some clue.\n\nGood suggestion. We will need to reference the reverse index belonging\nto individual packs in a few locations in pack-objects (for e.g.,\nwrite_reuse_object() calls offset_to_pack_pos(), and\npack_pos_to_offset(), both with arbitrary packs, not just the preferred\none).\n\nI left the comment vague; something along the lines of \"lots of routines\nin pack-objects will need these structures to be ready to use\".\n\nI think there's room for improvement there, since for e.g., `git\nrev-list --count --objects --use-bitmap-index` doesn't need to load the\nreverse indexes. But that's already the case with classic bitmaps, too,\nwhich eagerly call load_pack_revindex().\n\n> - in open_midx_bitmap_1(), we also unconditionally load the midx\n> reverse index. I think that will always happen before us here (we\n> cannot load_bitmap() a bitmap that has not been opened). So is this\n> load_midx_revindex() call always a noop?\n\nGreat catch. I removed the call to load_midx_revindex(), and replaced it\nwith a comment explaining why we don't need to call it (because we\nalready did).\n\n> > static int bitmap_position(struct bitmap_index *bitmap_git,\n> > \t\t\t const struct object_id *oid)\n> > {\n> > -\tint pos = bitmap_position_packfile(bitmap_git, oid);\n> > +\tint pos;\n> > +\tif (bitmap_is_midx(bitmap_git))\n> > +\t\tpos = bitmap_position_midx(bitmap_git, oid);\n> > +\telse\n> > +\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n> > \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n> > }\n>\n> Makes sense. Not new in your patch, but this \"int\" return is fudging the\n> same 32-bit space we were talking about elsewhere (i.e., \"pos\" really\n> could be 2^32, or even more due to extended objects).\n\n:-). It bothers me to no end, too, because of all of the recent effort\nto improve the reverse-index APIs to avoid exactly this issue. But I\ntend to agree that the concern is more theoretical than anything,\nbecause we're only using the MSB, so the remaining 2^31 possible objects\nstill seems pretty generous.\n\n> In practice I think even 2^31 objects is pretty out-of-reach, but it may\n> be worth changing the return type (and the callers), or even just\n> catching the overflow with an assertion.\n\nPossibly, but keep in mind that the former is basically the same\nrefactor as we did with the \"tell me whether this object was found via\nthis extra pointer\". But bitmap_position() has a lot more callers than\nthat, so the plumbing required would be a little more prevalent.\n\nSo I'd be content to just punt on it for now, if you'd be OK with it.\n\n> > \tif (pos < bitmap_num_objects(bitmap_git)) {\n> > -\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n> > +\t\tstruct packed_git *pack;\n> > +\t\toff_t ofs;\n> > +\n> > +\t\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n> > +\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> > +\n> > +\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> > +\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> > +\t\t} else {\n> > +\t\t\tpack = bitmap_git->pack;\n> > +\t\t\tofs = pack_pos_to_offset(pack, pos);\n> > +\t\t}\n> > +\n>\n> All of the hunks like this make perfect sense. The big problem would be\n> if we _missed_ a place that needed conversion to handle midx. But the\n> nice thing is that it would segfault quickly in such an instance. So\n> there I'm mostly relying on test coverage, plus our experience running\n> with this code at scale.\n\nYeah; I'm definitely happy to rely on our experience running this and\nrelated patches at GitHub for several months to give us confidence that\nwe didn't miss anything here.\n\n> > static void try_partial_reuse(struct bitmap_index *bitmap_git,\n> > +\t\t\t struct packed_git *pack,\n> > \t\t\t size_t pos,\n> > \t\t\t struct bitmap *reuse,\n> > \t\t\t struct pack_window **w_curs)\n> > {\n> > -\toff_t offset, header;\n> > +\toff_t offset, delta_obj_offset;\n>\n> I'm OK with all of this in one big patch. But I suspect you _could_\n> just put:\n>\n> if (bitmap_git->midx)\n> \treturn; /* partial reuse not implemented for midx yet */\n>\n> to start with, and then actually implement it later. I call out this\n> code in particular just because it's got a lot of subtleties (the\n> \"reuse\" bits are much more intimate with the assumptions of packs and\n> bitmaps than most other code).\n>\n> I'm not sure if it's worth the trouble at this point or not.\n\nYeah, I'd definitely err on the side of not splitting this up now,\nespecially since you've already gone through the whole patch and\nreviewed it. (Of course, if your response was \"this patch is way too\nbig, please split it up so I can more easily review it\", that would be a\ndifferent story).\n\nBut I appreciate the advice, since I have felt that a lot of these\nformat-level changes require a 3-patch arc where:\n\n - The first patch describes the new format in Documentation/technical.\n - The second patch implements support for reading files that are\n written in the new format.\n - And finally, the third patch implements support for writing such\n files.\n\n...and it's usually the second of those three patches that is the most\ncomplicated one by far. So this is a good way to split that patch up\ninto many pieces.\n\nOf course, that only works if you delay adding tests until after support\nis added for all parts of the new format, but that's more-or-less what\ndid here.\n\n> > +static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n> > +{\n> > +\tstruct multi_pack_index *m = bitmap_git->midx;\n> > +\tif (!m)\n> > +\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n> > +\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n> > +}\n>\n> This part is really subtle. We infer the preferred pack by looking at\n> the pack of the 0th bit position. In general that works, since that's\n> part of the definition of the preferred pack.\n>\n> Could this ever be fooled if we had a preferred pack with 0 objects in\n> it? I don't know why we would have such a thing, but just trying to\n> think of cases where our assumptions might not hold (and what bad things\n> could happen).\n\nAn empty preferred pack would cause a problem, yes. The solution is\ntwo-fold (and incorporated into the reroll that I plan on sending\nshortly):\n\n - When the user specifies --preferred-pack, the MIDX code must make\n sure that the given pack is non-empty. That's a new patch, and\n basically adds a new conditional (to check the pack itself) and a\n test (to make sure that we catch the case we are trying to prevent).\n\n - When the user doesn't specify --preferred-pack (and instead asks us\n to infer one for them) we want to select not just the oldest pack,\n but the oldest *non-empty* pack. That is folded into the \"midx:\n infer preferred pack when not given one\" patch.\n\nIn that patch, I made a note, but I think that it's subtle enough to\nmerit sharing here again. In the loop over all packs, the conditional for swapping out the oldest pack for the current one was something like:\n\n if (p->mtime < oldest->mtime)\n oldest = p;\n\nbut now we want it to be:\n\n if (!oldest->num_objects || p->mtime < oldest->mtime)\n oldest = p;\n\nto reject packs that have no objects. And we want to be extra careful in\nthe case where the only pack fed to the MIDX writer was empty. But we\ndon't have to do anything there, since there are no objects to write\nanyway, so any \"preferred_idx\" would be fine.\n\n> > +\tif (bitmap_is_midx(bitmap_git))\n> > +\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n> > +\telse\n> > +\t\tpack = bitmap_git->pack;\n> > +\tobjects_nr = pack->num_objects;\n> > +\n> > \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n> > \t\ti++;\n> >\n> > -\t/* Don't mark objects not in the packfile */\n> > +\t/*\n> > +\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n> > +\t * marks objects eligible for reuse, but the pack-reuse code only\n> > +\t * understands how to reuse a single pack. Since the preferred pack is\n> > +\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n> > +\t * we use it instead of another pack. In single-pack bitmaps, the choice\n> > +\t * is made for us.\n> > +\t */\n> > \tif (i > objects_nr / BITS_IN_EWORD)\n> > \t\ti = objects_nr / BITS_IN_EWORD;\n>\n> OK, so this clamps our \"quick\" contiguous set of bits to the number of\n> objects in the preferred pack. Makes sense. And then we hit the\n> object-by-object loop below...\n>\n> > @@ -1213,7 +1437,15 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n> > \t\t\t\tbreak;\n> >\n> > \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> > -\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n> > +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * Can't reuse from a non-preferred pack (see\n> > +\t\t\t\t * above).\n> > +\t\t\t\t */\n> > +\t\t\t\tif (pos + offset >= objects_nr)\n> > +\t\t\t\t\tcontinue;\n> > +\t\t\t}\n> > +\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n>\n> ...and this likewise makes sure we never go past that first pack. Good.\n>\n> I think this \"continue\" could actually be a \"break\", as the loop is\n> iterating over \"offset\" (and \"pos + offset\" always gets larger). In\n> fact, it could break out of the outer loop as well (which is\n> incrementing \"pos\"). It's probably a pretty small efficiency in\n> practice, though.\n\nYeah; you're right. And we'll save up to BITS_IN_EWORD cycles of this\nloop. (I wonder if smart-enough compilers will realize the same\noptimization that you did and turn that `continue` into a `break`\nautomatically, but that's neither here nor there).\n\n> > @@ -1511,8 +1749,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n> > \t\tstruct object_id oid;\n> > \t\tstruct object_entry *oe;\n> >\n> > -\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n> > -\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n> > +\t\tif (bitmap_is_midx(bitmap_git))\n> > +\t\t\tnth_midxed_object_oid(&oid,\n> > +\t\t\t\t\t bitmap_git->midx,\n> > +\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n> > +\t\telse\n> > +\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n> > +\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n> > \t\toe = packlist_find(mapping, &oid);\n>\n> Could this be using nth_bitmap_object_oid()? I guess not, because we are\n> feeding from pack_pos_to_*. I'm not sure if another helper function is\n> worth it (pack_pos_to_bitmap_index() or something?).\n\nYou're right that we can't call nth_bitmap_object_oid here directly,\nsadly. But I think your suggestion for pack_pos_to_bitmap_index() (or\nsimilar) would only benefit this caller, since most places that dispatch\nconditionally to either pack_pos_to_{midx,index} want to pass the result\nto a different function depending on which branch they took.\n\nDefinitely possible that I missed another case that would help, but that\nwas what I came up with after just a quick glance.\n\n> > @@ -1575,7 +1831,31 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n> > \t\t\t\tbreak;\n> >\n> > \t\t\toffset += ewah_bit_ctz64(word >> offset);\n> > -\t\t\tpos = base + offset;\n> > +\n> > +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\t\t\tuint32_t pack_pos;\n> > +\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n> > +\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n> > +\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n> > +\n> > +\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n> > +\n> > +\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n> > +\t\t\t\t\tstruct object_id oid;\n> > +\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n> > +\n> > +\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n> > +\t\t\t\t\t oid_to_hex(&oid),\n> > +\t\t\t\t\t pack_id,\n> > +\t\t\t\t\t (uintmax_t)offset);\n> > +\t\t\t\t}\n> > +\n> > +\t\t\t\tpos = pack_pos;\n> > +\t\t\t} else {\n> > +\t\t\t\tpack = bitmap_git->pack;\n> > +\t\t\t\tpos = base + offset;\n> > +\t\t\t}\n> > +\n> > \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n> > \t\t\t\t pack_pos_to_offset(pack, pos);\n> > \t\t}\n>\n> In the midx case, we have to go from midx-bitmap-pos to midx-index-pos,\n> to then get the pack/ofs combo, which then gives us a real \"pos\" in the\n> pack. I don't think there's a faster way to do it (and this is still\n> much faster than looking up objects in the pack only to check their\n> revindex).\n>\n> But then with the result, we compare the offset of \"pos\" and \"pos + 1\".\n> We need to know \"pos\" to find \"pos + 1\". But in the midx case, don't we\n> already have the offset of \"pos\" (it is \"offset\" in the bitmap_is_midx()\n> conditional, which is shadowing the completely unrelated \"offset\" in the\n> outer loop).\n>\n> We could reuse it, saving ourselves an extra round-trip of pack_pos to\n> index_pos to offset. It would just mean stuffing the \"total +=\" line\n> into the two sides of the conditional.\n\nYep; agreed. And it allows us to clean up a few little other things, so\nI squashed it in. Thanks for the suggestion!\n\n> > +off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos)\n> > +{\n> > +\tif (bitmap_is_midx(bitmap_git))\n> > +\t\treturn nth_midxed_offset(bitmap_git->midx,\n> > +\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, pos));\n> > +\treturn nth_packed_object_offset(bitmap_git->pack,\n> > +\t\t\t\t\tpack_pos_to_index(bitmap_git->pack, pos));\n> > +}\n>\n> Does anybody call this function? I don't see any users by the end of the\n> series.\n\nNope, great catch. I looked at callers of nth_midxed_offset and\nnth_packed_object_offset to see if they could use this function and\nweren't, but the spots I looked at didn't appear to be able that they\nwould be helped by the existence of this helper, so I just removed it.\n\nPhew! This was quite the email to respond to, but I suppose that's my\nfault for writing such a monstrous patch. Thank you for taking the time\nto read through it all so carefully. I think you got the short end of\nthe stick between writing this email and responding to it, so thank you\n:-).\n\nThanks,\nTaylor\n"},{"id":"430987","messageId":"YPpx0KoGUX0KfdSw@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPhXb9Zns8S6aIod@nand.local","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T07:37:52Z","receivedAt":"2021-07-23T07:37:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 01:20:47PM -0400, Taylor Blau wrote:\n\n> > > @@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> > > \t\tstruct commit *child;\n> > > \t\tint reused = 0;\n> > >\n> > > -\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n> > > -\t\t\t\t old_bitmap, mapping);\n> > > +\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n> > > +\t\t\t\t old_bitmap, mapping) < 0) {\n> > > +\t\t\tclosed = 0;\n> > > +\t\t\tbreak;\n> > > +\t\t}\n> > >\n> > > \t\tif (ent->selected) {\n> > > \t\t\tstore_selected(ent, commit);\n> >\n> > This is the right thing to do because we still want to free memory, stop\n> > progress, etc. I gave a look over what will run after breaking out of\n> > the loop, and compute_xor_offsets(), which you already handled, is the\n> > only thing we'd want to avoid running. Good.\n> \n> Right. The key is that we return \"closed ? 0 : -1\" (of course, being\n> careful to invert \"closed\" where \"1\" OK into a suitable return value for\n> bitmap_writer_build, where \"0\" means OK, and a negative number means\n> \"error\").\n> \n> While I'm thinking about that inversion, we *could* call this variable\n> \"open\" and set it to \"0\" until proven otherwise. Then the conditional\n> becomes \"if (!open)\", but the return value is still \"return open ? -1 :\n> 0\" (since I assume we'd want to use 0/1 values for \"open\" instead of -1,\n> meaning we'd have to do some translation).\n\nI thought about suggesting that it be called \"err\" or \"ret\" or\nsomething. And then we do not have to care that fill_bitmap_commit()\nonly returns an error in the non-closed state. We are simply propagating\nits error-return back up the stack.\n\nAnd then you can just write:\n\n ret = fill_bitmap_commit(...);\n if (ret < 0)\n\tbreak;\n\n ...\n return ret;\n\nwithout an extra conversion. I don't care that much either way, though\n(but if you like it and are re-rolling anyway... :) ).\n\n-Peff\n"},{"id":"430990","messageId":"YPpyLaM1Rc9+w6KT@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPhYFnudmHJ9lQek@nand.local","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T07:39:25Z","receivedAt":"2021-07-23T07:39:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 01:23:34PM -0400, Taylor Blau wrote:\n\n> > I don't know if it's better to have a poorly-formatted HTML file, or\n> > none at all. :)\n> >\n> > Personally, I would just read the source. And I have a slight concern\n> > that if we start \"cleaning it up\" to render as asciidoc, the source\n> > might end up a lot less readable (though I'd reserve judgement until\n> > actually seeing it).\n> \n> Yeah, the actual source is pretty readable (and it's what I had been\n> looking at, although it is sometimes convenient to have a version I can\n> read in my web browser). But it's definitely not good Asciidoc.\n> \n> I briefly considered cleaning it up, but decided against it. Usually I\n> would opt to clean it up, but this series is already so large that I\n> figured it would make a negative impact on the reviewer experience to\n> read a clean-up patch here.\n> \n> I wouldn't be opposed to coming back to it in the future, once the dust\n> settles. I guess we can consider this #leftoverbits until then.\n\nYeah, I definitely don't want to see that cleanup as a dependency for\nthis series. It's already long enough as it is. Coming back to it later\nis just fine with me.\n\nThe question here is: should we continue to omit it from the html build,\nsince it does not render well (i.e., should we simply drop this patch).\n\n-Peff\n"},{"id":"430991","messageId":"YPpzsU7GHpqXA/1H@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPhfJNubkJpOn4Sm@nand.local","subject":"Re: [PATCH v2 05/24] Documentation: describe MIDX-based bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T07:45:53Z","receivedAt":"2021-07-23T07:45:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 01:53:40PM -0400, Taylor Blau wrote:\n\n> > > +\tThe ordering between packs is done lexicographically by the pack name,\n> > > +\twith the exception of the preferred pack, which sorts ahead of all other\n> > > +\tpacks.\n> >\n> > Hmm, I'm not sure if this \"lexicographically\" part is true. Really we're\n> > building on the midx .rev format here. And that says \"defined by the\n> > MIDX's pack list\" (though I can't offhand remember if that is\n> > lexicographic, or if it is in the reverse-mtime order).\n> >\n> > At any rate, should we just be referencing the rev documentation?\n> \n> The packs are listed in lex order in the MIDX, but that is so we can\n> binary search that list to determine whether a pack is included in the\n> MIDX or not.\n> \n> I had to check, but we do use the lex order to resolve duplicate\n> objects, too. See (at the tip of this branch):\n> \n> QSORT(ctx.info, ctx.nr, pack_info_compare);\n> \n> from within midx.c:write_midx_internal(). Here, ctx.info contains the\n> list of packs, and pack_info_compare is a thin wrapper around\n> strcmp()-ing the pack_name values of two packed_git structures.\n\nAh, OK, thanks for checking.\n\n> Arguably, you'd get better EWAH compression of the bits between packs\n> if we sorted packs in reverse order according to their mtime. But I\n> suspect that it doesn't matter much in practice, since the number of\n> objects vastly outpaces the number of packs (but I haven't measured to\n> be certain, so take that with a grain of salt).\n\nAgreed, especially when the intended use is with geometric repacking to\nkeep reasonable-sized packs.\n\nEither way, I think heuristics to optimize the pack ordering can easily\ncome on top later. Let's keep this series focused on the fundamentals of\nhaving midx bitmaps at all.\n\n> In any case, I think that you're right that adding too much detail hurts\n> us here, so we should really be mentioning the MIDX's .rev-file\n> documentation (unfortunately, we can't linkgit it, so mentioning it by\n> name will have to suffice). I plan to reroll with something like this on\n> top:\n> \n> diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\n> index 25221c7ec8..04b3ec2178 100644\n> --- a/Documentation/technical/bitmap-format.txt\n> +++ b/Documentation/technical/bitmap-format.txt\n> @@ -26,9 +26,8 @@ An object is uniquely described by its bit position within a bitmap:\n> \n> \t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n> \n> -\tThe ordering between packs is done lexicographically by the pack name,\n> -\twith the exception of the preferred pack, which sorts ahead of all other\n> -\tpacks.\n> +\tThe ordering between packs is done according to the MIDX's .rev file.\n> +\tNotably, the preferred pack sorts ahead of all other packs.\n> \n> The on-disk representation (described below) of a bitmap is the same regardless\n> of whether or not that bitmap belongs to a packfile or a MIDX. The only\n\nThanks, that looks much better. We can't linkgit, but we only build HTML\nfor these. So just a link to pack-format.html would work, as they'd\ngenerally be found side-by-side in the filesystem. But since this\ndoesn't even really render as asciidoc, I'm not sure I care either way.\n(Obviously we could also mention pack-format.txt by name, but it's\nprobably already obvious-ish to a human that this is where you'd find\ninformation on the pack .rev format).\n\n-Peff\n"},{"id":"430993","messageId":"YPp98QgXW5PQHzyy@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPhz+iOMu4Q7zjY4@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T08:29:37Z","receivedAt":"2021-07-23T08:29:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 03:22:34PM -0400, Taylor Blau wrote:\n\n> > > This avoids a problem that would arise in subsequent patches due to the\n> > > combination of 'git repack' reopening the object store in-process and\n> > > the multi-pack index code not checking whether a pack already exists in\n> > > the object store when calling add_pack_to_midx().\n> > >\n> > > This would ultimately lead to a cycle being created along the\n> > > 'packed_git' struct's '->next' pointer. That is obviously bad, but it\n> > > has hard-to-debug downstream effects like saying a bitmap can't be\n> > > loaded for a pack because one already exists (for the same pack).\n> >\n> > I'm not sure I completely understand the bug that this causes.\n> \n> Off-hand, I can't quite remember either. But it is important; I do have\n> a distinct memory of dropping this patch and then watching a 'git repack\n> --write-midx' (that option will be introduced in a later series) fail\n> horribly.\n> \n> If I remember correctly, the bug has to do with loading a MIDX twice in\n> the same process. When we call add_packed_git() from within\n> prepare_midx_pack(), we load the pack without caring whether or not it's\n> already loaded. So loading a MIDX twice in the same process will fail.\n> \n> So really I think that this is papering over that bug: we're just\n> removing one of the times that we happened to load a MIDX from during\n> the writing phase.\n\nHmm, after staring at this for a bit, I've unconfused and re-confused\nmyself several times.\n\nHere are some interesting bits:\n\n - calling load_multi_pack_index() directly creates a new midx object.\n None of its m->packs[] array will be filled in. Nor is it reachable\n as r->objects->multi_pack_index.\n\n - in using that midx, we end up calling prepare_midx_pack() for\n various packs, which creates a new packed_git struct and adds it to\n r->objects->packed_git (via install_packed_git()).\n\nSo that's a bit weird already, because we have packed_git structs in\nr->objects that came from a midx that isn't r->objects->multi_pack_index.\nAnd then if we later call prepare_multi_pack_index(), for example as\npart of a pack reprepare, then we'd end up with duplicates.\n\nWhereas normally, when a direct load_multi_pack_index() was not called,\nour only midx would be r->objects->multi_pack_index, and so we'd avoid\nre-loading it.\n\nThat seems wrong and wasteful, but I don't see how it results in a\ncircular linked list. And it seems like it would already be the case for\nthis write path, independent of your series. Either way, the solution is\nprobably for prepare_midx_pack() to check for duplicates (which we can\ndo pretty cheaply these days due to the hashmap; see prepare_pack).\n\nBut I'm worried there is something else going on. Your commit message\nmentions add_pack_to_midx(). That's something we call as part of\nwrite_midx_internal(), and it does create other packed_git structs. But\nit never calls install_packed_git() on them; they just live in the\nwrite_midx_context. So I'm not sure how they'd interfere with things.\n\nAnd then there's one final oddity. Your patch assigns to ctx.m from\nr->objects->multi_pack_index. But later in write_midx_internal(), we\ncall close_midx(). In the original, it's in the middle of the function,\nbut one of your patches puts it at the end of the function. But that\nmeans we are closing r->objects->multi_pack_index.\n\nLooking at close_midx(), it does not actually zero the struct. So we'd\nstill have r->objects->multi_pack_index->data pointed to memory which\nhas been unmapped. That seems like an accident waiting to happen. I\nguess it doesn't usually cause problems because we'd typically write a\nmidx near the end of the process, and then not look up other objects?\n\nSo I'm concerned this is introducing a subtle bug that will bite us\nlater. And we should figure out what the actual thing it's fixing is, so\nwe can understand if there is a better way to fix it (e.g., by removing\nduplicates in prepare_midx_pack(), or if it is some interaction with the\nwriting code).\n\nI guess a good thing to try would be dropping this patch and seeing if\nthe tests break. ;)\n\n-Peff\n"},{"id":"430996","messageId":"YPqC19c2sFwuOCY9@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPiAhw2eP2MOksUF@nand.local","subject":"Re: [PATCH v2 09/24] midx: infer preferred pack when not given one","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T08:50:31Z","receivedAt":"2021-07-23T08:50:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 04:16:07PM -0400, Taylor Blau wrote:\n\n> > I dunno. Like I said, I was able to follow it, so maybe it is\n> > sufficient. I'm just not sure others would be able to.\n> \n> I think that others will follow it, too. But I agree that it is\n> confusing, since we're fixing a bug that doesn't yet exist. In reality,\n> I wrote this patch after sending v1, and then reordered its position to\n> come before the implementation of MIDX bitmaps for that reason.\n> \n> So in one sense, I prefer it this way because we don't ever introduce\n> the bug. But in another sense, it is very jarring to read about an\n> interaction that has no basis in the code (yet).\n> \n> I think that the best thing we could do without adding any significant\n> reordering would be to just call out the situation we're in. I added\n> this onto the end of the commit message which I think makes things a\n> little clearer:\n> \n> (Note that multi-pack reachability bitmaps have yet to be\n> implemented; so in that sense this patch is fixing a bug which does\n> not yet exist. But by having this patch beforehand, we can prevent\n> the bug from ever materializing.)\n\nI do like fixing it up front. Here's my attempt at rewriting the commit\nmessage. I tried to omit details about pack order, and instead refer to\nthe revindex code, and instead add more explanation of how this relates\nto the pack-reuse code.\n\nSomething like:\n\n In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30),\n the multi-pack index code learned how to select a pack which all\n duplicate objects are selected from. That is, if an object appears in\n multiple packs, select the copy in the preferred pack before using one\n from any other pack.\n\n Later in that same series, 38ff7cabb6 (pack-revindex: write multi-pack\n reverse indexes, 2021-03-30) learned to put the preferred pack at the\n start of the pack order when generating a midx \".rev\" file. So far,\n that .rev ordering has not mattered. But it will be very important\n once we start using the .rev ordering for midx bitmaps.\n\n There is code in pack-objects to reuse pack bytes verbatim when\n bitmaps tell us a significant portion of the beginning of the code\n should be in the output. This code relies on the pack mentioned by the\n 0th bit also being the pack that is preferred for duplicates (because\n we'd want to make sure both bases and deltas come from the same pack).\n For a pack .bitmap, this is trivially correct. For a midx bitmap, it\n is only true when some pack gets both duplicate-priority and is placed\n at the front of the .rev file. I.e., there must be _some_ preferred\n pack.\n\n So if the user did not specify a preferred pack, we pick one\n arbitrarily.\n\n There's no test here for a few reasons:\n\n - the midx bitmap feature does not yet exist; this is preemptively\n fixing a problem before introducing buggy code\n\n - whether things go wrong with the current rules depends on things\n like readdir() order, since that is used for some midx pack\n ordering. So any test might happen to succeed or fail based on\n factors outside of our control.\n\nThoughts?\n\n-Peff\n"},{"id":"431013","messageId":"YPqOjAkG7EZ6KXd6@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPinPFW50Mj8cVkP@nand.local","subject":"Re: [PATCH v2 13/24] pack-bitmap: read multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T09:40:28Z","receivedAt":"2021-07-23T09:40:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 07:01:16PM -0400, Taylor Blau wrote:\n\n> On Wed, Jul 21, 2021 at 07:32:49AM -0400, Jeff King wrote:\n> > On Mon, Jun 21, 2021 at 06:25:31PM -0400, Taylor Blau wrote:\n> > > +\tif (!is_pack_valid(packfile)) {\n> > > +\t\tclose(fd);\n> > > +\t\treturn -1;\n> > > +\t}\n> > > +\n> >\n> > What's this extra is_pack_valid() doing? I wouldn't expect many changes\n> > at all to this non-midx code path (aside from the \"did we already load a\n> > midx bitmap\" in the earlier part of the hunk, which makes sense).\n> \n> That looks like a mistake to me. I did a little digging and tried to\n> remember if it could have ever been useful, but I think that it's just a\n> stray change that has no value. Removed.\n\nThis turned out to be quite interesting. It _is_ a mistake to include it\nin this series. But it turns out to be quite valuable on its own. :)\n\nI just cleaned it up and sent it as its own separate patch:\n\n https://lore.kernel.org/git/YPqL%2FpZt6hNYN4hB@coredump.intra.peff.net/\n\nSo it's a happy accident that your series called attention to it. :)\n\n-Peff\n"},{"id":"431016","messageId":"YPqTTwqifHIWZRsn@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YPinPFW50Mj8cVkP@nand.local","subject":"Re: [PATCH v2 13/24] pack-bitmap: read multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-23T10:00:47Z","receivedAt":"2021-07-23T10:00:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 21, 2021 at 07:01:16PM -0400, Taylor Blau wrote:\n\n> > OK, this new function is used in load_bitmap(), which is used for both\n> > pack and midx bitmaps. So if we have a midx bitmap, we'll\n> > unconditionally load the revindex here. But:\n> >\n> > - why do we then load individual pack revindexes? I can believe it may\n> > be necessary to meet the assumptions of some other part of the code,\n> > but it would be nice to have a comment giving us some clue.\n> \n> Good suggestion. We will need to reference the reverse index belonging\n> to individual packs in a few locations in pack-objects (for e.g.,\n> write_reuse_object() calls offset_to_pack_pos(), and\n> pack_pos_to_offset(), both with arbitrary packs, not just the preferred\n> one).\n> \n> I left the comment vague; something along the lines of \"lots of routines\n> in pack-objects will need these structures to be ready to use\".\n\nMakes sense. I think we _could_ be lazy-loading them, but IIRC only some\nof the revindex functions are happy to lazy-load. It's definitely fine\nto punt on that for now with a comment.\n\n> I think there's room for improvement there, since for e.g., `git\n> rev-list --count --objects --use-bitmap-index` doesn't need to load the\n> reverse indexes. But that's already the case with classic bitmaps, too,\n> which eagerly call load_pack_revindex().\n\nRight. I think our solution there was to make loading the revindex\nreally cheap (open+mmap, rather than the in-core generation). I'm\ndefinitely happy to call that fast enough for now, and if somebody wants\nto benchmark and micro-optimize cases where we can avoid loading them,\nwe can do that later.\n\n> > In practice I think even 2^31 objects is pretty out-of-reach, but it may\n> > be worth changing the return type (and the callers), or even just\n> > catching the overflow with an assertion.\n> \n> Possibly, but keep in mind that the former is basically the same\n> refactor as we did with the \"tell me whether this object was found via\n> this extra pointer\". But bitmap_position() has a lot more callers than\n> that, so the plumbing required would be a little more prevalent.\n> \n> So I'd be content to just punt on it for now, if you'd be OK with it.\n\nYeah, I think it's fine to leave it out of this series. It's not new,\nand we can revisit it later.\n\n> > Could this ever be fooled if we had a preferred pack with 0 objects in\n> > it? I don't know why we would have such a thing, but just trying to\n> > think of cases where our assumptions might not hold (and what bad things\n> > could happen).\n> \n> An empty preferred pack would cause a problem, yes. The solution is\n> two-fold (and incorporated into the reroll that I plan on sending\n> shortly):\n> \n> - When the user specifies --preferred-pack, the MIDX code must make\n> sure that the given pack is non-empty. That's a new patch, and\n> basically adds a new conditional (to check the pack itself) and a\n> test (to make sure that we catch the case we are trying to prevent).\n> \n> - When the user doesn't specify --preferred-pack (and instead asks us\n> to infer one for them) we want to select not just the oldest pack,\n> but the oldest *non-empty* pack. That is folded into the \"midx:\n> infer preferred pack when not given one\" patch.\n\nOh good, I said something useful. ;) The fix you outlined sounds\nsensible.\n\n> > > +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> > > +\t\t\t\t/*\n> > > +\t\t\t\t * Can't reuse from a non-preferred pack (see\n> > > +\t\t\t\t * above).\n> > > +\t\t\t\t */\n> > > +\t\t\t\tif (pos + offset >= objects_nr)\n> > > +\t\t\t\t\tcontinue;\n> > > +\t\t\t}\n> > > +\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n> >\n> > ...and this likewise makes sure we never go past that first pack. Good.\n> >\n> > I think this \"continue\" could actually be a \"break\", as the loop is\n> > iterating over \"offset\" (and \"pos + offset\" always gets larger). In\n> > fact, it could break out of the outer loop as well (which is\n> > incrementing \"pos\"). It's probably a pretty small efficiency in\n> > practice, though.\n> \n> Yeah; you're right. And we'll save up to BITS_IN_EWORD cycles of this\n> loop. (I wonder if smart-enough compilers will realize the same\n> optimization that you did and turn that `continue` into a `break`\n> automatically, but that's neither here nor there).\n\nIf you break all the way out, then it saves iterating over all of those\nother words that are not in the first pack, too. I.e., if your bitmap\nhas 10 million bits (for a 10-million object clone), but your first pack\nonly has a million objects in it, we'll call try_partial_reuse() 9\nmillion extra times.\n\nFortunately, each call is super cheap, because the first thing it does\nis check if the requested bit is past the end of the pack. Which kind of\nmakes me wonder if we could simplify this further by just letting\ntry_partial_reuse() tell us when there's no point going further:\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 02948e8c78..b84b55c4f3 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1308,7 +1308,11 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \treturn NULL;\n }\n \n-static void try_partial_reuse(struct bitmap_index *bitmap_git,\n+/*\n+ * -1 means \"stop trying further objects\"; 0 means we may or may not have\n+ * reused, but you can keep feeding bits.\n+ */\n+static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t\t struct packed_git *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n@@ -1342,12 +1346,12 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t */\n \n \tif (pos >= pack->num_objects)\n-\t\treturn; /* not actually in the pack or MIDX preferred pack */\n+\t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n \n \toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n \ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n-\t\treturn; /* broken packfile, punt */\n+\t\treturn -1; /* broken packfile, punt */\n \n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n@@ -1364,9 +1368,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n \t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * We assume delta dependencies always point backwards. This\n@@ -1378,7 +1382,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * odd parameters.\n \t\t */\n \t\tif (base_pos >= pos)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * And finally, if we're not sending the base as part of our\n@@ -1389,13 +1393,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * object_entry code path handle it.\n \t\t */\n \t\tif (!bitmap_get(reuse, base_pos))\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n \tbitmap_set(reuse, pos);\n+\treturn 0;\n }\n \n static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n@@ -1449,22 +1454,20 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tfor (; i < result->word_alloc; ++i) {\n \t\teword_t word = result->words[i];\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n+\t\tint ret;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tif (bitmap_is_midx(bitmap_git)) {\n-\t\t\t\t/*\n-\t\t\t\t * Can't reuse from a non-preferred pack (see\n-\t\t\t\t * above).\n-\t\t\t\t */\n-\t\t\t\tif (pos + offset >= objects_nr)\n-\t\t\t\t\tcontinue;\n-\t\t\t}\n-\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n+\t\t\tret = try_partial_reuse(bitmap_git, pack, pos + offset,\n+\t\t\t\t\t\treuse, &w_curs);\n+\t\t\tif (ret < 0)\n+\t\t\t\tbreak;\n \t\t}\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n \t}\n \n \tunuse_pack(&w_curs);\n\nThe double-ret check is kind of ugly, though I suspect compilers\noptimize it pretty well. The alternative is a \"goto\" to a label just\npast the loop (also ugly, but easily explained with a comment).\n\n> > > @@ -1511,8 +1749,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n> > > \t\tstruct object_id oid;\n> > > \t\tstruct object_entry *oe;\n> > >\n> > > -\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n> > > -\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n> > > +\t\tif (bitmap_is_midx(bitmap_git))\n> > > +\t\t\tnth_midxed_object_oid(&oid,\n> > > +\t\t\t\t\t bitmap_git->midx,\n> > > +\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n> > > +\t\telse\n> > > +\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n> > > +\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n> > > \t\toe = packlist_find(mapping, &oid);\n> >\n> > Could this be using nth_bitmap_object_oid()? I guess not, because we are\n> > feeding from pack_pos_to_*. I'm not sure if another helper function is\n> > worth it (pack_pos_to_bitmap_index() or something?).\n> \n> You're right that we can't call nth_bitmap_object_oid here directly,\n> sadly. But I think your suggestion for pack_pos_to_bitmap_index() (or\n> similar) would only benefit this caller, since most places that dispatch\n> conditionally to either pack_pos_to_{midx,index} want to pass the result\n> to a different function depending on which branch they took.\n> \n> Definitely possible that I missed another case that would help, but that\n> was what I came up with after just a quick glance.\n\nYeah, looking around, I don't see another opportunity. So the benefits\nare pretty minimal. We could do:\n\n index_pos = bitmap_is_midx(bitmap_git) ?\n pack_pos_to_midx(bitmap_git->midx, i) :\n\t pack_pos_to_index(bitmap_git->pack, i);\n nth_bitmap_object_oid(&oid, bitmap_git, index_pos);\n\nbut that is not buying much. I'm content to leave it.\n\n-Peff\n"},{"id":"431207","messageId":"YP77DiffrCrxunvg@nand.local","threadId":"55464","inReplyTo":"YPgObwXjt/tzAJvV@coredump.intra.peff.net","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T18:12:30Z","receivedAt":"2021-07-26T18:12:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 21, 2021 at 08:09:19AM -0400, Jeff King wrote:\n> On Mon, Jun 21, 2021 at 06:25:34PM -0400, Taylor Blau wrote:\n>\n> > +static int add_ref_to_pending(const char *refname,\n> > +\t\t\t const struct object_id *oid,\n> > +\t\t\t int flag, void *cb_data)\n> > +{\n> > +\tstruct rev_info *revs = (struct rev_info*)cb_data;\n> > +\tstruct object *object;\n> > +\n> > +\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n> > +\t\twarning(\"symbolic ref is dangling: %s\", refname);\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > +\tobject = parse_object_or_die(oid, refname);\n> > +\tif (object->type != OBJ_COMMIT)\n> > +\t\treturn 0;\n> > +\n> > +\tadd_pending_object(revs, object, \"\");\n> > +\tif (bitmap_is_preferred_refname(revs->repo, refname))\n> > +\t\tobject->flags |= NEEDS_BITMAP;\n> > +\treturn 0;\n> > +}\n>\n> OK, so we'll look at each ref to get the set of commits that we want to\n> traverse to put into the bitmap. Which is roughly the same as what the\n> pack bitmap does. We only generate bitmaps for all-into-one repacks, so\n> it is traversing all of the reachable objects. It is a little different\n> in that the pack version is probably hitting reflogs, but IMHO we are\n> better off to ignore reflogs for the purposes of bitmaps (I would\n> suggest to do so in the pack-bitmap case, too, except that it is\n> combined with the \"what to pack\" traversal there, and by the time we see\n> each commit we don't know how we got there).\n\nRight. And we might end up ignoring a lot of these commits, too: the\nfor-each-ref is just a starting point to enumerate everything, but we\nonly care about parts of the object graph that are contained in a pack\nwhich is included in the MIDX we are writing (hence the bare \"return\"\nyou're commenting around below).\n\n> > +static void bitmap_show_commit(struct commit *commit, void *_data)\n> > +{\n> > +\tstruct bitmap_commit_cb *data = _data;\n> > +\tif (oid_pos(&commit->object.oid, data->ctx->entries,\n> > +\t\t data->ctx->entries_nr,\n> > +\t\t bitmap_oid_access) > -1) {\n>\n> This \"> -1\" struck me as a little bit funny. Perhaps \">= 0\" would be a\n> more obvious way of saying \"we found it\"?\n\nSure. (I looked for other uses of oid_pos() to see what is more\ncommon, but there really are vanishingly few uses.) Easier to read might\neven be:\n\n int pos = oid_pos(...);\n if (pos < 0)\n return;\n ALLOC_GROW(...);\n\nwhich is what I ended up going for.\n\n> > +\t/*\n> > +\t * Pass selected commits in topo order to match the behavior of\n> > +\t * pack-bitmaps when configured with delta islands.\n> > +\t */\n> > +\trevs.topo_order = 1;\n> > +\trevs.sort_order = REV_SORT_IN_GRAPH_ORDER;\n>\n> Hmm. Why do we want to match this side effect of delta islands here?\n>\n> The only impact this has is on the order of commits we feed for bitmap\n> selection (and during the actual generation phase, it may impact\n> visitation order).\n>\n> Now I'm of the opinion that topo order is probably the best thing for\n> bitmap generation (since the bitmaps themselves are connected to the\n> graph structure). But if it is the best thing, shouldn't we perhaps be\n> turning on topo-order for single-pack bitmaps, too?\n>\n> And if it isn't the best thing, then why would we want it here?\n\nHeh, you were the one that suggested I bring this over to MIDX-based\nbitmaps in the first place ;).\n\nThis comes from an investigation into why bitmap coverage had worsened\nfor some repositories using MIDX bitmaps at GitHub. The real reason was\nresolved and unrelated to this, but trying to match the behavior of MIDX\nbitmaps to our existing pack bitmap setup (which uses delta-islands) was\none strategy we tried while debugging.\n\nI actually suspect that it doesn't really matter what order we feed this\nlist to bitmap_writer_select_commits() in, because the first thing that\nit does is QSORT() the incoming list of commits in date order.\n\nBut it does mirror the behavior of our previous bitmap generation\nsettings, which has been running for years.\n\nSo... we could probably drop this hunk? I'd probably rather err on the\nsafe side and leave this alone since it matches a system that we know to\nwork well in practice.\n\n> > +\tif (prepare_revision_walk(&revs))\n> > +\t\tdie(_(\"revision walk setup failed\"));\n>\n> We call init_revisions(), and then go straight to\n> prepare_revision_walk() with no call to setup_revisions() between. It\n> doesn't seem to be clearly documented, but I think you're supposed to,\n> as it finalizes some bits like diff_setup_done().\n>\n> I suspect it works OK in practice, and I did find a few other spots that\n> do not call it (e.g., builtin/am.c:write_commit_patch). But most spots\n> do at least an empty setup_revisions(0, NULL, &rev, NULL).\n\nSure, thanks.\n\n> > +\t/*\n> > +\t * Build the MIDX-order index based on pdata.objects (which is already\n> > +\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n> > +\t * this order).\n> > +\t */\n> > +\tALLOC_ARRAY(index, pdata.nr_objects);\n> > +\tfor (i = 0; i < pdata.nr_objects; i++)\n> > +\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n>\n> This cast is correct because the pack_idx_entry is at the start of each\n> object_entry. But maybe:\n>\n> index[i] = &pdata.objects[i].idx;\n>\n> would be less scary looking?\n\nDefinitely, and thanks (for this spot and the other one you mentioned).\n\n> > +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n>\n> Not related to your patch, but I had to refresh my memory on what this\n> \"-1\" was for. It's \"max_bitmaps\", and is ignored if it's negative. But\n> the only callers pass \"-1\"! So we could get rid of it entirely.\n>\n> It probably makes sense to leave that cleanup out of this\n> already-complicated series. But maybe worth doing later on top.\n\nYeah, seems like an easy topic for somebody interested in any\n#leftoverbits could pick up. Once this lands, I'll be happy to take care\nof it myself, too.\n\n> > @@ -930,9 +1100,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> > \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> >\n> > +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> > +\t\t\t\terror(_(\"could not load pack %s\"),\n> > +\t\t\t\t ctx.m->pack_names[i]);\n> > +\t\t\t\tresult = 1;\n> > +\t\t\t\tgoto cleanup;\n> > +\t\t\t}\n>\n> It might be worth a comment here. I can easily believe that there is\n> some later part of the bitmap generation code that assumes the packs are\n> loaded. But somebody reading this is not likely to understand why it's\n> here.\n>\n> Should this be done conditionally only if we're writing a bitmap? (That\n> might also make it obvious why we are doing it).\n\nAh. Actually, I don't think this was necessary before, but we *do* need\nit now because we want to compare the pack mtime's for inferring a\npreferred pack when one wasn't given. And we also need to open the pack\nindexes, too, because we care about the object counts (to make sure that\nwe don't infer a preferred pack which has no objects).\n\nLuckily, any new packs will be loaded (and likewise have their indexes\nopen, too), via the the add_pack_to_midx() callback that we pass as an\nargument to for_each_file_in_pack_dir().\n\nBut we could do something like this instead:\n\n--- 8< ---\n\ndiff --git a/midx.c b/midx.c\nindex 8426e1a0b1..a70a6bca81 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1111,16 +1111,29 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n\n-\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n-\t\t\t\terror(_(\"could not load pack\"));\n-\t\t\t\tresult = 1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n+\t\t\tctx.info[ctx.nr].p = NULL;\n \t\t\tctx.info[ctx.nr].expired = 0;\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n+\t\t\t\t/*\n+\t\t\t\t * If generating a reverse index, need to have\n+\t\t\t\t * packed_git's loaded to compare their\n+\t\t\t\t * mtimes and object count.\n+\t\t\t\t */\n+\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\tresult = 1;\n+\t\t\t\t\tgoto cleanup;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t ctx.m->packs[i]->pack_name);\n+\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n+\t\t\t}\n+\n \t\t\tctx.nr++;\n \t\t}\n \t}\n\n--- >8 ---\n\n> > @@ -1075,9 +1271,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> > \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n> >\n> > -\tif (ctx.m)\n> > -\t\tclose_midx(ctx.m);\n> > -\n> > \tif (ctx.nr - dropped_packs == 0) {\n> > \t\terror(_(\"no pack files to index.\"));\n> > \t\tresult = 1;\n>\n> I'm not sure what this hunk is doing. We do pick up the close_midx()\n> call at the end of the function, amidst the other cleanup.\n>\n> I expect the answer is something like \"we need it open when we generate\n> the bitmaps\". But it makes me wonder if we could hit any cases where we\n> try to overwrite it while it's still open, which would cause problems on\n> Windows.\n\nThe reason is kind of annoying. If we're building a MIDX bitmap\nin-process (e.g., `git multi-pack-index write --bitmap`) then we'll call\nprepare_packed_git() to build our pseudo-packing list which we pass to\nthe bitmap generation machinery.\n\nBut prepare_packed_git() calls prepare_packed_git_one() ->\nfor_each_file_in_pack_dir() with the prepare_pack() callback -> which\nthe wants to see if the MIDX we have open already knows about a given\npack so we avoid opening it twice.\n\nBut even though the MIDX would have gone away by this point (with the\nprevious close_midx() call that is removed above), we still hold onto\na pointer to it via the object_store's `multi_pack_index` pointer. And\nthen all the way down in packfile.c:prepare_pack() we try to pass a\nnow-defunct pointer as the first argument to midx_contains_pack(), and\ncrash.\n\nAnd clearing out that `multi_pack_index` pointer is tricky, because the\nMIDX would have to compare the odb's `object_dir` with its own (which is\nbrittle in its own right), but also would have to see if that object\nstore is pointing at *it*, and not some other MIDX.\n\nSo we do have to keep it open there. Which makes me wonder how this\ncould possibly work on Windows, because holding the MIDX open will make\nthe commit_lock_file() definitely fail. But it seems OK in the\nWindows-based CI runs?\n\nPuzzled.\n\nThanks,\nTaylor\n"},{"id":"431212","messageId":"YP79mSWIfJWtIw2B@nand.local","threadId":"55464","inReplyTo":"YP77DiffrCrxunvg@nand.local","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T18:23:21Z","receivedAt":"2021-07-26T18:23:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 26, 2021 at 02:12:30PM -0400, Taylor Blau wrote:\n> So we do have to keep it open there. Which makes me wonder how this\n> could possibly work on Windows, because holding the MIDX open will make\n> the commit_lock_file() definitely fail. But it seems OK in the\n> Windows-based CI runs?\n>\n> Puzzled.\n\nThe below should do the trick; it'll keep the MIDX open just long enough\nto generate a bitmap (if one was requested), but will close any\nhandle(s) on an existing MIDX right before we move the temporary file\ninto place.\n\nIt has the added benefit of making that hunk about destroying stale\nreferences to packs be unnecessary.\n\nWatching the Actions run here to see how this runs on Windows:\n\n https://github.com/ttaylorr/git/actions/runs/1068457013\n\nBelow is the patch.\n\n--- >8 ---\n\ncommit c7b7ce0ebc793e311072929772a2d352600f3d54\nAuthor: Taylor Blau \nDate: Mon Jul 26 14:17:27 2021 -0400\n\n fixup! pack-bitmap: write multi-pack bitmaps\n\ndiff --git a/midx.c b/midx.c\nindex 76c94a0df2..297627f992 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1358,6 +1358,8 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t}\n \t}\n\n+\tclose_midx(ctx.m);\n+\n \tcommit_lock_file(&lk);\n\n \tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n@@ -1368,15 +1370,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tif (ctx.info[i].p) {\n \t\t\tclose_pack(ctx.info[i].p);\n \t\t\tfree(ctx.info[i].p);\n-\t\t\tif (ctx.m) {\n-\t\t\t\t/*\n-\t\t\t\t * Destroy a stale reference to the pack in\n-\t\t\t\t * 'ctx.m'.\n-\t\t\t\t */\n-\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n-\t\t\t\tif (orig < ctx.m->num_packs)\n-\t\t\t\t\tctx.m->packs[orig] = NULL;\n-\t\t\t}\n \t\t}\n \t\tfree(ctx.info[i].pack_name);\n \t}\n@@ -1386,7 +1379,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n-\tclose_midx(ctx.m);\n\n \treturn result;\n }\n"},{"id":"431218","messageId":"YP8Dc9rHLrUmYQMl@nand.local","threadId":"55464","inReplyTo":"YPpx0KoGUX0KfdSw@coredump.intra.peff.net","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T18:48:19Z","receivedAt":"2021-07-26T18:48:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 23, 2021 at 03:37:52AM -0400, Jeff King wrote:\n> I thought about suggesting that it be called \"err\" or \"ret\" or\n> something. And then we do not have to care that fill_bitmap_commit()\n> only returns an error in the non-closed state. We are simply propagating\n> its error-return back up the stack.\n\nHmm. For whatever the inconvience costs us, I do like that the variable\ncan be named specifically like \"open\" or \"closed\" as opposed to the more\ngeneric \"err\" or \"ret\".\n\nSo I'll probably keep it is unless you feel strongly (which I suspect\nyou do not).\n\nThanks,\nTaylor\n"},{"id":"431219","messageId":"YP8DoCEclqD3bXKP@nand.local","threadId":"55464","inReplyTo":"YPpyLaM1Rc9+w6KT@coredump.intra.peff.net","subject":"Re: [PATCH v2 04/24] Documentation: build 'technical/bitmap-format' by default","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T18:49:04Z","receivedAt":"2021-07-26T18:49:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 23, 2021 at 03:39:25AM -0400, Jeff King wrote:\n> The question here is: should we continue to omit it from the html build,\n> since it does not render well (i.e., should we simply drop this patch).\n\nI think that's a nice way of putting it. Since the HTML rendering is\nterrible, let's just drop this patch and leave cleaning it up as\n#leftoverbits.\n\nThanks,\nTaylor\n"},{"id":"431220","messageId":"YP8F9ttlMXwNZBam@nand.local","threadId":"55464","inReplyTo":"YPp98QgXW5PQHzyy@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T18:59:02Z","receivedAt":"2021-07-26T18:59:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 23, 2021 at 04:29:37AM -0400, Jeff King wrote:\n> On Wed, Jul 21, 2021 at 03:22:34PM -0400, Taylor Blau wrote:\n>\n> > > > This avoids a problem that would arise in subsequent patches due to the\n> > > > combination of 'git repack' reopening the object store in-process and\n> > > > the multi-pack index code not checking whether a pack already exists in\n> > > > the object store when calling add_pack_to_midx().\n> > > >\n> > > > This would ultimately lead to a cycle being created along the\n> > > > 'packed_git' struct's '->next' pointer. That is obviously bad, but it\n> > > > has hard-to-debug downstream effects like saying a bitmap can't be\n> > > > loaded for a pack because one already exists (for the same pack).\n> > >\n> > > I'm not sure I completely understand the bug that this causes.\n> >\n> > Off-hand, I can't quite remember either. But it is important; I do have\n> > a distinct memory of dropping this patch and then watching a 'git repack\n> > --write-midx' (that option will be introduced in a later series) fail\n> > horribly.\n> >\n> > If I remember correctly, the bug has to do with loading a MIDX twice in\n> > the same process. When we call add_packed_git() from within\n> > prepare_midx_pack(), we load the pack without caring whether or not it's\n> > already loaded. So loading a MIDX twice in the same process will fail.\n> >\n> > So really I think that this is papering over that bug: we're just\n> > removing one of the times that we happened to load a MIDX from during\n> > the writing phase.\n>\n> Hmm, after staring at this for a bit, I've unconfused and re-confused\n> myself several times.\n>\n> Here are some interesting bits:\n>\n> - calling load_multi_pack_index() directly creates a new midx object.\n> None of its m->packs[] array will be filled in. Nor is it reachable\n> as r->objects->multi_pack_index.\n>\n> - in using that midx, we end up calling prepare_midx_pack() for\n> various packs, which creates a new packed_git struct and adds it to\n> r->objects->packed_git (via install_packed_git()).\n>\n> So that's a bit weird already, because we have packed_git structs in\n> r->objects that came from a midx that isn't r->objects->multi_pack_index.\n> And then if we later call prepare_multi_pack_index(), for example as\n> part of a pack reprepare, then we'd end up with duplicates.\n\nAh, this jogged my memory: this is a relic from when we generated MIDX\nbitmaps in-process with the rest of the `repack` code. And when we did\nthat, we did have to call `reprepare_packed_git()` after writing the new\npacks but before moving them into place.\n\nSo that's where the `reprepare_packed_git()` came from, but we don't\nhave any of that code anymore, since we now generate MIDX bitmaps by\ninvoking:\n\n git multi-pack-index write --bitmap --stdin-packs --refs-snapshot\n\nas a sub-process of `git repack`; no need for any reprepare which is\nwhat was triggering this bug.\n\nTo be sure, I reverted this patch out of GitHub's fork, and reran the\ntests both in normal mode (just `make test`) and then once more with the\n`GIT_TEST_MULTI_PACK_INDEX{,_WRITE_BITMAP}` environment variables set.\nUnsurprisingly, it passed both times.\n\nI'm happy to keep digging further, but I think that I'm 99% satisfied\nhere. Digging further involves resurrecting a much older version of this\nseries (and others adjacent to it), and there are probably other bugs\nlurking that would be annoying to tease out.\n\nIn any case, let's drop this patch from the series. It's disappointing\nthat we can't run:\n\n git -c core.multiPackIndex= multi-pack-index write\n\nanymore, but I guess that's no worse than the state we were in before\nthis patch, so I'm content to let it live on.\n\nThanks,\nTaylor\n"},{"id":"431225","messageId":"YP8QmrSd0mS26jbP@nand.local","threadId":"55464","inReplyTo":"YPqC19c2sFwuOCY9@coredump.intra.peff.net","subject":"Re: [PATCH v2 09/24] midx: infer preferred pack when not given one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T19:44:26Z","receivedAt":"2021-07-26T19:44:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 23, 2021 at 04:50:31AM -0400, Jeff King wrote:\n> On Wed, Jul 21, 2021 at 04:16:07PM -0400, Taylor Blau wrote:\n>\n> > > I dunno. Like I said, I was able to follow it, so maybe it is\n> > > sufficient. I'm just not sure others would be able to.\n> >\n> > I think that others will follow it, too. But I agree that it is\n> > confusing, since we're fixing a bug that doesn't yet exist. In reality,\n> > I wrote this patch after sending v1, and then reordered its position to\n> > come before the implementation of MIDX bitmaps for that reason.\n> >\n> > So in one sense, I prefer it this way because we don't ever introduce\n> > the bug. But in another sense, it is very jarring to read about an\n> > interaction that has no basis in the code (yet).\n> >\n> > I think that the best thing we could do without adding any significant\n> > reordering would be to just call out the situation we're in. I added\n> > this onto the end of the commit message which I think makes things a\n> > little clearer:\n> >\n> > (Note that multi-pack reachability bitmaps have yet to be\n> > implemented; so in that sense this patch is fixing a bug which does\n> > not yet exist. But by having this patch beforehand, we can prevent\n> > the bug from ever materializing.)\n>\n> I do like fixing it up front. Here's my attempt at rewriting the commit\n> message. I tried to omit details about pack order, and instead refer to\n> the revindex code, and instead add more explanation of how this relates\n> to the pack-reuse code.\n>\n> Something like:\n>\n> [...]\n>\n> Thoughts?\n\nI like it, although reading it fresh I found the sentence beginning with\n\"So if the user did not specify a preferred pack\" to be a little\nconfusing. To connect it back to the previous paragraph, I added:\n\n ... in order to avoid a situation where no pack is marked as preferred\n (breaking our assumption about the pack representing the object at the\n 0th bit).\n\nand that read out much clearer (to me at least).\n\nThanks,\nTaylor\n"},{"id":"431235","messageId":"YP8csJZ+347qAjOe@nand.local","threadId":"55464","inReplyTo":"YPqTTwqifHIWZRsn@coredump.intra.peff.net","subject":"Re: [PATCH v2 13/24] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T20:36:00Z","receivedAt":"2021-07-26T20:36:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 23, 2021 at 06:00:47AM -0400, Jeff King wrote:\n> On Wed, Jul 21, 2021 at 07:01:16PM -0400, Taylor Blau wrote:\n> > > > +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n> > > > +\t\t\t\t/*\n> > > > +\t\t\t\t * Can't reuse from a non-preferred pack (see\n> > > > +\t\t\t\t * above).\n> > > > +\t\t\t\t */\n> > > > +\t\t\t\tif (pos + offset >= objects_nr)\n> > > > +\t\t\t\t\tcontinue;\n> > > > +\t\t\t}\n> > > > +\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n> > >\n> > > ...and this likewise makes sure we never go past that first pack. Good.\n> > >\n> > > I think this \"continue\" could actually be a \"break\", as the loop is\n> > > iterating over \"offset\" (and \"pos + offset\" always gets larger). In\n> > > fact, it could break out of the outer loop as well (which is\n> > > incrementing \"pos\"). It's probably a pretty small efficiency in\n> > > practice, though.\n> >\n> > Yeah; you're right. And we'll save up to BITS_IN_EWORD cycles of this\n> > loop. (I wonder if smart-enough compilers will realize the same\n> > optimization that you did and turn that `continue` into a `break`\n> > automatically, but that's neither here nor there).\n>\n> If you break all the way out, then it saves iterating over all of those\n> other words that are not in the first pack, too. I.e., if your bitmap\n> has 10 million bits (for a 10-million object clone), but your first pack\n> only has a million objects in it, we'll call try_partial_reuse() 9\n> million extra times.\n>\n> Fortunately, each call is super cheap, because the first thing it does\n> is check if the requested bit is past the end of the pack. Which kind of\n> makes me wonder if we could simplify this further by just letting\n> try_partial_reuse() tell us when there's no point going further:\n>\n> [snip suggested diff]\n\nAll looks pretty good to me. I think that a goto is a little easier to\nread than two identical \"if (ret < 0)\" checks. And having a comment\nmakes it clearer to me than the double if statements. So I'm content do\nto that instead.\n\nThanks,\nTaylor\n"},{"id":"431242","messageId":"YP8zsR+W8JeCWc1Q@nand.local","threadId":"55464","inReplyTo":"YP8F9ttlMXwNZBam@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-26T22:14:09Z","receivedAt":"2021-07-26T22:14:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 26, 2021 at 02:59:02PM -0400, Taylor Blau wrote:\n> On Fri, Jul 23, 2021 at 04:29:37AM -0400, Jeff King wrote:\n> > On Wed, Jul 21, 2021 at 03:22:34PM -0400, Taylor Blau wrote:\n> >\n> > > > > This avoids a problem that would arise in subsequent patches due to the\n> > > > > combination of 'git repack' reopening the object store in-process and\n> > > > > the multi-pack index code not checking whether a pack already exists in\n> > > > > the object store when calling add_pack_to_midx().\n> > > > >\n> > > > > This would ultimately lead to a cycle being created along the\n> > > > > 'packed_git' struct's '->next' pointer. That is obviously bad, but it\n> > > > > has hard-to-debug downstream effects like saying a bitmap can't be\n> > > > > loaded for a pack because one already exists (for the same pack).\n> > > >\n> > > > I'm not sure I completely understand the bug that this causes.\n> > >\n> > > Off-hand, I can't quite remember either. But it is important; I do have\n> > > a distinct memory of dropping this patch and then watching a 'git repack\n> > > --write-midx' (that option will be introduced in a later series) fail\n> > > horribly.\n> > >\n> > > If I remember correctly, the bug has to do with loading a MIDX twice in\n> > > the same process. When we call add_packed_git() from within\n> > > prepare_midx_pack(), we load the pack without caring whether or not it's\n> > > already loaded. So loading a MIDX twice in the same process will fail.\n> > >\n> > > So really I think that this is papering over that bug: we're just\n> > > removing one of the times that we happened to load a MIDX from during\n> > > the writing phase.\n> >\n> > Hmm, after staring at this for a bit, I've unconfused and re-confused\n> > myself several times.\n> >\n> > Here are some interesting bits:\n> >\n> > - calling load_multi_pack_index() directly creates a new midx object.\n> > None of its m->packs[] array will be filled in. Nor is it reachable\n> > as r->objects->multi_pack_index.\n> >\n> > - in using that midx, we end up calling prepare_midx_pack() for\n> > various packs, which creates a new packed_git struct and adds it to\n> > r->objects->packed_git (via install_packed_git()).\n> >\n> > So that's a bit weird already, because we have packed_git structs in\n> > r->objects that came from a midx that isn't r->objects->multi_pack_index.\n> > And then if we later call prepare_multi_pack_index(), for example as\n> > part of a pack reprepare, then we'd end up with duplicates.\n>\n> Ah, this jogged my memory: this is a relic from when we generated MIDX\n> bitmaps in-process with the rest of the `repack` code. And when we did\n> that, we did have to call `reprepare_packed_git()` after writing the new\n> packs but before moving them into place.\n\nActually, I take that back. You were right from the start: the way the\ncode is written we *can* end up calling both:\n\n - load_multi_pack_index, from write_midx_internal(), which sets up a\n MIDX, but does not update r->objects->multi_pack_index to point at\n it.\n\n - ...and prepare_multi_pack_index_one (via prepare_bitmap_git ->\n open_bitmap -> open_midx_bitmap -> get_multi_pack_index ->\n prepare_packed_git) which *also* creates a new MIDX, *and*\n updates the_repository->objects->multi_pack_index to point at it.\n\n(The latter codepath is from the check in write_midx_internal() to see\nif we already have a MIDX bitmap when the MIDX we are trying to write\nalready exists on disk.)\n\nSo in this scenario, we have two copies of the same MIDX open, and the\nrepository's single pack is opened in one of the MIDXs, but not both.\nOne copy of the pack is pointed at via r->objects->packed_git. Then when\nwe fall back to open_pack_bitmap(), we call get_all_packs(), which calls\nprepare_midx_pack(), which installs the second MIDX's copy of the same\npack into the r->objects->packed_git, and we have a cycle.\n\nI think there are a few ways to fix this bug. The most obvious is to\nmake install_packed_git() check for the existence of the pack in the\nhashmap of installed packs before (re-)installing it. But that could be\nquadratic if the hashmap has too many collisions (and the lookup tends\ntowards being linear in the number of keys rather than constant).\n\nBut I think that a more straightforward way would be to open the MIDX we\nuse when generating the MIDX with prepare_multi_pack_index_one() instead\nof load_multi_pack_index() so that the resulting MIDX is pointed at by\nr->objects->multi_pack_index. That would prevent the latter call from\ndeep within the callstack of prepare_bitmap_git() from opening another\ncopy and then (mistakenly) re-installing the same pack twice.\n\nThoughts?\n\nThanks,\nTaylor\n"},{"id":"431317","messageId":"YQA+PaWb7tweEKuk@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YP77DiffrCrxunvg@nand.local","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T17:11:25Z","receivedAt":"2021-07-27T17:11:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 26, 2021 at 02:12:30PM -0400, Taylor Blau wrote:\n\n> > This \"> -1\" struck me as a little bit funny. Perhaps \">= 0\" would be a\n> > more obvious way of saying \"we found it\"?\n> \n> Sure. (I looked for other uses of oid_pos() to see what is more\n> common, but there really are vanishingly few uses.) Easier to read might\n> even be:\n> \n> int pos = oid_pos(...);\n> if (pos < 0)\n> return;\n> ALLOC_GROW(...);\n> \n> which is what I ended up going for.\n\nSure, that's better still.\n\n> [topo-sorting commits fed to bitmap writer]\n>\n> This comes from an investigation into why bitmap coverage had worsened\n> for some repositories using MIDX bitmaps at GitHub. The real reason was\n> resolved and unrelated to this, but trying to match the behavior of MIDX\n> bitmaps to our existing pack bitmap setup (which uses delta-islands) was\n> one strategy we tried while debugging.\n>\n> I actually suspect that it doesn't really matter what order we feed this\n> list to bitmap_writer_select_commits() in, because the first thing that\n> it does is QSORT() the incoming list of commits in date order.\n\nHmm, yes, I agree that it shouldn't matter for that reason (though\narguably topo order would still be better than a strict date order, it\ndoes nothing now).\n\nI remember looking into reasons why a single-pack bitmap versus a\nmidx-of-a-single-pack bitmap might not be identical, but the interesting\nthings turned out to be elsewhere. Did this actually change anything at\nall? If so, then perhaps the \"it doesn't matter\" is not as true as we\nare thinking. I could believe that it has a tiny impact when breaking\ntimes between identical committer dates, though.\n\n> But it does mirror the behavior of our previous bitmap generation\n> settings, which has been running for years.\n> \n> So... we could probably drop this hunk? I'd probably rather err on the\n> safe side and leave this alone since it matches a system that we know to\n> work well in practice.\n\nI'd rather drop it, if we think it's doing nothing. While I do value\nhistory in production as a sign of stability, upstream review is a good\ntime to make sure we understand all of the \"why\", and to clean things up\n(e.g., another example is the questionable close_midx() stuff discussed\nelsewhere).\n\nAnd if we do suspect it is doing something, then IMHO the right thing is\nprobably still to drop it, and to introduce the feature identically to\nboth the midx and pack bitmap generation code paths. But that should be\na separate topic (and may actually involve fixing the QSORT to put\nthings in topo order rather than just date order).\n\n> > > @@ -930,9 +1100,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > > \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> > > \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> > >\n> > > +\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> > > +\t\t\t\terror(_(\"could not load pack %s\"),\n> > > +\t\t\t\t ctx.m->pack_names[i]);\n> > > +\t\t\t\tresult = 1;\n> > > +\t\t\t\tgoto cleanup;\n> > > +\t\t\t}\n> >\n> > It might be worth a comment here. I can easily believe that there is\n> > some later part of the bitmap generation code that assumes the packs are\n> > loaded. But somebody reading this is not likely to understand why it's\n> > here.\n> >\n> > Should this be done conditionally only if we're writing a bitmap? (That\n> > might also make it obvious why we are doing it).\n> \n> Ah. Actually, I don't think this was necessary before, but we *do* need\n> it now because we want to compare the pack mtime's for inferring a\n> preferred pack when one wasn't given. And we also need to open the pack\n> indexes, too, because we care about the object counts (to make sure that\n> we don't infer a preferred pack which has no objects).\n> \n> Luckily, any new packs will be loaded (and likewise have their indexes\n> open, too), via the the add_pack_to_midx() callback that we pass as an\n> argument to for_each_file_in_pack_dir().\n\nHmm, OK. Your second paragraph makes it sound like we _don't_ need to do\nthis. But the key is \"new packs\". In add_pack_to_midx() we skip any\npacks that are already in the existing midx, assuming they've already\nbeen added. And we probably must do that, otherwise we end up with\nduplicate structs that are not actually shared by ctx->m.\n\n> But we could do something like this instead:\n> \n> --- 8< ---\n> \n> diff --git a/midx.c b/midx.c\n> index 8426e1a0b1..a70a6bca81 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1111,16 +1111,29 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> \n> -\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> -\t\t\t\terror(_(\"could not load pack\"));\n> -\t\t\t\tresult = 1;\n> -\t\t\t\tgoto cleanup;\n> -\t\t\t}\n> -\n> \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n> \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n> -\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n> +\t\t\tctx.info[ctx.nr].p = NULL;\n> \t\t\tctx.info[ctx.nr].expired = 0;\n> +\n> +\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n> +\t\t\t\t/*\n> +\t\t\t\t * If generating a reverse index, need to have\n> +\t\t\t\t * packed_git's loaded to compare their\n> +\t\t\t\t * mtimes and object count.\n> +\t\t\t\t */\n> +\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n> +\t\t\t\t\terror(_(\"could not load pack\"));\n> +\t\t\t\t\tresult = 1;\n> +\t\t\t\t\tgoto cleanup;\n> +\t\t\t\t}\n> +\n> +\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n> +\t\t\t\t\tdie(_(\"could not open index for %s\"),\n> +\t\t\t\t\t ctx.m->packs[i]->pack_name);\n> +\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n> +\t\t\t}\n> +\n\nYeah, was what I was suggesting: make it conditional on bitmaps (well, a\n.rev index, which is more precise), and put in comments. :)\n\nIt's interesting that your earlier iteration didn't call\nopen_pack_index(). Is it necessary, or not? From your description, it\nseems like it should be. But maybe some later step lazy-loads it? Even\nif so, I can see how prepare_midx_pack() would still be required\n(because we want to make sure we are using the same struct).\n\n> [closing the midx]\n> \n> The reason is kind of annoying. If we're building a MIDX bitmap\n> in-process (e.g., `git multi-pack-index write --bitmap`) then we'll call\n> prepare_packed_git() to build our pseudo-packing list which we pass to\n> the bitmap generation machinery.\n> \n> But prepare_packed_git() calls prepare_packed_git_one() ->\n> for_each_file_in_pack_dir() with the prepare_pack() callback -> which\n> the wants to see if the MIDX we have open already knows about a given\n> pack so we avoid opening it twice.\n> \n> But even though the MIDX would have gone away by this point (with the\n> previous close_midx() call that is removed above), we still hold onto\n> a pointer to it via the object_store's `multi_pack_index` pointer. And\n> then all the way down in packfile.c:prepare_pack() we try to pass a\n> now-defunct pointer as the first argument to midx_contains_pack(), and\n> crash.\n> \n> And clearing out that `multi_pack_index` pointer is tricky, because the\n> MIDX would have to compare the odb's `object_dir` with its own (which is\n> brittle in its own right), but also would have to see if that object\n> store is pointing at *it*, and not some other MIDX.\n> \n> So we do have to keep it open there. Which makes me wonder how this\n> could possibly work on Windows, because holding the MIDX open will make\n> the commit_lock_file() definitely fail. But it seems OK in the\n> Windows-based CI runs?\n\nForgetting Windows for a moment, it seems like the same \"the pointer is\nbogus and we crash\" is now just pushed down to the end of the function,\nrather than the middle. I.e., it is safe to close the midx we got from\nload_multi_pack_index(), but not one we got from\nprepare_multi_pack_index_one(). So it's hard to reason about this at all\nuntil that problem from patch 08 is resolved.\n\n-Peff\n"},{"id":"431318","messageId":"YQA+VjLMV0icRvut@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YP8Dc9rHLrUmYQMl@nand.local","subject":"Re: [PATCH v2 02/24] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T17:11:50Z","receivedAt":"2021-07-27T17:11:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 26, 2021 at 02:48:19PM -0400, Taylor Blau wrote:\n\n> On Fri, Jul 23, 2021 at 03:37:52AM -0400, Jeff King wrote:\n> > I thought about suggesting that it be called \"err\" or \"ret\" or\n> > something. And then we do not have to care that fill_bitmap_commit()\n> > only returns an error in the non-closed state. We are simply propagating\n> > its error-return back up the stack.\n> \n> Hmm. For whatever the inconvience costs us, I do like that the variable\n> can be named specifically like \"open\" or \"closed\" as opposed to the more\n> generic \"err\" or \"ret\".\n> \n> So I'll probably keep it is unless you feel strongly (which I suspect\n> you do not).\n\nI don't.\n\n-Peff\n"},{"id":"431320","messageId":"YQA/jt0mhP0QjA26@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YP8F9ttlMXwNZBam@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T17:17:02Z","receivedAt":"2021-07-27T17:17:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 26, 2021 at 02:59:02PM -0400, Taylor Blau wrote:\n\n> > Hmm, after staring at this for a bit, I've unconfused and re-confused\n> > myself several times.\n> >\n> > Here are some interesting bits:\n> >\n> > - calling load_multi_pack_index() directly creates a new midx object.\n> > None of its m->packs[] array will be filled in. Nor is it reachable\n> > as r->objects->multi_pack_index.\n> >\n> > - in using that midx, we end up calling prepare_midx_pack() for\n> > various packs, which creates a new packed_git struct and adds it to\n> > r->objects->packed_git (via install_packed_git()).\n> >\n> > So that's a bit weird already, because we have packed_git structs in\n> > r->objects that came from a midx that isn't r->objects->multi_pack_index.\n> > And then if we later call prepare_multi_pack_index(), for example as\n> > part of a pack reprepare, then we'd end up with duplicates.\n> \n> Ah, this jogged my memory: this is a relic from when we generated MIDX\n> bitmaps in-process with the rest of the `repack` code. And when we did\n> that, we did have to call `reprepare_packed_git()` after writing the new\n> packs but before moving them into place.\n> \n> So that's where the `reprepare_packed_git()` came from, but we don't\n> have any of that code anymore, since we now generate MIDX bitmaps by\n> invoking:\n> \n> git multi-pack-index write --bitmap --stdin-packs --refs-snapshot\n> \n> as a sub-process of `git repack`; no need for any reprepare which is\n> what was triggering this bug.\n\nOK, that makes sense, especially given the \"close_midx() leaves the\npointer bogus\" stuff discussed elsewhere.\n\n> To be sure, I reverted this patch out of GitHub's fork, and reran the\n> tests both in normal mode (just `make test`) and then once more with the\n> `GIT_TEST_MULTI_PACK_INDEX{,_WRITE_BITMAP}` environment variables set.\n> Unsurprisingly, it passed both times.\n> \n> I'm happy to keep digging further, but I think that I'm 99% satisfied\n> here. Digging further involves resurrecting a much older version of this\n> series (and others adjacent to it), and there are probably other bugs\n> lurking that would be annoying to tease out.\n> \n> In any case, let's drop this patch from the series. It's disappointing\n> that we can't run:\n> \n> git -c core.multiPackIndex= multi-pack-index write\n> \n> anymore, but I guess that's no worse than the state we were in before\n> this patch, so I'm content to let it live on.\n\nGreat. If we can drop it, I think that is the best path forward. I think\nthat may simplify things for the writing patch, too, then. It should not\nmatter if we move close_midx() anymore, because we will not be closing\nthe main r->objects->multi_pack_index struct.\n\nI do suspect we could be skipping the load _and_ close of the midx\nentirely in write_midx_internal(), and just using whatever the caller\nhas passed in (and arguably just having most callers pass in the regular\nmidx struct if they want us to reuse parts of it). That might be a\ncleanup we can leave for later, but it might be necessary to touch these\nbits anyway (if there is still some kind of close_midx() ordering gotcha\nin the function).\n\n-Peff\n"},{"id":"431322","messageId":"YQBCjSmdOPfrnNnK@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YP8zsR+W8JeCWc1Q@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T17:29:49Z","receivedAt":"2021-07-27T17:29:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 26, 2021 at 06:14:09PM -0400, Taylor Blau wrote:\n\n> > Ah, this jogged my memory: this is a relic from when we generated MIDX\n> > bitmaps in-process with the rest of the `repack` code. And when we did\n> > that, we did have to call `reprepare_packed_git()` after writing the new\n> > packs but before moving them into place.\n> \n> Actually, I take that back. You were right from the start: the way the\n> code is written we *can* end up calling both:\n> \n> - load_multi_pack_index, from write_midx_internal(), which sets up a\n> MIDX, but does not update r->objects->multi_pack_index to point at\n> it.\n> \n> - ...and prepare_multi_pack_index_one (via prepare_bitmap_git ->\n> open_bitmap -> open_midx_bitmap -> get_multi_pack_index ->\n> prepare_packed_git) which *also* creates a new MIDX, *and*\n> updates the_repository->objects->multi_pack_index to point at it.\n> \n> (The latter codepath is from the check in write_midx_internal() to see\n> if we already have a MIDX bitmap when the MIDX we are trying to write\n> already exists on disk.)\n> \n> So in this scenario, we have two copies of the same MIDX open, and the\n> repository's single pack is opened in one of the MIDXs, but not both.\n> One copy of the pack is pointed at via r->objects->packed_git. Then when\n> we fall back to open_pack_bitmap(), we call get_all_packs(), which calls\n> prepare_midx_pack(), which installs the second MIDX's copy of the same\n> pack into the r->objects->packed_git, and we have a cycle.\n\nRight, I understand how that ends up with duplicate structs for each\npack. But how do we get a cycle out of that?\n\n> I think there are a few ways to fix this bug. The most obvious is to\n> make install_packed_git() check for the existence of the pack in the\n> hashmap of installed packs before (re-)installing it. But that could be\n> quadratic if the hashmap has too many collisions (and the lookup tends\n> towards being linear in the number of keys rather than constant).\n\nI think it may be worth doing that anyway. You can assume the hashmap\nwill behave reasonably.\n\nBut it would mean that the \"multi_pack_index\" flag in packed_git does\nnot specify _which_ midx is pointing to it. At the very least, it would\nneed to become a ref-count (so when one midx goes away, it does not lose\nits \"I am part of a midx\" flag). And possibly it would need to actually\nknow the complete list of midx structs it's associated with (I haven't\nlooked at all of the uses of that flag).\n\nThat makes things sufficiently tricky that I would prefer not to\nuntangle it as part of this series.\n\n> But I think that a more straightforward way would be to open the MIDX we\n> use when generating the MIDX with prepare_multi_pack_index_one() instead\n> of load_multi_pack_index() so that the resulting MIDX is pointed at by\n> r->objects->multi_pack_index. That would prevent the latter call from\n> deep within the callstack of prepare_bitmap_git() from opening another\n> copy and then (mistakenly) re-installing the same pack twice.\n\nBut now the internal midx writing code can never call close_midx() on\nthat, because it does not own it to close. Can we simply drop the\nclose_midx() call there?\n\nThis would all make much more sense to me if write_midx_internal()\nsimply took a conceptually read-only midx as a parameter, and the caller\npassed in the appropriate one (probably even using\nprepare_multi_pack_index_one() to get it).\n\n-Peff\n"},{"id":"431323","messageId":"YQBEIrRfcq5dhpZn@nand.local","threadId":"55464","inReplyTo":"YQBCjSmdOPfrnNnK@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T17:36:34Z","receivedAt":"2021-07-27T17:36:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jul 27, 2021 at 01:29:49PM -0400, Jeff King wrote:\n> On Mon, Jul 26, 2021 at 06:14:09PM -0400, Taylor Blau wrote:\n> > So in this scenario, we have two copies of the same MIDX open, and the\n> > repository's single pack is opened in one of the MIDXs, but not both.\n> > One copy of the pack is pointed at via r->objects->packed_git. Then when\n> > we fall back to open_pack_bitmap(), we call get_all_packs(), which calls\n> > prepare_midx_pack(), which installs the second MIDX's copy of the same\n> > pack into the r->objects->packed_git, and we have a cycle.\n>\n> Right, I understand how that ends up with duplicate structs for each\n> pack. But how do we get a cycle out of that?\n\nSorry, it isn't a true cycle where p->next == p.\n\n> But now the internal midx writing code can never call close_midx() on\n> that, because it does not own it to close. Can we simply drop the\n> close_midx() call there?\n>\n> This would all make much more sense to me if write_midx_internal()\n> simply took a conceptually read-only midx as a parameter, and the caller\n> passed in the appropriate one (probably even using\n> prepare_multi_pack_index_one() to get it).\n\nNo, we can't drop the close_midx() call there because we must close the\nMIDX file on Windows before moving a new one into place. My feeling is\nwe should always be working on the r->objects->multi_pack_index pointer,\nand calling close_object_store() there instead of close_midx().\n\nDoes that seem like a reasonable approach to you?\n\nThanks,\nTaylor\n"},{"id":"431326","messageId":"YQBFi70c1wfXdKQf@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YQBEIrRfcq5dhpZn@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T17:42:35Z","receivedAt":"2021-07-27T17:42:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 01:36:34PM -0400, Taylor Blau wrote:\n\n> > But now the internal midx writing code can never call close_midx() on\n> > that, because it does not own it to close. Can we simply drop the\n> > close_midx() call there?\n> >\n> > This would all make much more sense to me if write_midx_internal()\n> > simply took a conceptually read-only midx as a parameter, and the caller\n> > passed in the appropriate one (probably even using\n> > prepare_multi_pack_index_one() to get it).\n> \n> No, we can't drop the close_midx() call there because we must close the\n> MIDX file on Windows before moving a new one into place. My feeling is\n> we should always be working on the r->objects->multi_pack_index pointer,\n> and calling close_object_store() there instead of close_midx().\n> \n> Does that seem like a reasonable approach to you?\n\nYes, though I'd have said that it is the responsibility of the caller\n(who knows we are operating with r->objects->multi_pack_index) to do\nthat closing. But maybe it's not possible if the rename-into-place\nhappens at too low a level.\n\nBTW, yet another weirdness: close_object_store() will call close_midx()\non the outermost midx struct, ignoring o->multi_pack_index->next\nentirely. So that's a leak, but also means we may not be closing the\nmidx we're interested in (since write_midx_internal() takes an\nobject-dir parameter, and we could be pointing to some other\nobject-dir's midx).\n\n-Peff\n"},{"id":"431329","messageId":"YQBGvEQoZpw49Z7L@nand.local","threadId":"55464","inReplyTo":"YQBFi70c1wfXdKQf@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T17:47:40Z","receivedAt":"2021-07-27T17:47:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jul 27, 2021 at 01:42:35PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 01:36:34PM -0400, Taylor Blau wrote:\n>\n> > > But now the internal midx writing code can never call close_midx() on\n> > > that, because it does not own it to close. Can we simply drop the\n> > > close_midx() call there?\n> > >\n> > > This would all make much more sense to me if write_midx_internal()\n> > > simply took a conceptually read-only midx as a parameter, and the caller\n> > > passed in the appropriate one (probably even using\n> > > prepare_multi_pack_index_one() to get it).\n> >\n> > No, we can't drop the close_midx() call there because we must close the\n> > MIDX file on Windows before moving a new one into place. My feeling is\n> > we should always be working on the r->objects->multi_pack_index pointer,\n> > and calling close_object_store() there instead of close_midx().\n> >\n> > Does that seem like a reasonable approach to you?\n>\n> Yes, though I'd have said that it is the responsibility of the caller\n> (who knows we are operating with r->objects->multi_pack_index) to do\n> that closing. But maybe it's not possible if the rename-into-place\n> happens at too low a level.\n\nRight; write_midx_internal() needs to access the MIDX right up until the\npoint that we write the new one into place, so the only place to close\nit is in write_midx_internal().\n\n> BTW, yet another weirdness: close_object_store() will call close_midx()\n> on the outermost midx struct, ignoring o->multi_pack_index->next\n> entirely. So that's a leak, but also means we may not be closing the\n> midx we're interested in (since write_midx_internal() takes an\n> object-dir parameter, and we could be pointing to some other\n> object-dir's midx).\n\nYuck, this is a mess. I'm tempted to say that we should be closing the\nMIDX that we're operating on inside of write_midx_internal() so we can\nwrite, but then declaring the whole object store to be bunk and calling\nclose_object_store() before leaving the function. Of course, one of\nthose steps should be closing the inner-most MIDX before closing the\nnext one and so on.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"431331","messageId":"YQBIqO5j0VHXL6V7@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YQBGvEQoZpw49Z7L@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T17:55:52Z","receivedAt":"2021-07-27T17:55:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 01:47:40PM -0400, Taylor Blau wrote:\n\n> > BTW, yet another weirdness: close_object_store() will call close_midx()\n> > on the outermost midx struct, ignoring o->multi_pack_index->next\n> > entirely. So that's a leak, but also means we may not be closing the\n> > midx we're interested in (since write_midx_internal() takes an\n> > object-dir parameter, and we could be pointing to some other\n> > object-dir's midx).\n> \n> Yuck, this is a mess. I'm tempted to say that we should be closing the\n> MIDX that we're operating on inside of write_midx_internal() so we can\n> write, but then declaring the whole object store to be bunk and calling\n> close_object_store() before leaving the function. Of course, one of\n> those steps should be closing the inner-most MIDX before closing the\n> next one and so on.\n\nThat gets even weirder when you look at other callers of\nwrite_midx_internal(). For instance, expire_midx_packs() is calling\nload_multi_pack_index() directly, and then passing it to\nwrite_midx_internal().\n\nSo closing the whole object store there is likewise weird.\n\nI actually think having write_midx_internal() open up a new midx is\nreasonable-ish. It's just that:\n\n - it's weird when it stuffs duplicate packs into the\n r->objects->packed_git list. But AFAICT that's not actually hurting\n anything?\n\n - we do need to make sure that the midx is closed (not just our copy,\n but any other open copies that happen to be in the same process) in\n order for things to work on Windows.\n\nSo I guess because of the second point, the internal midx write probably\nneeds to be calling close_object_store(). But because other callers use\nload_multi_pack_index(), it probably needs to be closing the one that is\npassed in, too! But of course not double-closing it if it did come from\nthe regular object store. One easy easy way to avoid that is to just\nopen a separate one.\n\nI have some spicy takes on how midx's should have been designed, but I\nthink it's probably not productive to rant about it at this point. ;)\n\n-Peff\n"},{"id":"431336","messageId":"YQBnE+ft/MR3zs1t@nand.local","threadId":"55464","inReplyTo":"YQBIqO5j0VHXL6V7@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T20:05:39Z","receivedAt":"2021-07-27T20:05:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jul 27, 2021 at 01:55:52PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 01:47:40PM -0400, Taylor Blau wrote:\n>\n> > > BTW, yet another weirdness: close_object_store() will call close_midx()\n> > > on the outermost midx struct, ignoring o->multi_pack_index->next\n> > > entirely. So that's a leak, but also means we may not be closing the\n> > > midx we're interested in (since write_midx_internal() takes an\n> > > object-dir parameter, and we could be pointing to some other\n> > > object-dir's midx).\n> >\n> > Yuck, this is a mess. I'm tempted to say that we should be closing the\n> > MIDX that we're operating on inside of write_midx_internal() so we can\n> > write, but then declaring the whole object store to be bunk and calling\n> > close_object_store() before leaving the function. Of course, one of\n> > those steps should be closing the inner-most MIDX before closing the\n> > next one and so on.\n>\n> That gets even weirder when you look at other callers of\n> write_midx_internal(). For instance, expire_midx_packs() is calling\n> load_multi_pack_index() directly, and then passing it to\n> write_midx_internal().\n>\n> So closing the whole object store there is likewise weird.\n>\n> I actually think having write_midx_internal() open up a new midx is\n> reasonable-ish. It's just that:\n>\n> - it's weird when it stuffs duplicate packs into the\n> r->objects->packed_git list. But AFAICT that's not actually hurting\n> anything?\n\nIt is hurting us when we try to write a MIDX bitmap, because we try to\nsee if one already exists. And to do that, we call prepare_bitmap_git(),\nwhich tries to call open_pack_bitmap_1 on *each* pack in the packed_git\nlist. Critically, prepare_bitmap_git() errors out if it is called with a\nbitmap_git that has a non-NULL `->pack` pointer.\n\nSo they aren't a cycle in the sense that `p->next == p`, but it does\ncause problems for us nonetheless.\n\nI stepped away from my computer for an hour or so and thought about\nthis, and I think that the solution is two-fold:\n\n - We should be more careful about freeing up the ->next pointers of a\n MIDX, and releasing the memory we allocated to hold each MIDX struct\n in the first place.\n\n - We should always be operating on the repository's\n r->objects->multi_pack_index, or any other MIDX that can be reached\n via walking the `->next` pointers. If we do that consistently, then\n we'll only have at most one instance of a MIDX struct corresponding\n to each MIDX file on disk.\n\nIn the reroll that I'll send shortly, those are:\n\n - https://github.com/ttaylorr/git/commit/61a617715f3827401522c7b08b50bb6866f2a4e9\n - https://github.com/ttaylorr/git/commit/fd15ecf47c57ce4ff0d31621c2c9f61ff7a74939\n\nrespectively. It resolves my issue locally which was that I was\npreviously unable to run:\n\n GIT_TEST_MULTI_PACK_INDEX=1 \\\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 \\\n t7700-repack.sh\n\nwithout t7700.13 failing on me (it previously complained about a\n\"duplicate\" .bitmap file, which is a side-effect of placing duplicates\nin the packed_git list, not a true duplicate .bitmap on disk).\n\nI'm waiting on a CI run [1], but I'm relatively happy with the result. I\nthink we could do something similar to the MIDX code like we did to the\ncommit-graph code in [2], but I'm reasonably happy with where we are\nnow.\n\nThanks,\nTaylor\n\n[1]: https://github.com/ttaylorr/git/actions/runs/1072513087\n[2]: https://lore.kernel.org/git/cover.1580424766.git.me@ttaylorr.com/\n"},{"id":"431337","messageId":"YQBtfRP0svLL6VDl@nand.local","threadId":"55464","inReplyTo":"YQA+PaWb7tweEKuk@coredump.intra.peff.net","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T20:33:01Z","receivedAt":"2021-07-27T20:33:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jul 27, 2021 at 01:11:25PM -0400, Jeff King wrote:\n> On Mon, Jul 26, 2021 at 02:12:30PM -0400, Taylor Blau wrote:\n> > But it does mirror the behavior of our previous bitmap generation\n> > settings, which has been running for years.\n> >\n> > So... we could probably drop this hunk? I'd probably rather err on the\n> > safe side and leave this alone since it matches a system that we know to\n> > work well in practice.\n>\n> I'd rather drop it, if we think it's doing nothing. While I do value\n> history in production as a sign of stability, upstream review is a good\n> time to make sure we understand all of the \"why\", and to clean things up\n> (e.g., another example is the questionable close_midx() stuff discussed\n> elsewhere).\n\nOK, I think that's a very reasonable way of thinking about it, so I'd\nrather just get rid of it (not to mention that I really doubt it's doing\nmuch of anything in the first place).\n\n> > Luckily, any new packs will be loaded (and likewise have their indexes\n> > open, too), via the the add_pack_to_midx() callback that we pass as an\n> > argument to for_each_file_in_pack_dir().\n>\n> Hmm, OK. Your second paragraph makes it sound like we _don't_ need to do\n> this. But the key is \"new packs\". In add_pack_to_midx() we skip any\n> packs that are already in the existing midx, assuming they've already\n> been added. And we probably must do that, otherwise we end up with\n> duplicate structs that are not actually shared by ctx->m.\n\nExactly.\n\n> It's interesting that your earlier iteration didn't call\n> open_pack_index(). Is it necessary, or not? From your description, it\n> seems like it should be. But maybe some later step lazy-loads it? Even\n> if so, I can see how prepare_midx_pack() would still be required\n> (because we want to make sure we are using the same struct).\n\nIt's only necessary now (at least for determining a preferred pack if\nthe caller didn't specify one with `--preferred-pack`) because we care\nabout reading the `num_objects` field, which the index must be loaded\nfor.\n\nThanks,\nTaylor\n"},{"id":"431341","messageId":"cover.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH v3 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:21Z","receivedAt":"2021-07-27T21:21:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll of my series to implement multi-pack reachability\nbitmaps, based on reviews from Ævar and Peff.\n\nNotable changes since last time are summarized here (though a complete\nrange-diff is below as well):\n\n - Preventing multiple copies of the same MIDX from being opened, see the new\n patches 8-9 for details.\n - Ensuring preferred packs are non-empty.\n - Simplifying a handful of routines to read from MIDX bitmaps.\n - General code clean-up, removing a few stray hunks, some commit message\n tweaking.\n\nThis reroll also dropped three patches present in v2, namely:\n\n - A patch to build Documentation/technical/bitmap-format.txt (the document is\n poorly formatted and the generated HTML isn't readable).\n - A patch to make some MIDX-related functions non-static (the required\n functions are instead exposed in the patches that first make use of them).\n - A patch to respect `core.multiPackIndex` when writing MIDXs (see patches 8-9\n for the replacement).\n\nThanks in advance for your review. I think Peff still wanted to read through\npatches 16-25, but that the first 15 or so should be in pretty good shape by\nnow.\n\nJeff King (2):\n t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n\nTaylor Blau (23):\n pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n pack-bitmap-write.c: free existing bitmaps\n Documentation: describe MIDX-based bitmaps\n midx: clear auxiliary .rev after replacing the MIDX\n midx: reject empty `--preferred-pack`'s\n midx: infer preferred pack when not given one\n midx: close linked MIDXs, avoid leaking memory\n midx: avoid opening multiple MIDXs when writing\n pack-bitmap.c: introduce 'bitmap_num_objects()'\n pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n pack-bitmap.c: avoid redundant calls to try_partial_reuse\n pack-bitmap: read multi-pack bitmaps\n pack-bitmap: write multi-pack bitmaps\n t5310: move some tests to lib-bitmap.sh\n t/helper/test-read-midx.c: add --checksum mode\n t5326: test multi-pack bitmap behavior\n t5319: don't write MIDX bitmaps in t5319\n t7700: update to work with MIDX bitmap test knob\n midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n p5310: extract full and partial bitmap tests\n p5326: perf tests for MIDX bitmaps\n\n Documentation/git-multi-pack-index.txt | 18 +-\n Documentation/technical/bitmap-format.txt | 71 ++-\n Documentation/technical/multi-pack-index.txt | 10 +-\n builtin/multi-pack-index.c | 2 +\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 12 +-\n ci/run-build-and-tests.sh | 1 +\n midx.c | 321 +++++++++++-\n midx.h | 5 +\n pack-bitmap-write.c | 79 ++-\n pack-bitmap.c | 499 ++++++++++++++++---\n pack-bitmap.h | 9 +-\n packfile.c | 2 +-\n t/README | 4 +\n t/helper/test-read-midx.c | 16 +-\n t/lib-bitmap.sh | 240 +++++++++\n t/perf/lib-bitmap.sh | 69 +++\n t/perf/p5310-pack-bitmaps.sh | 65 +--\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++\n t/t0410-partial-clone.sh | 12 +-\n t/t5310-pack-bitmaps.sh | 231 +--------\n t/t5319-multi-pack-index.sh | 20 +-\n t/t5326-multi-pack-bitmaps.sh | 277 ++++++++++\n t/t7700-repack.sh | 18 +-\n 24 files changed, 1596 insertions(+), 436 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\nRange-diff against v2:\n 1: a18baeb0b4 ! 1: fa4cbed48e pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n @@ pack-bitmap.c: void count_bitmap_commit_list(struct bitmap_index *bitmap_git,\n +\t\tbitmaps_nr++;\n +\t}\n +\n -+\tif (!bitmap_type)\n ++\tif (bitmap_type == OBJ_NONE)\n +\t\tdie(\"object %s not found in type bitmaps\",\n +\t\t oid_to_hex(&obj->oid));\n +\n 2: 3e637d9ec8 = 2: 2b15c1fc5c pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n 3: 490d733d12 = 3: 2ad513a230 pack-bitmap-write.c: free existing bitmaps\n 4: b0bb2e8051 < -: ---------- Documentation: build 'technical/bitmap-format' by default\n 5: 64a260e0c6 ! 4: 8da5de7c24 Documentation: describe MIDX-based bitmaps\n @@ Documentation/technical/bitmap-format.txt\n +\n +\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n +\n -+\tThe ordering between packs is done lexicographically by the pack name,\n -+\twith the exception of the preferred pack, which sorts ahead of all other\n -+\tpacks.\n ++\tThe ordering between packs is done according to the MIDX's .rev file.\n ++\tNotably, the preferred pack sorts ahead of all other packs.\n +\n +The on-disk representation (described below) of a bitmap is the same regardless\n +of whether or not that bitmap belongs to a packfile or a MIDX. The only\n 6: b3a12424d7 < -: ---------- midx: make a number of functions non-static\n 7: 1448ca0d2b = 5: 49297f57ed midx: clear auxiliary .rev after replacing the MIDX\n 8: dfd1daacc5 < -: ---------- midx: respect 'core.multiPackIndex' when writing\n -: ---------- > 6: c5513f2a75 midx: reject empty `--preferred-pack`'s\n 9: 9495f6869d ! 7: 53ef0a6d67 midx: infer preferred pack when not given one\n @@ Commit message\n Not specifying a preferred pack can cause serious problems with\n multi-pack reachability bitmaps, because these bitmaps rely on having at\n least one pack from which all duplicates are selected. Not having such a\n - pack causes problems with the pack reuse code (e.g., like assuming that\n - a base object was sent from that pack via reuse when in fact the base\n - was selected from a different pack).\n + pack causes problems with the code in pack-objects to reuse packs\n + verbatim (e.g., that code assumes that a delta object in a chunk of pack\n + sent verbatim will have its base object sent from the same pack).\n \n So why does not marking a pack preferred cause problems here? The reason\n is roughly as follows:\n @@ Commit message\n later).\n \n - The psuedo pack-order (described in\n - Documentation/technical/bitmap-format.txt) is computed by\n + Documentation/technical/pack-format.txt under the section\n + \"multi-pack-index reverse indexes\") is computed by\n midx_pack_order(), and sorts by pack ID and pack offset, with\n preferred packs sorting first.\n \n @@ Commit message\n order, which the bitmap code will treat as the preferred one) did *not*\n have all duplicate objects resolved in its favor, resulting in breakage.\n \n - The fix is simple: pick a (semi-arbitrary) preferred pack when none was\n - specified. This forces that pack to have duplicates resolved in its\n - favor, and (critically) to sort first in pseudo-pack order.\n - Unfortunately, testing this behavior portably isn't possible, since it\n - depends on readdir() order which isn't guaranteed by POSIX.\n + The fix is simple: pick a (semi-arbitrary, non-empty) preferred pack\n + when none was specified. This forces that pack to have duplicates\n + resolved in its favor, and (critically) to sort first in pseudo-pack\n + order. Unfortunately, testing this behavior portably isn't possible,\n + since it depends on readdir() order which isn't guaranteed by POSIX.\n +\n + (Note that multi-pack reachability bitmaps have yet to be implemented;\n + so in that sense this patch is fixing a bug which does not yet exist.\n + But by having this patch beforehand, we can prevent the bug from ever\n + materializing.)\n \n Signed-off-by: Taylor Blau \n \n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n +\t\t\t\tpreferred_pack_name);\n +\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n -+\t\ttime_t oldest = ctx.info[0].p->mtime;\n ++\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n +\t\tctx.preferred_pack_idx = 0;\n +\n +\t\tif (packs_to_drop && packs_to_drop->nr)\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\t * (and not another pack containing a duplicate)\n +\t\t */\n +\t\tfor (i = 1; i < ctx.nr; i++) {\n -+\t\t\ttime_t mtime = ctx.info[i].p->mtime;\n -+\t\t\tif (mtime < oldest) {\n -+\t\t\t\toldest = mtime;\n ++\t\t\tstruct packed_git *p = ctx.info[i].p;\n ++\n ++\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n ++\t\t\t\toldest = p;\n +\t\t\t\tctx.preferred_pack_idx = i;\n +\t\t\t}\n +\t\t}\n ++\n ++\t\tif (!oldest->num_objects) {\n ++\t\t\t/*\n ++\t\t\t * If all packs are empty; unset the preferred index.\n ++\t\t\t * This is acceptable since there will be no duplicate\n ++\t\t\t * objects to resolve, so the preferred value doesn't\n ++\t\t\t * matter.\n ++\t\t\t */\n ++\t\t\tctx.preferred_pack_idx = -1;\n ++\t\t}\n +\t} else {\n +\t\t/*\n +\t\t * otherwise don't mark any pack as preferred to avoid\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\tctx.preferred_pack_idx = -1;\n \t}\n \n - \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n + \tif (ctx.preferred_pack_idx > -1) {\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n -: ---------- > 8: 114773d9cd midx: close linked MIDXs, avoid leaking memory\n -: ---------- > 9: 40cff5beb5 midx: avoid opening multiple MIDXs when writing\n10: 373aa47528 = 10: ca7f726abf pack-bitmap.c: introduce 'bitmap_num_objects()'\n11: ac1f46aa1f ! 11: 67e6897a34 pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n @@ pack-bitmap.c: static inline uint8_t read_u8(const unsigned char *buffer, size_t\n \n #define MAX_XOR_OFFSET 160\n \n -+static void nth_bitmap_object_oid(struct bitmap_index *index,\n -+\t\t\t\t struct object_id *oid,\n -+\t\t\t\t uint32_t n)\n ++static int nth_bitmap_object_oid(struct bitmap_index *index,\n ++\t\t\t\t struct object_id *oid,\n ++\t\t\t\t uint32_t n)\n +{\n -+\tnth_packed_object_id(oid, index->pack, n);\n ++\treturn nth_packed_object_id(oid, index->pack, n);\n +}\n +\n static int load_bitmap_entries_v1(struct bitmap_index *index)\n @@ pack-bitmap.c: static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n -\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n --\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n --\t\t\t\t (unsigned)commit_idx_pos);\n -+\t\tnth_bitmap_object_oid(index, &oid, commit_idx_pos);\n ++\t\tif (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)\n + \t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n + \t\t\t\t (unsigned)commit_idx_pos);\n \n - \t\tbitmap = read_bitmap_1(index);\n - \t\tif (!bitmap)\n @@ pack-bitmap.c: static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n12: c474d2eda5 ! 12: 743a1a138e pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n @@ Commit message\n 'pack.preferBitmapTips' configuration. This patch prepares the\n multi-pack bitmap code to respect this configuration, too.\n \n - Since the multi-pack bitmap code already does a traversal of all\n - references (in order to discover the set of reachable commits in the\n - multi-pack index), it is more efficient to check whether or not each\n - reference is a suffix of any value of 'pack.preferBitmapTips' rather\n - than do an additional traversal.\n + The yet-to-be implemented code will find that it is more efficient to\n + check whether each reference contains a prefix found in the configured\n + set of values rather than doing an additional traversal.\n \n - Implement a function 'bitmap_is_preferred_refname()' which does just\n - that. The caller will be added in a subsequent patch.\n + Implement a function 'bitmap_is_preferred_refname()' which will perform\n + that check. Its caller will be added in a subsequent patch.\n \n Signed-off-by: Taylor Blau \n \n -: ---------- > 13: a3b641b3e6 pack-bitmap.c: avoid redundant calls to try_partial_reuse\n13: 7d44ba6299 ! 14: 141ff83275 pack-bitmap: read multi-pack bitmaps\n @@ Commit message\n in a MIDX.\n \n Note that there are currently no writers who write multi-pack bitmaps,\n - and that this will be implemented in the subsequent commit.\n + and that this will be implemented in the subsequent commit. Note also\n + that get_midx_checksum() and get_midx_filename() are made non-static so\n + they can be called from pack-bitmap.c.\n \n Signed-off-by: Taylor Blau \n \n @@ builtin/pack-objects.c: static void write_reused_pack(struct hashfile *f)\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \n + ## midx.c ##\n +@@ midx.c: static uint8_t oid_version(void)\n + \t}\n + }\n + \n +-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n ++const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n + {\n + \treturn m->data + m->data_len - the_hash_algo->rawsz;\n + }\n + \n +-static char *get_midx_filename(const char *object_dir)\n ++char *get_midx_filename(const char *object_dir)\n + {\n + \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n + }\n +\n + ## midx.h ##\n +@@ midx.h: struct multi_pack_index {\n + #define MIDX_PROGRESS (1 << 0)\n + #define MIDX_WRITE_REV_INDEX (1 << 1)\n + \n ++const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n ++char *get_midx_filename(const char *object_dir);\n + char *get_midx_rev_filename(struct multi_pack_index *m);\n + \n + struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n +\n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: void bitmap_writer_show_progress(int show)\n }\n @@ pack-bitmap.c: static int load_bitmap_header(struct bitmap_index *index)\n \tindex->map_pos += header_size;\n \treturn 0;\n }\n -@@ pack-bitmap.c: static void nth_bitmap_object_oid(struct bitmap_index *index,\n - \t\t\t\t struct object_id *oid,\n - \t\t\t\t uint32_t n)\n +@@ pack-bitmap.c: static int nth_bitmap_object_oid(struct bitmap_index *index,\n + \t\t\t\t struct object_id *oid,\n + \t\t\t\t uint32_t n)\n {\n --\tnth_packed_object_id(oid, index->pack, n);\n +\tif (index->midx)\n -+\t\tnth_midxed_object_oid(oid, index->midx, n);\n -+\telse\n -+\t\tnth_packed_object_id(oid, index->pack, n);\n ++\t\treturn nth_midxed_object_oid(oid, index->midx, n) ? 0 : -1;\n + \treturn nth_packed_object_id(oid, index->pack, n);\n }\n \n - static int load_bitmap_entries_v1(struct bitmap_index *index)\n @@ pack-bitmap.c: static int load_bitmap_entries_v1(struct bitmap_index *index)\n \treturn 0;\n }\n @@ pack-bitmap.c: static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, st\n \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n \t\tclose(fd);\n \t\treturn -1;\n - \t}\n - \n -+\tif (!is_pack_valid(packfile)) {\n -+\t\tclose(fd);\n -+\t\treturn -1;\n -+\t}\n -+\n - \tbitmap_git->pack = packfile;\n - \tbitmap_git->map_size = xsize_t(st.st_size);\n - \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n @@ pack-bitmap.c: static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \treturn 0;\n }\n @@ pack-bitmap.c: static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, st\n +\t\tuint32_t i;\n +\t\tint ret;\n +\n -+\t\tret = load_midx_revindex(bitmap_git->midx);\n -+\t\tif (ret)\n -+\t\t\treturn ret;\n -+\n ++\t\t/*\n ++\t\t * The multi-pack-index's .rev file is already loaded via\n ++\t\t * open_pack_bitmap_1().\n ++\t\t *\n ++\t\t * But we still need to open the individual pack .rev files,\n ++\t\t * since we will need to make use of them in pack-objects.\n ++\t\t */\n +\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n +\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n +\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n @@ pack-bitmap.c: static int open_pack_bitmap(struct repository *r,\n \n -\tif (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git))\n +\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git))\n ++\t\treturn bitmap_git;\n ++\n ++\tfree_bitmap_index(bitmap_git);\n ++\treturn NULL;\n ++}\n ++\n ++struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n ++\t\t\t\t\t struct multi_pack_index *midx)\n ++{\n ++\tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n ++\n ++\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(bitmap_git))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \n \tobject_array_clear(&revs->pending);\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n - }\n - \n - static void try_partial_reuse(struct bitmap_index *bitmap_git,\n -+\t\t\t struct packed_git *pack,\n - \t\t\t size_t pos,\n - \t\t\t struct bitmap *reuse,\n - \t\t\t struct pack_window **w_curs)\n + * reused, but you can keep feeding bits.\n + */\n + static int try_partial_reuse(struct bitmap_index *bitmap_git,\n ++\t\t\t struct packed_git *pack,\n + \t\t\t size_t pos,\n + \t\t\t struct bitmap *reuse,\n + \t\t\t struct pack_window **w_curs)\n {\n -\toff_t offset, header;\n +\toff_t offset, delta_obj_offset;\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tunsigned long size;\n \n -\tif (pos >= bitmap_num_objects(bitmap_git))\n --\t\treturn; /* not actually in the pack or MIDX */\n +-\t\treturn -1; /* not actually in the pack or MIDX */\n +\t/*\n +\t * try_partial_reuse() is called either on (a) objects in the\n +\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n -\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n -\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n +\tif (pos >= pack->num_objects)\n -+\t\treturn; /* not actually in the pack or MIDX preferred pack */\n ++\t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n +\n +\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n +\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n - \t\treturn; /* broken packfile, punt */\n + \t\treturn -1; /* broken packfile, punt */\n \n -@@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n +@@ pack-bitmap.c: static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n @@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n +\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n +\t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n - \t\t\treturn;\n + \t\t\treturn 0;\n -\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n +\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n - \t\t\treturn;\n + \t\t\treturn 0;\n \n \t\t/*\n -@@ pack-bitmap.c: static void try_partial_reuse(struct bitmap_index *bitmap_git,\n - \tbitmap_set(reuse, pos);\n +@@ pack-bitmap.c: static int try_partial_reuse(struct bitmap_index *bitmap_git,\n + \treturn 0;\n }\n \n +static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitma\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n --\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n -+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n -+\t\t\t\t/*\n -+\t\t\t\t * Can't reuse from a non-preferred pack (see\n -+\t\t\t\t * above).\n -+\t\t\t\t */\n -+\t\t\t\tif (pos + offset >= objects_nr)\n -+\t\t\t\t\tcontinue;\n -+\t\t\t}\n -+\t\t\ttry_partial_reuse(bitmap_git, pack, pos + offset, reuse, &w_curs);\n - \t\t}\n - \t}\n - \n +-\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n +-\t\t\t\t\t &w_curs) < 0) {\n ++\t\t\tif (try_partial_reuse(bitmap_git, pack, pos + offset,\n ++\t\t\t\t\t reuse, &w_curs) < 0) {\n + \t\t\t\t/*\n + \t\t\t\t * try_partial_reuse indicated we couldn't reuse\n + \t\t\t\t * any bits, so there is no point in trying more\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_\n {\n \tstruct bitmap *result = bitmap_git->result;\n -\tstruct packed_git *pack = bitmap_git->pack;\n -+\tstruct packed_git *pack;\n \toff_t total = 0;\n \tstruct ewah_iterator it;\n \teword_t filter;\n @@ pack-bitmap.c: static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n + \t\t\tcontinue;\n + \n + \t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n +-\t\t\tsize_t pos;\n +-\n + \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n -\t\t\tpos = base + offset;\n +-\t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n +-\t\t\t\t pack_pos_to_offset(pack, pos);\n +\n +\t\t\tif (bitmap_is_midx(bitmap_git)) {\n +\t\t\t\tuint32_t pack_pos;\n +\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n -+\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n +\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n +\n -+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n ++\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n ++\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n +\n +\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n +\t\t\t\t\tstruct object_id oid;\n +\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n +\n -+\t\t\t\t\tdie(_(\"could not find %s in pack #%\"PRIu32\" at offset %\"PRIuMAX),\n ++\t\t\t\t\tdie(_(\"could not find %s in pack %s at offset %\"PRIuMAX),\n +\t\t\t\t\t oid_to_hex(&oid),\n -+\t\t\t\t\t pack_id,\n ++\t\t\t\t\t pack->pack_name,\n +\t\t\t\t\t (uintmax_t)offset);\n +\t\t\t\t}\n +\n -+\t\t\t\tpos = pack_pos;\n ++\t\t\t\ttotal += pack_pos_to_offset(pack, pack_pos + 1) - offset;\n +\t\t\t} else {\n -+\t\t\t\tpack = bitmap_git->pack;\n -+\t\t\t\tpos = base + offset;\n ++\t\t\t\tsize_t pos = base + offset;\n ++\t\t\t\ttotal += pack_pos_to_offset(bitmap_git->pack, pos + 1) -\n ++\t\t\t\t\t pack_pos_to_offset(bitmap_git->pack, pos);\n +\t\t\t}\n -+\n - \t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n - \t\t\t\t pack_pos_to_offset(pack, pos);\n \t\t}\n + \t}\n + \n @@ pack-bitmap.c: off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n \treturn total;\n }\n @@ pack-bitmap.c: off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n +{\n +\treturn !!bitmap_git->midx;\n +}\n -+\n -+off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos)\n -+{\n -+\tif (bitmap_is_midx(bitmap_git))\n -+\t\treturn nth_midxed_offset(bitmap_git->midx,\n -+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, pos));\n -+\treturn nth_packed_object_offset(bitmap_git->pack,\n -+\t\t\t\t\tpack_pos_to_index(bitmap_git->pack, pos));\n -+}\n +\n const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\n \n ## pack-bitmap.h ##\n +@@ pack-bitmap.h: typedef int (*show_reachable_fn)(\n + struct bitmap_index;\n + \n + struct bitmap_index *prepare_bitmap_git(struct repository *r);\n ++struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n ++\t\t\t\t\t struct multi_pack_index *midx);\n + void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n + \t\t\t uint32_t *trees, uint32_t *blobs, uint32_t *tags);\n + void traverse_bitmap_commit_list(struct bitmap_index *,\n @@ pack-bitmap.h: void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n @@ pack-bitmap.h: void bitmap_writer_finish(struct pack_idx_entry **index,\n +char *pack_bitmap_filename(struct packed_git *p);\n +\n +int bitmap_is_midx(struct bitmap_index *bitmap_git);\n -+off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n14: a8cec2463d ! 15: 54600b5814 pack-bitmap: write multi-pack bitmaps\n @@ Documentation/git-multi-pack-index.txt: SYNOPSIS\n DESCRIPTION\n -----------\n @@ Documentation/git-multi-pack-index.txt: write::\n - \t\tmultiple packs contain the same object. If not given,\n - \t\tties are broken in favor of the pack with the lowest\n - \t\tmtime.\n + \t\tmultiple packs contain the same object. `` must\n + \t\tcontain at least one object. If not given, ties are\n + \t\tbroken in favor of the pack with the lowest mtime.\n +\n +\t--[no-]bitmap::\n +\t\tControl whether or not a multi-pack bitmap is written.\n @@ Documentation/git-multi-pack-index.txt: EXAMPLES\n +corresponding bitmap.\n ++\n +-------------------------------------------------------------\n -+$ git multi-pack-index write --preferred-pack --bitmap\n ++$ git multi-pack-index write --preferred-pack= --bitmap\n +-------------------------------------------------------------\n +\n * Write a MIDX file for the packfiles in an alternate object store.\n @@ midx.c\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n -@@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n - static void clear_midx_files_ext(struct repository *r, const char *ext,\n - \t\t\t\t unsigned char *keep_hash);\n +@@ midx.c: static int midx_checksum_valid(struct multi_pack_index *m)\n + \treturn hashfile_checksum_valid(m->data, m->data_len);\n + }\n \n +static void prepare_midx_packing_data(struct packing_data *pdata,\n +\t\t\t\t struct write_midx_context *ctx)\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +static void bitmap_show_commit(struct commit *commit, void *_data)\n +{\n +\tstruct bitmap_commit_cb *data = _data;\n -+\tif (oid_pos(&commit->object.oid, data->ctx->entries,\n -+\t\t data->ctx->entries_nr,\n -+\t\t bitmap_oid_access) > -1) {\n -+\t\tALLOC_GROW(data->commits, data->commits_nr + 1,\n -+\t\t\t data->commits_alloc);\n -+\t\tdata->commits[data->commits_nr++] = commit;\n -+\t}\n ++\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n ++\t\t\t data->ctx->entries_nr,\n ++\t\t\t bitmap_oid_access);\n ++\tif (pos < 0)\n ++\t\treturn;\n ++\n ++\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n ++\tdata->commits[data->commits_nr++] = commit;\n +}\n +\n +static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n +\t\t\t\t\t\t struct write_midx_context *ctx)\n +{\n +\tstruct rev_info revs;\n -+\tstruct bitmap_commit_cb cb;\n ++\tstruct bitmap_commit_cb cb = {0};\n +\n -+\tmemset(&cb, 0, sizeof(struct bitmap_commit_cb));\n +\tcb.ctx = ctx;\n +\n +\trepo_init_revisions(the_repository, &revs, NULL);\n ++\tsetup_revisions(0, NULL, &revs, NULL);\n +\tfor_each_ref(add_ref_to_pending, &revs);\n +\n +\t/*\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +\tfetch_if_missing = 0;\n +\trevs.exclude_promisor_objects = 1;\n +\n -+\t/*\n -+\t * Pass selected commits in topo order to match the behavior of\n -+\t * pack-bitmaps when configured with delta islands.\n -+\t */\n -+\trevs.topo_order = 1;\n -+\trevs.sort_order = REV_SORT_IN_GRAPH_ORDER;\n -+\n +\tif (prepare_revision_walk(&revs))\n +\t\tdie(_(\"revision walk setup failed\"));\n +\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +\t */\n +\tALLOC_ARRAY(index, pdata.nr_objects);\n +\tfor (i = 0; i < pdata.nr_objects; i++)\n -+\t\tindex[i] = (struct pack_idx_entry *)&pdata.objects[i];\n ++\t\tindex[i] = &pdata.objects[i].idx;\n +\n +\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n +\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +\t * bitmap_writer_finish().\n +\t */\n +\tfor (i = 0; i < pdata.nr_objects; i++)\n -+\t\tindex[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i];\n ++\t\tindex[ctx->pack_order[i]] = &pdata.objects[i].idx;\n +\n +\tbitmap_writer_select_commits(commits, commits_nr, -1);\n +\tret = bitmap_writer_build(&pdata);\n @@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *mid\n +\treturn ret;\n +}\n +\n - static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n + static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n - \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n +@@ midx.c: static int write_midx_internal(const char *object_dir,\n \n -+\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n -+\t\t\t\terror(_(\"could not load pack %s\"),\n -+\t\t\t\t ctx.m->pack_names[i]);\n -+\t\t\t\tresult = 1;\n -+\t\t\t\tgoto cleanup;\n -+\t\t\t}\n -+\n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n -\t\t\tctx.info[ctx.nr].p = NULL;\n +\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\tctx.info[ctx.nr].expired = 0;\n - \t\t\tctx.nr++;\n - \t\t}\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n + \n + \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n +@@ midx.c: static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\tint bitmap_exists;\n +\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n +\n -+\t\tbitmap_git = prepare_bitmap_git(the_repository);\n ++\t\tbitmap_git = prepare_midx_bitmap_git(the_repository, ctx.m);\n +\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n +\t\tfree_bitmap_index(bitmap_git);\n +\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \n \tif (preferred_pack_name) {\n \t\tint found = 0;\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n +@@ midx.c: static int write_midx_internal(const char *object_dir,\n \t\tif (!found)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n \t\t\t\tpreferred_pack_name);\n -\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n +\t} else if (ctx.nr &&\n +\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n - \t\ttime_t oldest = ctx.info[0].p->mtime;\n + \t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n \t\tctx.preferred_pack_idx = 0;\n \n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n +@@ midx.c: static int write_midx_internal(const char *object_dir,\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n +@@ midx.c: static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\t\tgoto cleanup;\n +\t\t}\n +\t}\n ++\n ++\tclose_midx(ctx.m);\n \n \tcommit_lock_file(&lk);\n \n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n cleanup:\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \t\tif (ctx.info[i].p) {\n - \t\t\tclose_pack(ctx.info[i].p);\n - \t\t\tfree(ctx.info[i].p);\n -+\t\t\tif (ctx.m) {\n -+\t\t\t\t/*\n -+\t\t\t\t * Destroy a stale reference to the pack in\n -+\t\t\t\t * 'ctx.m'.\n -+\t\t\t\t */\n -+\t\t\t\tuint32_t orig = ctx.info[i].orig_pack_int_id;\n -+\t\t\t\tif (orig < ctx.m->num_packs)\n -+\t\t\t\t\tctx.m->packs[orig] = NULL;\n -+\t\t\t}\n - \t\t}\n - \t\tfree(ctx.info[i].pack_name);\n - \t}\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n +@@ midx.c: static int write_midx_internal(const char *object_dir,\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n -+\tif (ctx.m)\n -+\t\tclose_midx(ctx.m);\n +\n \treturn result;\n }\n15: c63eb637c8 = 16: 168b7b0976 t5310: move some tests to lib-bitmap.sh\n16: bedb7afb37 = 17: 60ec8b3466 t/helper/test-read-midx.c: add --checksum mode\n17: fbfac4ae8e = 18: 3258ccfc1c t5326: test multi-pack bitmap behavior\n18: 2a5df1832a = 19: 47c7e6bb9b t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n19: 2d24c5b7ad = 20: 6a708858b1 t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n20: 4cbfaa0e97 = 21: 1eaa744b24 t5319: don't write MIDX bitmaps in t5319\n21: 839a7a79eb = 22: a4a899e31f t7700: update to work with MIDX bitmap test knob\n22: 00418d5b09 ! 23: 50865e52a3 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n \t\t !is_bare_repository())\n \t\t\twrite_bitmaps = 0;\n --\t}\n +\t} else if (write_bitmaps &&\n +\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n -+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n ++\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n +\t\twrite_bitmaps = 0;\n + \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n - \n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tupdate_server_info(0);\n \tremove_temporary_files();\n23: 98fa73a76a = 24: 0f1fd6e7d4 p5310: extract full and partial bitmap tests\n24: ec0f53b424 = 25: 82e8133bf4 p5326: perf tests for MIDX bitmaps\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"431342","messageId":"fa4cbed48e558d797cac318a2692e300ff7d32e2.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 01/25] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:24Z","receivedAt":"2021-07-27T21:21:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The special `--test-bitmap` mode of `git rev-list` is used to compare\nthe result of an object traversal with a bitmap to check its integrity.\nThis mode does not, however, assert that the types of reachable objects\nare stored correctly.\n\nHarden this mode by teaching it to also check that each time an object's\nbit is marked, the corresponding bit should be set in exactly one of the\ntype bitmaps (whose type matches the object's true type).\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex bfc10148f5..a73960a55d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1319,10 +1319,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,\n struct bitmap_test_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\n+\tstruct bitmap *commits;\n+\tstruct bitmap *trees;\n+\tstruct bitmap *blobs;\n+\tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n };\n \n+static void test_bitmap_type(struct bitmap_test_data *tdata,\n+\t\t\t struct object *obj, int pos)\n+{\n+\tenum object_type bitmap_type = OBJ_NONE;\n+\tint bitmaps_nr = 0;\n+\n+\tif (bitmap_get(tdata->commits, pos)) {\n+\t\tbitmap_type = OBJ_COMMIT;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->trees, pos)) {\n+\t\tbitmap_type = OBJ_TREE;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->blobs, pos)) {\n+\t\tbitmap_type = OBJ_BLOB;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->tags, pos)) {\n+\t\tbitmap_type = OBJ_TAG;\n+\t\tbitmaps_nr++;\n+\t}\n+\n+\tif (bitmap_type == OBJ_NONE)\n+\t\tdie(\"object %s not found in type bitmaps\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmaps_nr > 1)\n+\t\tdie(\"object %s does not have a unique type\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmap_type != obj->type)\n+\t\tdie(\"object %s: real type %s, expected: %s\",\n+\t\t oid_to_hex(&obj->oid),\n+\t\t type_name(obj->type),\n+\t\t type_name(bitmap_type));\n+}\n+\n static void test_show_object(struct object *object, const char *name,\n \t\t\t void *data)\n {\n@@ -1332,6 +1374,7 @@ static void test_show_object(struct object *object, const char *name,\n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&object->oid));\n+\ttest_bitmap_type(tdata, object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1346,6 +1389,7 @@ static void test_show_commit(struct commit *commit, void *data)\n \t\t\t\t &commit->object.oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&commit->object.oid));\n+\ttest_bitmap_type(tdata, &commit->object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1393,6 +1437,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \ttdata.bitmap_git = bitmap_git;\n \ttdata.base = bitmap_new();\n+\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n \ttdata.seen = 0;\n \n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431343","messageId":"2ad513a2300eec4bb4068a88cf63cea9aa6968fe.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 03/25] pack-bitmap-write.c: free existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:29Z","receivedAt":"2021-07-27T21:21:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new bitmap, the bitmap writer code attempts to read the\nexisting bitmap (if one is present). This is done in order to quickly\npermute the bits of any bitmaps for commits which appear in the existing\nbitmap, and were also selected for the new bitmap.\n\nBut since this code was added in 341fa34887 (pack-bitmap-write: use\nexisting bitmaps, 2020-12-08), the resources associated with opening an\nexisting bitmap were never released.\n\nIt's fine to ignore this, but it's bad hygiene. It will also cause a\nproblem for the multi-pack-index builtin, which will be responsible not\nonly for writing bitmaps, but also for expiring any old multi-pack\nbitmaps.\n\nIf an existing bitmap was reused here, it will also be expired. That\nwill cause a problem on platforms which require file resources to be\nclosed before unlinking them, like Windows. Avoid this by ensuring we\nclose reused bitmaps with free_bitmap_index() before removing them.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d374f7884b..142fd0adb8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tclear_prio_queue(&queue);\n \tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree_bitmap_index(old_bitmap);\n \tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431344","messageId":"8da5de7c24d274e697e0f5ae0c65720ff9e5fa4c.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 04/25] Documentation: describe MIDX-based bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:32Z","receivedAt":"2021-07-27T21:21:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Update the technical documentation to describe the multi-pack bitmap\nformat. This patch merely introduces the new format, and describes its\nhigh-level ideas. Git does not yet know how to read nor write these\nmulti-pack variants, and so the subsequent patches will:\n\n - Introduce code to interpret multi-pack bitmaps, according to this\n document.\n\n - Then, introduce code to write multi-pack bitmaps from the 'git\n multi-pack-index write' sub-command.\n\nFinally, the implementation will gain tests in subsequent patches (as\nopposed to inline with the patch teaching Git how to write multi-pack\nbitmaps) to avoid a cyclic dependency.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 71 ++++++++++++++++----\n Documentation/technical/multi-pack-index.txt | 10 +--\n 2 files changed, 60 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f8c18a0f7a..04b3ec2178 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -1,6 +1,44 @@\n GIT bitmap v1 format\n ====================\n \n+== Pack and multi-pack bitmaps\n+\n+Bitmaps store reachability information about the set of objects in a packfile,\n+or a multi-pack index (MIDX). The former is defined obviously, and the latter is\n+defined as the union of objects in packs contained in the MIDX.\n+\n+A bitmap may belong to either one pack, or the repository's multi-pack index (if\n+it exists). A repository may have at most one bitmap.\n+\n+An object is uniquely described by its bit position within a bitmap:\n+\n+\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n+\tthe __n__th object in pack order. For a function `offset` which maps\n+\tobjects to their byte offset within a pack, pack order is defined as\n+\tfollows:\n+\n+\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n+\n+\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n+\t__n__th object in MIDX order. With an additional function `pack` which\n+\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n+\tis defined as follows:\n+\n+\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n+\n+\tThe ordering between packs is done according to the MIDX's .rev file.\n+\tNotably, the preferred pack sorts ahead of all other packs.\n+\n+The on-disk representation (described below) of a bitmap is the same regardless\n+of whether or not that bitmap belongs to a packfile or a MIDX. The only\n+difference is the interpretation of the bits, which is described above.\n+\n+Certain bitmap extensions are supported (see: Appendix B). No extensions are\n+required for bitmaps corresponding to packfiles. For bitmaps that correspond to\n+MIDXs, both the bit-cache and rev-cache extensions are required.\n+\n+== On-disk format\n+\n \t- A header appears at the beginning:\n \n \t\t4-byte signature: {'B', 'I', 'T', 'M'}\n@@ -14,17 +52,19 @@ GIT bitmap v1 format\n \t\t\tThe following flags are supported:\n \n \t\t\t- BITMAP_OPT_FULL_DAG (0x1) REQUIRED\n-\t\t\tThis flag must always be present. It implies that the bitmap\n-\t\t\tindex has been generated for a packfile with full closure\n-\t\t\t(i.e. where every single object in the packfile can find\n-\t\t\t its parent links inside the same packfile). This is a\n-\t\t\trequirement for the bitmap index format, also present in JGit,\n-\t\t\tthat greatly reduces the complexity of the implementation.\n+\t\t\tThis flag must always be present. It implies that the\n+\t\t\tbitmap index has been generated for a packfile or\n+\t\t\tmulti-pack index (MIDX) with full closure (i.e. where\n+\t\t\tevery single object in the packfile/MIDX can find its\n+\t\t\tparent links inside the same packfile/MIDX). This is a\n+\t\t\trequirement for the bitmap index format, also present in\n+\t\t\tJGit, that greatly reduces the complexity of the\n+\t\t\timplementation.\n \n \t\t\t- BITMAP_OPT_HASH_CACHE (0x4)\n \t\t\tIf present, the end of the bitmap file contains\n \t\t\t`N` 32-bit name-hash values, one per object in the\n-\t\t\tpack. The format and meaning of the name-hash is\n+\t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n \t\t4-byte entry count (network byte order)\n@@ -33,7 +73,8 @@ GIT bitmap v1 format\n \n \t\t20-byte checksum\n \n-\t\t\tThe SHA1 checksum of the pack this bitmap index belongs to.\n+\t\t\tThe SHA1 checksum of the pack/MIDX this bitmap index\n+\t\t\tbelongs to.\n \n \t- 4 EWAH bitmaps that act as type indexes\n \n@@ -50,7 +91,7 @@ GIT bitmap v1 format\n \t\t\t- Tags\n \n \t\tIn each bitmap, the `n`th bit is set to true if the `n`th object\n-\t\tin the packfile is of that type.\n+\t\tin the packfile or multi-pack index is of that type.\n \n \t\tThe obvious consequence is that the OR of all 4 bitmaps will result\n \t\tin a full set (all bits set), and the AND of all 4 bitmaps will\n@@ -62,8 +103,9 @@ GIT bitmap v1 format\n \t\tEach entry contains the following:\n \n \t\t- 4-byte object position (network byte order)\n-\t\t\tThe position **in the index for the packfile** where the\n-\t\t\tbitmap for this commit is found.\n+\t\t\tThe position **in the index for the packfile or\n+\t\t\tmulti-pack index** where the bitmap for this commit is\n+\t\t\tfound.\n \n \t\t- 1-byte XOR-offset\n \t\t\tThe xor offset used to compress this bitmap. For an entry\n@@ -146,10 +188,11 @@ Name-hash cache\n ---------------\n \n If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains\n-a cache of 32-bit values, one per object in the pack. The value at\n+a cache of 32-bit values, one per object in the pack/MIDX. The value at\n position `i` is the hash of the pathname at which the `i`th object\n-(counting in index order) in the pack can be found. This can be fed\n-into the delta heuristics to compare objects with similar pathnames.\n+(counting in index or multi-pack index order) in the pack/MIDX can be found.\n+This can be fed into the delta heuristics to compare objects with similar\n+pathnames.\n \n The hash algorithm used is:\n \ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex fb688976c4..1a73c3ee20 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -71,14 +71,10 @@ Future Work\n still reducing the number of binary searches required for object\n lookups.\n \n-- The reachability bitmap is currently paired directly with a single\n- packfile, using the pack-order as the object order to hopefully\n- compress the bitmaps well using run-length encoding. This could be\n- extended to pair a reachability bitmap with a multi-pack-index. If\n- the multi-pack-index is extended to store a \"stable object order\"\n+- If the multi-pack-index is extended to store a \"stable object order\"\n (a function Order(hash) = integer that is constant for a given hash,\n- even as the multi-pack-index is updated) then a reachability bitmap\n- could point to a multi-pack-index and be updated independently.\n+ even as the multi-pack-index is updated) then MIDX bitmaps could be\n+ updated independently of the MIDX.\n \n - Packfiles can be marked as \"special\" using empty files that share\n the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431345","messageId":"2b15c1fc5cd2df1f93879829ecc1ffe6de283b5d.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 02/25] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:27Z","receivedAt":"2021-07-27T21:21:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The set of objects covered by a bitmap must be closed under\nreachability, since it must be the case that there is a valid bit\nposition assigned for every possible reachable object (otherwise the\nbitmaps would be incomplete).\n\nPack bitmaps are never written from 'git repack' unless repacking\nall-into-one, and so we never write non-closed bitmaps (except in the\ncase of partial clones where we aren't guaranteed to have all objects).\n\nBut multi-pack bitmaps change this, since it isn't known whether the\nset of objects in the MIDX is closed under reachability until walking\nthem. Plumb through a bit that is set when a reachable object isn't\nfound.\n\nAs soon as a reachable object isn't found in the set of objects to\ninclude in the bitmap, bitmap_writer_build() knows that the set is not\nclosed, and so it now fails gracefully.\n\nA test is added in t0410 to trigger a bitmap write without full\nreachability closure by removing local copies of some reachable objects\nfrom a promisor remote.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 +-\n pack-bitmap-write.c | 76 ++++++++++++++++++++++++++++------------\n pack-bitmap.h | 2 +-\n t/t0410-partial-clone.sh | 9 ++++-\n 4 files changed, 64 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex de00adbb9e..8a523624a1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1256,7 +1256,8 @@ static void write_pack_file(void)\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n-\t\t\t\tbitmap_writer_build(&to_pack);\n+\t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n+\t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..d374f7884b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit)\n \twriter.selected_nr++;\n }\n \n-static uint32_t find_object_pos(const struct object_id *oid)\n+static uint32_t find_object_pos(const struct object_id *oid, int *found)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n \n \tif (!entry) {\n-\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\tif (found)\n+\t\t\t*found = 0;\n+\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\t\treturn 0;\n \t}\n \n+\tif (found)\n+\t\t*found = 1;\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n@@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n \n-static void fill_bitmap_tree(struct bitmap *bitmap,\n-\t\t\t struct tree *tree)\n+static int fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n {\n+\tint found;\n \tuint32_t pos;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t * If our bit is already set, then there is nothing to do. Both this\n \t * tree and all of its children will be set.\n \t */\n-\tpos = find_object_pos(&tree->object.oid);\n+\tpos = find_object_pos(&tree->object.oid, &found);\n+\tif (!found)\n+\t\treturn -1;\n \tif (bitmap_get(bitmap, pos))\n-\t\treturn;\n+\t\treturn 0;\n \tbitmap_set(bitmap, pos);\n \n \tif (parse_tree(tree) < 0)\n@@ -355,11 +363,15 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \twhile (tree_entry(&desc, &entry)) {\n \t\tswitch (object_type(entry.mode)) {\n \t\tcase OBJ_TREE:\n-\t\t\tfill_bitmap_tree(bitmap,\n-\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tif (fill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid)) < 0)\n+\t\t\t\treturn -1;\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n-\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tpos = find_object_pos(&entry.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(bitmap, pos);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\t/* Gitlink, etc; not reachable */\n@@ -368,15 +380,18 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t}\n \n \tfree_tree_buffer(tree);\n+\treturn 0;\n }\n \n-static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue,\n-\t\t\t struct prio_queue *tree_queue,\n-\t\t\t struct bitmap_index *old_bitmap,\n-\t\t\t const uint32_t *mapping)\n+static int fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n+\tint found;\n+\tuint32_t pos;\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n@@ -401,11 +416,16 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\tif (!found)\n+\t\t\treturn -1;\n+\t\tbitmap_set(ent->bitmap, pos);\n \t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n-\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n \t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n \t\t\t\tbitmap_set(ent->bitmap, pos);\n \t\t\t\tprio_queue_put(queue, p->item);\n@@ -413,8 +433,12 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t}\n \t}\n \n-\twhile (tree_queue->nr)\n-\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n+\twhile (tree_queue->nr) {\n+\t\tif (fill_bitmap_tree(ent->bitmap,\n+\t\t\t\t prio_queue_get(tree_queue)) < 0)\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -432,7 +456,7 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+int bitmap_writer_build(struct packing_data *to_pack)\n {\n \tstruct bitmap_builder bb;\n \tsize_t i;\n@@ -441,6 +465,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n \tuint32_t *mapping;\n+\tint closed = 1; /* until proven otherwise */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n-\t\t\t\t old_bitmap, mapping);\n+\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping) < 0) {\n+\t\t\tclosed = 0;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -499,7 +527,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \tstop_progress(&writer.progress);\n \n-\tcompute_xor_offsets();\n+\tif (closed)\n+\t\tcompute_xor_offsets();\n+\treturn closed ? 0 : -1;\n }\n \n /**\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 99d733eb26..020cd8d868 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -87,7 +87,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n-void bitmap_writer_build(struct packing_data *to_pack);\n+int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex a211a66c67..bbcc51ee8e 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '\n repack_and_check () {\n \trm -rf repo2 &&\n \tcp -r repo repo2 &&\n-\tgit -C repo2 repack $1 -d &&\n+\tif test x\"$1\" = \"x--must-fail\"\n+\tthen\n+\t\tshift\n+\t\ttest_must_fail git -C repo2 repack $1 -d\n+\telse\n+\t\tgit -C repo2 repack $1 -d\n+\tfi &&\n \tgit -C repo2 fsck &&\n \n \tgit -C repo2 cat-file -e $2 &&\n@@ -561,6 +567,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' '\n \tprintf \"$THREE\\n\" | pack_as_from_promisor &&\n \tdelete_object repo \"$ONE\" &&\n \n+\trepack_and_check --must-fail -ab \"$TWO\" \"$THREE\" &&\n \trepack_and_check -a \"$TWO\" \"$THREE\" &&\n \trepack_and_check -A \"$TWO\" \"$THREE\" &&\n \trepack_and_check -l \"$TWO\" \"$THREE\"\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431346","messageId":"c5513f2a75d065427395c7f7e3b261819184ffaa.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 06/25] midx: reject empty `--preferred-pack`'s","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:38Z","receivedAt":"2021-07-27T21:21:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The soon-to-be-implemented multi-pack bitmap treats object in the first\nbit position specially by assuming that all objects in the pack it was\nselected from are also represented from that pack in the MIDX. In other\nwords, the pack from which the first object was selected must also have\nall of its other objects selected from that same pack in the MIDX in\ncase of any duplicates.\n\nBut this assumption relies on the fact that there is at least one object\nin that pack to begin with; otherwise the object in the first bit\nposition isn't from a preferred pack, in which case we can no longer\nassume that all objects in that pack were also selected from the same\npack.\n\nGuard this assumption by checking the number of objects in the given\npreferred pack, and failing if the given pack is empty.\n\nTo make sure we can safely perform this check, open any packs which are\ncontained in an existing MIDX via prepare_midx_pack(). The same is done\nfor new packs via the add_pack_to_midx() callback, but packs picked up\nfrom a previous MIDX will not yet have these opened.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 6 +++---\n midx.c | 29 ++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 17 +++++++++++++++\n 3 files changed, 49 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex ffd601bc17..c9b063d31e 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -37,9 +37,9 @@ write::\n --\n \t--preferred-pack=::\n \t\tOptionally specify the tie-breaking pack used when\n-\t\tmultiple packs contain the same object. If not given,\n-\t\tties are broken in favor of the pack with the lowest\n-\t\tmtime.\n+\t\tmultiple packs contain the same object. `` must\n+\t\tcontain at least one object. If not given, ties are\n+\t\tbroken in favor of the pack with the lowest mtime.\n --\n \n verify::\ndiff --git a/midx.c b/midx.c\nindex 3193426d24..092dbf45b6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -934,6 +934,25 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n \t\t\tctx.info[ctx.nr].p = NULL;\n \t\t\tctx.info[ctx.nr].expired = 0;\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n+\t\t\t\t/*\n+\t\t\t\t * If generating a reverse index, need to have\n+\t\t\t\t * packed_git's loaded to compare their\n+\t\t\t\t * mtimes and object count.\n+\t\t\t\t */\n+\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\tresult = 1;\n+\t\t\t\t\tgoto cleanup;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t ctx.m->packs[i]->pack_name);\n+\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n+\t\t\t}\n+\n \t\t\tctx.nr++;\n \t\t}\n \t}\n@@ -961,6 +980,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t}\n \t}\n \n+\tif (ctx.preferred_pack_idx > -1) {\n+\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tif (!preferred->num_objects) {\n+\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n+\t\t\t preferred->pack_name);\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n \t\t\t\t\t ctx.preferred_pack_idx);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 7609f1ea64..1f0a2ae852 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -277,6 +277,23 @@ test_expect_success 'midx picks objects from preferred pack' '\n \t)\n '\n \n+test_expect_success 'preferred packs must be non-empty' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ttest_commit base &&\n+\t\tgit repack -ad &&\n+\n+\t\tempty=\"$(git pack-objects $objdir/pack/pack err &&\n+\t\tgrep \"with no objects\" err\n+\t)\n+'\n+\n test_expect_success 'verify multi-pack-index success' '\n \tgit multi-pack-index verify --object-dir=$objdir\n '\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431347","messageId":"40cff5beb50cdfbd13ae7f6017152f2628b25814.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:46Z","receivedAt":"2021-07-27T21:21:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Opening multiple instance of the same MIDX can lead to problems like two\nseparate packed_git structures which represent the same pack being added\nto the repository's object store.\n\nThe above scenario can happen because prepare_midx_pack() checks if\n`m->packs[pack_int_id]` is NULL in order to determine if a pack has been\nopened and installed in the repository before. But a caller can\nconstruct two copies of the same MIDX by calling get_multi_pack_index()\nand load_multi_pack_index() since the former manipulates the\nobject store directly but the latter is a lower-level routine which\nallocates a new MIDX for each call.\n\nSo if prepare_midx_pack() is called on multiple MIDXs with the same\npack_int_id, then that pack will be installed twice in the object\nstore's packed_git pointer.\n\nThis can lead to problems in, for e.g., the pack-bitmap code, which does\nsomething like the following (in pack-bitmap.c:open_pack_bitmap()):\n\n struct bitmap_index *bitmap_git = ...;\n for (p = get_all_packs(r); p; p = p->next) {\n if (open_pack_bitmap_1(bitmap_git, p) == 0)\n ret = 0;\n }\n\nwhich is a problem if two copies of the same pack exist in the\npacked_git list because pack-bitmap.c:open_pack_bitmap_1() contains a\nconditional like the following:\n\n if (bitmap_git->pack || bitmap_git->midx) {\n /* ignore extra bitmap file; we can only handle one */\n warning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n close(fd);\n return -1;\n }\n\nAvoid this scenario by not letting write_midx_internal() open a MIDX\nthat isn't also pointed at by the object store. So long as this is the\ncase, other routines should prefer to open MIDXs with\nget_multi_pack_index() or reprepare_packed_git() instead of creating\ninstances on their own. Because get_multi_pack_index() returns\n`r->object_store->multi_pack_index` if it is non-NULL, we'll only have\none instance of a MIDX open at one time, avoiding these problems.\n\nTo encourage this, drop the `struct multi_pack_index *` parameter from\n`write_midx_internal()`, and rely instead on the `object_dir` to find\n(or initialize) the correct MIDX instance.\n\nLikewise, replace the call to `close_midx()` with\n`close_object_store()`, since we're about to replace the MIDX with a new\none and should invalidate the object store's memory of any MIDX that\nmight have existed beforehand.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 26 ++++++++++++++++----------\n 1 file changed, 16 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 18e1949613..d67d7f383d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n \t\t\t unsigned flags)\n@@ -904,6 +904,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n+\tstruct multi_pack_index *cur;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -914,10 +915,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name);\n \n-\tif (m)\n-\t\tctx.m = m;\n-\telse\n-\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\tctx.m = cur;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tif (!ctx.m)\n+\t\tctx.m = get_local_multi_pack_index(the_repository);\n \n \tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n \t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n@@ -1182,8 +1187,7 @@ int write_midx_file(const char *object_dir,\n \t\t const char *preferred_pack_name,\n \t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t flags);\n+\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n }\n \n struct clear_midx_data {\n@@ -1460,8 +1464,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \n \tfree(count);\n \n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n+\tif (packs_to_drop.nr) {\n+\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tm = NULL;\n+\t}\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1650,7 +1656,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431348","messageId":"ca7f726abf825c30105ae330a12e0c7d20684076.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 10/25] pack-bitmap.c: introduce 'bitmap_num_objects()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:49Z","receivedAt":"2021-07-27T21:21:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to return how many objects are\ncontained in a bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 37 +++++++++++++++++++++----------------\n 1 file changed, 21 insertions(+), 16 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex a73960a55d..54dc4f7915 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn b;\n }\n \n+static uint32_t bitmap_num_objects(struct bitmap_index *index)\n+{\n+\treturn index->pack->num_objects;\n+}\n+\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n-\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tsize_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));\n \t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n@@ -399,7 +404,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -451,7 +456,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -668,7 +673,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\tif (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -826,7 +831,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == type &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos))\n@@ -853,7 +858,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < pack->num_objects) {\n+\tif (pos < bitmap_num_objects(bitmap_git)) {\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n@@ -863,7 +868,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - pack->num_objects];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -905,7 +910,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t}\n \n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == OBJ_BLOB &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos) &&\n@@ -1131,8 +1136,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_git->pack->num_objects)\n-\t\treturn; /* not actually in the pack */\n+\tif (pos >= bitmap_num_objects(bitmap_git))\n+\t\treturn; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n@@ -1198,6 +1203,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n+\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n \n \tassert(result);\n \n@@ -1205,8 +1211,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\ti++;\n \n \t/* Don't mark objects not in the packfile */\n-\tif (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)\n-\t\ti = bitmap_git->pack->num_objects / BITS_IN_EWORD;\n+\tif (i > objects_nr / BITS_IN_EWORD)\n+\t\ti = objects_nr / BITS_IN_EWORD;\n \n \treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n@@ -1290,7 +1296,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n-\t\t\tbitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\t\tbitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcount++;\n \t}\n \n@@ -1511,7 +1517,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n-\tnum_objects = bitmap_git->pack->num_objects;\n+\tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -1594,7 +1600,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \toff_t total = 0;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n@@ -1606,7 +1611,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \tfor (i = 0; i < eindex->count; i++) {\n \t\tstruct object *obj = eindex->objects[i];\n \n-\t\tif (!bitmap_get(result, pack->num_objects + i))\n+\t\tif (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431349","messageId":"67e6897a34cb5283daac84053c468701216f90c5.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 11/25] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:51Z","receivedAt":"2021-07-27T21:21:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to fetch the nth OID contained in\nthe bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 54dc4f7915..9d0dd1cde7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)\n \n #define MAX_XOR_OFFSET 160\n \n+static int nth_bitmap_object_oid(struct bitmap_index *index,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t uint32_t n)\n+{\n+\treturn nth_packed_object_id(oid, index->pack, n);\n+}\n+\n static int load_bitmap_entries_v1(struct bitmap_index *index)\n {\n \tuint32_t i;\n@@ -242,7 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\tif (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)\n \t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n \t\t\t\t (unsigned)commit_idx_pos);\n \n@@ -862,8 +869,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack,\n-\t\t\t\t\t pack_pos_to_index(pack, pos));\n+\t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n+\t\t\t\t\t pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431350","messageId":"743a1a138ef236e876f2c18257e5f4213c3acbc9.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 12/25] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:54Z","receivedAt":"2021-07-27T21:21:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a recent commit, pack-objects learned support for the\n'pack.preferBitmapTips' configuration. This patch prepares the\nmulti-pack bitmap code to respect this configuration, too.\n\nThe yet-to-be implemented code will find that it is more efficient to\ncheck whether each reference contains a prefix found in the configured\nset of values rather than doing an additional traversal.\n\nImplement a function 'bitmap_is_preferred_refname()' which will perform\nthat check. Its caller will be added in a subsequent patch.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 16 ++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9d0dd1cde7..6b12c96e32 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1652,3 +1652,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\n }\n+\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname)\n+{\n+\tconst struct string_list *preferred_tips = bitmap_preferred_tips(r);\n+\tstruct string_list_item *item;\n+\n+\tif (!preferred_tips)\n+\t\treturn 0;\n+\n+\tfor_each_string_list_item(item, preferred_tips) {\n+\t\tif (starts_with(refname, item->string))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 020cd8d868..52ea10de51 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint16_t options);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n #endif\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431351","messageId":"114773d9cdc1e83992afe8c007477940ee8ba03a.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 08/25] midx: close linked MIDXs, avoid leaking memory","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:43Z","receivedAt":"2021-07-27T21:21:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a repository has at least one alternate, the MIDX belonging to each\nalternate is accessed through the `next` pointer on the main object\nstore's copy of the MIDX. close_midx() didn't bother to close any\nof the linked MIDXs. It likewise didn't free the memory pointed to by\n`m`, leaving uninitialized bytes with live pointers to them left around\nin the heap.\n\nClean this up by closing linked MIDXs, and freeing up the memory pointed\nto by each of them. When callers call close_midx(), then they can\ndiscard the entire linked list of MIDXs and set their pointer to the\nhead of that list to NULL.\n\nThis isn't strictly required for the upcoming patches, but it makes it\nmuch more difficult (though still possible, for e.g., by calling\n`close_midx(m->next)` which leaves `m->next` pointing at uninitialized\nbytes) to have pointers to uninitialized memory.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 8956492b9c..18e1949613 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -195,6 +195,8 @@ void close_midx(struct multi_pack_index *m)\n \tif (!m)\n \t\treturn;\n \n+\tclose_midx(m->next);\n+\n \tmunmap((unsigned char *)m->data, m->data_len);\n \n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -203,6 +205,7 @@ void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n \tFREE_AND_NULL(m->pack_names);\n+\tfree(m);\n }\n \n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431352","messageId":"53ef0a6d67469078f237e2719052ab05f642cf2a.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 07/25] midx: infer preferred pack when not given one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:41Z","receivedAt":"2021-07-27T21:21:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the\nmulti-pack index code learned how to select a pack which all duplicate\nobjects are selected from. That is, if an object appears in multiple\npacks, select the copy in the preferred pack before breaking ties\naccording to the other rules like pack mtime and readdir() order.\n\nNot specifying a preferred pack can cause serious problems with\nmulti-pack reachability bitmaps, because these bitmaps rely on having at\nleast one pack from which all duplicates are selected. Not having such a\npack causes problems with the code in pack-objects to reuse packs\nverbatim (e.g., that code assumes that a delta object in a chunk of pack\nsent verbatim will have its base object sent from the same pack).\n\nSo why does not marking a pack preferred cause problems here? The reason\nis roughly as follows:\n\n - Ties are broken (when handling duplicate objects) by sorting\n according to midx_oid_compare(), which sorts objects by OID,\n preferred-ness, pack mtime, and finally pack ID (more on that\n later).\n\n - The psuedo pack-order (described in\n Documentation/technical/pack-format.txt under the section\n \"multi-pack-index reverse indexes\") is computed by\n midx_pack_order(), and sorts by pack ID and pack offset, with\n preferred packs sorting first.\n\n - But! Pack IDs come from incrementing the pack count in\n add_pack_to_midx(), which is a callback to\n for_each_file_in_pack_dir(), meaning that pack IDs are assigned in\n readdir() order.\n\nWhen specifying a preferred pack, all of that works fine, because\nduplicate objects are correctly resolved in favor of the copy in the\npreferred pack, and the preferred pack sorts first in the object order.\n\n\"Sorting first\" is critical, because the bitmap code relies on finding\nout which pack holds the first object in the MIDX's pseudo pack-order to\ndetermine which pack is preferred.\n\nBut if we didn't specify a preferred pack, and the pack which comes\nfirst in readdir() order does not also have the lowest timestamp, then\nit's possible that that pack (the one that sorts first in pseudo-pack\norder, which the bitmap code will treat as the preferred one) did *not*\nhave all duplicate objects resolved in its favor, resulting in breakage.\n\nThe fix is simple: pick a (semi-arbitrary, non-empty) preferred pack\nwhen none was specified. This forces that pack to have duplicates\nresolved in its favor, and (critically) to sort first in pseudo-pack\norder. Unfortunately, testing this behavior portably isn't possible,\nsince it depends on readdir() order which isn't guaranteed by POSIX.\n\n(Note that multi-pack reachability bitmaps have yet to be implemented;\nso in that sense this patch is fixing a bug which does not yet exist.\nBut by having this patch beforehand, we can prevent the bug from ever\nmaterializing.)\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 44 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 092dbf45b6..8956492b9c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -969,15 +969,57 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n+\t\tint found = 0;\n \t\tfor (i = 0; i < ctx.nr; i++) {\n \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tfound = 1;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n+\n+\t\tif (!found)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tctx.preferred_pack_idx = 0;\n+\n+\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n+\n+\t\t/*\n+\t\t * set a preferred pack when writing a bitmap to ensure that\n+\t\t * the pack from which the first object is selected in pseudo\n+\t\t * pack-order has all of its objects selected from that pack\n+\t\t * (and not another pack containing a duplicate)\n+\t\t */\n+\t\tfor (i = 1; i < ctx.nr; i++) {\n+\t\t\tstruct packed_git *p = ctx.info[i].p;\n+\n+\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n+\t\t\t\toldest = p;\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!oldest->num_objects) {\n+\t\t\t/*\n+\t\t\t * If all packs are empty; unset the preferred index.\n+\t\t\t * This is acceptable since there will be no duplicate\n+\t\t\t * objects to resolve, so the preferred value doesn't\n+\t\t\t * matter.\n+\t\t\t */\n+\t\t\tctx.preferred_pack_idx = -1;\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * otherwise don't mark any pack as preferred to avoid\n+\t\t * interfering with expiration logic below\n+\t\t */\n+\t\tctx.preferred_pack_idx = -1;\n \t}\n \n \tif (ctx.preferred_pack_idx > -1) {\n@@ -1058,11 +1100,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n-\n-\t\tif (!preferred)\n-\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n-\t\telse {\n+\t\tif (preferred) {\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431353","messageId":"141ff8327523e88bf22fe13dceed31782ea52213.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 14/25] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:59Z","receivedAt":"2021-07-27T21:21:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This prepares the code in pack-bitmap to interpret the new multi-pack\nbitmaps described in Documentation/technical/bitmap-format.txt, which\nmostly involves converting bit positions to accommodate looking them up\nin a MIDX.\n\nNote that there are currently no writers who write multi-pack bitmaps,\nand that this will be implemented in the subsequent commit. Note also\nthat get_midx_checksum() and get_midx_filename() are made non-static so\nthey can be called from pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 5 +\n midx.c | 4 +-\n midx.h | 2 +\n pack-bitmap-write.c | 2 +-\n pack-bitmap.c | 357 ++++++++++++++++++++++++++++++++++++-----\n pack-bitmap.h | 6 +\n packfile.c | 2 +-\n 7 files changed, 336 insertions(+), 42 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8a523624a1..e11d3ac2e5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\t/*\n+\t\t\t * Can use bit positions directly, even for MIDX\n+\t\t\t * bitmaps. See comment in try_partial_reuse()\n+\t\t\t * for why.\n+\t\t\t */\n \t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\ndiff --git a/midx.c b/midx.c\nindex d67d7f383d..db21727c62 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -48,12 +48,12 @@ static uint8_t oid_version(void)\n \t}\n }\n \n-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - the_hash_algo->rawsz;\n }\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex 8684cf0fef..1172df1a71 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -42,6 +42,8 @@ struct multi_pack_index {\n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n \n+const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+char *get_midx_filename(const char *object_dir);\n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 142fd0adb8..9c55c1531e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)\n }\n \n /**\n- * Build the initial type index for the packfile\n+ * Build the initial type index for the packfile or multi-pack-index\n */\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1442f0c8f2..f599646e19 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -13,6 +13,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"midx.h\"\n #include \"config.h\"\n \n /*\n@@ -35,8 +36,15 @@ struct stored_bitmap {\n * the active bitmap index is the largest one.\n */\n struct bitmap_index {\n-\t/* Packfile to which this bitmap index belongs to */\n+\t/*\n+\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n+\t * to.\n+\t *\n+\t * Exactly one of these must be non-NULL; this specifies the object\n+\t * order used to interpret this bitmap.\n+\t */\n \tstruct packed_git *pack;\n+\tstruct multi_pack_index *midx;\n \n \t/*\n \t * Mark the first `reuse_objects` in the packfile as reused:\n@@ -71,6 +79,9 @@ struct bitmap_index {\n \t/* If not NULL, this is a name-hash cache pointing into map. */\n \tuint32_t *hashes;\n \n+\t/* The checksum of the packfile or MIDX; points into map. */\n+\tconst unsigned char *checksum;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -138,6 +149,8 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n+\tif (index->midx)\n+\t\treturn index->midx->num_objects;\n \treturn index->pack->num_objects;\n }\n \n@@ -175,6 +188,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n+\tindex->checksum = header->checksum;\n \tindex->map_pos += header_size;\n \treturn 0;\n }\n@@ -227,6 +241,8 @@ static int nth_bitmap_object_oid(struct bitmap_index *index,\n \t\t\t\t struct object_id *oid,\n \t\t\t\t uint32_t n)\n {\n+\tif (index->midx)\n+\t\treturn nth_midxed_object_oid(oid, index->midx, n) ? 0 : -1;\n \treturn nth_packed_object_id(oid, index->pack, n);\n }\n \n@@ -274,7 +290,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \treturn 0;\n }\n \n-static char *pack_bitmap_filename(struct packed_git *p)\n+char *midx_bitmap_filename(struct multi_pack_index *midx)\n+{\n+\treturn xstrfmt(\"%s-%s.bitmap\",\n+\t\t get_midx_filename(midx->object_dir),\n+\t\t hash_to_hex(get_midx_checksum(midx)));\n+}\n+\n+char *pack_bitmap_filename(struct packed_git *p)\n {\n \tsize_t len;\n \n@@ -283,6 +306,57 @@ static char *pack_bitmap_filename(struct packed_git *p)\n \treturn xstrfmt(\"%.*s.bitmap\", (int)len, p->pack_name);\n }\n \n+static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct stat st;\n+\tchar *idx_name = midx_bitmap_filename(midx);\n+\tint fd = git_open(idx_name);\n+\n+\tfree(idx_name);\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n+\t\twarning(\"ignoring extra bitmap file: %s\",\n+\t\t\tget_midx_filename(midx->object_dir));\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tbitmap_git->midx = midx;\n+\tbitmap_git->map_size = xsize_t(st.st_size);\n+\tbitmap_git->map_pos = 0;\n+\tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ,\n+\t\t\t\tMAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\tif (load_bitmap_header(bitmap_git) < 0)\n+\t\tgoto cleanup;\n+\n+\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum))\n+\t\tgoto cleanup;\n+\n+\tif (load_midx_revindex(bitmap_git->midx) < 0) {\n+\t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n+\t\tgoto cleanup;\n+\t}\n+\treturn 0;\n+\n+cleanup:\n+\tmunmap(bitmap_git->map, bitmap_git->map_size);\n+\tbitmap_git->map_size = 0;\n+\tbitmap_git->map = NULL;\n+\treturn -1;\n+}\n+\n static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git *packfile)\n {\n \tint fd;\n@@ -304,7 +378,8 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\treturn -1;\n \t}\n \n-\tif (bitmap_git->pack) {\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n \t\tclose(fd);\n \t\treturn -1;\n@@ -326,13 +401,39 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \treturn 0;\n }\n \n-static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n+static int load_reverse_index(struct bitmap_index *bitmap_git)\n+{\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tuint32_t i;\n+\t\tint ret;\n+\n+\t\t/*\n+\t\t * The multi-pack-index's .rev file is already loaded via\n+\t\t * open_pack_bitmap_1().\n+\t\t *\n+\t\t * But we still need to open the individual pack .rev files,\n+\t\t * since we will need to make use of them in pack-objects.\n+\t\t */\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n+\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn load_pack_revindex(bitmap_git->pack);\n+}\n+\n+static int load_bitmap(struct bitmap_index *bitmap_git)\n {\n \tassert(bitmap_git->map);\n \n \tbitmap_git->bitmaps = kh_init_oid_map();\n \tbitmap_git->ext_index.positions = kh_init_oid_pos();\n-\tif (load_pack_revindex(bitmap_git->pack))\n+\n+\tif (load_reverse_index(bitmap_git))\n \t\tgoto failed;\n \n \tif (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) ||\n@@ -376,11 +477,47 @@ static int open_pack_bitmap(struct repository *r,\n \treturn ret;\n }\n \n+static int open_midx_bitmap(struct repository *r,\n+\t\t\t struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *midx;\n+\n+\tassert(!bitmap_git->map);\n+\n+\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n+\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\t\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n+static int open_bitmap(struct repository *r,\n+\t\t struct bitmap_index *bitmap_git)\n+{\n+\tassert(!bitmap_git->map);\n+\n+\tif (!open_midx_bitmap(r, bitmap_git))\n+\t\treturn 0;\n+\treturn open_pack_bitmap(r, bitmap_git);\n+}\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git))\n+\t\treturn bitmap_git;\n+\n+\tfree_bitmap_index(bitmap_git);\n+\treturn NULL;\n+}\n+\n+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n+\n+\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(bitmap_git))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -430,10 +567,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \treturn pos;\n }\n \n+static int bitmap_position_midx(struct bitmap_index *bitmap_git,\n+\t\t\t\tconst struct object_id *oid)\n+{\n+\tuint32_t want, got;\n+\tif (!bsearch_midx(oid, bitmap_git->midx, &want))\n+\t\treturn -1;\n+\n+\tif (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0)\n+\t\treturn -1;\n+\treturn got;\n+}\n+\n static int bitmap_position(struct bitmap_index *bitmap_git,\n \t\t\t const struct object_id *oid)\n {\n-\tint pos = bitmap_position_packfile(bitmap_git, oid);\n+\tint pos;\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpos = bitmap_position_midx(bitmap_git, oid);\n+\telse\n+\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n }\n \n@@ -744,6 +897,7 @@ static void show_objects_for_type(\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n+\t\t\tstruct packed_git *pack;\n \t\t\tstruct object_id oid;\n \t\t\tuint32_t hash = 0, index_pos;\n \t\t\toff_t ofs;\n@@ -753,14 +907,28 @@ static void show_objects_for_type(\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n-\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\t\t\tuint32_t pack_id;\n+\n+\t\t\t\tindex_pos = pack_pos_to_midx(m, pos + offset);\n+\t\t\t\tofs = nth_midxed_offset(m, index_pos);\n+\t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n+\n+\t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t} else {\n+\t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\t\tnth_bitmap_object_oid(bitmap_git, &oid, index_pos);\n+\n+\t\t\t\tpack = bitmap_git->pack;\n+\t\t\t}\n \n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n }\n@@ -772,8 +940,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n-\t\t\treturn 1;\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n+\t\t\t\treturn 1;\n+\t\t} else {\n+\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -859,14 +1032,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git,\n static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\t\t uint32_t pos)\n {\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tunsigned long size;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = &size;\n \n \tif (pos < bitmap_num_objects(bitmap_git)) {\n-\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tstruct packed_git *pack;\n+\t\toff_t ofs;\n+\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\n+\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\t\t} else {\n+\t\t\tpack = bitmap_git->pack;\n+\t\t\tofs = pack_pos_to_offset(pack, pos);\n+\t\t}\n+\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n \t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n@@ -1047,7 +1232,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t/* try to open a bitmapped pack, but don't parse it yet\n \t * because we may not need to use it */\n \tCALLOC_ARRAY(bitmap_git, 1);\n-\tif (open_pack_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (open_bitmap(revs->repo, bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tfor (i = 0; i < revs->pending.nr; ++i) {\n@@ -1091,7 +1276,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_pack_bitmap(bitmap_git) < 0)\n+\tif (load_bitmap(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tobject_array_clear(&revs->pending);\n@@ -1139,19 +1324,43 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n * reused, but you can keep feeding bits.\n */\n static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t struct packed_git *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n-\toff_t offset, header;\n+\toff_t offset, delta_obj_offset;\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn -1; /* not actually in the pack or MIDX */\n+\t/*\n+\t * try_partial_reuse() is called either on (a) objects in the\n+\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n+\t * objects in the preferred pack of a multi-pack bitmap.\n+\t * Importantly, the latter can pretend as if only a single pack\n+\t * exists because:\n+\t *\n+\t * - The first pack->num_objects bits of a MIDX bitmap are\n+\t * reserved for the preferred pack, and\n+\t *\n+\t * - Ties due to duplicate objects are always resolved in\n+\t * favor of the preferred pack.\n+\t *\n+\t * Therefore we do not need to ever ask the MIDX for its copy of\n+\t * an object by OID, since it will always select it from the\n+\t * preferred pack. Likewise, the selected copy of the base\n+\t * object for any deltas will reside in the same pack.\n+\t *\n+\t * This means that we can reuse pos when looking up the bit in\n+\t * the reuse bitmap, too, since bits corresponding to the\n+\t * preferred pack precede all bits from other packs.\n+\t */\n \n-\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n-\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n+\tif (pos >= pack->num_objects)\n+\t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n+\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n+\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn -1; /* broken packfile, punt */\n \n@@ -1167,11 +1376,11 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t &offset, type, header);\n+\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn 0;\n-\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n \t\t\treturn 0;\n \n \t\t/*\n@@ -1205,24 +1414,48 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \treturn 0;\n }\n \n+static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *m = bitmap_git->midx;\n+\tif (!m)\n+\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n+\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n+}\n+\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t struct packed_git **packfile_out,\n \t\t\t\t uint32_t *entries,\n \t\t\t\t struct bitmap **reuse_out)\n {\n+\tstruct packed_git *pack;\n \tstruct bitmap *result = bitmap_git->result;\n \tstruct bitmap *reuse;\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n-\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n+\tuint32_t objects_nr;\n \n \tassert(result);\n \n+\tload_reverse_index(bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\telse\n+\t\tpack = bitmap_git->pack;\n+\tobjects_nr = pack->num_objects;\n+\n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n \n-\t/* Don't mark objects not in the packfile */\n+\t/*\n+\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n+\t * marks objects eligible for reuse, but the pack-reuse code only\n+\t * understands how to reuse a single pack. Since the preferred pack is\n+\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n+\t * we use it instead of another pack. In single-pack bitmaps, the choice\n+\t * is made for us.\n+\t */\n \tif (i > objects_nr / BITS_IN_EWORD)\n \t\ti = objects_nr / BITS_IN_EWORD;\n \n@@ -1238,8 +1471,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n-\t\t\t\t\t &w_curs) < 0) {\n+\t\t\tif (try_partial_reuse(bitmap_git, pack, pos + offset,\n+\t\t\t\t\t reuse, &w_curs) < 0) {\n \t\t\t\t/*\n \t\t\t\t * try_partial_reuse indicated we couldn't reuse\n \t\t\t\t * any bits, so there is no point in trying more\n@@ -1268,7 +1501,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = bitmap_git->pack;\n+\t*packfile_out = pack;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n@@ -1542,6 +1775,12 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n+\tif (!bitmap_is_midx(bitmap_git))\n+\t\tload_reverse_index(bitmap_git);\n+\telse if (load_midx_revindex(bitmap_git->midx) < 0)\n+\t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n+\t\t \"extension\");\n+\n \tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n@@ -1549,8 +1788,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tstruct object_id oid;\n \t\tstruct object_entry *oe;\n \n-\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n-\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n+\t\tif (bitmap_is_midx(bitmap_git))\n+\t\t\tnth_midxed_object_oid(&oid,\n+\t\t\t\t\t bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n+\t\telse\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n@@ -1576,6 +1820,19 @@ void free_bitmap_index(struct bitmap_index *b)\n \tfree(b->ext_index.hashes);\n \tbitmap_free(b->result);\n \tbitmap_free(b->haves);\n+\tif (bitmap_is_midx(b)) {\n+\t\t/*\n+\t\t * Multi-pack bitmaps need to have resources associated with\n+\t\t * their on-disk reverse indexes unmapped so that stale .rev and\n+\t\t * .bitmap files can be removed.\n+\t\t *\n+\t\t * Unlike pack-based bitmaps, multi-pack bitmaps can be read and\n+\t\t * written in the same 'git multi-pack-index write --bitmap'\n+\t\t * process. Close resources so they can be removed safely on\n+\t\t * platforms like Windows.\n+\t\t */\n+\t\tclose_midx_revindex(b->midx);\n+\t}\n \tfree(b);\n }\n \n@@ -1590,7 +1847,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\t enum object_type object_type)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \toff_t total = 0;\n \tstruct ewah_iterator it;\n \teword_t filter;\n@@ -1607,15 +1863,35 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n-\t\t\tsize_t pos;\n-\n \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tpos = base + offset;\n-\t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n-\t\t\t\t pack_pos_to_offset(pack, pos);\n+\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tuint32_t pack_pos;\n+\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n+\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\n+\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\n+\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n+\t\t\t\t\tstruct object_id oid;\n+\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n+\n+\t\t\t\t\tdie(_(\"could not find %s in pack %s at offset %\"PRIuMAX),\n+\t\t\t\t\t oid_to_hex(&oid),\n+\t\t\t\t\t pack->pack_name,\n+\t\t\t\t\t (uintmax_t)offset);\n+\t\t\t\t}\n+\n+\t\t\t\ttotal += pack_pos_to_offset(pack, pack_pos + 1) - offset;\n+\t\t\t} else {\n+\t\t\t\tsize_t pos = base + offset;\n+\t\t\t\ttotal += pack_pos_to_offset(bitmap_git->pack, pos + 1) -\n+\t\t\t\t\t pack_pos_to_offset(bitmap_git->pack, pos);\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1666,6 +1942,11 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n \treturn total;\n }\n \n+int bitmap_is_midx(struct bitmap_index *bitmap_git)\n+{\n+\treturn !!bitmap_git->midx;\n+}\n+\n const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 52ea10de51..81664f933f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,6 +44,8 @@ typedef int (*show_reachable_fn)(\n struct bitmap_index;\n \n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *midx);\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n \t\t\t uint32_t *trees, uint32_t *blobs, uint32_t *tags);\n void traverse_bitmap_commit_list(struct bitmap_index *,\n@@ -92,6 +94,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n \t\t\t uint16_t options);\n+char *midx_bitmap_filename(struct multi_pack_index *midx);\n+char *pack_bitmap_filename(struct packed_git *p);\n+\n+int bitmap_is_midx(struct bitmap_index *bitmap_git);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n int bitmap_is_preferred_refname(struct repository *r, const char *refname);\ndiff --git a/packfile.c b/packfile.c\nindex 9ef6d98292..371f5488cf 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n-\t ends_with(file_name, \".rev\"))\n+\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431354","messageId":"a3b641b3e66a6fe8257064e976e271a5750aff9f.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 13/25] pack-bitmap.c: avoid redundant calls to try_partial_reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:56Z","receivedAt":"2021-07-27T21:21:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"try_partial_reuse() is used to mark any bits in the beginning of a\nbitmap whose objects can be reused verbatim from the pack they came\nfrom.\n\nCurrently this function returns void, and signals nothing to the caller\nwhen bits could not be reused. But multi-pack bitmaps would benefit from\nhaving such a signal, because they may try to pass objects which are in\nbounds, but from a pack other than the preferred one.\n\nAny extra calls are noops because of a conditional in\nreuse_partial_packfile_from_bitmap(), but those loop iterations can be\navoided by letting try_partial_reuse() indicate when it can't accept any\nmore bits for reuse, and then listening to that signal.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 40 +++++++++++++++++++++++++++++-----------\n 1 file changed, 29 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6b12c96e32..1442f0c8f2 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1134,22 +1134,26 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \treturn NULL;\n }\n \n-static void try_partial_reuse(struct bitmap_index *bitmap_git,\n-\t\t\t size_t pos,\n-\t\t\t struct bitmap *reuse,\n-\t\t\t struct pack_window **w_curs)\n+/*\n+ * -1 means \"stop trying further objects\"; 0 means we may or may not have\n+ * reused, but you can keep feeding bits.\n+ */\n+static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t size_t pos,\n+\t\t\t struct bitmap *reuse,\n+\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, header;\n \tenum object_type type;\n \tunsigned long size;\n \n \tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn; /* not actually in the pack or MIDX */\n+\t\treturn -1; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n \tif (type < 0)\n-\t\treturn; /* broken packfile, punt */\n+\t\treturn -1; /* broken packfile, punt */\n \n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n@@ -1166,9 +1170,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n \t\t\t\t\t &offset, type, header);\n \t\tif (!base_offset)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * We assume delta dependencies always point backwards. This\n@@ -1180,7 +1184,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * odd parameters.\n \t\t */\n \t\tif (base_pos >= pos)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * And finally, if we're not sending the base as part of our\n@@ -1191,13 +1195,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * object_entry code path handle it.\n \t\t */\n \t\tif (!bitmap_get(reuse, base_pos))\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n \tbitmap_set(reuse, pos);\n+\treturn 0;\n }\n \n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n@@ -1233,10 +1238,23 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n+\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n+\t\t\t\t\t &w_curs) < 0) {\n+\t\t\t\t/*\n+\t\t\t\t * try_partial_reuse indicated we couldn't reuse\n+\t\t\t\t * any bits, so there is no point in trying more\n+\t\t\t\t * bits in the current word, or any other words\n+\t\t\t\t * in result.\n+\t\t\t\t *\n+\t\t\t\t * Jump out of both loops to avoid future\n+\t\t\t\t * unnecessary calls to try_partial_reuse.\n+\t\t\t\t */\n+\t\t\t\tgoto done;\n+\t\t\t}\n \t\t}\n \t}\n \n+done:\n \tunuse_pack(&w_curs);\n \n \t*entries = bitmap_popcount(reuse);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431355","messageId":"60ec8b3466e7f94610a45bdd1c79feb06e439429.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 17/25] t/helper/test-read-midx.c: add --checksum mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:07Z","receivedAt":"2021-07-27T21:21:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subsequent tests will want to check for the existence of a multi-pack\nbitmap which matches the multi-pack-index stored in the pack directory.\n\nThe multi-pack bitmap includes the hex checksum of the MIDX it\ncorresponds to in its filename (for example,\n'$packdir/multi-pack-index-.bitmap'). As a result, some tests\nwant a way to learn what '' is.\n\nThis helper addresses that need by printing the checksum of the\nrepository's multi-pack-index.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 16 +++++++++++++++-\n t/lib-bitmap.sh | 4 ++++\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 7c2eb11a8e..cb0d27049a 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \treturn 0;\n }\n \n+static int read_midx_checksum(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m;\n+\n+\tsetup_git_directory();\n+\tm = load_multi_pack_index(object_dir, 1);\n+\tif (!m)\n+\t\treturn 1;\n+\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n+\telse if (!strcmp(argv[1], \"--checksum\"))\n+\t\treturn read_midx_checksum(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex ecb5d0e05d..09cd036f4d 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -260,3 +260,7 @@ have_delta () {\n \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n \ttest_cmp expect actual\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"${1:-.git/objects}\"\n+}\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431356","messageId":"168b7b0976c7d3ab2082843b7c322252eed11a3f.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 16/25] t5310: move some tests to lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:04Z","receivedAt":"2021-07-27T21:21:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We'll soon be adding a test script that will cover many of the same\nbitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many\nof the applicable tests as we can so we don't have to rewrite them.\n\nThere should be no functional change to t5310; we still run the same\noperations in the same order.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 236 ++++++++++++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 227 +-------------------------------------\n 2 files changed, 240 insertions(+), 223 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex fe3f98be24..ecb5d0e05d 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,3 +1,6 @@\n+# Helpers for scripts testing bitamp functionality; see t5310 for\n+# example usage.\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -24,3 +27,236 @@ test_bitmap_traversal () {\n \ttest_cmp \"$1.normalized\" \"$2.normalized\" &&\n \trm -f \"$1.normalized\" \"$2.normalized\"\n }\n+\n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other second\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-second * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for second (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n+#\n+# second: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n+setup_bitmap_history() {\n+\ttest_expect_success 'setup repo with moderate-sized history' '\n+\t\ttest_commit_bulk --id=file 10 &&\n+\t\tgit branch -M second &&\n+\t\tgit checkout -b other HEAD~5 &&\n+\t\ttest_commit_bulk --id=side 10 &&\n+\n+\t\t# add complicated history setup, including merges and\n+\t\t# ambiguous merge-bases\n+\n+\t\tgit checkout -b merge-left other~2 &&\n+\t\tgit merge second~2 -m \"merge-left\" &&\n+\n+\t\tgit checkout -b merge-right second~1 &&\n+\t\tgit merge other~1 -m \"merge-right\" &&\n+\n+\t\tgit checkout -b octo-second second &&\n+\t\tgit merge merge-left merge-right -m \"octopus-second\" &&\n+\n+\t\tgit checkout -b octo-other other &&\n+\t\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\t\tgit checkout other &&\n+\t\tgit merge octo-other -m \"pull octopus\" &&\n+\n+\t\tgit checkout second &&\n+\t\tgit merge octo-second -m \"pull octopus\" &&\n+\n+\t\t# Remove these branches so they are not selected\n+\t\t# as bitmap tips\n+\t\tgit branch -D merge-left &&\n+\t\tgit branch -D merge-right &&\n+\t\tgit branch -D octo-other &&\n+\t\tgit branch -D octo-second &&\n+\n+\t\t# add padding to make these merges less interesting\n+\t\t# and avoid having them selected for bitmaps\n+\t\ttest_commit_bulk --id=file 100 &&\n+\t\tgit checkout other &&\n+\t\ttest_commit_bulk --id=side 100 &&\n+\t\tgit checkout second &&\n+\n+\t\tbitmaptip=$(git rev-parse second) &&\n+\t\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n+\t\tgit tag tagged-blob $blob\n+\t'\n+}\n+\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n+\t\tgit rev-list --count other...second >expect &&\n+\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n+\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n+\t\ttest_bitmap_traversal expect actual\n+\t'\n+\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n+\t\tgrep $blob actual\n+\t'\n+}\n+\n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"second\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n+basic_bitmap_tests () {\n+\ttip=\"$1\"\n+\ttest_expect_success 'rev-list --test-bitmap verifies bitmaps' \"\n+\t\tgit rev-list --test-bitmap \"${tip:-HEAD}\"\n+\t\"\n+\n+\trev_list_tests 'full bitmap'\n+\n+\ttest_expect_success 'clone from bitmapped repository' '\n+\t\trm -fr clone.git &&\n+\t\tgit clone --no-local --bare . clone.git &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'partial clone from bitmapped repository' '\n+\t\ttest_config uploadpack.allowfilter true &&\n+\t\trm -fr partial-clone.git &&\n+\t\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n+\t\t(\n+\t\t\tcd partial-clone.git &&\n+\t\t\tpack=$(echo objects/pack/*.pack) &&\n+\t\t\tgit verify-pack -v \"$pack\" >have &&\n+\t\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n+\t\t\t# we expect this single blob because of the direct ref\n+\t\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n+\t\t\ttest_cmp expect blobs\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'setup further non-bitmapped commits' '\n+\t\ttest_commit_bulk --id=further 10\n+\t'\n+\n+\trev_list_tests 'partial bitmap'\n+\n+\ttest_expect_success 'fetch (partial bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'enumerating progress counts pack-reused objects' '\n+\t\tcount=$(git rev-list --objects --all --count) &&\n+\t\tgit repack -adb &&\n+\n+\t\t# check first with only reused objects; confirm that our\n+\t\t# progress showed the right number, and also that we did\n+\t\t# pack-reuse as expected. Check only the final \"done\"\n+\t\t# line of the meter (there may be an arbitrary number of\n+\t\t# intermediate lines ending with CR).\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $count, done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr &&\n+\n+\t\t# now the same but with one non-reused object\n+\t\tgit commit --allow-empty -m \"an extra commit object\" &&\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr\n+\t'\n+}\n+\n+# have_delta \n+#\n+# Note that because this relies on cat-file, it might find _any_ copy of an\n+# object in the repository. The caller is responsible for making sure\n+# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n+have_delta () {\n+\techo $2 >expect &&\n+\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n+\ttest_cmp expect actual\n+}\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex b02838750e..4318f84d53 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -25,93 +25,10 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-# To ensure the logic for \"maximal commits\" is exercised, make\n-# the repository a bit more complicated.\n-#\n-# other second\n-# * *\n-# (99 commits) (99 commits)\n-# * *\n-# |\\ /|\n-# | * octo-other octo-second * |\n-# |/|\\_________ ____________/|\\|\n-# | \\ \\/ __________/ |\n-# | | ________/\\ / |\n-# * |/ * merge-right *\n-# | _|__________/ \\____________ |\n-# |/ | \\|\n-# (l1) * * merge-left * (r1)\n-# | / \\________________________ |\n-# |/ \\|\n-# (l2) * * (r2)\n-# \\___________________________ |\n-# \\|\n-# * (base)\n-#\n-# We only push bits down the first-parent history, which\n-# makes some of these commits unimportant!\n-#\n-# The important part for the maximal commit algorithm is how\n-# the bitmasks are extended. Assuming starting bit positions\n-# for second (bit 0) and other (bit 1), the bitmasks at the\n-# end should be:\n-#\n-# second: 1 (maximal, selected)\n-# other: 01 (maximal, selected)\n-# (base): 11 (maximal)\n-#\n-# This complicated history was important for a previous\n-# version of the walk that guarantees never walking a\n-# commit multiple times. That goal might be important\n-# again, so preserve this complicated case. For now, this\n-# test will guarantee that the bitmaps are computed\n-# correctly, even with the repeat calculations.\n+setup_bitmap_history\n \n-test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 10 &&\n-\tgit branch -M second &&\n-\tgit checkout -b other HEAD~5 &&\n-\ttest_commit_bulk --id=side 10 &&\n-\n-\t# add complicated history setup, including merges and\n-\t# ambiguous merge-bases\n-\n-\tgit checkout -b merge-left other~2 &&\n-\tgit merge second~2 -m \"merge-left\" &&\n-\n-\tgit checkout -b merge-right second~1 &&\n-\tgit merge other~1 -m \"merge-right\" &&\n-\n-\tgit checkout -b octo-second second &&\n-\tgit merge merge-left merge-right -m \"octopus-second\" &&\n-\n-\tgit checkout -b octo-other other &&\n-\tgit merge merge-left merge-right -m \"octopus-other\" &&\n-\n-\tgit checkout other &&\n-\tgit merge octo-other -m \"pull octopus\" &&\n-\n-\tgit checkout second &&\n-\tgit merge octo-second -m \"pull octopus\" &&\n-\n-\t# Remove these branches so they are not selected\n-\t# as bitmap tips\n-\tgit branch -D merge-left &&\n-\tgit branch -D merge-right &&\n-\tgit branch -D octo-other &&\n-\tgit branch -D octo-second &&\n-\n-\t# add padding to make these merges less interesting\n-\t# and avoid having them selected for bitmaps\n-\ttest_commit_bulk --id=file 100 &&\n-\tgit checkout other &&\n-\ttest_commit_bulk --id=side 100 &&\n-\tgit checkout second &&\n-\n-\tbitmaptip=$(git rev-parse second) &&\n-\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n-\tgit tag tagged-blob $blob &&\n-\tgit config repack.writebitmaps true\n+test_expect_success 'setup writing bitmaps during repack' '\n+\tgit config repack.writeBitmaps true\n '\n \n test_expect_success 'full repack creates bitmaps' '\n@@ -123,109 +40,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n-test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-\tgit rev-list --test-bitmap HEAD\n-'\n-\n-rev_list_tests_head () {\n-\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch~5..$branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n-\t\tgit rev-list --count -n 1 $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n-\t\tgit rev-list --count other...second >expect &&\n-\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n-\t\tgit rev-list --count $branch -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count --objects $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n-\t\tgit rev-list --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list $branch >expect &&\n-\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list --objects $branch >expect &&\n-\t\ttest_bitmap_traversal expect actual\n-\t'\n-\n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n-\t\tgrep $blob actual\n-\t'\n-}\n-\n-rev_list_tests () {\n-\tstate=$1\n-\n-\tfor branch in \"second\" \"other\"\n-\tdo\n-\t\trev_list_tests_head\n-\tdone\n-}\n-\n-rev_list_tests 'full bitmap'\n-\n-test_expect_success 'clone from bitmapped repository' '\n-\tgit clone --no-local --bare . clone.git &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'partial clone from bitmapped repository' '\n-\ttest_config uploadpack.allowfilter true &&\n-\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n-\t(\n-\t\tcd partial-clone.git &&\n-\t\tpack=$(echo objects/pack/*.pack) &&\n-\t\tgit verify-pack -v \"$pack\" >have &&\n-\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n-\t\t# we expect this single blob because of the direct ref\n-\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n-\t\ttest_cmp expect blobs\n-\t)\n-'\n-\n-test_expect_success 'setup further non-bitmapped commits' '\n-\ttest_commit_bulk --id=further 10\n-'\n-\n-rev_list_tests 'partial bitmap'\n-\n-test_expect_success 'fetch (partial bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+basic_bitmap_tests\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -461,40 +276,6 @@ test_expect_success 'truncated bitmap fails gracefully (cache)' '\n \ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n-test_expect_success 'enumerating progress counts pack-reused objects' '\n-\tcount=$(git rev-list --objects --all --count) &&\n-\tgit repack -adb &&\n-\n-\t# check first with only reused objects; confirm that our progress\n-\t# showed the right number, and also that we did pack-reuse as expected.\n-\t# Check only the final \"done\" line of the meter (there may be an\n-\t# arbitrary number of intermediate lines ending with CR).\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $count, done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr &&\n-\n-\t# now the same but with one non-reused object\n-\tgit commit --allow-empty -m \"an extra commit object\" &&\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr\n-'\n-\n-# have_delta \n-#\n-# Note that because this relies on cat-file, it might find _any_ copy of an\n-# object in the repository. The caller is responsible for making sure\n-# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n-have_delta () {\n-\techo $2 >expect &&\n-\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n-\ttest_cmp expect actual\n-}\n-\n # Create a state of history with these properties:\n #\n # - refs that allow a client to fetch some new history, while sharing some old\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431357","messageId":"3258ccfc1cc99038e43a37bd2d53c9d30a4f22ae.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 18/25] t5326: test multi-pack bitmap behavior","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:10Z","receivedAt":"2021-07-27T21:21:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch introduces a new test, t5326, which tests the basic\nfunctionality of multi-pack bitmaps.\n\nSome trivial behavior is tested, such as:\n\n - Whether bitmaps can be generated with more than one pack.\n - Whether clones can be served with all objects in the bitmap.\n - Whether follow-up fetches can be served with some objects outside of\n the server's bitmap\n\nThese use lib-bitmap's tests (which in turn were pulled from t5310), and\nwe cover cases where the MIDX represents both a single pack and multiple\npacks.\n\nIn addition, some non-trivial and MIDX-specific behavior is tested, too,\nincluding:\n\n - Whether multi-pack bitmaps behave correctly with respect to the\n pack-reuse machinery when the base for some object is selected from\n a different pack than the delta.\n - Whether multi-pack bitmaps correctly respect the\n pack.preferBitmapTips configuration.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 277 ++++++++++++++++++++++++++++++++++\n 1 file changed, 277 insertions(+)\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..c1b7d633e2\n--- /dev/null\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,277 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality'\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+setup_bitmap_history\n+\n+test_expect_success 'enable core.multiPackIndex' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_expect_success 'create single-pack midx with bitmaps' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success 'create new additional packs' '\n+\tfor i in $(test_seq 1 16)\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\t\tgit repack -d\n+\tdone &&\n+\n+\tgit checkout -b other2 HEAD~8 &&\n+\tfor i in $(test_seq 1 8)\n+\tdo\n+\t\ttest_commit \"side-$i\" &&\n+\t\tgit repack -d\n+\tdone &&\n+\tgit checkout second\n+'\n+\n+test_expect_success 'create multi-pack midx with bitmaps' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\tls $objdir/pack/pack-*.pack >packs &&\n+\ttest_line_count = 25 packs &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\ttest_commit respect--no-bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\tgit multi-pack-index write --no-bitmap &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+test_expect_success 'setup midx with base from later pack' '\n+\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t# prefers to delete contents out of a base rather than add to a shorter\n+\t# object.\n+\ttest_seq 1 128 >a &&\n+\ttest_seq 1 130 >b &&\n+\n+\tgit add a b &&\n+\tgit commit -m \"initial commit\" &&\n+\n+\ta=$(git rev-parse HEAD:a) &&\n+\tb=$(git rev-parse HEAD:b) &&\n+\n+\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$a\n+\t$b\n+\tEOF\n+\t) &&\n+\n+\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t# any other object.\n+\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$b\n+\t$(git rev-parse HEAD)\n+\t$(git rev-parse HEAD^{tree})\n+\tEOF\n+\t) &&\n+\n+\tgit prune-packed &&\n+\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t# reuse.\n+\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\thave_delta $a $b &&\n+\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+'\n+\n+rev_list_tests 'full bitmap with backwards delta'\n+\n+test_expect_success 'clone with bitmaps enabled' '\n+\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\tgit rev-parse HEAD >expect &&\n+\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+bitmap_reuse_tests() {\n+\tfrom=$1\n+\tto=$2\n+\n+\ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk 16 &&\n+\t\t\tgit tag old-tip &&\n+\n+\t\t\tgit config core.multiPackIndex true &&\n+\t\t\tif test \"MIDX\" = \"$from\"\n+\t\t\tthen\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk --id=further 16 &&\n+\t\t\tgit tag new-tip &&\n+\n+\t\t\tif test \"MIDX\" = \"$to\"\n+\t\t\tthen\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit for-each-ref &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n+\t\t)\n+\t'\n+}\n+\n+bitmap_reuse_tests 'pack' 'MIDX'\n+bitmap_reuse_tests 'MIDX' 'pack'\n+bitmap_reuse_tests 'MIDX' 'MIDX'\n+\n+test_expect_success 'missing object closure fails gracefully' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit loose &&\n+\t\ttest_commit packed &&\n+\n+\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t# commit.\n+\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t$(git rev-parse packed)\n+\t\tEOF\n+\n+\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\tgrep \"doesn.t have full closure\" err &&\n+\t\ttest_path_is_missing $midx\n+\t)\n+'\n+\n+test_expect_success 'setup partial bitmaps' '\n+\ttest_commit packed &&\n+\tgit repack &&\n+\ttest_commit loose &&\n+\tgit multi-pack-index write --bitmap 2>err &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+'\n+\n+basic_bitmap_tests HEAD~\n+\n+test_expect_success 'removing a MIDX clears stale bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm $midx &&\n+\n+\t\t# Then write a new MIDX.\n+\t\ttest_commit new &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_missing $stale_bitmap\n+\t)\n+'\n+\n+test_expect_success 'pack.preferBitmapTips' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\tbitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431358","messageId":"49297f57ed60cecdf7516f79f3977d94f962b214.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:19:35Z","receivedAt":"2021-07-27T21:21:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nclean up any auxiliary files (currently just the MIDX's `.rev` file, but\nsoon to include a `.bitmap`, too) corresponding to the MIDX it's\nreplacing.\n\nThis step should happen after the new MIDX is written into place, since\ndoing so beforehand means that the old MIDX could be read without its\ncorresponding .rev file.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 9a35b0255d..3193426d24 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n-\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431359","messageId":"54600b58143f482dcddbaba5edbed70481a90c01.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 15/25] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:02Z","receivedAt":"2021-07-27T21:22:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Write multi-pack bitmaps in the format described by\nDocumentation/technical/bitmap-format.txt, inferring their presence with\nthe absence of '--bitmap'.\n\nTo write a multi-pack bitmap, this patch attempts to reuse as much of\nthe existing machinery from pack-objects as possible. Specifically, the\nMIDX code prepares a packing_data struct that pretends as if a single\npackfile has been generated containing all of the objects contained\nwithin the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 12 +-\n builtin/multi-pack-index.c | 2 +\n midx.c | 208 ++++++++++++++++++++++++-\n midx.h | 1 +\n 4 files changed, 214 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex c9b063d31e..ed52459a9d 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n-\t[--preferred-pack=] \n+\t[--preferred-pack=] [--[no-]bitmap] \n \n DESCRIPTION\n -----------\n@@ -40,6 +40,9 @@ write::\n \t\tmultiple packs contain the same object. `` must\n \t\tcontain at least one object. If not given, ties are\n \t\tbroken in favor of the pack with the lowest mtime.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\n@@ -81,6 +84,13 @@ EXAMPLES\n $ git multi-pack-index write\n -----------------------------------------------\n \n+* Write a MIDX file for the packfiles in the current .git folder with a\n+corresponding bitmap.\n++\n+-------------------------------------------------------------\n+$ git multi-pack-index write --preferred-pack= --bitmap\n+-------------------------------------------------------------\n+\n * Write a MIDX file for the packfiles in an alternate object store.\n +\n -----------------------------------------------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5d3ea445fd..bf6fa982e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n \t\t\t N_(\"preferred-pack\"),\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/midx.c b/midx.c\nindex db21727c62..ce6cc62c20 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,10 @@\n #include \"repository.h\"\n #include \"chunk-format.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -893,6 +897,166 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t\t data->ctx->entries_nr,\n+\t\t\t bitmap_oid_access);\n+\tif (pos < 0)\n+\t\treturn;\n+\n+\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n+\tdata->commits[data->commits_nr++] = commit;\n+}\n+\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb = {0};\n+\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tsetup_revisions(0, NULL, &revs, NULL);\n+\tfor_each_ref(add_ref_to_pending, &revs);\n+\n+\t/*\n+\t * Skipping promisor objects here is intentional, since it only excludes\n+\t * them from the list of reachable commits that we want to select from\n+\t * when computing the selection of MIDX'd commits to receive bitmaps.\n+\t *\n+\t * Reachability bitmaps do require that their objects be closed under\n+\t * reachability, but fetching any objects missing from promisors at this\n+\t * point is too late. But, if one of those objects can be reached from\n+\t * an another object that is included in the bitmap, then we will\n+\t * complain later that we don't have reachability closure (and fail\n+\t * appropriately).\n+\t */\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t unsigned flags)\n+{\n+\tstruct packing_data pdata;\n+\tstruct pack_idx_entry **index;\n+\tstruct commit **commits = NULL;\n+\tuint32_t i, commits_nr;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n+\tint ret;\n+\n+\tprepare_midx_packing_data(&pdata, ctx);\n+\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata.nr_objects);\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[i] = &pdata.objects[i].idx;\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_finish expects objects in lex order, but pack_order\n+\t * gives us exactly that. use it directly instead of re-sorting the\n+\t * array.\n+\t *\n+\t * This changes the order of objects in 'index' between\n+\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n+\t *\n+\t * The same re-ordering takes place in the single-pack bitmap code via\n+\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n+\t * happens between bitmap_writer_build_type_index() and\n+\t * bitmap_writer_finish().\n+\t */\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[ctx->pack_order[i]] = &pdata.objects[i].idx;\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(&pdata);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\treturn ret;\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -940,7 +1104,7 @@ static int write_midx_internal(const char *object_dir,\n \n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\tctx.info[ctx.nr].expired = 0;\n \n \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n@@ -974,8 +1138,26 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n-\t\tgoto cleanup;\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_midx_bitmap_git(the_repository, ctx.m);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n+\t\t\t\t\t\t NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n \n \tif (preferred_pack_name) {\n \t\tint found = 0;\n@@ -991,7 +1173,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tif (!found)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n \t\t\t\tpreferred_pack_name);\n-\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t} else if (ctx.nr &&\n+\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1123,9 +1306,6 @@ static int write_midx_internal(const char *object_dir,\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n-\tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n@@ -1156,14 +1336,24 @@ static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tif (flags & MIDX_WRITE_BITMAP) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\t\terror(_(\"could not write multi-pack bitmap\"));\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tclose_midx(ctx.m);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n \tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n cleanup:\n@@ -1180,6 +1370,7 @@ static int write_midx_internal(const char *object_dir,\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n+\n \treturn result;\n }\n \n@@ -1240,6 +1431,7 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".bitmap\", NULL);\n \tclear_midx_files_ext(r, \".rev\", NULL);\n \n \tfree(midx);\ndiff --git a/midx.h b/midx.h\nindex 1172df1a71..350f4d0a7b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -41,6 +41,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n+#define MIDX_WRITE_BITMAP (1 << 2)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n char *get_midx_filename(const char *object_dir);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431360","messageId":"47c7e6bb9bb7dd959e68214823cc1636c8576503.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 19/25] t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:12Z","receivedAt":"2021-07-27T21:22:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap causes tests which repack in a partial clone to\nfail because they are missing objects. Missing objects is an expected\ncomponent of tests in t0410, so disable this knob altogether. Graceful\ndegradation when writing a bitmap with missing objects is tested in\nt5326.\n\nSigned-off-by: Taylor Blau \n---\n t/t0410-partial-clone.sh | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex bbcc51ee8e..bba679685f 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -4,6 +4,9 @@ test_description='partial clone'\n \n . ./test-lib.sh\n \n+# missing promisor objects cause repacks which write bitmaps to fail\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n delete_object () {\n \trm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431361","messageId":"6a708858b1048f83c6165c452ab238cb017b7713.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 20/25] t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:15Z","receivedAt":"2021-07-27T21:22:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap confuses many of the tests in t5310, which\nexpect to control whether and how bitmaps are written. Since the\nrelevant MIDX-bitmap tests here are covered already in t5326, let's just\ndisable the flag for the whole t5310 script.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 4318f84d53..673baa5c3c 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n . \"$TEST_DIRECTORY\"/lib-bundle.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n+# their place.\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431362","messageId":"1eaa744b2411a55f2cc70d4e0b70809351080347.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 21/25] t5319: don't write MIDX bitmaps in t5319","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:18Z","receivedAt":"2021-07-27T21:22:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This test is specifically about generating a midx still respecting a\npack-based bitmap file. Generating a MIDX bitmap would confuse the test.\nLet's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to\nmake sure we don't do so.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1f0a2ae852..7b685957c6 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -504,7 +504,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431363","messageId":"a4a899e31f71c46df7cb784366fd114325124343.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 22/25] t7700: update to work with MIDX bitmap test knob","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:20Z","receivedAt":"2021-07-27T21:22:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A number of these tests are focused only on pack-based bitmaps and need\nto be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where\nnecessary.\n\nSigned-off-by: Taylor Blau \n---\n t/t7700-repack.sh | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 25b235c063..98eda3bfeb 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit repack -Adbl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '\n \n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n-\tgit -C bare.git repack -ad &&\n+\trm -f bare.git/objects/pack/*.bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431364","messageId":"50865e52a37590d0bef541fd96c95a0416d7d585.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 23/25] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:23Z","receivedAt":"2021-07-27T21:22:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment\nvariable to also write a multi-pack bitmap when\n'GIT_TEST_MULTI_PACK_INDEX' is set.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 12 ++++++++++--\n ci/run-build-and-tests.sh | 1 +\n midx.h | 2 ++\n t/README | 4 ++++\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..82ab668272 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n \t\t !is_bare_repository())\n \t\t\twrite_bitmaps = 0;\n+\t} else if (write_bitmaps &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n+\t\twrite_bitmaps = 0;\n \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tupdate_server_info(0);\n \tremove_temporary_files();\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n+\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t}\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..7ee9ba9325 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -23,6 +23,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_WRITE_REV_INDEX=1\ndiff --git a/midx.h b/midx.h\nindex 350f4d0a7b..aa3da557bb 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,8 @@ struct pack_entry;\n struct repository;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex 9e70122302..12014aa988 100644\n--- a/t/README\n+++ b/t/README\n@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n+'--bitmap' option on all invocations of 'git multi-pack-index write',\n+and ignores pack-objects' '--write-bitmap-index'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431365","messageId":"0f1fd6e7d49d98ae7f7829b4debbef6382dfc3e1.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 24/25] p5310: extract full and partial bitmap tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:26Z","receivedAt":"2021-07-27T21:22:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A new p5326 introduced by the next patch will want these same tests,\ninterjecting its own setup in between. Move them out so that both perf\ntests can reuse them.\n\nSigned-off-by: Taylor Blau \n---\n t/perf/lib-bitmap.sh | 69 ++++++++++++++++++++++++++++++++++++\n t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------\n 2 files changed, 72 insertions(+), 62 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n\ndiff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh\nnew file mode 100644\nindex 0000000000..63d3bc7cec\n--- /dev/null\n+++ b/t/perf/lib-bitmap.sh\n@@ -0,0 +1,69 @@\n+# Helper functions for testing bitmap performance; see p5310.\n+\n+test_full_bitmap () {\n+\ttest_perf 'simulated clone' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'simulated fetch' '\n+\t\thave=$(git rev-list HEAD~100 -1) &&\n+\t\t{\n+\t\t\techo HEAD &&\n+\t\t\techo ^$have\n+\t\t} | git pack-objects --revs --stdout >/dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (commits)' '\n+\t\tgit rev-list --all --use-bitmap-index >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (objects)' '\n+\t\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tag negated via --not --all (objects)' '\n+\t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with negative tag (objects)' '\n+\t\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:none' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:none >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:limit=1k' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:limit=1k >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with tree:0' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+\n+\ttest_perf 'simulated partial clone' '\n+\t\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n+\t'\n+}\n+\n+test_partial_bitmap () {\n+\ttest_perf 'clone (partial bitmap)' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (partial bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tree filter (partial bitmap)' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+}\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 452be01056..7ad4f237bc 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -2,6 +2,7 @@\n \n test_description='Tests pack performance using bitmaps'\n . ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n test_perf_large_repo\n \n@@ -25,56 +26,7 @@ test_perf 'repack to disk' '\n \tgit repack -ad\n '\n \n-test_perf 'simulated clone' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'simulated fetch' '\n-\thave=$(git rev-list HEAD~100 -1) &&\n-\t{\n-\t\techo HEAD &&\n-\t\techo ^$have\n-\t} | git pack-objects --revs --stdout >/dev/null\n-'\n-\n-test_perf 'pack to file (bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n-'\n-\n-test_perf 'rev-list (commits)' '\n-\tgit rev-list --all --use-bitmap-index >/dev/null\n-'\n-\n-test_perf 'rev-list (objects)' '\n-\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with tag negated via --not --all (objects)' '\n-\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with negative tag (objects)' '\n-\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:none' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:none >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:limit=1k' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:limit=1k >/dev/null\n-'\n-\n-test_perf 'rev-list count with tree:0' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n-\n-test_perf 'simulated partial clone' '\n-\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n-'\n+test_full_bitmap\n \n test_expect_success 'create partial bitmap state' '\n \t# pick a commit to represent the repo tip in the past\n@@ -97,17 +49,6 @@ test_expect_success 'create partial bitmap state' '\n \tgit update-ref HEAD $orig_tip\n '\n \n-test_perf 'clone (partial bitmap)' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'pack to file (partial bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n-'\n-\n-test_perf 'rev-list with tree filter (partial bitmap)' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n+test_partial_bitmap\n \n test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"431366","messageId":"82e8133bf4f6ecf2ca509f6d9e2e0d369d7f19e3.1627420428.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"[PATCH v3 25/25] p5326: perf tests for MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-27T21:20:28Z","receivedAt":"2021-07-27T21:22:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These new performance tests demonstrate effectively the same behavior as\np5310, but use a multi-pack bitmap instead of a single-pack one.\n\nNotably, p5326 does not create a MIDX bitmap with multiple packs. This\nis so we can measure a direct comparison between it and p5310. Any\ndifference between the two is measuring just the overhead of using MIDX\nbitmaps.\n\nHere are the results of p5310 and p5326 together, measured at the same\ntime and on the same machine (using a Xenon W-2255 CPU):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5310.2: repack to disk 96.78(93.39+11.33)\n 5310.3: simulated clone 9.98(9.79+0.19)\n 5310.4: simulated fetch 1.75(4.26+0.19)\n 5310.5: pack to file (bitmap) 28.20(27.87+8.70)\n 5310.6: rev-list (commits) 0.41(0.36+0.05)\n 5310.7: rev-list (objects) 1.61(1.54+0.07)\n 5310.8: rev-list count with blob:none 0.25(0.21+0.04)\n 5310.9: rev-list count with blob:limit=1k 2.65(2.54+0.10)\n 5310.10: rev-list count with tree:0 0.23(0.19+0.04)\n 5310.11: simulated partial clone 4.34(4.21+0.12)\n 5310.13: clone (partial bitmap) 11.05(12.21+0.48)\n 5310.14: pack to file (partial bitmap) 31.25(34.22+3.70)\n 5310.15: rev-list with tree filter (partial bitmap) 0.26(0.22+0.04)\n\nversus the same tests (this time using a multi-pack index):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5326.2: setup multi-pack index 78.99(75.29+11.58)\n 5326.3: simulated clone 11.78(11.56+0.22)\n 5326.4: simulated fetch 1.70(4.49+0.13)\n 5326.5: pack to file (bitmap) 28.02(27.72+8.76)\n 5326.6: rev-list (commits) 0.42(0.36+0.06)\n 5326.7: rev-list (objects) 1.65(1.58+0.06)\n 5326.8: rev-list count with blob:none 0.26(0.21+0.05)\n 5326.9: rev-list count with blob:limit=1k 2.97(2.86+0.10)\n 5326.10: rev-list count with tree:0 0.25(0.20+0.04)\n 5326.11: simulated partial clone 5.65(5.49+0.16)\n 5326.13: clone (partial bitmap) 12.22(13.43+0.38)\n 5326.14: pack to file (partial bitmap) 30.05(31.57+7.25)\n 5326.15: rev-list with tree filter (partial bitmap) 0.24(0.20+0.04)\n\nThere is slight overhead in \"simulated clone\", \"simulated partial\nclone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\ndue to using the MIDX's reverse index to map between bit positions and\nMIDX positions.\n\nThis can be reproduced by running \"git repack -adb\" along with \"git\nmulti-pack-index write --bitmap\" in a large-ish repository. Then run:\n\n $ perf record -o pack.perf git -c core.multiPackIndex=false \\\n pack-objects --all --stdout >/dev/null /dev/null \n---\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..5845109ac7\n--- /dev/null\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,43 @@\n+#!/bin/sh\n+\n+test_description='Tests performance using midx bitmaps'\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n+\n+test_perf_large_repo\n+\n+test_expect_success 'enable multi-pack index' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_perf 'setup multi-pack index' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap\n+'\n+\n+test_full_bitmap\n+\n+test_expect_success 'create partial bitmap state' '\n+\t# pick a commit to represent the repo tip in the past\n+\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t# now pretend we have just one tip\n+\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\tgit update-ref HEAD $cutoff &&\n+\n+\t# and then repack, which will leave us with a nice\n+\t# big bitmap pack of the \"old\" history, and all of\n+\t# the new history will be loose, as if it had been pushed\n+\t# up incrementally and exploded via unpack-objects\n+\tgit repack -Ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\n+\t# and now restore our original tip, as if the pushes\n+\t# had happened\n+\tgit update-ref HEAD $orig_tip\n+'\n+\n+test_partial_bitmap\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"431410","messageId":"YQGX7SMu4UoTJ2VK@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YQBnE+ft/MR3zs1t@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-28T17:46:21Z","receivedAt":"2021-07-28T17:46:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 04:05:39PM -0400, Taylor Blau wrote:\n\n> > I actually think having write_midx_internal() open up a new midx is\n> > reasonable-ish. It's just that:\n> >\n> > - it's weird when it stuffs duplicate packs into the\n> > r->objects->packed_git list. But AFAICT that's not actually hurting\n> > anything?\n> \n> It is hurting us when we try to write a MIDX bitmap, because we try to\n> see if one already exists. And to do that, we call prepare_bitmap_git(),\n> which tries to call open_pack_bitmap_1 on *each* pack in the packed_git\n> list. Critically, prepare_bitmap_git() errors out if it is called with a\n> bitmap_git that has a non-NULL `->pack` pointer.\n\nIt doesn't error out. It does produce a warning(), though, if it ignores\na bitmap (and that warning is doubly confusing because it is ignoring\nbitmap X because it has already loaded and will use that exact same X!).\n\nThis causes t7700.13 to fail because it is being picky about stderr\nbeing empty.\n\nSo the overall behavior is correct, but I agree it's sufficiently ugly\nthat we should make sure it doesn't happen.\n\n Side note: IMHO the \"check all packs to see if there are any other\n bitmaps to warn about\" behavior is kind of pointless, and we should\n consider just returning as soon as we have one. This is already\n somewhat the case after your midx-bitmap patches, as we will not even\n bother to look for a pack bitmap after finding a midx bitmap. That is\n a good thing, because it means you can keep pack bitmaps around for\n flexibility. But let's leave any changes to the pack-only behavior out\n of this series for simplicity.\n\n> I stepped away from my computer for an hour or so and thought about\n> this, and I think that the solution is two-fold:\n> \n> - We should be more careful about freeing up the ->next pointers of a\n> MIDX, and releasing the memory we allocated to hold each MIDX struct\n> in the first place.\n\nYeah. This is a bug already before your series. I suspect nobody noticed\nbecause it's very rare for us to call close_midx() at all, and it only\nmatters if there's an alternate odb with a midx. (The common call to\nclose_midx() is in these write paths, but it is always using a single\nmidx file).\n\n> - We should always be operating on the repository's\n> r->objects->multi_pack_index, or any other MIDX that can be reached\n> via walking the `->next` pointers. If we do that consistently, then\n> we'll only have at most one instance of a MIDX struct corresponding\n> to each MIDX file on disk.\n\nCertainly that makes sense to me in terms of the Windows \"must close the\ncurrent midx before writing\" behavior. We have to realize that we're\noperating in the current repo.\n\nBut we do allow an \"--object-dir\" option to \"multi-pack-index write\",\nand I don't see any other code explicitly requiring that it be part of\nthe current repository. What I'm wondering is whether this would be\nbreaking:\n\n cd $REPO/..\n git multi-pack-index --object-dir $REPO/.git/objects write\n\nor:\n\n cd /some/other/repo\n git multi-pack-index --object-dir $REPO/.git/objects write\n\nThe latter does seem to work, but the former segfaults (usually -- if\nthere's already a midx it is OK).\n\nIf it is broken now, this may be a good time to explicitly forbid it.\nIt does seem to make the --object-dir mostly pointless, though it would\nstill work for operating on a midx in one of your alternates. I'm not\nsure I understand the original point of that option, and if the current\nbehavior is sufficient.\n\nIf it turns out that we can't forbid writing midx's besides the ones in\nr->objects, it may be sufficient to just make any assumptions\nconditional. I.e., _if_ it's one of the ones mentioned by r->objects,\nthen close it, but otherwise leave it open. But if we can get away with\nrestricting ourselves as you described, I think the result will be much\nsimpler, and we should prefer that.\n\n-Peff\n"},{"id":"431412","messageId":"YQGZZTXjSuZkHJgm@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YQBtfRP0svLL6VDl@nand.local","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-28T17:52:37Z","receivedAt":"2021-07-28T17:52:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 04:33:01PM -0400, Taylor Blau wrote:\n\n> > It's interesting that your earlier iteration didn't call\n> > open_pack_index(). Is it necessary, or not? From your description, it\n> > seems like it should be. But maybe some later step lazy-loads it? Even\n> > if so, I can see how prepare_midx_pack() would still be required\n> > (because we want to make sure we are using the same struct).\n> \n> It's only necessary now (at least for determining a preferred pack if\n> the caller didn't specify one with `--preferred-pack`) because we care\n> about reading the `num_objects` field, which the index must be loaded\n> for.\n\nI guess I'm a little confused about \"now\" in your sentence. I understand\nthat it's not necessary before your series to have loaded all of the\nindex files ahead of time. But didn't we need to do so in v2 of your\nseries, which has the preferred-pack logic?\n\nIf so, then was the v2 version buggy, since it only called\nprepare_midx_pack() and not open_pack_index()? And then v3 is fixing\nthat? Or is something else opening the pack index for us?\n\n-Peff\n"},{"id":"431510","messageId":"YQMB32fvSiH9julg@nand.local","threadId":"55464","inReplyTo":"40cff5beb50cdfbd13ae7f6017152f2628b25814.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-29T19:30:39Z","receivedAt":"2021-07-29T19:30:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jul 27, 2021 at 05:19:46PM -0400, Taylor Blau wrote:\n> @@ -914,10 +915,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n> \t\t\t midx_name);\n>\n> -\tif (m)\n> -\t\tctx.m = m;\n> -\telse\n> -\t\tctx.m = load_multi_pack_index(object_dir, 1);\n> +\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n> +\t\tif (!strcmp(object_dir, cur->object_dir)) {\n> +\t\t\tctx.m = cur;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\tif (!ctx.m)\n> +\t\tctx.m = get_local_multi_pack_index(the_repository);\n\nOops, the `if (!ctx.m)` part of this diff is just plain wrong.\n\nI think that I had in my mind that some callers don't pass object_dir,\nand so that we should fall-back to the local MIDX in that case. And so I\nprobably meant to write `if (!object_dir && !ctx.m)` instead.\n\nBut, all of the callers *do* pass the result of get_object_directory(),\nso we don't need to do anything of the sort.\n\nOn a related note, though, a side-effect of this change is that this\nmakes it no longer possible to do\n\n git multi-pack-index write --object-dir=/not/an/alternate.git/objects\n\nsince get_local_multi_pack_index() will only populate the MIDXs in\nalternate object stores. We never enforced that `--object-dir` must\npoint to an alternate, but the documentation uses `` to describe\nthe argument to this flag, and accepting arbitrary non-alternate paths\nseems like a footgun to me.\n\nSo I'm OK with \"breaking\" that behavior, as long as nobody complains\nloudly. Obviously it makes the fix easier to write, but I'd argue that\nthe behavior we're losing is worth getting rid of anyway.\n\nThanks,\nTaylor\n"},{"id":"431512","messageId":"YQMCfnlr6BAXC/c0@nand.local","threadId":"55464","inReplyTo":"YQGZZTXjSuZkHJgm@coredump.intra.peff.net","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-29T19:33:18Z","receivedAt":"2021-07-29T19:33:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 28, 2021 at 01:52:37PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 04:33:01PM -0400, Taylor Blau wrote:\n>\n> > > It's interesting that your earlier iteration didn't call\n> > > open_pack_index(). Is it necessary, or not? From your description, it\n> > > seems like it should be. But maybe some later step lazy-loads it? Even\n> > > if so, I can see how prepare_midx_pack() would still be required\n> > > (because we want to make sure we are using the same struct).\n> >\n> > It's only necessary now (at least for determining a preferred pack if\n> > the caller didn't specify one with `--preferred-pack`) because we care\n> > about reading the `num_objects` field, which the index must be loaded\n> > for.\n>\n> I guess I'm a little confused about \"now\" in your sentence. I understand\n> that it's not necessary before your series to have loaded all of the\n> index files ahead of time. But didn't we need to do so in v2 of your\n> series, which has the preferred-pack logic?\n>\n> If so, then was the v2 version buggy, since it only called\n> prepare_midx_pack() and not open_pack_index()? And then v3 is fixing\n> that? Or is something else opening the pack index for us?\n\nIn earlier versions of this series, I don't think we needed to have the\nindexes loaded by this point, since (before v3) we didn't care about\nignoring the empty packs when finding a default preferred-pack.\n\nBut now we do, and so we need to call open_pack_index() ourselves.\nConfusingly, we only need to do that on packs that *are* included in the\nMIDX, since prepare_midx_pack() doesn't do it for us, but\nadd_pack_to_midx() does.\n\nThanks,\nTaylor\n"},{"id":"431514","messageId":"YQMFIljXl7sAAA/L@nand.local","threadId":"55464","inReplyTo":"YQGX7SMu4UoTJ2VK@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-29T19:44:34Z","receivedAt":"2021-07-29T19:44:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 28, 2021 at 01:46:21PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 04:05:39PM -0400, Taylor Blau wrote:\n>\n> > > I actually think having write_midx_internal() open up a new midx is\n> > > reasonable-ish. It's just that:\n> > >\n> > > - it's weird when it stuffs duplicate packs into the\n> > > r->objects->packed_git list. But AFAICT that's not actually hurting\n> > > anything?\n> >\n> > It is hurting us when we try to write a MIDX bitmap, because we try to\n> > see if one already exists. And to do that, we call prepare_bitmap_git(),\n> > which tries to call open_pack_bitmap_1 on *each* pack in the packed_git\n> > list. Critically, prepare_bitmap_git() errors out if it is called with a\n> > bitmap_git that has a non-NULL `->pack` pointer.\n>\n> It doesn't error out. It does produce a warning(), though, if it ignores\n> a bitmap (and that warning is doubly confusing because it is ignoring\n> bitmap X because it has already loaded and will use that exact same X!).\n>\n> This causes t7700.13 to fail because it is being picky about stderr\n> being empty.\n\nRight, sorry for suggesting that the error was more severe than it\nactually is.\n\n> So the overall behavior is correct, but I agree it's sufficiently ugly\n> that we should make sure it doesn't happen.\n\n100% agreed. I think the most unfortunate thing is the state of\nr->objects->packed_git, since it's utterly bizarre to have the same pack\nopened twice and have both of those copies in the list. That is\ndefinitely worth preventing.\n\n> Side note: IMHO the \"check all packs to see if there are any other\n> bitmaps to warn about\" behavior is kind of pointless, and we should\n> consider just returning as soon as we have one. This is already\n> somewhat the case after your midx-bitmap patches, as we will not even\n> bother to look for a pack bitmap after finding a midx bitmap. That is\n> a good thing, because it means you can keep pack bitmaps around for\n> flexibility. But let's leave any changes to the pack-only behavior out\n> of this series for simplicity.\n\nI agree. I'd be in favor of something like\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f599646e19..5450ffb04c 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -378,13 +378,6 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\treturn -1;\n \t}\n\n-\tif (bitmap_git->pack || bitmap_git->midx) {\n-\t\t/* ignore extra bitmap file; we can only handle one */\n-\t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n-\t\tclose(fd);\n-\t\treturn -1;\n-\t}\n-\n \tbitmap_git->pack = packfile;\n \tbitmap_git->map_size = xsize_t(st.st_size);\n \tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);\n@@ -465,16 +458,15 @@ static int open_pack_bitmap(struct repository *r,\n \t\t\t struct bitmap_index *bitmap_git)\n {\n \tstruct packed_git *p;\n-\tint ret = -1;\n\n \tassert(!bitmap_git->map);\n\n \tfor (p = get_all_packs(r); p; p = p->next) {\n-\t\tif (open_pack_bitmap_1(bitmap_git, p) == 0)\n-\t\t\tret = 0;\n+\t\tif (!open_pack_bitmap_1(bitmap_git, p))\n+\t\t\treturn 0;\n \t}\n\n-\treturn ret;\n+\treturn -1;\n }\n\n static int open_midx_bitmap(struct repository *r,\n\n...but agree that we should wait until after the dust has settled on\nthis already-complex series.\n\n> > - We should always be operating on the repository's\n> > r->objects->multi_pack_index, or any other MIDX that can be reached\n> > via walking the `->next` pointers. If we do that consistently, then\n> > we'll only have at most one instance of a MIDX struct corresponding\n> > to each MIDX file on disk.\n>\n> Certainly that makes sense to me in terms of the Windows \"must close the\n> current midx before writing\" behavior. We have to realize that we're\n> operating in the current repo.\n>\n> But we do allow an \"--object-dir\" option to \"multi-pack-index write\",\n> and I don't see any other code explicitly requiring that it be part of\n> the current repository. What I'm wondering is whether this would be\n> breaking:\n>\n> cd $REPO/..\n> git multi-pack-index --object-dir $REPO/.git/objects write\n>\n> or:\n>\n> cd /some/other/repo\n> git multi-pack-index --object-dir $REPO/.git/objects write\n>\n> The latter does seem to work, but the former segfaults (usually -- if\n> there's already a midx it is OK).\n\nThe former should work, but doesn't, because (as you pointed out to me\nin our regular weekly discussion off-list) that the \"multi-pack-index\"\nentry in git.c's commands array has the RUN_SETUP_GENTLY option, and\nprobably should have RUN_SETUP so that we complain with die() instead of\nBUG.\n\nAnd the latter will continue to work, but only if in your scenario that\n$REPO is an alternate of /some/other/repo.\n\nI wrote a little bit more in [1] about this behavior, but the upshot is\nthat we used to technically support passing *any* directory to\n`--object-dir`, including directories that didn't belong to an\nalternated repository.\n\nAnd that will cease to work after the patch that [1] is in response to\nis applied. But for the reasons that I explain there, I think that is a\nsufficient outcome, because the behavior is kind of bizarre to begin\nwith.\n\n[1]: https://lore.kernel.org/git/YQMB32fvSiH9julg@nand.local/\n\nThanks,\nTaylor\n"},{"id":"432597","messageId":"YRV9tXGTxziK3V7l@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YQMFIljXl7sAAA/L@nand.local","subject":"Re: [PATCH v2 08/24] midx: respect 'core.multiPackIndex' when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T19:59:49Z","receivedAt":"2021-08-12T19:59:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 03:44:34PM -0400, Taylor Blau wrote:\n\n> > Side note: IMHO the \"check all packs to see if there are any other\n> > bitmaps to warn about\" behavior is kind of pointless, and we should\n> > consider just returning as soon as we have one. This is already\n> > somewhat the case after your midx-bitmap patches, as we will not even\n> > bother to look for a pack bitmap after finding a midx bitmap. That is\n> > a good thing, because it means you can keep pack bitmaps around for\n> > flexibility. But let's leave any changes to the pack-only behavior out\n> > of this series for simplicity.\n> \n> I agree. I'd be in favor of something like\n> [...patch...]\n\nYep, that looks good. I'd be quite happy if you sent that once the dust\nis settled.\n\n> > But we do allow an \"--object-dir\" option to \"multi-pack-index write\",\n> > and I don't see any other code explicitly requiring that it be part of\n> > the current repository. What I'm wondering is whether this would be\n> > breaking:\n> >\n> > cd $REPO/..\n> > git multi-pack-index --object-dir $REPO/.git/objects write\n> >\n> > or:\n> >\n> > cd /some/other/repo\n> > git multi-pack-index --object-dir $REPO/.git/objects write\n> >\n> > The latter does seem to work, but the former segfaults (usually -- if\n> > there's already a midx it is OK).\n> \n> The former should work, but doesn't, because (as you pointed out to me\n> in our regular weekly discussion off-list) that the \"multi-pack-index\"\n> entry in git.c's commands array has the RUN_SETUP_GENTLY option, and\n> probably should have RUN_SETUP so that we complain with die() instead of\n> BUG.\n> \n> And the latter will continue to work, but only if in your scenario that\n> $REPO is an alternate of /some/other/repo.\n> \n> I wrote a little bit more in [1] about this behavior, but the upshot is\n> that we used to technically support passing *any* directory to\n> `--object-dir`, including directories that didn't belong to an\n> alternated repository.\n> \n> And that will cease to work after the patch that [1] is in response to\n> is applied. But for the reasons that I explain there, I think that is a\n> sufficient outcome, because the behavior is kind of bizarre to begin\n> with.\n\nYeah, I think I am comfortable with the change at this point. The only\ncase that will be broken is one that is quite ridiculous, and I am\nsurprised worked in the first place. Thanks for talking it through.\n\n-Peff\n"},{"id":"432598","messageId":"YRV95Bx3z6U06Qqd@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YQMCfnlr6BAXC/c0@nand.local","subject":"Re: [PATCH v2 14/24] pack-bitmap: write multi-pack bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T20:00:36Z","receivedAt":"2021-08-12T20:00:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 03:33:18PM -0400, Taylor Blau wrote:\n\n> > > It's only necessary now (at least for determining a preferred pack if\n> > > the caller didn't specify one with `--preferred-pack`) because we care\n> > > about reading the `num_objects` field, which the index must be loaded\n> > > for.\n> >\n> > I guess I'm a little confused about \"now\" in your sentence. I understand\n> > that it's not necessary before your series to have loaded all of the\n> > index files ahead of time. But didn't we need to do so in v2 of your\n> > series, which has the preferred-pack logic?\n> >\n> > If so, then was the v2 version buggy, since it only called\n> > prepare_midx_pack() and not open_pack_index()? And then v3 is fixing\n> > that? Or is something else opening the pack index for us?\n> \n> In earlier versions of this series, I don't think we needed to have the\n> indexes loaded by this point, since (before v3) we didn't care about\n> ignoring the empty packs when finding a default preferred-pack.\n> \n> But now we do, and so we need to call open_pack_index() ourselves.\n> Confusingly, we only need to do that on packs that *are* included in the\n> MIDX, since prepare_midx_pack() doesn't do it for us, but\n> add_pack_to_midx() does.\n\nAh, that was the part I was missing: the default preferred-pack stuff is\nonly in v3. That makes sense.\n\n-Peff\n"},{"id":"432599","messageId":"YRWBZJDCVyUOhk2F@coredump.intra.peff.net","threadId":"55464","inReplyTo":"40cff5beb50cdfbd13ae7f6017152f2628b25814.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T20:15:32Z","receivedAt":"2021-08-12T20:15:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:19:46PM -0400, Taylor Blau wrote:\n\n> Opening multiple instance of the same MIDX can lead to problems like two\n> separate packed_git structures which represent the same pack being added\n> to the repository's object store.\n> [...]\n\nThanks, I think this approach fixes all of the potential problems from\nour earlier discussion. You already noted the \"!ctx->m\" thing in a\nfollow-up. But also...\n\n> Likewise, replace the call to `close_midx()` with\n> `close_object_store()`, since we're about to replace the MIDX with a new\n> one and should invalidate the object store's memory of any MIDX that\n> might have existed beforehand.\n\nYes, I agree we need to do this, but I don't see the change in the\npatch. Did something get lost in the rebasing/squashing process?\n\nI think we'd need something like this:\n\ndiff --git a/midx.c b/midx.c\nindex 6dfafe7a8c..bfb6afea2e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1123,8 +1123,7 @@ static int write_midx_internal(const char *object_dir,\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n-\tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n+\tclose_object_store(the_repository->objects);\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n\nthough I'm not sure:\n\n - if this should be unconditional or dependent on ctx.m (I think the\n latter, because if we are renaming over any open midx, we would have\n filled in ctx.m earlier).\n\n - if this should go below the \"no pack files to index\" check (i.e., is\n there any point in closing if we know we will not write?). In fact,\n its purpose might be more obvious right before finalize_hashfile(),\n but I am OK either way on that.\n\n-Peff\n"},{"id":"432600","messageId":"YRWDBdpizLH/gX1a@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YRWBZJDCVyUOhk2F@coredump.intra.peff.net","subject":"Re: [PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T20:22:29Z","receivedAt":"2021-08-12T20:22:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 12, 2021 at 04:15:32PM -0400, Jeff King wrote:\n\n> I think we'd need something like this:\n> \n> diff --git a/midx.c b/midx.c\n> index 6dfafe7a8c..bfb6afea2e 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1123,8 +1123,7 @@ static int write_midx_internal(const char *object_dir,\n> \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n> \n> -\tif (ctx.m)\n> -\t\tclose_midx(ctx.m);\n> +\tclose_object_store(the_repository->objects);\n> \n> \tif (ctx.nr - dropped_packs == 0) {\n> \t\terror(_(\"no pack files to index.\"));\n> \n> though I'm not sure:\n> \n> - if this should be unconditional or dependent on ctx.m (I think the\n> latter, because if we are renaming over any open midx, we would have\n> filled in ctx.m earlier).\n> \n> - if this should go below the \"no pack files to index\" check (i.e., is\n> there any point in closing if we know we will not write?). In fact,\n> its purpose might be more obvious right before finalize_hashfile(),\n> but I am OK either way on that.\n\nAh, this close_midx() actually gets moved and made unconditional later\nin the series. But it still needs to be close_object_store() instead.\n\nAlso, my mention of finalize_hashfile() is wrong. It's\ncommit_lock_file() that does the actual rename, and indeed, that's where\nyou moved it to in the end, which is good.\n\n-Peff\n\n\n\n\n> \n> -Peff\n"},{"id":"432602","messageId":"YRWDyyhJAot8hxTD@coredump.intra.peff.net","threadId":"55464","inReplyTo":"168b7b0976c7d3ab2082843b7c322252eed11a3f.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 16/25] t5310: move some tests to lib-bitmap.sh","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T20:25:47Z","receivedAt":"2021-08-12T20:25:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:20:04PM -0400, Taylor Blau wrote:\n\n> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> index fe3f98be24..ecb5d0e05d 100644\n> --- a/t/lib-bitmap.sh\n> +++ b/t/lib-bitmap.sh\n> @@ -1,3 +1,6 @@\n> +# Helpers for scripts testing bitamp functionality; see t5310 for\n> +# example usage.\n\nBitamp. :) Not worth a re-roll on its own, but I think we'll want one\nmore round to fix the close_object_store() stuff earlier in the series.\n\nThe rest of the patch looks good.\n\n-Peff\n"},{"id":"432603","messageId":"YRWFFgfXnyiRKdC1@coredump.intra.peff.net","threadId":"55464","inReplyTo":"60ec8b3466e7f94610a45bdd1c79feb06e439429.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 17/25] t/helper/test-read-midx.c: add --checksum mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T20:31:18Z","receivedAt":"2021-08-12T20:31:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:20:07PM -0400, Taylor Blau wrote:\n\n> Subsequent tests will want to check for the existence of a multi-pack\n> bitmap which matches the multi-pack-index stored in the pack directory.\n> \n> The multi-pack bitmap includes the hex checksum of the MIDX it\n> corresponds to in its filename (for example,\n> '$packdir/multi-pack-index-.bitmap'). As a result, some tests\n> want a way to learn what '' is.\n> \n> This helper addresses that need by printing the checksum of the\n> repository's multi-pack-index.\n\nMakes sense. It might be nice to have a generic tool for pulling hashes\nout of checksum files. Perhaps even a tool that is shipped with Git for\noperating on such files (for in-the-field debugging and diagnosis). But\nthat can definitely be separate from this series (if ever).\n\n> t/helper/test-read-midx.c | 16 +++++++++++++++-\n> t/lib-bitmap.sh | 4 ++++\n> 2 files changed, 19 insertions(+), 1 deletion(-)\n\nThe patch itself looks fine to me. One curiosity:\n\n> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> index ecb5d0e05d..09cd036f4d 100644\n> --- a/t/lib-bitmap.sh\n> +++ b/t/lib-bitmap.sh\n> @@ -260,3 +260,7 @@ have_delta () {\n> \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n> \ttest_cmp expect actual\n> }\n> +\n> +midx_checksum () {\n> +\ttest-tool read-midx --checksum \"${1:-.git/objects}\"\n> +}\n\nThis default \".git/objects\" will only _usually_ be the right thing. :)\nIf the actual C code accepted a missing object-dir, it could use the\ncorrect object directory discovered by setup_git_directory().\n\nProbably not a big deal either way, though.\n\n-Peff\n"},{"id":"432606","messageId":"YRWMYg2rvv7HjGE+@coredump.intra.peff.net","threadId":"55464","inReplyTo":"3258ccfc1cc99038e43a37bd2d53c9d30a4f22ae.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T21:02:26Z","receivedAt":"2021-08-12T21:02:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:20:10PM -0400, Taylor Blau wrote:\n\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> new file mode 100755\n> index 0000000000..c1b7d633e2\n> --- /dev/null\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -0,0 +1,277 @@\n> +#!/bin/sh\n> +\n> +test_description='exercise basic multi-pack bitmap functionality'\n> +. ./test-lib.sh\n> +. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n> +\n> +# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n> +# automatic ones.\n> +GIT_TEST_MULTI_PACK_INDEX=0\n> +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n\nThis latter variable doesn't do anything at this point in the series.\nProbably not a big deal (it is simply a noop until then), but if it's\nnot hard, it may make sense to bump the \"respect ... WRITE_BITMAP\" patch\nearlier in the series.\n\n> +test_expect_success 'create single-pack midx with bitmaps' '\n> +\tgit repack -ad &&\n> +\tgit multi-pack-index write --bitmap &&\n> +\ttest_path_is_file $midx &&\n> +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n> +'\n> +\n> +basic_bitmap_tests\n\nWe can't use a midx bitmap without a .rev file. The basic_bitmap_tests\nfunction covers that, but I wonder if we should also check:\n\n test_path_is_file $midx-$(midx_checksum $objdir).rev\n\nin that first test.\n\n> +test_expect_success 'create new additional packs' '\n> +\tfor i in $(test_seq 1 16)\n> +\tdo\n> +\t\ttest_commit \"$i\" &&\n> +\t\tgit repack -d\n> +\tdone &&\n\nThis loop needs an \"|| return 1\" inside to catch &&-chain problems (not\nthat we expect \"repack -d\" to fail, but just on principle).\n\n> +\tgit checkout -b other2 HEAD~8 &&\n> +\tfor i in $(test_seq 1 8)\n> +\tdo\n> +\t\ttest_commit \"side-$i\" &&\n> +\t\tgit repack -d\n> +\tdone &&\n\nDitto here.\n\n> +test_expect_success 'create multi-pack midx with bitmaps' '\n> +\tgit multi-pack-index write --bitmap &&\n> +\n> +\tls $objdir/pack/pack-*.pack >packs &&\n> +\ttest_line_count = 25 packs &&\n> +\n> +\ttest_path_is_file $midx &&\n> +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n> +'\n\nPossible spot for checking the .rev file again (though really, it is\nbelt-and-suspenders at this point).\n\n> +basic_bitmap_tests\n\nI love how the earlier refactoring made it easy to test the single- and\nmulti-pack cases thoroughly.\n\n> +test_expect_success '--no-bitmap is respected when bitmaps exist' '\n> +\tgit multi-pack-index write --bitmap &&\n> +\n> +\ttest_commit respect--no-bitmap &&\n> +\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n\nDo we need to set this env variable? We've already set it to 0 at the\ntop of the script.\n\n> +\ttest_path_is_file $midx &&\n> +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\n> +\tgit multi-pack-index write --no-bitmap &&\n> +\n> +\ttest_path_is_file $midx &&\n> +\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n> +'\n\nOK, so we expect \"--no-bitmap\" to drop the bitmap (just like it does for\na regular pack bitmap). Makes sense. We probably should check:\n\n test_path_is_missing $midx-$(midx_checksum $objdir).rev\n\nhere, too (unlike the other spots, it isn't redundant; we could leave a\nstale file around and likely nobody would notice).\n\n> +test_expect_success 'setup midx with base from later pack' '\n> +\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n> +\t# prefers to delete contents out of a base rather than add to a shorter\n> +\t# object.\n> +\ttest_seq 1 128 >a &&\n> +\ttest_seq 1 130 >b &&\n> +\n> +\tgit add a b &&\n> +\tgit commit -m \"initial commit\" &&\n> +\n> +\ta=$(git rev-parse HEAD:a) &&\n> +\tb=$(git rev-parse HEAD:b) &&\n> +\n> +\t# In the first pack, \"a\" is stored as a delta to \"b\".\n> +\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n> +\t$a\n> +\t$b\n> +\tEOF\n> +\t) &&\n\nThis is brittle with respect to Git's delta heuristics, of course, but I\ndon't think there's a better way to do it with pack-objects. And this is\nnot the first test to make similar assumptions. I think you can\nconstruct a known set of deltas using lib-pack.sh. It may get a bit\ncomplicated. As an alternative, maybe it makes sense to confirm that the\ndeltas are set up as expected? You can do it with cat-file\n--batch-check.\n\n> +test_expect_success 'removing a MIDX clears stale bitmaps' '\n> +\trm -fr repo &&\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\ttest_commit base &&\n> +\t\tgit repack &&\n> +\t\tgit multi-pack-index write --bitmap &&\n> +\n> +\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n> +\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n> +\t\trm $midx &&\n> +\n> +\t\t# Then write a new MIDX.\n> +\t\ttest_commit new &&\n> +\t\tgit repack &&\n> +\t\tgit multi-pack-index write --bitmap &&\n> +\n> +\t\ttest_path_is_file $midx &&\n> +\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\ttest_path_is_missing $stale_bitmap\n> +\t)\n\nAnother spot where we might want to check that the stale .rev file has\ngone away (and optionally that the new one was written; I haven't noted\nall of those, though).\n\n> +test_expect_success 'pack.preferBitmapTips' '\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit_bulk --message=\"%s\" 103 &&\n> +\n> +\t\tgit log --format=\"%H\" >commits.raw &&\n> +\t\tsort commits &&\n> +\n> +\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> +\t\tgit update-ref --stdin +\n> +\t\tgit multi-pack-index write --bitmap &&\n> +\t\ttest_path_is_file $midx &&\n> +\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\n> +\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\tcomm -13 bitmaps commits >before &&\n> +\t\ttest_line_count = 1 before &&\n> +\n> +\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n> +\t\t\t +\n> +\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n> +\t\trm -fr $midx &&\n> +\n> +\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n> +\t\t\tmulti-pack-index write --bitmap &&\n> +\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\tcomm -13 bitmaps commits >after &&\n> +\n> +\t\t! test_cmp before after\n> +\t)\n> +'\n\nOK, so we are not depending on any _specific_ commits to get bitmapped,\nbut just confirming that we have some impact. That may be the best we\ncan do given that we are subject to the bitmap code's heuristics (and\nanyway, this is exactly what the pack version does).\n\nAny other parts of the patch that I didn't quote looked very good to me.\nI'm happy to have such a thorough set of tests.\n\n-Peff\n"},{"id":"432607","messageId":"YRWNm+HlgBE5E0ia@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YRWMYg2rvv7HjGE+@coredump.intra.peff.net","subject":"Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T21:07:39Z","receivedAt":"2021-08-12T21:07:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 12, 2021 at 05:02:26PM -0400, Jeff King wrote:\n\n> > +# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n> > +# automatic ones.\n> > +GIT_TEST_MULTI_PACK_INDEX=0\n> > +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> \n> This latter variable doesn't do anything at this point in the series.\n> Probably not a big deal (it is simply a noop until then), but if it's\n> not hard, it may make sense to bump the \"respect ... WRITE_BITMAP\" patch\n> earlier in the series.\n\nReading the other patches, I guess you ordering was to \"fix\" each of the\ntests preemptively, and then add the knob at the end. That's OK by me.\nFor an alternate test-mode like this, I usually wouldn't worry about\nbisectability, but it doesn't hurt. Somebody reading the commits later\nwon't have any trouble finding the definition of the WRITE_BITMAP\nvariable added in the subsequent patch.\n\n> > +test_expect_success '--no-bitmap is respected when bitmaps exist' '\n> > +\tgit multi-pack-index write --bitmap &&\n> > +\n> > +\ttest_commit respect--no-bitmap &&\n> > +\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n> \n> Do we need to set this env variable? We've already set it to 0 at the\n> top of the script.\n\nBy the way, there were a few more of these later in the script that\ncould be cleaned up, too.\n\n-Peff\n"},{"id":"432608","messageId":"YRWOI8nmhPuCp2TU@coredump.intra.peff.net","threadId":"55464","inReplyTo":"50865e52a37590d0bef541fd96c95a0416d7d585.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 23/25] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T21:09:55Z","receivedAt":"2021-08-12T21:09:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:20:23PM -0400, Taylor Blau wrote:\n\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 5f9bc74adc..82ab668272 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n> \t\t !is_bare_repository())\n> \t\t\twrite_bitmaps = 0;\n> +\t} else if (write_bitmaps &&\n> +\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n> +\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n> +\t\twrite_bitmaps = 0;\n> \t}\n\nThis hunk confused me for a minute, since we are turning write_bitmaps\n_off_ if we see a positive \"write midx bitmap\". But I guess the point is\nto turn off the pack bitmap, and then in the later hunk:\n\n> @@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> \t\tupdate_server_info(0);\n> \tremove_temporary_files();\n> \n> -\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n> -\t\twrite_midx_file(get_object_directory(), NULL, 0);\n> +\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n> +\t\tunsigned flags = 0;\n> +\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n> +\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n> +\t\twrite_midx_file(get_object_directory(), NULL, flags);\n> +\t}\n\n...we'd turn on the midx one. Makes sense.\n\n-Peff\n"},{"id":"432609","messageId":"YRWQPDd2S6Hi/2Ge@coredump.intra.peff.net","threadId":"55464","inReplyTo":"82e8133bf4f6ecf2ca509f6d9e2e0d369d7f19e3.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 25/25] p5326: perf tests for MIDX bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T21:18:52Z","receivedAt":"2021-08-12T21:19:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:20:28PM -0400, Taylor Blau wrote:\n\n> These new performance tests demonstrate effectively the same behavior as\n> p5310, but use a multi-pack bitmap instead of a single-pack one.\n> \n> Notably, p5326 does not create a MIDX bitmap with multiple packs. This\n> is so we can measure a direct comparison between it and p5310. Any\n> difference between the two is measuring just the overhead of using MIDX\n> bitmaps.\n> \n> Here are the results of p5310 and p5326 together, measured at the same\n> time and on the same machine (using a Xenon W-2255 CPU):\n\nNeat. I think having separate perf regression tests for regular and mix\nbitmaps will be useful, but being able to compare the pack and mix\nversions is a cherry on top.\n\nThere was one funny number:\n\n> 5310.2: repack to disk 96.78(93.39+11.33)\n> 5326.2: setup multi-pack index 78.99(75.29+11.58)\n\nIn p5310, that step is repacking and writing bitmaps. With the midx,\nit's repacking, then writing a midx with bitmaps. I'd expect the latter\nto be strictly slower than the former, but here it's faster.\n\nRunning the code locally, I got similar results (with p5310 just a tiny\nbit faster). So it may have just been noise or some other timing issue.\n\n As an aside, I think that test is a little bit bogus due to\n GIT_PERF_REPEAT_COUNT; the first trial will generate bitmaps from\n scratch, and then subsequent runs will reuse partial results. It\n probably should \"rm -f .git/objects/*.bitmap\" within the test. We can\n deal with that separately, though.\n\n-Peff\n"},{"id":"432610","messageId":"YRWQ6G/dYJC5Of1q@coredump.intra.peff.net","threadId":"55464","inReplyTo":"cover.1627420428.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/25] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T21:21:44Z","receivedAt":"2021-08-12T21:21:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 27, 2021 at 05:19:21PM -0400, Taylor Blau wrote:\n\n> Thanks in advance for your review. I think Peff still wanted to read through\n> patches 16-25, but that the first 15 or so should be in pretty good shape by\n> now.\n\nI think this is looking pretty close. There's the close_midx() thing\ndiscussed in patch 9 that I think we need to deal with. In the tests, I\nfound some little nits. Nothing serious, but some of it at least is\nworth fixing.\n\nSo I think with one more fairly trivial re-roll, we can think about\nmerging this to 'next'.\n\nThanks for your patience with my slow reviews, and for all your work on\nthis. It's really quite a complicated topic. :)\n\n-Peff\n"},{"id":"432611","messageId":"YRWQpMXzF1Q8RmVu@nand.local","threadId":"55464","inReplyTo":"YRWDBdpizLH/gX1a@coredump.intra.peff.net","subject":"Re: [PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-12T21:20:36Z","receivedAt":"2021-08-12T21:22:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 12, 2021 at 04:22:29PM -0400, Jeff King wrote:\n> On Thu, Aug 12, 2021 at 04:15:32PM -0400, Jeff King wrote:\n>\n> > I think we'd need something like this:\n> >\n> > diff --git a/midx.c b/midx.c\n> > index 6dfafe7a8c..bfb6afea2e 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -1123,8 +1123,7 @@ static int write_midx_internal(const char *object_dir,\n> > \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n> > \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n> >\n> > -\tif (ctx.m)\n> > -\t\tclose_midx(ctx.m);\n> > +\tclose_object_store(the_repository->objects);\n> >\n> > \tif (ctx.nr - dropped_packs == 0) {\n> > \t\terror(_(\"no pack files to index.\"));\n> >\n> > though I'm not sure:\n> >\n> > - if this should be unconditional or dependent on ctx.m (I think the\n> > latter, because if we are renaming over any open midx, we would have\n> > filled in ctx.m earlier).\n> >\n> > - if this should go below the \"no pack files to index\" check (i.e., is\n> > there any point in closing if we know we will not write?). In fact,\n> > its purpose might be more obvious right before finalize_hashfile(),\n> > but I am OK either way on that.\n>\n> Ah, this close_midx() actually gets moved and made unconditional later\n> in the series. But it still needs to be close_object_store() instead.\n\nExactly; this first patch should read:\n\n if (ctx.m)\n close_object_store(the_repository->objects);\n\nand then the latter patch (15/25) we drop the conditional and move our\ncall down until after the MIDX bitmap is written, but before we call\ncommit_lock_file().\n\nThanks,\nTaylor\n"},{"id":"432612","messageId":"YRWTGzdS1Ab80JnH@nand.local","threadId":"55464","inReplyTo":"YRWFFgfXnyiRKdC1@coredump.intra.peff.net","subject":"Re: [PATCH v3 17/25] t/helper/test-read-midx.c: add --checksum mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-12T21:31:07Z","receivedAt":"2021-08-12T21:32:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 12, 2021 at 04:31:18PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 05:20:07PM -0400, Taylor Blau wrote:\n>\n> > Subsequent tests will want to check for the existence of a multi-pack\n> > bitmap which matches the multi-pack-index stored in the pack directory.\n> >\n> > The multi-pack bitmap includes the hex checksum of the MIDX it\n> > corresponds to in its filename (for example,\n> > '$packdir/multi-pack-index-.bitmap'). As a result, some tests\n> > want a way to learn what '' is.\n> >\n> > This helper addresses that need by printing the checksum of the\n> > repository's multi-pack-index.\n>\n> Makes sense. It might be nice to have a generic tool for pulling hashes\n> out of checksum files. Perhaps even a tool that is shipped with Git for\n> operating on such files (for in-the-field debugging and diagnosis). But\n> that can definitely be separate from this series (if ever).\n\nYeah. That would definitely be in the spirit of \"we should have more\ntest-tool-like helpers exposed via user-facing plumbing\". And I agree\nthat it would be nice, but I definitely agree that it's a topic for a\nlater date ;).\n\n> > t/helper/test-read-midx.c | 16 +++++++++++++++-\n> > t/lib-bitmap.sh | 4 ++++\n> > 2 files changed, 19 insertions(+), 1 deletion(-)\n>\n> The patch itself looks fine to me. One curiosity:\n>\n> > diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> > index ecb5d0e05d..09cd036f4d 100644\n> > --- a/t/lib-bitmap.sh\n> > +++ b/t/lib-bitmap.sh\n> > @@ -260,3 +260,7 @@ have_delta () {\n> > \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n> > \ttest_cmp expect actual\n> > }\n> > +\n> > +midx_checksum () {\n> > +\ttest-tool read-midx --checksum \"${1:-.git/objects}\"\n> > +}\n>\n> This default \".git/objects\" will only _usually_ be the right thing. :)\n> If the actual C code accepted a missing object-dir, it could use the\n> correct object directory discovered by setup_git_directory().\n>\n> Probably not a big deal either way, though.\n\nYeah. We could just sidestep the whole thing by not having the\n`.git/objects` default, since all callers of midx_checksum pass an\nargument, so that fallback is dead code anyway. Thanks for noting, I'll\nremove it for the next round.\n\nThanks,\nTaylor\n"},{"id":"432614","messageId":"YRWi3KHM2iunQ02f@nand.local","threadId":"55464","inReplyTo":"YRWMYg2rvv7HjGE+@coredump.intra.peff.net","subject":"Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-12T22:38:20Z","receivedAt":"2021-08-12T22:38:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 12, 2021 at 05:02:26PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 05:20:10PM -0400, Taylor Blau wrote:\n>\n> > diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> > new file mode 100755\n> > index 0000000000..c1b7d633e2\n> > --- /dev/null\n> > +++ b/t/t5326-multi-pack-bitmaps.sh\n> > @@ -0,0 +1,277 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='exercise basic multi-pack bitmap functionality'\n> > +. ./test-lib.sh\n> > +. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n> > +\n> > +# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n> > +# automatic ones.\n> > +GIT_TEST_MULTI_PACK_INDEX=0\n> > +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n>\n> This latter variable doesn't do anything at this point in the series.\n> Probably not a big deal (it is simply a noop until then), but if it's\n> not hard, it may make sense to bump the \"respect ... WRITE_BITMAP\" patch\n> earlier in the series.\n\nIf my memory serves me correctly, I think the very first version of this\npatch didn't have a GIT_TEST_MULTI_PACK_INDEX{,_WRITE_BITMAP}=0 at the\ntop, and so individual invocations needed to set it in their own\nenvironment. Presumably at some point I added this, but forgot to clean\nup the redundant ones. I removed the ones you mentioned in your\nresponse, and a few others.\n\n> > +test_expect_success 'create single-pack midx with bitmaps' '\n> > +\tgit repack -ad &&\n> > +\tgit multi-pack-index write --bitmap &&\n> > +\ttest_path_is_file $midx &&\n> > +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n> > +'\n> > +\n> > +basic_bitmap_tests\n>\n> We can't use a midx bitmap without a .rev file. The basic_bitmap_tests\n> function covers that, but I wonder if we should also check:\n>\n> test_path_is_file $midx-$(midx_checksum $objdir).rev\n>\n> in that first test.\n\nGood idea. These tests probably preceded the invention of .rev files, so\na lot of them needed updating. I made sure to add them where\nappropriate.\n\n> > +test_expect_success 'create new additional packs' '\n> > +\tfor i in $(test_seq 1 16)\n> > +\tdo\n> > +\t\ttest_commit \"$i\" &&\n> > +\t\tgit repack -d\n> > +\tdone &&\n>\n> This loop needs an \"|| return 1\" inside to catch &&-chain problems (not\n> that we expect \"repack -d\" to fail, but just on principle).\n\nNice catch, thanks.\n\n> I love how the earlier refactoring made it easy to test the single- and\n> multi-pack cases thoroughly.\n\nLikewise :-).\n\n> > +test_expect_success 'setup midx with base from later pack' '\n> > +\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n> > +\t# prefers to delete contents out of a base rather than add to a shorter\n> > +\t# object.\n> > +\ttest_seq 1 128 >a &&\n> > +\ttest_seq 1 130 >b &&\n> > +\n> > +\tgit add a b &&\n> > +\tgit commit -m \"initial commit\" &&\n> > +\n> > +\ta=$(git rev-parse HEAD:a) &&\n> > +\tb=$(git rev-parse HEAD:b) &&\n> > +\n> > +\t# In the first pack, \"a\" is stored as a delta to \"b\".\n> > +\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n> > +\t$a\n> > +\t$b\n> > +\tEOF\n> > +\t) &&\n>\n> This is brittle with respect to Git's delta heuristics, of course, but I\n> don't think there's a better way to do it with pack-objects. And this is\n> not the first test to make similar assumptions. I think you can\n> construct a known set of deltas using lib-pack.sh. It may get a bit\n> complicated. As an alternative, maybe it makes sense to confirm that the\n> deltas are set up as expected? You can do it with cat-file\n> --batch-check.\n\nYeah, I definitely agree that this test is brittle. But it would fail if\nour assumptions about what gets delta'd with what changes, because we do\ncheck that 'a' is a delta on top of 'b' (see the call to have_delta\ntowards the end of this test). That have_delta helper does use\n`--batch-check=%(deltabase)`, which is (I think) the cat-file invocation\nyou're mentioning.\n\nThanks,\nTaylor\n"},{"id":"432615","messageId":"YRWjluDhPlMyQfmx@nand.local","threadId":"55464","inReplyTo":"YRWQ6G/dYJC5Of1q@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-12T22:41:26Z","receivedAt":"2021-08-12T22:41:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 12, 2021 at 05:21:44PM -0400, Jeff King wrote:\n> On Tue, Jul 27, 2021 at 05:19:21PM -0400, Taylor Blau wrote:\n>\n> > Thanks in advance for your review. I think Peff still wanted to read through\n> > patches 16-25, but that the first 15 or so should be in pretty good shape by\n> > now.\n>\n> I think this is looking pretty close. There's the close_midx() thing\n> discussed in patch 9 that I think we need to deal with. In the tests, I\n> found some little nits. Nothing serious, but some of it at least is\n> worth fixing.\n\nThanks; I think I fixed everything up that you mentioned. The big deals\nwere about close_object_store() versus close_midx(), and the changes to\nt5326 in patch 18/25. I think the remaining comments on patches 19-25\nwere thinking aloud instead of recommending any changes.\n\n> So I think with one more fairly trivial re-roll, we can think about\n> merging this to 'next'.\n\nGreat. I have all of that prepared locally, but I'll wait until after\nMonday to send it since I don't want to dilute the conversation away\nfrom release hardening (I certainly don't mind your review, I just\nfigure folks would appreciate me *not* sending 25 new messages to their\ninboxes ;-)).\n\n> Thanks for your patience with my slow reviews, and for all your work on\n> this. It's really quite a complicated topic. :)\n\nThanks for reviewing. I tried my best to break this series up from all\nof the ones that it depends on, but there was only so much I could do to\nisolate the complexity. I'm glad that it had a thorough set of eyes on\nit.\n\nI'll look forward to sending a hopefully-final reroll shortly after the\nrelease so we can move onto a few more cosmetic topics on top to\nintegrate MIDX bitmaps more closely with `repack` and so on.\n\nThanks,\nTaylor\n"},{"id":"432617","messageId":"YRWtaBtZfRKvKrpk@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YRWi3KHM2iunQ02f@nand.local","subject":"Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-12T23:23:20Z","receivedAt":"2021-08-12T23:23:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 12, 2021 at 06:38:20PM -0400, Taylor Blau wrote:\n\n> > This is brittle with respect to Git's delta heuristics, of course, but I\n> > don't think there's a better way to do it with pack-objects. And this is\n> > not the first test to make similar assumptions. I think you can\n> > construct a known set of deltas using lib-pack.sh. It may get a bit\n> > complicated. As an alternative, maybe it makes sense to confirm that the\n> > deltas are set up as expected? You can do it with cat-file\n> > --batch-check.\n> \n> Yeah, I definitely agree that this test is brittle. But it would fail if\n> our assumptions about what gets delta'd with what changes, because we do\n> check that 'a' is a delta on top of 'b' (see the call to have_delta\n> towards the end of this test). That have_delta helper does use\n> `--batch-check=%(deltabase)`, which is (I think) the cat-file invocation\n> you're mentioning.\n\nDoh, I totally missed that. I was expecting to verify it earlier in the\ntest as a pre-condition, but it works just fine where it is. So yeah,\nyou are already doing the thing I was suggesting.\n\n-Peff\n"},{"id":"433568","messageId":"92dc0bbc0d0e3297f9eb6f51e8d6f40b367a77ca.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 01/25] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:15:51Z","receivedAt":"2021-08-24T16:16:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The special `--test-bitmap` mode of `git rev-list` is used to compare\nthe result of an object traversal with a bitmap to check its integrity.\nThis mode does not, however, assert that the types of reachable objects\nare stored correctly.\n\nHarden this mode by teaching it to also check that each time an object's\nbit is marked, the corresponding bit should be set in exactly one of the\ntype bitmaps (whose type matches the object's true type).\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d999616c9e..9b11af87aa 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1325,10 +1325,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,\n struct bitmap_test_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\n+\tstruct bitmap *commits;\n+\tstruct bitmap *trees;\n+\tstruct bitmap *blobs;\n+\tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n };\n \n+static void test_bitmap_type(struct bitmap_test_data *tdata,\n+\t\t\t struct object *obj, int pos)\n+{\n+\tenum object_type bitmap_type = OBJ_NONE;\n+\tint bitmaps_nr = 0;\n+\n+\tif (bitmap_get(tdata->commits, pos)) {\n+\t\tbitmap_type = OBJ_COMMIT;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->trees, pos)) {\n+\t\tbitmap_type = OBJ_TREE;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->blobs, pos)) {\n+\t\tbitmap_type = OBJ_BLOB;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->tags, pos)) {\n+\t\tbitmap_type = OBJ_TAG;\n+\t\tbitmaps_nr++;\n+\t}\n+\n+\tif (bitmap_type == OBJ_NONE)\n+\t\tdie(\"object %s not found in type bitmaps\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmaps_nr > 1)\n+\t\tdie(\"object %s does not have a unique type\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmap_type != obj->type)\n+\t\tdie(\"object %s: real type %s, expected: %s\",\n+\t\t oid_to_hex(&obj->oid),\n+\t\t type_name(obj->type),\n+\t\t type_name(bitmap_type));\n+}\n+\n static void test_show_object(struct object *object, const char *name,\n \t\t\t void *data)\n {\n@@ -1338,6 +1380,7 @@ static void test_show_object(struct object *object, const char *name,\n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&object->oid));\n+\ttest_bitmap_type(tdata, object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1352,6 +1395,7 @@ static void test_show_commit(struct commit *commit, void *data)\n \t\t\t\t &commit->object.oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&commit->object.oid));\n+\ttest_bitmap_type(tdata, &commit->object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1399,6 +1443,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \ttdata.bitmap_git = bitmap_git;\n \ttdata.base = bitmap_new();\n+\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n \ttdata.seen = 0;\n \n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433570","messageId":"cover.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:15:47Z","receivedAt":"2021-08-24T16:16:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is what I anticipate to be a final reroll of my series to implement\nmulti-pack reachability bitmaps, based on review feedback from Peff.\n\nMost of the change since last time are cosmetic test clean-ups. The previous\nreroll of this series incorporated feedback from a discussion[1] surrounding the\n`multi-pack-index` builtin's `--object-dir` argument. This reroll fixes a bug\ndiscussed here[2] where we should have been calling close_object_store() but\nweren't; the remainder of that bug has already been dealt with.\n\nThanks everybody for dealing with multiple versions of this quite lengthy and\ncomplicated series. Hopefully we are done in this round and can move on to\nintegrating this with `git repack`, which will complete the MIDX bitmaps topic.\n\n[1]: https://lore.kernel.org/git/YQMFIljXl7sAAA%2FL@nand.local/\n[2]: https://lore.kernel.org/git/YRWBZJDCVyUOhk2F@coredump.intra.peff.net/\n\nJeff King (2):\n t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n\nTaylor Blau (23):\n pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n pack-bitmap-write.c: free existing bitmaps\n Documentation: describe MIDX-based bitmaps\n midx: clear auxiliary .rev after replacing the MIDX\n midx: reject empty `--preferred-pack`'s\n midx: infer preferred pack when not given one\n midx: close linked MIDXs, avoid leaking memory\n midx: avoid opening multiple MIDXs when writing\n pack-bitmap.c: introduce 'bitmap_num_objects()'\n pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n pack-bitmap.c: avoid redundant calls to try_partial_reuse\n pack-bitmap: read multi-pack bitmaps\n pack-bitmap: write multi-pack bitmaps\n t5310: move some tests to lib-bitmap.sh\n t/helper/test-read-midx.c: add --checksum mode\n t5326: test multi-pack bitmap behavior\n t5319: don't write MIDX bitmaps in t5319\n t7700: update to work with MIDX bitmap test knob\n midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n p5310: extract full and partial bitmap tests\n p5326: perf tests for MIDX bitmaps\n\n Documentation/git-multi-pack-index.txt | 18 +-\n Documentation/technical/bitmap-format.txt | 71 ++-\n Documentation/technical/multi-pack-index.txt | 10 +-\n builtin/multi-pack-index.c | 2 +\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 12 +-\n ci/run-build-and-tests.sh | 1 +\n midx.c | 319 +++++++++++-\n midx.h | 5 +\n pack-bitmap-write.c | 79 ++-\n pack-bitmap.c | 499 ++++++++++++++++---\n pack-bitmap.h | 9 +-\n packfile.c | 2 +-\n t/README | 4 +\n t/helper/test-read-midx.c | 16 +-\n t/lib-bitmap.sh | 240 +++++++++\n t/perf/lib-bitmap.sh | 69 +++\n t/perf/p5310-pack-bitmaps.sh | 65 +--\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++\n t/t0410-partial-clone.sh | 12 +-\n t/t5310-pack-bitmaps.sh | 231 +--------\n t/t5319-multi-pack-index.sh | 20 +-\n t/t5326-multi-pack-bitmaps.sh | 286 +++++++++++\n t/t7700-repack.sh | 18 +-\n 24 files changed, 1603 insertions(+), 436 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\nRange-diff against v3:\n 1: fa4cbed48e = 1: 92dc0bbc0d pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n 2: 2b15c1fc5c = 2: 979276bc74 pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n 3: 2ad513a230 = 3: 8f00493955 pack-bitmap-write.c: free existing bitmaps\n 4: 8da5de7c24 = 4: bc7db926d8 Documentation: describe MIDX-based bitmaps\n 5: 49297f57ed = 5: 771741844b midx: clear auxiliary .rev after replacing the MIDX\n 6: c5513f2a75 = 6: dab5dbf228 midx: reject empty `--preferred-pack`'s\n 7: 53ef0a6d67 = 7: 31f4517de0 midx: infer preferred pack when not given one\n 8: 114773d9cd = 8: aa3bd96d9b midx: close linked MIDXs, avoid leaking memory\n 9: 40cff5beb5 ! 9: c9fea31fa8 midx: avoid opening multiple MIDXs when writing\n @@ Commit message\n one and should invalidate the object store's memory of any MIDX that\n might have existed beforehand.\n \n + Note that this now forbids passing object directories that don't belong\n + to alternate repositories over `--object-dir`, since before we would\n + have happily opened a MIDX in any directory, but now restrict ourselves\n + to only those reachable by `r->objects->multi_pack_index` (and alternate\n + MIDXs that we can see by walking the `next` pointer).\n +\n + As far as I can tell, supporting arbitrary directories with\n + `--object-dir` was a historical accident, since even the documentation\n + says `` when referring to the value passed to this option.\n +\n + A future patch could clean this up and provide a warning() when a\n + non-alternate directory was given, since we'll still write a new MIDX\n + there, we just won't reuse any MIDX that might happen to already exist\n + in that directory.\n +\n Signed-off-by: Taylor Blau \n \n ## midx.c ##\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\t\tbreak;\n +\t\t}\n +\t}\n -+\tif (!ctx.m)\n -+\t\tctx.m = get_local_multi_pack_index(the_repository);\n \n \tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n \t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n +@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n + \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n + \n + \tif (ctx.m)\n +-\t\tclose_midx(ctx.m);\n ++\t\tclose_object_store(the_repository->objects);\n + \n + \tif (ctx.nr - dropped_packs == 0) {\n + \t\terror(_(\"no pack files to index.\"));\n @@ midx.c: int write_midx_file(const char *object_dir,\n \t\t const char *preferred_pack_name,\n \t\t unsigned flags)\n10: ca7f726abf ! 10: ee72fb7e38 pack-bitmap.c: introduce 'bitmap_num_objects()'\n @@ pack-bitmap.c: static void show_extended_objects(struct bitmap_index *bitmap_git\n \n \t\tobj = eindex->objects[i];\n @@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n - \t * individually.\n + \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n -\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n11: 67e6897a34 = 11: ede0bf1ce1 pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n12: 743a1a138e = 12: df6844def0 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n13: a3b641b3e6 = 13: 4e06f051a7 pack-bitmap.c: avoid redundant calls to try_partial_reuse\n14: 141ff83275 = 14: a0d73eb3d3 pack-bitmap: read multi-pack bitmaps\n15: 54600b5814 ! 15: 9d83ad77ab pack-bitmap: write multi-pack bitmaps\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n -\tif (ctx.m)\n --\t\tclose_midx(ctx.m);\n +-\t\tclose_object_store(the_repository->objects);\n -\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n +\t\t}\n +\t}\n +\n -+\tclose_midx(ctx.m);\n ++\tclose_object_store(the_repository->objects);\n \n \tcommit_lock_file(&lk);\n \n16: 168b7b0976 ! 16: a92af89884 t5310: move some tests to lib-bitmap.sh\n @@ Commit message\n \n ## t/lib-bitmap.sh ##\n @@\n -+# Helpers for scripts testing bitamp functionality; see t5310 for\n ++# Helpers for scripts testing bitmap functionality; see t5310 for\n +# example usage.\n +\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n17: 60ec8b3466 ! 17: d47aa4a919 t/helper/test-read-midx.c: add --checksum mode\n @@ t/lib-bitmap.sh: have_delta () {\n }\n +\n +midx_checksum () {\n -+\ttest-tool read-midx --checksum \"${1:-.git/objects}\"\n ++\ttest-tool read-midx --checksum \"$1\"\n +}\n18: 3258ccfc1c ! 18: 9d9d9f28a6 t5326: test multi-pack bitmap behavior\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\tgit repack -ad &&\n +\tgit multi-pack-index write --bitmap &&\n +\ttest_path_is_file $midx &&\n -+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n +'\n +\n +basic_bitmap_tests\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\tfor i in $(test_seq 1 16)\n +\tdo\n +\t\ttest_commit \"$i\" &&\n -+\t\tgit repack -d\n ++\t\tgit repack -d || return 1\n +\tdone &&\n +\n +\tgit checkout -b other2 HEAD~8 &&\n +\tfor i in $(test_seq 1 8)\n +\tdo\n +\t\ttest_commit \"side-$i\" &&\n -+\t\tgit repack -d\n ++\t\tgit repack -d || return 1\n +\tdone &&\n +\tgit checkout second\n +'\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\ttest_line_count = 25 packs &&\n +\n +\ttest_path_is_file $midx &&\n -+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n +'\n +\n +basic_bitmap_tests\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\tgit multi-pack-index write --bitmap &&\n +\n +\ttest_commit respect--no-bitmap &&\n -+\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n ++\tgit repack -d &&\n +\n +\ttest_path_is_file $midx &&\n +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n +\n +\tgit multi-pack-index write --no-bitmap &&\n +\n +\ttest_path_is_file $midx &&\n -+\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n ++\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n ++\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n +'\n +\n +test_expect_success 'setup midx with base from later pack' '\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\t\t\tgit config core.multiPackIndex true &&\n +\t\t\tif test \"MIDX\" = \"$from\"\n +\t\t\tthen\n -+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&\n ++\t\t\t\tgit repack -Ad &&\n +\t\t\t\tgit multi-pack-index write --bitmap\n +\t\t\telse\n -+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n ++\t\t\t\tgit repack -Adb\n +\t\t\tfi\n +\t\t)\n +\t'\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\n +\t\t\tif test \"MIDX\" = \"$to\"\n +\t\t\tthen\n -+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&\n ++\t\t\t\tgit repack -d &&\n +\t\t\t\tgit multi-pack-index write --bitmap\n +\t\t\telse\n -+\t\t\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb\n ++\t\t\t\tgit repack -Adb\n +\t\t\tfi\n +\t\t)\n +\t'\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\ttest_commit loose &&\n +\tgit multi-pack-index write --bitmap 2>err &&\n +\ttest_path_is_file $midx &&\n -+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n +'\n +\n +basic_bitmap_tests HEAD~\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\n +\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n +\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n ++\t\tstale_rev=$midx-$(midx_checksum $objdir).rev &&\n +\t\trm $midx &&\n +\n +\t\t# Then write a new MIDX.\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\n +\t\ttest_path_is_file $midx &&\n +\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n -+\t\ttest_path_is_missing $stale_bitmap\n ++\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n ++\t\ttest_path_is_missing $stale_bitmap &&\n ++\t\ttest_path_is_missing $stale_rev\n +\t)\n +'\n +\n @@ t/t5326-multi-pack-bitmaps.sh (new)\n +\t\tgit multi-pack-index write --bitmap &&\n +\t\ttest_path_is_file $midx &&\n +\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n +\n +\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +\t\tcomm -13 bitmaps commits >before &&\n19: 47c7e6bb9b = 19: 3e0da7e5ed t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n20: 6a708858b1 = 20: 4e0d49a2dd t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n21: 1eaa744b24 = 21: 47eba8ecf9 t5319: don't write MIDX bitmaps in t5319\n22: a4a899e31f = 22: 3d78afa2ad t7700: update to work with MIDX bitmap test knob\n23: 50865e52a3 = 23: c2f94e033d midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n24: 0f1fd6e7d4 = 24: 6b03016c99 p5310: extract full and partial bitmap tests\n25: 82e8133bf4 = 25: d98faa4c2c p5326: perf tests for MIDX bitmaps\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"433569","messageId":"979276bc74a9e703da844d461503f6720663ae31.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 02/25] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:15:54Z","receivedAt":"2021-08-24T16:16:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The set of objects covered by a bitmap must be closed under\nreachability, since it must be the case that there is a valid bit\nposition assigned for every possible reachable object (otherwise the\nbitmaps would be incomplete).\n\nPack bitmaps are never written from 'git repack' unless repacking\nall-into-one, and so we never write non-closed bitmaps (except in the\ncase of partial clones where we aren't guaranteed to have all objects).\n\nBut multi-pack bitmaps change this, since it isn't known whether the\nset of objects in the MIDX is closed under reachability until walking\nthem. Plumb through a bit that is set when a reachable object isn't\nfound.\n\nAs soon as a reachable object isn't found in the set of objects to\ninclude in the bitmap, bitmap_writer_build() knows that the set is not\nclosed, and so it now fails gracefully.\n\nA test is added in t0410 to trigger a bitmap write without full\nreachability closure by removing local copies of some reachable objects\nfrom a promisor remote.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 +-\n pack-bitmap-write.c | 76 ++++++++++++++++++++++++++++------------\n pack-bitmap.h | 2 +-\n t/t0410-partial-clone.sh | 9 ++++-\n 4 files changed, 64 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex de00adbb9e..8a523624a1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1256,7 +1256,8 @@ static void write_pack_file(void)\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n-\t\t\t\tbitmap_writer_build(&to_pack);\n+\t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n+\t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..d374f7884b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit)\n \twriter.selected_nr++;\n }\n \n-static uint32_t find_object_pos(const struct object_id *oid)\n+static uint32_t find_object_pos(const struct object_id *oid, int *found)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n \n \tif (!entry) {\n-\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\tif (found)\n+\t\t\t*found = 0;\n+\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\t\treturn 0;\n \t}\n \n+\tif (found)\n+\t\t*found = 1;\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n@@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n \n-static void fill_bitmap_tree(struct bitmap *bitmap,\n-\t\t\t struct tree *tree)\n+static int fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n {\n+\tint found;\n \tuint32_t pos;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t * If our bit is already set, then there is nothing to do. Both this\n \t * tree and all of its children will be set.\n \t */\n-\tpos = find_object_pos(&tree->object.oid);\n+\tpos = find_object_pos(&tree->object.oid, &found);\n+\tif (!found)\n+\t\treturn -1;\n \tif (bitmap_get(bitmap, pos))\n-\t\treturn;\n+\t\treturn 0;\n \tbitmap_set(bitmap, pos);\n \n \tif (parse_tree(tree) < 0)\n@@ -355,11 +363,15 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \twhile (tree_entry(&desc, &entry)) {\n \t\tswitch (object_type(entry.mode)) {\n \t\tcase OBJ_TREE:\n-\t\t\tfill_bitmap_tree(bitmap,\n-\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tif (fill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid)) < 0)\n+\t\t\t\treturn -1;\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n-\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tpos = find_object_pos(&entry.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(bitmap, pos);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\t/* Gitlink, etc; not reachable */\n@@ -368,15 +380,18 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t}\n \n \tfree_tree_buffer(tree);\n+\treturn 0;\n }\n \n-static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue,\n-\t\t\t struct prio_queue *tree_queue,\n-\t\t\t struct bitmap_index *old_bitmap,\n-\t\t\t const uint32_t *mapping)\n+static int fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n+\tint found;\n+\tuint32_t pos;\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n@@ -401,11 +416,16 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\tif (!found)\n+\t\t\treturn -1;\n+\t\tbitmap_set(ent->bitmap, pos);\n \t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n-\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n \t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n \t\t\t\tbitmap_set(ent->bitmap, pos);\n \t\t\t\tprio_queue_put(queue, p->item);\n@@ -413,8 +433,12 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t}\n \t}\n \n-\twhile (tree_queue->nr)\n-\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n+\twhile (tree_queue->nr) {\n+\t\tif (fill_bitmap_tree(ent->bitmap,\n+\t\t\t\t prio_queue_get(tree_queue)) < 0)\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -432,7 +456,7 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+int bitmap_writer_build(struct packing_data *to_pack)\n {\n \tstruct bitmap_builder bb;\n \tsize_t i;\n@@ -441,6 +465,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n \tuint32_t *mapping;\n+\tint closed = 1; /* until proven otherwise */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n-\t\t\t\t old_bitmap, mapping);\n+\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping) < 0) {\n+\t\t\tclosed = 0;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -499,7 +527,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \tstop_progress(&writer.progress);\n \n-\tcompute_xor_offsets();\n+\tif (closed)\n+\t\tcompute_xor_offsets();\n+\treturn closed ? 0 : -1;\n }\n \n /**\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 99d733eb26..020cd8d868 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -87,7 +87,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n-void bitmap_writer_build(struct packing_data *to_pack);\n+int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex a211a66c67..bbcc51ee8e 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '\n repack_and_check () {\n \trm -rf repo2 &&\n \tcp -r repo repo2 &&\n-\tgit -C repo2 repack $1 -d &&\n+\tif test x\"$1\" = \"x--must-fail\"\n+\tthen\n+\t\tshift\n+\t\ttest_must_fail git -C repo2 repack $1 -d\n+\telse\n+\t\tgit -C repo2 repack $1 -d\n+\tfi &&\n \tgit -C repo2 fsck &&\n \n \tgit -C repo2 cat-file -e $2 &&\n@@ -561,6 +567,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' '\n \tprintf \"$THREE\\n\" | pack_as_from_promisor &&\n \tdelete_object repo \"$ONE\" &&\n \n+\trepack_and_check --must-fail -ab \"$TWO\" \"$THREE\" &&\n \trepack_and_check -a \"$TWO\" \"$THREE\" &&\n \trepack_and_check -A \"$TWO\" \"$THREE\" &&\n \trepack_and_check -l \"$TWO\" \"$THREE\"\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433571","messageId":"8f00493955662a5c7ec8d50a1b9c9ffe2ef7f343.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 03/25] pack-bitmap-write.c: free existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:15:56Z","receivedAt":"2021-08-24T16:16:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new bitmap, the bitmap writer code attempts to read the\nexisting bitmap (if one is present). This is done in order to quickly\npermute the bits of any bitmaps for commits which appear in the existing\nbitmap, and were also selected for the new bitmap.\n\nBut since this code was added in 341fa34887 (pack-bitmap-write: use\nexisting bitmaps, 2020-12-08), the resources associated with opening an\nexisting bitmap were never released.\n\nIt's fine to ignore this, but it's bad hygiene. It will also cause a\nproblem for the multi-pack-index builtin, which will be responsible not\nonly for writing bitmaps, but also for expiring any old multi-pack\nbitmaps.\n\nIf an existing bitmap was reused here, it will also be expired. That\nwill cause a problem on platforms which require file resources to be\nclosed before unlinking them, like Windows. Avoid this by ensuring we\nclose reused bitmaps with free_bitmap_index() before removing them.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d374f7884b..142fd0adb8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tclear_prio_queue(&queue);\n \tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree_bitmap_index(old_bitmap);\n \tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433572","messageId":"bc7db926d8ad6c523dc303344fb52811b9b0b2c3.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 04/25] Documentation: describe MIDX-based bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:15:59Z","receivedAt":"2021-08-24T16:16:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Update the technical documentation to describe the multi-pack bitmap\nformat. This patch merely introduces the new format, and describes its\nhigh-level ideas. Git does not yet know how to read nor write these\nmulti-pack variants, and so the subsequent patches will:\n\n - Introduce code to interpret multi-pack bitmaps, according to this\n document.\n\n - Then, introduce code to write multi-pack bitmaps from the 'git\n multi-pack-index write' sub-command.\n\nFinally, the implementation will gain tests in subsequent patches (as\nopposed to inline with the patch teaching Git how to write multi-pack\nbitmaps) to avoid a cyclic dependency.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 71 ++++++++++++++++----\n Documentation/technical/multi-pack-index.txt | 10 +--\n 2 files changed, 60 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f8c18a0f7a..04b3ec2178 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -1,6 +1,44 @@\n GIT bitmap v1 format\n ====================\n \n+== Pack and multi-pack bitmaps\n+\n+Bitmaps store reachability information about the set of objects in a packfile,\n+or a multi-pack index (MIDX). The former is defined obviously, and the latter is\n+defined as the union of objects in packs contained in the MIDX.\n+\n+A bitmap may belong to either one pack, or the repository's multi-pack index (if\n+it exists). A repository may have at most one bitmap.\n+\n+An object is uniquely described by its bit position within a bitmap:\n+\n+\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n+\tthe __n__th object in pack order. For a function `offset` which maps\n+\tobjects to their byte offset within a pack, pack order is defined as\n+\tfollows:\n+\n+\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n+\n+\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n+\t__n__th object in MIDX order. With an additional function `pack` which\n+\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n+\tis defined as follows:\n+\n+\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n+\n+\tThe ordering between packs is done according to the MIDX's .rev file.\n+\tNotably, the preferred pack sorts ahead of all other packs.\n+\n+The on-disk representation (described below) of a bitmap is the same regardless\n+of whether or not that bitmap belongs to a packfile or a MIDX. The only\n+difference is the interpretation of the bits, which is described above.\n+\n+Certain bitmap extensions are supported (see: Appendix B). No extensions are\n+required for bitmaps corresponding to packfiles. For bitmaps that correspond to\n+MIDXs, both the bit-cache and rev-cache extensions are required.\n+\n+== On-disk format\n+\n \t- A header appears at the beginning:\n \n \t\t4-byte signature: {'B', 'I', 'T', 'M'}\n@@ -14,17 +52,19 @@ GIT bitmap v1 format\n \t\t\tThe following flags are supported:\n \n \t\t\t- BITMAP_OPT_FULL_DAG (0x1) REQUIRED\n-\t\t\tThis flag must always be present. It implies that the bitmap\n-\t\t\tindex has been generated for a packfile with full closure\n-\t\t\t(i.e. where every single object in the packfile can find\n-\t\t\t its parent links inside the same packfile). This is a\n-\t\t\trequirement for the bitmap index format, also present in JGit,\n-\t\t\tthat greatly reduces the complexity of the implementation.\n+\t\t\tThis flag must always be present. It implies that the\n+\t\t\tbitmap index has been generated for a packfile or\n+\t\t\tmulti-pack index (MIDX) with full closure (i.e. where\n+\t\t\tevery single object in the packfile/MIDX can find its\n+\t\t\tparent links inside the same packfile/MIDX). This is a\n+\t\t\trequirement for the bitmap index format, also present in\n+\t\t\tJGit, that greatly reduces the complexity of the\n+\t\t\timplementation.\n \n \t\t\t- BITMAP_OPT_HASH_CACHE (0x4)\n \t\t\tIf present, the end of the bitmap file contains\n \t\t\t`N` 32-bit name-hash values, one per object in the\n-\t\t\tpack. The format and meaning of the name-hash is\n+\t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n \t\t4-byte entry count (network byte order)\n@@ -33,7 +73,8 @@ GIT bitmap v1 format\n \n \t\t20-byte checksum\n \n-\t\t\tThe SHA1 checksum of the pack this bitmap index belongs to.\n+\t\t\tThe SHA1 checksum of the pack/MIDX this bitmap index\n+\t\t\tbelongs to.\n \n \t- 4 EWAH bitmaps that act as type indexes\n \n@@ -50,7 +91,7 @@ GIT bitmap v1 format\n \t\t\t- Tags\n \n \t\tIn each bitmap, the `n`th bit is set to true if the `n`th object\n-\t\tin the packfile is of that type.\n+\t\tin the packfile or multi-pack index is of that type.\n \n \t\tThe obvious consequence is that the OR of all 4 bitmaps will result\n \t\tin a full set (all bits set), and the AND of all 4 bitmaps will\n@@ -62,8 +103,9 @@ GIT bitmap v1 format\n \t\tEach entry contains the following:\n \n \t\t- 4-byte object position (network byte order)\n-\t\t\tThe position **in the index for the packfile** where the\n-\t\t\tbitmap for this commit is found.\n+\t\t\tThe position **in the index for the packfile or\n+\t\t\tmulti-pack index** where the bitmap for this commit is\n+\t\t\tfound.\n \n \t\t- 1-byte XOR-offset\n \t\t\tThe xor offset used to compress this bitmap. For an entry\n@@ -146,10 +188,11 @@ Name-hash cache\n ---------------\n \n If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains\n-a cache of 32-bit values, one per object in the pack. The value at\n+a cache of 32-bit values, one per object in the pack/MIDX. The value at\n position `i` is the hash of the pathname at which the `i`th object\n-(counting in index order) in the pack can be found. This can be fed\n-into the delta heuristics to compare objects with similar pathnames.\n+(counting in index or multi-pack index order) in the pack/MIDX can be found.\n+This can be fed into the delta heuristics to compare objects with similar\n+pathnames.\n \n The hash algorithm used is:\n \ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex fb688976c4..1a73c3ee20 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -71,14 +71,10 @@ Future Work\n still reducing the number of binary searches required for object\n lookups.\n \n-- The reachability bitmap is currently paired directly with a single\n- packfile, using the pack-order as the object order to hopefully\n- compress the bitmaps well using run-length encoding. This could be\n- extended to pair a reachability bitmap with a multi-pack-index. If\n- the multi-pack-index is extended to store a \"stable object order\"\n+- If the multi-pack-index is extended to store a \"stable object order\"\n (a function Order(hash) = integer that is constant for a given hash,\n- even as the multi-pack-index is updated) then a reachability bitmap\n- could point to a multi-pack-index and be updated independently.\n+ even as the multi-pack-index is updated) then MIDX bitmaps could be\n+ updated independently of the MIDX.\n \n - Packfiles can be marked as \"special\" using empty files that share\n the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433573","messageId":"771741844be3570395abfda813ed5ef2fa78332e.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:02Z","receivedAt":"2021-08-24T16:16:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nclean up any auxiliary files (currently just the MIDX's `.rev` file, but\nsoon to include a `.bitmap`, too) corresponding to the MIDX it's\nreplacing.\n\nThis step should happen after the new MIDX is written into place, since\ndoing so beforehand means that the old MIDX could be read without its\ncorresponding .rev file.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 321c6fdd2f..73b199ca49 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n-\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433574","messageId":"dab5dbf228a5e6d698d28e643e99f6b5492d40b6.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 06/25] midx: reject empty `--preferred-pack`'s","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:04Z","receivedAt":"2021-08-24T16:16:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The soon-to-be-implemented multi-pack bitmap treats object in the first\nbit position specially by assuming that all objects in the pack it was\nselected from are also represented from that pack in the MIDX. In other\nwords, the pack from which the first object was selected must also have\nall of its other objects selected from that same pack in the MIDX in\ncase of any duplicates.\n\nBut this assumption relies on the fact that there is at least one object\nin that pack to begin with; otherwise the object in the first bit\nposition isn't from a preferred pack, in which case we can no longer\nassume that all objects in that pack were also selected from the same\npack.\n\nGuard this assumption by checking the number of objects in the given\npreferred pack, and failing if the given pack is empty.\n\nTo make sure we can safely perform this check, open any packs which are\ncontained in an existing MIDX via prepare_midx_pack(). The same is done\nfor new packs via the add_pack_to_midx() callback, but packs picked up\nfrom a previous MIDX will not yet have these opened.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 6 +++---\n midx.c | 29 ++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 17 +++++++++++++++\n 3 files changed, 49 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex ffd601bc17..c9b063d31e 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -37,9 +37,9 @@ write::\n --\n \t--preferred-pack=::\n \t\tOptionally specify the tie-breaking pack used when\n-\t\tmultiple packs contain the same object. If not given,\n-\t\tties are broken in favor of the pack with the lowest\n-\t\tmtime.\n+\t\tmultiple packs contain the same object. `` must\n+\t\tcontain at least one object. If not given, ties are\n+\t\tbroken in favor of the pack with the lowest mtime.\n --\n \n verify::\ndiff --git a/midx.c b/midx.c\nindex 73b199ca49..551e5c2ee5 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -934,6 +934,25 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n \t\t\tctx.info[ctx.nr].p = NULL;\n \t\t\tctx.info[ctx.nr].expired = 0;\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n+\t\t\t\t/*\n+\t\t\t\t * If generating a reverse index, need to have\n+\t\t\t\t * packed_git's loaded to compare their\n+\t\t\t\t * mtimes and object count.\n+\t\t\t\t */\n+\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\tresult = 1;\n+\t\t\t\t\tgoto cleanup;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t ctx.m->packs[i]->pack_name);\n+\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n+\t\t\t}\n+\n \t\t\tctx.nr++;\n \t\t}\n \t}\n@@ -961,6 +980,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t}\n \t}\n \n+\tif (ctx.preferred_pack_idx > -1) {\n+\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tif (!preferred->num_objects) {\n+\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n+\t\t\t preferred->pack_name);\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n \t\t\t\t\t ctx.preferred_pack_idx);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 3d4d9f10c3..9b184bd45e 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -277,6 +277,23 @@ test_expect_success 'midx picks objects from preferred pack' '\n \t)\n '\n \n+test_expect_success 'preferred packs must be non-empty' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ttest_commit base &&\n+\t\tgit repack -ad &&\n+\n+\t\tempty=\"$(git pack-objects $objdir/pack/pack err &&\n+\t\tgrep \"with no objects\" err\n+\t)\n+'\n+\n test_expect_success 'verify multi-pack-index success' '\n \tgit multi-pack-index verify --object-dir=$objdir\n '\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433575","messageId":"31f4517de0bc188a7b0d53e6092dde82c4b2855a.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 07/25] midx: infer preferred pack when not given one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:07Z","receivedAt":"2021-08-24T16:16:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the\nmulti-pack index code learned how to select a pack which all duplicate\nobjects are selected from. That is, if an object appears in multiple\npacks, select the copy in the preferred pack before breaking ties\naccording to the other rules like pack mtime and readdir() order.\n\nNot specifying a preferred pack can cause serious problems with\nmulti-pack reachability bitmaps, because these bitmaps rely on having at\nleast one pack from which all duplicates are selected. Not having such a\npack causes problems with the code in pack-objects to reuse packs\nverbatim (e.g., that code assumes that a delta object in a chunk of pack\nsent verbatim will have its base object sent from the same pack).\n\nSo why does not marking a pack preferred cause problems here? The reason\nis roughly as follows:\n\n - Ties are broken (when handling duplicate objects) by sorting\n according to midx_oid_compare(), which sorts objects by OID,\n preferred-ness, pack mtime, and finally pack ID (more on that\n later).\n\n - The psuedo pack-order (described in\n Documentation/technical/pack-format.txt under the section\n \"multi-pack-index reverse indexes\") is computed by\n midx_pack_order(), and sorts by pack ID and pack offset, with\n preferred packs sorting first.\n\n - But! Pack IDs come from incrementing the pack count in\n add_pack_to_midx(), which is a callback to\n for_each_file_in_pack_dir(), meaning that pack IDs are assigned in\n readdir() order.\n\nWhen specifying a preferred pack, all of that works fine, because\nduplicate objects are correctly resolved in favor of the copy in the\npreferred pack, and the preferred pack sorts first in the object order.\n\n\"Sorting first\" is critical, because the bitmap code relies on finding\nout which pack holds the first object in the MIDX's pseudo pack-order to\ndetermine which pack is preferred.\n\nBut if we didn't specify a preferred pack, and the pack which comes\nfirst in readdir() order does not also have the lowest timestamp, then\nit's possible that that pack (the one that sorts first in pseudo-pack\norder, which the bitmap code will treat as the preferred one) did *not*\nhave all duplicate objects resolved in its favor, resulting in breakage.\n\nThe fix is simple: pick a (semi-arbitrary, non-empty) preferred pack\nwhen none was specified. This forces that pack to have duplicates\nresolved in its favor, and (critically) to sort first in pseudo-pack\norder. Unfortunately, testing this behavior portably isn't possible,\nsince it depends on readdir() order which isn't guaranteed by POSIX.\n\n(Note that multi-pack reachability bitmaps have yet to be implemented;\nso in that sense this patch is fixing a bug which does not yet exist.\nBut by having this patch beforehand, we can prevent the bug from ever\nmaterializing.)\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 44 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 551e5c2ee5..e5b17483af 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -969,15 +969,57 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n+\t\tint found = 0;\n \t\tfor (i = 0; i < ctx.nr; i++) {\n \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tfound = 1;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n+\n+\t\tif (!found)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tctx.preferred_pack_idx = 0;\n+\n+\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n+\n+\t\t/*\n+\t\t * set a preferred pack when writing a bitmap to ensure that\n+\t\t * the pack from which the first object is selected in pseudo\n+\t\t * pack-order has all of its objects selected from that pack\n+\t\t * (and not another pack containing a duplicate)\n+\t\t */\n+\t\tfor (i = 1; i < ctx.nr; i++) {\n+\t\t\tstruct packed_git *p = ctx.info[i].p;\n+\n+\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n+\t\t\t\toldest = p;\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!oldest->num_objects) {\n+\t\t\t/*\n+\t\t\t * If all packs are empty; unset the preferred index.\n+\t\t\t * This is acceptable since there will be no duplicate\n+\t\t\t * objects to resolve, so the preferred value doesn't\n+\t\t\t * matter.\n+\t\t\t */\n+\t\t\tctx.preferred_pack_idx = -1;\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * otherwise don't mark any pack as preferred to avoid\n+\t\t * interfering with expiration logic below\n+\t\t */\n+\t\tctx.preferred_pack_idx = -1;\n \t}\n \n \tif (ctx.preferred_pack_idx > -1) {\n@@ -1058,11 +1100,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n-\n-\t\tif (!preferred)\n-\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n-\t\telse {\n+\t\tif (preferred) {\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433576","messageId":"aa3bd96d9bba8bbb3f31d91a482234519b668b5d.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 08/25] midx: close linked MIDXs, avoid leaking memory","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:09Z","receivedAt":"2021-08-24T16:16:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a repository has at least one alternate, the MIDX belonging to each\nalternate is accessed through the `next` pointer on the main object\nstore's copy of the MIDX. close_midx() didn't bother to close any\nof the linked MIDXs. It likewise didn't free the memory pointed to by\n`m`, leaving uninitialized bytes with live pointers to them left around\nin the heap.\n\nClean this up by closing linked MIDXs, and freeing up the memory pointed\nto by each of them. When callers call close_midx(), then they can\ndiscard the entire linked list of MIDXs and set their pointer to the\nhead of that list to NULL.\n\nThis isn't strictly required for the upcoming patches, but it makes it\nmuch more difficult (though still possible, for e.g., by calling\n`close_midx(m->next)` which leaves `m->next` pointing at uninitialized\nbytes) to have pointers to uninitialized memory.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex e5b17483af..0a515d8711 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -195,6 +195,8 @@ void close_midx(struct multi_pack_index *m)\n \tif (!m)\n \t\treturn;\n \n+\tclose_midx(m->next);\n+\n \tmunmap((unsigned char *)m->data, m->data_len);\n \n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -203,6 +205,7 @@ void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n \tFREE_AND_NULL(m->pack_names);\n+\tfree(m);\n }\n \n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433578","messageId":"c9fea31fa875188de5aeb66ad0dd5b64f4c941ca.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 09/25] midx: avoid opening multiple MIDXs when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:12Z","receivedAt":"2021-08-24T16:16:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Opening multiple instance of the same MIDX can lead to problems like two\nseparate packed_git structures which represent the same pack being added\nto the repository's object store.\n\nThe above scenario can happen because prepare_midx_pack() checks if\n`m->packs[pack_int_id]` is NULL in order to determine if a pack has been\nopened and installed in the repository before. But a caller can\nconstruct two copies of the same MIDX by calling get_multi_pack_index()\nand load_multi_pack_index() since the former manipulates the\nobject store directly but the latter is a lower-level routine which\nallocates a new MIDX for each call.\n\nSo if prepare_midx_pack() is called on multiple MIDXs with the same\npack_int_id, then that pack will be installed twice in the object\nstore's packed_git pointer.\n\nThis can lead to problems in, for e.g., the pack-bitmap code, which does\nsomething like the following (in pack-bitmap.c:open_pack_bitmap()):\n\n struct bitmap_index *bitmap_git = ...;\n for (p = get_all_packs(r); p; p = p->next) {\n if (open_pack_bitmap_1(bitmap_git, p) == 0)\n ret = 0;\n }\n\nwhich is a problem if two copies of the same pack exist in the\npacked_git list because pack-bitmap.c:open_pack_bitmap_1() contains a\nconditional like the following:\n\n if (bitmap_git->pack || bitmap_git->midx) {\n /* ignore extra bitmap file; we can only handle one */\n warning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n close(fd);\n return -1;\n }\n\nAvoid this scenario by not letting write_midx_internal() open a MIDX\nthat isn't also pointed at by the object store. So long as this is the\ncase, other routines should prefer to open MIDXs with\nget_multi_pack_index() or reprepare_packed_git() instead of creating\ninstances on their own. Because get_multi_pack_index() returns\n`r->object_store->multi_pack_index` if it is non-NULL, we'll only have\none instance of a MIDX open at one time, avoiding these problems.\n\nTo encourage this, drop the `struct multi_pack_index *` parameter from\n`write_midx_internal()`, and rely instead on the `object_dir` to find\n(or initialize) the correct MIDX instance.\n\nLikewise, replace the call to `close_midx()` with\n`close_object_store()`, since we're about to replace the MIDX with a new\none and should invalidate the object store's memory of any MIDX that\nmight have existed beforehand.\n\nNote that this now forbids passing object directories that don't belong\nto alternate repositories over `--object-dir`, since before we would\nhave happily opened a MIDX in any directory, but now restrict ourselves\nto only those reachable by `r->objects->multi_pack_index` (and alternate\nMIDXs that we can see by walking the `next` pointer).\n\nAs far as I can tell, supporting arbitrary directories with\n`--object-dir` was a historical accident, since even the documentation\nsays `` when referring to the value passed to this option.\n\nA future patch could clean this up and provide a warning() when a\nnon-alternate directory was given, since we'll still write a new MIDX\nthere, we just won't reuse any MIDX that might happen to already exist\nin that directory.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 26 +++++++++++++++-----------\n 1 file changed, 15 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 0a515d8711..3dacb31f9d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n \t\t\t unsigned flags)\n@@ -904,6 +904,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n+\tstruct multi_pack_index *cur;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -914,10 +915,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name);\n \n-\tif (m)\n-\t\tctx.m = m;\n-\telse\n-\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\tctx.m = cur;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n \n \tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n \t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n@@ -1119,7 +1122,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n \tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n+\t\tclose_object_store(the_repository->objects);\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -1182,8 +1185,7 @@ int write_midx_file(const char *object_dir,\n \t\t const char *preferred_pack_name,\n \t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t flags);\n+\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n }\n \n struct clear_midx_data {\n@@ -1461,8 +1463,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \n \tfree(count);\n \n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n+\tif (packs_to_drop.nr) {\n+\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tm = NULL;\n+\t}\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1651,7 +1655,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433577","messageId":"ee72fb7e38f19391b50f2cb0d2f4988c5f40403e.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 10/25] pack-bitmap.c: introduce 'bitmap_num_objects()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:15Z","receivedAt":"2021-08-24T16:16:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to return how many objects are\ncontained in a bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 37 +++++++++++++++++++++----------------\n 1 file changed, 21 insertions(+), 16 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9b11af87aa..65356f9657 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn b;\n }\n \n+static uint32_t bitmap_num_objects(struct bitmap_index *index)\n+{\n+\treturn index->pack->num_objects;\n+}\n+\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n-\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tsize_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));\n \t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n@@ -404,7 +409,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -456,7 +461,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -673,7 +678,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\tif (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -832,7 +837,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == type &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos))\n@@ -859,7 +864,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < pack->num_objects) {\n+\tif (pos < bitmap_num_objects(bitmap_git)) {\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n@@ -869,7 +874,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - pack->num_objects];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -911,7 +916,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t}\n \n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == OBJ_BLOB &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos) &&\n@@ -1137,8 +1142,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_git->pack->num_objects)\n-\t\treturn; /* not actually in the pack */\n+\tif (pos >= bitmap_num_objects(bitmap_git))\n+\t\treturn; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n@@ -1204,6 +1209,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n+\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n \n \tassert(result);\n \n@@ -1211,8 +1217,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\ti++;\n \n \t/* Don't mark objects not in the packfile */\n-\tif (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)\n-\t\ti = bitmap_git->pack->num_objects / BITS_IN_EWORD;\n+\tif (i > objects_nr / BITS_IN_EWORD)\n+\t\ti = objects_nr / BITS_IN_EWORD;\n \n \treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n@@ -1296,7 +1302,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n-\t\t\tbitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\t\tbitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcount++;\n \t}\n \n@@ -1517,7 +1523,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n-\tnum_objects = bitmap_git->pack->num_objects;\n+\tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -1600,7 +1606,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \toff_t total = 0;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n@@ -1612,7 +1617,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \tfor (i = 0; i < eindex->count; i++) {\n \t\tstruct object *obj = eindex->objects[i];\n \n-\t\tif (!bitmap_get(result, pack->num_objects + i))\n+\t\tif (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433579","messageId":"ede0bf1ce1f1f42783d59928dd033d17c2ad6314.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 11/25] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:17Z","receivedAt":"2021-08-24T16:16:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to fetch the nth OID contained in\nthe bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 65356f9657..612f62da97 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)\n \n #define MAX_XOR_OFFSET 160\n \n+static int nth_bitmap_object_oid(struct bitmap_index *index,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t uint32_t n)\n+{\n+\treturn nth_packed_object_id(oid, index->pack, n);\n+}\n+\n static int load_bitmap_entries_v1(struct bitmap_index *index)\n {\n \tuint32_t i;\n@@ -242,7 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\tif (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)\n \t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n \t\t\t\t (unsigned)commit_idx_pos);\n \n@@ -868,8 +875,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack,\n-\t\t\t\t\t pack_pos_to_index(pack, pos));\n+\t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n+\t\t\t\t\t pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433580","messageId":"df6844def0afb90b992a203ded5523c88ddf002e.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 12/25] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:20Z","receivedAt":"2021-08-24T16:16:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a recent commit, pack-objects learned support for the\n'pack.preferBitmapTips' configuration. This patch prepares the\nmulti-pack bitmap code to respect this configuration, too.\n\nThe yet-to-be implemented code will find that it is more efficient to\ncheck whether each reference contains a prefix found in the configured\nset of values rather than doing an additional traversal.\n\nImplement a function 'bitmap_is_preferred_refname()' which will perform\nthat check. Its caller will be added in a subsequent patch.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 16 ++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 612f62da97..d5296750eb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1658,3 +1658,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\n }\n+\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname)\n+{\n+\tconst struct string_list *preferred_tips = bitmap_preferred_tips(r);\n+\tstruct string_list_item *item;\n+\n+\tif (!preferred_tips)\n+\t\treturn 0;\n+\n+\tfor_each_string_list_item(item, preferred_tips) {\n+\t\tif (starts_with(refname, item->string))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 020cd8d868..52ea10de51 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint16_t options);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n #endif\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433581","messageId":"4e06f051a7d56a906d1db97513c88d4b3029742e.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 13/25] pack-bitmap.c: avoid redundant calls to try_partial_reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:22Z","receivedAt":"2021-08-24T16:16:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"try_partial_reuse() is used to mark any bits in the beginning of a\nbitmap whose objects can be reused verbatim from the pack they came\nfrom.\n\nCurrently this function returns void, and signals nothing to the caller\nwhen bits could not be reused. But multi-pack bitmaps would benefit from\nhaving such a signal, because they may try to pass objects which are in\nbounds, but from a pack other than the preferred one.\n\nAny extra calls are noops because of a conditional in\nreuse_partial_packfile_from_bitmap(), but those loop iterations can be\navoided by letting try_partial_reuse() indicate when it can't accept any\nmore bits for reuse, and then listening to that signal.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 40 +++++++++++++++++++++++++++++-----------\n 1 file changed, 29 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d5296750eb..4e37f5d574 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1140,22 +1140,26 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \treturn NULL;\n }\n \n-static void try_partial_reuse(struct bitmap_index *bitmap_git,\n-\t\t\t size_t pos,\n-\t\t\t struct bitmap *reuse,\n-\t\t\t struct pack_window **w_curs)\n+/*\n+ * -1 means \"stop trying further objects\"; 0 means we may or may not have\n+ * reused, but you can keep feeding bits.\n+ */\n+static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t size_t pos,\n+\t\t\t struct bitmap *reuse,\n+\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, header;\n \tenum object_type type;\n \tunsigned long size;\n \n \tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn; /* not actually in the pack or MIDX */\n+\t\treturn -1; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n \tif (type < 0)\n-\t\treturn; /* broken packfile, punt */\n+\t\treturn -1; /* broken packfile, punt */\n \n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n@@ -1172,9 +1176,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n \t\t\t\t\t &offset, type, header);\n \t\tif (!base_offset)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * We assume delta dependencies always point backwards. This\n@@ -1186,7 +1190,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * odd parameters.\n \t\t */\n \t\tif (base_pos >= pos)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * And finally, if we're not sending the base as part of our\n@@ -1197,13 +1201,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * object_entry code path handle it.\n \t\t */\n \t\tif (!bitmap_get(reuse, base_pos))\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n \tbitmap_set(reuse, pos);\n+\treturn 0;\n }\n \n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n@@ -1239,10 +1244,23 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n+\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n+\t\t\t\t\t &w_curs) < 0) {\n+\t\t\t\t/*\n+\t\t\t\t * try_partial_reuse indicated we couldn't reuse\n+\t\t\t\t * any bits, so there is no point in trying more\n+\t\t\t\t * bits in the current word, or any other words\n+\t\t\t\t * in result.\n+\t\t\t\t *\n+\t\t\t\t * Jump out of both loops to avoid future\n+\t\t\t\t * unnecessary calls to try_partial_reuse.\n+\t\t\t\t */\n+\t\t\t\tgoto done;\n+\t\t\t}\n \t\t}\n \t}\n \n+done:\n \tunuse_pack(&w_curs);\n \n \t*entries = bitmap_popcount(reuse);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433582","messageId":"a0d73eb3d3720b66d63077390f2eb0afb71b193d.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 14/25] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:25Z","receivedAt":"2021-08-24T16:16:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This prepares the code in pack-bitmap to interpret the new multi-pack\nbitmaps described in Documentation/technical/bitmap-format.txt, which\nmostly involves converting bit positions to accommodate looking them up\nin a MIDX.\n\nNote that there are currently no writers who write multi-pack bitmaps,\nand that this will be implemented in the subsequent commit. Note also\nthat get_midx_checksum() and get_midx_filename() are made non-static so\nthey can be called from pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 5 +\n midx.c | 4 +-\n midx.h | 2 +\n pack-bitmap-write.c | 2 +-\n pack-bitmap.c | 357 ++++++++++++++++++++++++++++++++++++-----\n pack-bitmap.h | 6 +\n packfile.c | 2 +-\n 7 files changed, 336 insertions(+), 42 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8a523624a1..e11d3ac2e5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\t/*\n+\t\t\t * Can use bit positions directly, even for MIDX\n+\t\t\t * bitmaps. See comment in try_partial_reuse()\n+\t\t\t * for why.\n+\t\t\t */\n \t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\ndiff --git a/midx.c b/midx.c\nindex 3dacb31f9d..2dceaf9565 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -48,12 +48,12 @@ static uint8_t oid_version(void)\n \t}\n }\n \n-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - the_hash_algo->rawsz;\n }\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex 8684cf0fef..1172df1a71 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -42,6 +42,8 @@ struct multi_pack_index {\n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n \n+const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+char *get_midx_filename(const char *object_dir);\n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 142fd0adb8..9c55c1531e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)\n }\n \n /**\n- * Build the initial type index for the packfile\n+ * Build the initial type index for the packfile or multi-pack-index\n */\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4e37f5d574..fa69ed7a6d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -13,6 +13,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"midx.h\"\n #include \"config.h\"\n \n /*\n@@ -35,8 +36,15 @@ struct stored_bitmap {\n * the active bitmap index is the largest one.\n */\n struct bitmap_index {\n-\t/* Packfile to which this bitmap index belongs to */\n+\t/*\n+\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n+\t * to.\n+\t *\n+\t * Exactly one of these must be non-NULL; this specifies the object\n+\t * order used to interpret this bitmap.\n+\t */\n \tstruct packed_git *pack;\n+\tstruct multi_pack_index *midx;\n \n \t/*\n \t * Mark the first `reuse_objects` in the packfile as reused:\n@@ -71,6 +79,9 @@ struct bitmap_index {\n \t/* If not NULL, this is a name-hash cache pointing into map. */\n \tuint32_t *hashes;\n \n+\t/* The checksum of the packfile or MIDX; points into map. */\n+\tconst unsigned char *checksum;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -138,6 +149,8 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n+\tif (index->midx)\n+\t\treturn index->midx->num_objects;\n \treturn index->pack->num_objects;\n }\n \n@@ -175,6 +188,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n+\tindex->checksum = header->checksum;\n \tindex->map_pos += header_size;\n \treturn 0;\n }\n@@ -227,6 +241,8 @@ static int nth_bitmap_object_oid(struct bitmap_index *index,\n \t\t\t\t struct object_id *oid,\n \t\t\t\t uint32_t n)\n {\n+\tif (index->midx)\n+\t\treturn nth_midxed_object_oid(oid, index->midx, n) ? 0 : -1;\n \treturn nth_packed_object_id(oid, index->pack, n);\n }\n \n@@ -274,7 +290,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \treturn 0;\n }\n \n-static char *pack_bitmap_filename(struct packed_git *p)\n+char *midx_bitmap_filename(struct multi_pack_index *midx)\n+{\n+\treturn xstrfmt(\"%s-%s.bitmap\",\n+\t\t get_midx_filename(midx->object_dir),\n+\t\t hash_to_hex(get_midx_checksum(midx)));\n+}\n+\n+char *pack_bitmap_filename(struct packed_git *p)\n {\n \tsize_t len;\n \n@@ -283,6 +306,57 @@ static char *pack_bitmap_filename(struct packed_git *p)\n \treturn xstrfmt(\"%.*s.bitmap\", (int)len, p->pack_name);\n }\n \n+static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct stat st;\n+\tchar *idx_name = midx_bitmap_filename(midx);\n+\tint fd = git_open(idx_name);\n+\n+\tfree(idx_name);\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n+\t\twarning(\"ignoring extra bitmap file: %s\",\n+\t\t\tget_midx_filename(midx->object_dir));\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tbitmap_git->midx = midx;\n+\tbitmap_git->map_size = xsize_t(st.st_size);\n+\tbitmap_git->map_pos = 0;\n+\tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ,\n+\t\t\t\tMAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\tif (load_bitmap_header(bitmap_git) < 0)\n+\t\tgoto cleanup;\n+\n+\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum))\n+\t\tgoto cleanup;\n+\n+\tif (load_midx_revindex(bitmap_git->midx) < 0) {\n+\t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n+\t\tgoto cleanup;\n+\t}\n+\treturn 0;\n+\n+cleanup:\n+\tmunmap(bitmap_git->map, bitmap_git->map_size);\n+\tbitmap_git->map_size = 0;\n+\tbitmap_git->map = NULL;\n+\treturn -1;\n+}\n+\n static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git *packfile)\n {\n \tint fd;\n@@ -304,7 +378,8 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\treturn -1;\n \t}\n \n-\tif (bitmap_git->pack) {\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n \t\tclose(fd);\n \t\treturn -1;\n@@ -331,13 +406,39 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \treturn 0;\n }\n \n-static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n+static int load_reverse_index(struct bitmap_index *bitmap_git)\n+{\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tuint32_t i;\n+\t\tint ret;\n+\n+\t\t/*\n+\t\t * The multi-pack-index's .rev file is already loaded via\n+\t\t * open_pack_bitmap_1().\n+\t\t *\n+\t\t * But we still need to open the individual pack .rev files,\n+\t\t * since we will need to make use of them in pack-objects.\n+\t\t */\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n+\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn load_pack_revindex(bitmap_git->pack);\n+}\n+\n+static int load_bitmap(struct bitmap_index *bitmap_git)\n {\n \tassert(bitmap_git->map);\n \n \tbitmap_git->bitmaps = kh_init_oid_map();\n \tbitmap_git->ext_index.positions = kh_init_oid_pos();\n-\tif (load_pack_revindex(bitmap_git->pack))\n+\n+\tif (load_reverse_index(bitmap_git))\n \t\tgoto failed;\n \n \tif (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) ||\n@@ -381,11 +482,47 @@ static int open_pack_bitmap(struct repository *r,\n \treturn ret;\n }\n \n+static int open_midx_bitmap(struct repository *r,\n+\t\t\t struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *midx;\n+\n+\tassert(!bitmap_git->map);\n+\n+\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n+\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\t\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n+static int open_bitmap(struct repository *r,\n+\t\t struct bitmap_index *bitmap_git)\n+{\n+\tassert(!bitmap_git->map);\n+\n+\tif (!open_midx_bitmap(r, bitmap_git))\n+\t\treturn 0;\n+\treturn open_pack_bitmap(r, bitmap_git);\n+}\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git))\n+\t\treturn bitmap_git;\n+\n+\tfree_bitmap_index(bitmap_git);\n+\treturn NULL;\n+}\n+\n+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n+\n+\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(bitmap_git))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -435,10 +572,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \treturn pos;\n }\n \n+static int bitmap_position_midx(struct bitmap_index *bitmap_git,\n+\t\t\t\tconst struct object_id *oid)\n+{\n+\tuint32_t want, got;\n+\tif (!bsearch_midx(oid, bitmap_git->midx, &want))\n+\t\treturn -1;\n+\n+\tif (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0)\n+\t\treturn -1;\n+\treturn got;\n+}\n+\n static int bitmap_position(struct bitmap_index *bitmap_git,\n \t\t\t const struct object_id *oid)\n {\n-\tint pos = bitmap_position_packfile(bitmap_git, oid);\n+\tint pos;\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpos = bitmap_position_midx(bitmap_git, oid);\n+\telse\n+\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n }\n \n@@ -749,6 +902,7 @@ static void show_objects_for_type(\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n+\t\t\tstruct packed_git *pack;\n \t\t\tstruct object_id oid;\n \t\t\tuint32_t hash = 0, index_pos;\n \t\t\toff_t ofs;\n@@ -758,14 +912,28 @@ static void show_objects_for_type(\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n-\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\t\t\tuint32_t pack_id;\n+\n+\t\t\t\tindex_pos = pack_pos_to_midx(m, pos + offset);\n+\t\t\t\tofs = nth_midxed_offset(m, index_pos);\n+\t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n+\n+\t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t} else {\n+\t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\t\tnth_bitmap_object_oid(bitmap_git, &oid, index_pos);\n+\n+\t\t\t\tpack = bitmap_git->pack;\n+\t\t\t}\n \n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n }\n@@ -777,8 +945,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n-\t\t\treturn 1;\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n+\t\t\t\treturn 1;\n+\t\t} else {\n+\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -865,14 +1038,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git,\n static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\t\t uint32_t pos)\n {\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tunsigned long size;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = &size;\n \n \tif (pos < bitmap_num_objects(bitmap_git)) {\n-\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tstruct packed_git *pack;\n+\t\toff_t ofs;\n+\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\n+\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\t\t} else {\n+\t\t\tpack = bitmap_git->pack;\n+\t\t\tofs = pack_pos_to_offset(pack, pos);\n+\t\t}\n+\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n \t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n@@ -1053,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t/* try to open a bitmapped pack, but don't parse it yet\n \t * because we may not need to use it */\n \tCALLOC_ARRAY(bitmap_git, 1);\n-\tif (open_pack_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (open_bitmap(revs->repo, bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tfor (i = 0; i < revs->pending.nr; ++i) {\n@@ -1097,7 +1282,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_pack_bitmap(bitmap_git) < 0)\n+\tif (load_bitmap(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tobject_array_clear(&revs->pending);\n@@ -1145,19 +1330,43 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n * reused, but you can keep feeding bits.\n */\n static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t struct packed_git *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n-\toff_t offset, header;\n+\toff_t offset, delta_obj_offset;\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn -1; /* not actually in the pack or MIDX */\n+\t/*\n+\t * try_partial_reuse() is called either on (a) objects in the\n+\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n+\t * objects in the preferred pack of a multi-pack bitmap.\n+\t * Importantly, the latter can pretend as if only a single pack\n+\t * exists because:\n+\t *\n+\t * - The first pack->num_objects bits of a MIDX bitmap are\n+\t * reserved for the preferred pack, and\n+\t *\n+\t * - Ties due to duplicate objects are always resolved in\n+\t * favor of the preferred pack.\n+\t *\n+\t * Therefore we do not need to ever ask the MIDX for its copy of\n+\t * an object by OID, since it will always select it from the\n+\t * preferred pack. Likewise, the selected copy of the base\n+\t * object for any deltas will reside in the same pack.\n+\t *\n+\t * This means that we can reuse pos when looking up the bit in\n+\t * the reuse bitmap, too, since bits corresponding to the\n+\t * preferred pack precede all bits from other packs.\n+\t */\n \n-\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n-\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n+\tif (pos >= pack->num_objects)\n+\t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n+\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n+\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn -1; /* broken packfile, punt */\n \n@@ -1173,11 +1382,11 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t &offset, type, header);\n+\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn 0;\n-\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n \t\t\treturn 0;\n \n \t\t/*\n@@ -1211,24 +1420,48 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \treturn 0;\n }\n \n+static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *m = bitmap_git->midx;\n+\tif (!m)\n+\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n+\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n+}\n+\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t struct packed_git **packfile_out,\n \t\t\t\t uint32_t *entries,\n \t\t\t\t struct bitmap **reuse_out)\n {\n+\tstruct packed_git *pack;\n \tstruct bitmap *result = bitmap_git->result;\n \tstruct bitmap *reuse;\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n-\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n+\tuint32_t objects_nr;\n \n \tassert(result);\n \n+\tload_reverse_index(bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\telse\n+\t\tpack = bitmap_git->pack;\n+\tobjects_nr = pack->num_objects;\n+\n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n \n-\t/* Don't mark objects not in the packfile */\n+\t/*\n+\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n+\t * marks objects eligible for reuse, but the pack-reuse code only\n+\t * understands how to reuse a single pack. Since the preferred pack is\n+\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n+\t * we use it instead of another pack. In single-pack bitmaps, the choice\n+\t * is made for us.\n+\t */\n \tif (i > objects_nr / BITS_IN_EWORD)\n \t\ti = objects_nr / BITS_IN_EWORD;\n \n@@ -1244,8 +1477,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n-\t\t\t\t\t &w_curs) < 0) {\n+\t\t\tif (try_partial_reuse(bitmap_git, pack, pos + offset,\n+\t\t\t\t\t reuse, &w_curs) < 0) {\n \t\t\t\t/*\n \t\t\t\t * try_partial_reuse indicated we couldn't reuse\n \t\t\t\t * any bits, so there is no point in trying more\n@@ -1274,7 +1507,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = bitmap_git->pack;\n+\t*packfile_out = pack;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n@@ -1548,6 +1781,12 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n+\tif (!bitmap_is_midx(bitmap_git))\n+\t\tload_reverse_index(bitmap_git);\n+\telse if (load_midx_revindex(bitmap_git->midx) < 0)\n+\t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n+\t\t \"extension\");\n+\n \tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n@@ -1555,8 +1794,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tstruct object_id oid;\n \t\tstruct object_entry *oe;\n \n-\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n-\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n+\t\tif (bitmap_is_midx(bitmap_git))\n+\t\t\tnth_midxed_object_oid(&oid,\n+\t\t\t\t\t bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n+\t\telse\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n@@ -1582,6 +1826,19 @@ void free_bitmap_index(struct bitmap_index *b)\n \tfree(b->ext_index.hashes);\n \tbitmap_free(b->result);\n \tbitmap_free(b->haves);\n+\tif (bitmap_is_midx(b)) {\n+\t\t/*\n+\t\t * Multi-pack bitmaps need to have resources associated with\n+\t\t * their on-disk reverse indexes unmapped so that stale .rev and\n+\t\t * .bitmap files can be removed.\n+\t\t *\n+\t\t * Unlike pack-based bitmaps, multi-pack bitmaps can be read and\n+\t\t * written in the same 'git multi-pack-index write --bitmap'\n+\t\t * process. Close resources so they can be removed safely on\n+\t\t * platforms like Windows.\n+\t\t */\n+\t\tclose_midx_revindex(b->midx);\n+\t}\n \tfree(b);\n }\n \n@@ -1596,7 +1853,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\t enum object_type object_type)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \toff_t total = 0;\n \tstruct ewah_iterator it;\n \teword_t filter;\n@@ -1613,15 +1869,35 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n-\t\t\tsize_t pos;\n-\n \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tpos = base + offset;\n-\t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n-\t\t\t\t pack_pos_to_offset(pack, pos);\n+\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tuint32_t pack_pos;\n+\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n+\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\n+\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\n+\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n+\t\t\t\t\tstruct object_id oid;\n+\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n+\n+\t\t\t\t\tdie(_(\"could not find %s in pack %s at offset %\"PRIuMAX),\n+\t\t\t\t\t oid_to_hex(&oid),\n+\t\t\t\t\t pack->pack_name,\n+\t\t\t\t\t (uintmax_t)offset);\n+\t\t\t\t}\n+\n+\t\t\t\ttotal += pack_pos_to_offset(pack, pack_pos + 1) - offset;\n+\t\t\t} else {\n+\t\t\t\tsize_t pos = base + offset;\n+\t\t\t\ttotal += pack_pos_to_offset(bitmap_git->pack, pos + 1) -\n+\t\t\t\t\t pack_pos_to_offset(bitmap_git->pack, pos);\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1672,6 +1948,11 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n \treturn total;\n }\n \n+int bitmap_is_midx(struct bitmap_index *bitmap_git)\n+{\n+\treturn !!bitmap_git->midx;\n+}\n+\n const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 52ea10de51..81664f933f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,6 +44,8 @@ typedef int (*show_reachable_fn)(\n struct bitmap_index;\n \n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *midx);\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n \t\t\t uint32_t *trees, uint32_t *blobs, uint32_t *tags);\n void traverse_bitmap_commit_list(struct bitmap_index *,\n@@ -92,6 +94,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n \t\t\t uint16_t options);\n+char *midx_bitmap_filename(struct multi_pack_index *midx);\n+char *pack_bitmap_filename(struct packed_git *p);\n+\n+int bitmap_is_midx(struct bitmap_index *bitmap_git);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n int bitmap_is_preferred_refname(struct repository *r, const char *refname);\ndiff --git a/packfile.c b/packfile.c\nindex 9ef6d98292..371f5488cf 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n-\t ends_with(file_name, \".rev\"))\n+\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433583","messageId":"9d83ad77abff76b0c9e43603295461009cefa415.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 15/25] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:27Z","receivedAt":"2021-08-24T16:16:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Write multi-pack bitmaps in the format described by\nDocumentation/technical/bitmap-format.txt, inferring their presence with\nthe absence of '--bitmap'.\n\nTo write a multi-pack bitmap, this patch attempts to reuse as much of\nthe existing machinery from pack-objects as possible. Specifically, the\nMIDX code prepares a packing_data struct that pretends as if a single\npackfile has been generated containing all of the objects contained\nwithin the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 12 +-\n builtin/multi-pack-index.c | 2 +\n midx.c | 208 ++++++++++++++++++++++++-\n midx.h | 1 +\n 4 files changed, 214 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex c9b063d31e..ed52459a9d 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n-\t[--preferred-pack=] \n+\t[--preferred-pack=] [--[no-]bitmap] \n \n DESCRIPTION\n -----------\n@@ -40,6 +40,9 @@ write::\n \t\tmultiple packs contain the same object. `` must\n \t\tcontain at least one object. If not given, ties are\n \t\tbroken in favor of the pack with the lowest mtime.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\n@@ -81,6 +84,13 @@ EXAMPLES\n $ git multi-pack-index write\n -----------------------------------------------\n \n+* Write a MIDX file for the packfiles in the current .git folder with a\n+corresponding bitmap.\n++\n+-------------------------------------------------------------\n+$ git multi-pack-index write --preferred-pack= --bitmap\n+-------------------------------------------------------------\n+\n * Write a MIDX file for the packfiles in an alternate object store.\n +\n -----------------------------------------------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 8ff0dee2ec..73c0113b48 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n \t\t\t N_(\"preferred-pack\"),\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/midx.c b/midx.c\nindex 2dceaf9565..4574e6d411 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,10 @@\n #include \"repository.h\"\n #include \"chunk-format.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -893,6 +897,166 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t\t data->ctx->entries_nr,\n+\t\t\t bitmap_oid_access);\n+\tif (pos < 0)\n+\t\treturn;\n+\n+\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n+\tdata->commits[data->commits_nr++] = commit;\n+}\n+\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb = {0};\n+\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tsetup_revisions(0, NULL, &revs, NULL);\n+\tfor_each_ref(add_ref_to_pending, &revs);\n+\n+\t/*\n+\t * Skipping promisor objects here is intentional, since it only excludes\n+\t * them from the list of reachable commits that we want to select from\n+\t * when computing the selection of MIDX'd commits to receive bitmaps.\n+\t *\n+\t * Reachability bitmaps do require that their objects be closed under\n+\t * reachability, but fetching any objects missing from promisors at this\n+\t * point is too late. But, if one of those objects can be reached from\n+\t * an another object that is included in the bitmap, then we will\n+\t * complain later that we don't have reachability closure (and fail\n+\t * appropriately).\n+\t */\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t unsigned flags)\n+{\n+\tstruct packing_data pdata;\n+\tstruct pack_idx_entry **index;\n+\tstruct commit **commits = NULL;\n+\tuint32_t i, commits_nr;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n+\tint ret;\n+\n+\tprepare_midx_packing_data(&pdata, ctx);\n+\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata.nr_objects);\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[i] = &pdata.objects[i].idx;\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_finish expects objects in lex order, but pack_order\n+\t * gives us exactly that. use it directly instead of re-sorting the\n+\t * array.\n+\t *\n+\t * This changes the order of objects in 'index' between\n+\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n+\t *\n+\t * The same re-ordering takes place in the single-pack bitmap code via\n+\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n+\t * happens between bitmap_writer_build_type_index() and\n+\t * bitmap_writer_finish().\n+\t */\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[ctx->pack_order[i]] = &pdata.objects[i].idx;\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(&pdata);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\treturn ret;\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -938,7 +1102,7 @@ static int write_midx_internal(const char *object_dir,\n \n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\tctx.info[ctx.nr].expired = 0;\n \n \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n@@ -972,8 +1136,26 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n-\t\tgoto cleanup;\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_midx_bitmap_git(the_repository, ctx.m);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n+\t\t\t\t\t\t NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n \n \tif (preferred_pack_name) {\n \t\tint found = 0;\n@@ -989,7 +1171,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tif (!found)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n \t\t\t\tpreferred_pack_name);\n-\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t} else if (ctx.nr &&\n+\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1121,9 +1304,6 @@ static int write_midx_internal(const char *object_dir,\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n-\tif (ctx.m)\n-\t\tclose_object_store(the_repository->objects);\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n@@ -1154,14 +1334,24 @@ static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tif (flags & MIDX_WRITE_BITMAP) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\t\terror(_(\"could not write multi-pack bitmap\"));\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tclose_object_store(the_repository->objects);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n \tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n \n cleanup:\n@@ -1178,6 +1368,7 @@ static int write_midx_internal(const char *object_dir,\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n+\n \treturn result;\n }\n \n@@ -1238,6 +1429,7 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".bitmap\", NULL);\n \tclear_midx_files_ext(r, \".rev\", NULL);\n \n \tfree(midx);\ndiff --git a/midx.h b/midx.h\nindex 1172df1a71..350f4d0a7b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -41,6 +41,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n+#define MIDX_WRITE_BITMAP (1 << 2)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n char *get_midx_filename(const char *object_dir);\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433584","messageId":"a92af898846531878e291acbb5cd7ed2ee18d644.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 16/25] t5310: move some tests to lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:31Z","receivedAt":"2021-08-24T16:17:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We'll soon be adding a test script that will cover many of the same\nbitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many\nof the applicable tests as we can so we don't have to rewrite them.\n\nThere should be no functional change to t5310; we still run the same\noperations in the same order.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 236 ++++++++++++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 227 +-------------------------------------\n 2 files changed, 240 insertions(+), 223 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex fe3f98be24..77464da6fd 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,3 +1,6 @@\n+# Helpers for scripts testing bitmap functionality; see t5310 for\n+# example usage.\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -24,3 +27,236 @@ test_bitmap_traversal () {\n \ttest_cmp \"$1.normalized\" \"$2.normalized\" &&\n \trm -f \"$1.normalized\" \"$2.normalized\"\n }\n+\n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other second\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-second * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for second (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n+#\n+# second: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n+setup_bitmap_history() {\n+\ttest_expect_success 'setup repo with moderate-sized history' '\n+\t\ttest_commit_bulk --id=file 10 &&\n+\t\tgit branch -M second &&\n+\t\tgit checkout -b other HEAD~5 &&\n+\t\ttest_commit_bulk --id=side 10 &&\n+\n+\t\t# add complicated history setup, including merges and\n+\t\t# ambiguous merge-bases\n+\n+\t\tgit checkout -b merge-left other~2 &&\n+\t\tgit merge second~2 -m \"merge-left\" &&\n+\n+\t\tgit checkout -b merge-right second~1 &&\n+\t\tgit merge other~1 -m \"merge-right\" &&\n+\n+\t\tgit checkout -b octo-second second &&\n+\t\tgit merge merge-left merge-right -m \"octopus-second\" &&\n+\n+\t\tgit checkout -b octo-other other &&\n+\t\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\t\tgit checkout other &&\n+\t\tgit merge octo-other -m \"pull octopus\" &&\n+\n+\t\tgit checkout second &&\n+\t\tgit merge octo-second -m \"pull octopus\" &&\n+\n+\t\t# Remove these branches so they are not selected\n+\t\t# as bitmap tips\n+\t\tgit branch -D merge-left &&\n+\t\tgit branch -D merge-right &&\n+\t\tgit branch -D octo-other &&\n+\t\tgit branch -D octo-second &&\n+\n+\t\t# add padding to make these merges less interesting\n+\t\t# and avoid having them selected for bitmaps\n+\t\ttest_commit_bulk --id=file 100 &&\n+\t\tgit checkout other &&\n+\t\ttest_commit_bulk --id=side 100 &&\n+\t\tgit checkout second &&\n+\n+\t\tbitmaptip=$(git rev-parse second) &&\n+\t\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n+\t\tgit tag tagged-blob $blob\n+\t'\n+}\n+\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n+\t\tgit rev-list --count other...second >expect &&\n+\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n+\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n+\t\ttest_bitmap_traversal expect actual\n+\t'\n+\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n+\t\tgrep $blob actual\n+\t'\n+}\n+\n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"second\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n+basic_bitmap_tests () {\n+\ttip=\"$1\"\n+\ttest_expect_success 'rev-list --test-bitmap verifies bitmaps' \"\n+\t\tgit rev-list --test-bitmap \"${tip:-HEAD}\"\n+\t\"\n+\n+\trev_list_tests 'full bitmap'\n+\n+\ttest_expect_success 'clone from bitmapped repository' '\n+\t\trm -fr clone.git &&\n+\t\tgit clone --no-local --bare . clone.git &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'partial clone from bitmapped repository' '\n+\t\ttest_config uploadpack.allowfilter true &&\n+\t\trm -fr partial-clone.git &&\n+\t\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n+\t\t(\n+\t\t\tcd partial-clone.git &&\n+\t\t\tpack=$(echo objects/pack/*.pack) &&\n+\t\t\tgit verify-pack -v \"$pack\" >have &&\n+\t\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n+\t\t\t# we expect this single blob because of the direct ref\n+\t\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n+\t\t\ttest_cmp expect blobs\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'setup further non-bitmapped commits' '\n+\t\ttest_commit_bulk --id=further 10\n+\t'\n+\n+\trev_list_tests 'partial bitmap'\n+\n+\ttest_expect_success 'fetch (partial bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'enumerating progress counts pack-reused objects' '\n+\t\tcount=$(git rev-list --objects --all --count) &&\n+\t\tgit repack -adb &&\n+\n+\t\t# check first with only reused objects; confirm that our\n+\t\t# progress showed the right number, and also that we did\n+\t\t# pack-reuse as expected. Check only the final \"done\"\n+\t\t# line of the meter (there may be an arbitrary number of\n+\t\t# intermediate lines ending with CR).\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $count, done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr &&\n+\n+\t\t# now the same but with one non-reused object\n+\t\tgit commit --allow-empty -m \"an extra commit object\" &&\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr\n+\t'\n+}\n+\n+# have_delta \n+#\n+# Note that because this relies on cat-file, it might find _any_ copy of an\n+# object in the repository. The caller is responsible for making sure\n+# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n+have_delta () {\n+\techo $2 >expect &&\n+\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n+\ttest_cmp expect actual\n+}\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex b02838750e..4318f84d53 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -25,93 +25,10 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-# To ensure the logic for \"maximal commits\" is exercised, make\n-# the repository a bit more complicated.\n-#\n-# other second\n-# * *\n-# (99 commits) (99 commits)\n-# * *\n-# |\\ /|\n-# | * octo-other octo-second * |\n-# |/|\\_________ ____________/|\\|\n-# | \\ \\/ __________/ |\n-# | | ________/\\ / |\n-# * |/ * merge-right *\n-# | _|__________/ \\____________ |\n-# |/ | \\|\n-# (l1) * * merge-left * (r1)\n-# | / \\________________________ |\n-# |/ \\|\n-# (l2) * * (r2)\n-# \\___________________________ |\n-# \\|\n-# * (base)\n-#\n-# We only push bits down the first-parent history, which\n-# makes some of these commits unimportant!\n-#\n-# The important part for the maximal commit algorithm is how\n-# the bitmasks are extended. Assuming starting bit positions\n-# for second (bit 0) and other (bit 1), the bitmasks at the\n-# end should be:\n-#\n-# second: 1 (maximal, selected)\n-# other: 01 (maximal, selected)\n-# (base): 11 (maximal)\n-#\n-# This complicated history was important for a previous\n-# version of the walk that guarantees never walking a\n-# commit multiple times. That goal might be important\n-# again, so preserve this complicated case. For now, this\n-# test will guarantee that the bitmaps are computed\n-# correctly, even with the repeat calculations.\n+setup_bitmap_history\n \n-test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 10 &&\n-\tgit branch -M second &&\n-\tgit checkout -b other HEAD~5 &&\n-\ttest_commit_bulk --id=side 10 &&\n-\n-\t# add complicated history setup, including merges and\n-\t# ambiguous merge-bases\n-\n-\tgit checkout -b merge-left other~2 &&\n-\tgit merge second~2 -m \"merge-left\" &&\n-\n-\tgit checkout -b merge-right second~1 &&\n-\tgit merge other~1 -m \"merge-right\" &&\n-\n-\tgit checkout -b octo-second second &&\n-\tgit merge merge-left merge-right -m \"octopus-second\" &&\n-\n-\tgit checkout -b octo-other other &&\n-\tgit merge merge-left merge-right -m \"octopus-other\" &&\n-\n-\tgit checkout other &&\n-\tgit merge octo-other -m \"pull octopus\" &&\n-\n-\tgit checkout second &&\n-\tgit merge octo-second -m \"pull octopus\" &&\n-\n-\t# Remove these branches so they are not selected\n-\t# as bitmap tips\n-\tgit branch -D merge-left &&\n-\tgit branch -D merge-right &&\n-\tgit branch -D octo-other &&\n-\tgit branch -D octo-second &&\n-\n-\t# add padding to make these merges less interesting\n-\t# and avoid having them selected for bitmaps\n-\ttest_commit_bulk --id=file 100 &&\n-\tgit checkout other &&\n-\ttest_commit_bulk --id=side 100 &&\n-\tgit checkout second &&\n-\n-\tbitmaptip=$(git rev-parse second) &&\n-\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n-\tgit tag tagged-blob $blob &&\n-\tgit config repack.writebitmaps true\n+test_expect_success 'setup writing bitmaps during repack' '\n+\tgit config repack.writeBitmaps true\n '\n \n test_expect_success 'full repack creates bitmaps' '\n@@ -123,109 +40,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n-test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-\tgit rev-list --test-bitmap HEAD\n-'\n-\n-rev_list_tests_head () {\n-\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch~5..$branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n-\t\tgit rev-list --count -n 1 $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n-\t\tgit rev-list --count other...second >expect &&\n-\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n-\t\tgit rev-list --count $branch -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count --objects $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n-\t\tgit rev-list --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list $branch >expect &&\n-\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list --objects $branch >expect &&\n-\t\ttest_bitmap_traversal expect actual\n-\t'\n-\n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n-\t\tgrep $blob actual\n-\t'\n-}\n-\n-rev_list_tests () {\n-\tstate=$1\n-\n-\tfor branch in \"second\" \"other\"\n-\tdo\n-\t\trev_list_tests_head\n-\tdone\n-}\n-\n-rev_list_tests 'full bitmap'\n-\n-test_expect_success 'clone from bitmapped repository' '\n-\tgit clone --no-local --bare . clone.git &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'partial clone from bitmapped repository' '\n-\ttest_config uploadpack.allowfilter true &&\n-\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n-\t(\n-\t\tcd partial-clone.git &&\n-\t\tpack=$(echo objects/pack/*.pack) &&\n-\t\tgit verify-pack -v \"$pack\" >have &&\n-\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n-\t\t# we expect this single blob because of the direct ref\n-\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n-\t\ttest_cmp expect blobs\n-\t)\n-'\n-\n-test_expect_success 'setup further non-bitmapped commits' '\n-\ttest_commit_bulk --id=further 10\n-'\n-\n-rev_list_tests 'partial bitmap'\n-\n-test_expect_success 'fetch (partial bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+basic_bitmap_tests\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -461,40 +276,6 @@ test_expect_success 'truncated bitmap fails gracefully (cache)' '\n \ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n-test_expect_success 'enumerating progress counts pack-reused objects' '\n-\tcount=$(git rev-list --objects --all --count) &&\n-\tgit repack -adb &&\n-\n-\t# check first with only reused objects; confirm that our progress\n-\t# showed the right number, and also that we did pack-reuse as expected.\n-\t# Check only the final \"done\" line of the meter (there may be an\n-\t# arbitrary number of intermediate lines ending with CR).\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $count, done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr &&\n-\n-\t# now the same but with one non-reused object\n-\tgit commit --allow-empty -m \"an extra commit object\" &&\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr\n-'\n-\n-# have_delta \n-#\n-# Note that because this relies on cat-file, it might find _any_ copy of an\n-# object in the repository. The caller is responsible for making sure\n-# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n-have_delta () {\n-\techo $2 >expect &&\n-\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n-\ttest_cmp expect actual\n-}\n-\n # Create a state of history with these properties:\n #\n # - refs that allow a client to fetch some new history, while sharing some old\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433585","messageId":"d47aa4a91933b7424bf720be7ec4911a398833b1.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 17/25] t/helper/test-read-midx.c: add --checksum mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:33Z","receivedAt":"2021-08-24T16:17:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subsequent tests will want to check for the existence of a multi-pack\nbitmap which matches the multi-pack-index stored in the pack directory.\n\nThe multi-pack bitmap includes the hex checksum of the MIDX it\ncorresponds to in its filename (for example,\n'$packdir/multi-pack-index-.bitmap'). As a result, some tests\nwant a way to learn what '' is.\n\nThis helper addresses that need by printing the checksum of the\nrepository's multi-pack-index.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 16 +++++++++++++++-\n t/lib-bitmap.sh | 4 ++++\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 7c2eb11a8e..cb0d27049a 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \treturn 0;\n }\n \n+static int read_midx_checksum(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m;\n+\n+\tsetup_git_directory();\n+\tm = load_multi_pack_index(object_dir, 1);\n+\tif (!m)\n+\t\treturn 1;\n+\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n+\telse if (!strcmp(argv[1], \"--checksum\"))\n+\t\treturn read_midx_checksum(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 77464da6fd..21d0392dda 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -260,3 +260,7 @@ have_delta () {\n \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n \ttest_cmp expect actual\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"$1\"\n+}\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433586","messageId":"3e0da7e5ed0da144a5305c0bb211ecebd71e798b.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 19/25] t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:38Z","receivedAt":"2021-08-24T16:17:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap causes tests which repack in a partial clone to\nfail because they are missing objects. Missing objects is an expected\ncomponent of tests in t0410, so disable this knob altogether. Graceful\ndegradation when writing a bitmap with missing objects is tested in\nt5326.\n\nSigned-off-by: Taylor Blau \n---\n t/t0410-partial-clone.sh | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex bbcc51ee8e..bba679685f 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -4,6 +4,9 @@ test_description='partial clone'\n \n . ./test-lib.sh\n \n+# missing promisor objects cause repacks which write bitmaps to fail\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n delete_object () {\n \trm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433587","messageId":"4e0d49a2dd6856763486bf7931199494a521d29c.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 20/25] t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:41Z","receivedAt":"2021-08-24T16:17:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap confuses many of the tests in t5310, which\nexpect to control whether and how bitmaps are written. Since the\nrelevant MIDX-bitmap tests here are covered already in t5326, let's just\ndisable the flag for the whole t5310 script.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 4318f84d53..673baa5c3c 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n . \"$TEST_DIRECTORY\"/lib-bundle.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n+# their place.\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433588","messageId":"9d9d9f28a6703a49aa9c62892985dea32543d880.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 18/25] t5326: test multi-pack bitmap behavior","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:36Z","receivedAt":"2021-08-24T16:17:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch introduces a new test, t5326, which tests the basic\nfunctionality of multi-pack bitmaps.\n\nSome trivial behavior is tested, such as:\n\n - Whether bitmaps can be generated with more than one pack.\n - Whether clones can be served with all objects in the bitmap.\n - Whether follow-up fetches can be served with some objects outside of\n the server's bitmap\n\nThese use lib-bitmap's tests (which in turn were pulled from t5310), and\nwe cover cases where the MIDX represents both a single pack and multiple\npacks.\n\nIn addition, some non-trivial and MIDX-specific behavior is tested, too,\nincluding:\n\n - Whether multi-pack bitmaps behave correctly with respect to the\n pack-reuse machinery when the base for some object is selected from\n a different pack than the delta.\n - Whether multi-pack bitmaps correctly respect the\n pack.preferBitmapTips configuration.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 286 ++++++++++++++++++++++++++++++++++\n 1 file changed, 286 insertions(+)\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..4ad7c2c969\n--- /dev/null\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,286 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality'\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+setup_bitmap_history\n+\n+test_expect_success 'enable core.multiPackIndex' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_expect_success 'create single-pack midx with bitmaps' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success 'create new additional packs' '\n+\tfor i in $(test_seq 1 16)\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\n+\tgit checkout -b other2 HEAD~8 &&\n+\tfor i in $(test_seq 1 8)\n+\tdo\n+\t\ttest_commit \"side-$i\" &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit checkout second\n+'\n+\n+test_expect_success 'create multi-pack midx with bitmaps' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\tls $objdir/pack/pack-*.pack >packs &&\n+\ttest_line_count = 25 packs &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\ttest_commit respect--no-bitmap &&\n+\tgit repack -d &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\n+\tgit multi-pack-index write --no-bitmap &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n+'\n+\n+test_expect_success 'setup midx with base from later pack' '\n+\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t# prefers to delete contents out of a base rather than add to a shorter\n+\t# object.\n+\ttest_seq 1 128 >a &&\n+\ttest_seq 1 130 >b &&\n+\n+\tgit add a b &&\n+\tgit commit -m \"initial commit\" &&\n+\n+\ta=$(git rev-parse HEAD:a) &&\n+\tb=$(git rev-parse HEAD:b) &&\n+\n+\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$a\n+\t$b\n+\tEOF\n+\t) &&\n+\n+\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t# any other object.\n+\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$b\n+\t$(git rev-parse HEAD)\n+\t$(git rev-parse HEAD^{tree})\n+\tEOF\n+\t) &&\n+\n+\tgit prune-packed &&\n+\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t# reuse.\n+\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\thave_delta $a $b &&\n+\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+'\n+\n+rev_list_tests 'full bitmap with backwards delta'\n+\n+test_expect_success 'clone with bitmaps enabled' '\n+\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\tgit rev-parse HEAD >expect &&\n+\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+bitmap_reuse_tests() {\n+\tfrom=$1\n+\tto=$2\n+\n+\ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk 16 &&\n+\t\t\tgit tag old-tip &&\n+\n+\t\t\tgit config core.multiPackIndex true &&\n+\t\t\tif test \"MIDX\" = \"$from\"\n+\t\t\tthen\n+\t\t\t\tgit repack -Ad &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tgit repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk --id=further 16 &&\n+\t\t\tgit tag new-tip &&\n+\n+\t\t\tif test \"MIDX\" = \"$to\"\n+\t\t\tthen\n+\t\t\t\tgit repack -d &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tgit repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit for-each-ref &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n+\t\t)\n+\t'\n+}\n+\n+bitmap_reuse_tests 'pack' 'MIDX'\n+bitmap_reuse_tests 'MIDX' 'pack'\n+bitmap_reuse_tests 'MIDX' 'MIDX'\n+\n+test_expect_success 'missing object closure fails gracefully' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit loose &&\n+\t\ttest_commit packed &&\n+\n+\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t# commit.\n+\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t$(git rev-parse packed)\n+\t\tEOF\n+\n+\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\tgrep \"doesn.t have full closure\" err &&\n+\t\ttest_path_is_missing $midx\n+\t)\n+'\n+\n+test_expect_success 'setup partial bitmaps' '\n+\ttest_commit packed &&\n+\tgit repack &&\n+\ttest_commit loose &&\n+\tgit multi-pack-index write --bitmap 2>err &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+'\n+\n+basic_bitmap_tests HEAD~\n+\n+test_expect_success 'removing a MIDX clears stale bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\tstale_rev=$midx-$(midx_checksum $objdir).rev &&\n+\t\trm $midx &&\n+\n+\t\t# Then write a new MIDX.\n+\t\ttest_commit new &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\ttest_path_is_missing $stale_bitmap &&\n+\t\ttest_path_is_missing $stale_rev\n+\t)\n+'\n+\n+test_expect_success 'pack.preferBitmapTips' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\tbitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433589","messageId":"47eba8ecf93e6e220ffbb98ebdb7e958db2c2d53.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 21/25] t5319: don't write MIDX bitmaps in t5319","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:44Z","receivedAt":"2021-08-24T16:17:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This test is specifically about generating a midx still respecting a\npack-based bitmap file. Generating a MIDX bitmap would confuse the test.\nLet's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to\nmake sure we don't do so.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 9b184bd45e..a81375d920 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -504,7 +504,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433591","messageId":"c2f94e033d6905e5bb5b27ae237f6a0db16aae53.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 23/25] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:49Z","receivedAt":"2021-08-24T16:17:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment\nvariable to also write a multi-pack bitmap when\n'GIT_TEST_MULTI_PACK_INDEX' is set.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 12 ++++++++++--\n ci/run-build-and-tests.sh | 1 +\n midx.h | 2 ++\n t/README | 4 ++++\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..82ab668272 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n \t\t !is_bare_repository())\n \t\t\twrite_bitmaps = 0;\n+\t} else if (write_bitmaps &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n+\t\twrite_bitmaps = 0;\n \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tupdate_server_info(0);\n \tremove_temporary_files();\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n+\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t}\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..7ee9ba9325 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -23,6 +23,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_WRITE_REV_INDEX=1\ndiff --git a/midx.h b/midx.h\nindex 350f4d0a7b..aa3da557bb 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,8 @@ struct pack_entry;\n struct repository;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex 9e70122302..12014aa988 100644\n--- a/t/README\n+++ b/t/README\n@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n+'--bitmap' option on all invocations of 'git multi-pack-index write',\n+and ignores pack-objects' '--write-bitmap-index'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433590","messageId":"d98faa4c2c4e58d2579f9a2ed9a3d3518695e49d.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 25/25] p5326: perf tests for MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:53Z","receivedAt":"2021-08-24T16:17:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These new performance tests demonstrate effectively the same behavior as\np5310, but use a multi-pack bitmap instead of a single-pack one.\n\nNotably, p5326 does not create a MIDX bitmap with multiple packs. This\nis so we can measure a direct comparison between it and p5310. Any\ndifference between the two is measuring just the overhead of using MIDX\nbitmaps.\n\nHere are the results of p5310 and p5326 together, measured at the same\ntime and on the same machine (using a Xenon W-2255 CPU):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5310.2: repack to disk 96.78(93.39+11.33)\n 5310.3: simulated clone 9.98(9.79+0.19)\n 5310.4: simulated fetch 1.75(4.26+0.19)\n 5310.5: pack to file (bitmap) 28.20(27.87+8.70)\n 5310.6: rev-list (commits) 0.41(0.36+0.05)\n 5310.7: rev-list (objects) 1.61(1.54+0.07)\n 5310.8: rev-list count with blob:none 0.25(0.21+0.04)\n 5310.9: rev-list count with blob:limit=1k 2.65(2.54+0.10)\n 5310.10: rev-list count with tree:0 0.23(0.19+0.04)\n 5310.11: simulated partial clone 4.34(4.21+0.12)\n 5310.13: clone (partial bitmap) 11.05(12.21+0.48)\n 5310.14: pack to file (partial bitmap) 31.25(34.22+3.70)\n 5310.15: rev-list with tree filter (partial bitmap) 0.26(0.22+0.04)\n\nversus the same tests (this time using a multi-pack index):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5326.2: setup multi-pack index 78.99(75.29+11.58)\n 5326.3: simulated clone 11.78(11.56+0.22)\n 5326.4: simulated fetch 1.70(4.49+0.13)\n 5326.5: pack to file (bitmap) 28.02(27.72+8.76)\n 5326.6: rev-list (commits) 0.42(0.36+0.06)\n 5326.7: rev-list (objects) 1.65(1.58+0.06)\n 5326.8: rev-list count with blob:none 0.26(0.21+0.05)\n 5326.9: rev-list count with blob:limit=1k 2.97(2.86+0.10)\n 5326.10: rev-list count with tree:0 0.25(0.20+0.04)\n 5326.11: simulated partial clone 5.65(5.49+0.16)\n 5326.13: clone (partial bitmap) 12.22(13.43+0.38)\n 5326.14: pack to file (partial bitmap) 30.05(31.57+7.25)\n 5326.15: rev-list with tree filter (partial bitmap) 0.24(0.20+0.04)\n\nThere is slight overhead in \"simulated clone\", \"simulated partial\nclone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\ndue to using the MIDX's reverse index to map between bit positions and\nMIDX positions.\n\nThis can be reproduced by running \"git repack -adb\" along with \"git\nmulti-pack-index write --bitmap\" in a large-ish repository. Then run:\n\n $ perf record -o pack.perf git -c core.multiPackIndex=false \\\n pack-objects --all --stdout >/dev/null /dev/null \n---\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..5845109ac7\n--- /dev/null\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,43 @@\n+#!/bin/sh\n+\n+test_description='Tests performance using midx bitmaps'\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n+\n+test_perf_large_repo\n+\n+test_expect_success 'enable multi-pack index' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_perf 'setup multi-pack index' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap\n+'\n+\n+test_full_bitmap\n+\n+test_expect_success 'create partial bitmap state' '\n+\t# pick a commit to represent the repo tip in the past\n+\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t# now pretend we have just one tip\n+\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\tgit update-ref HEAD $cutoff &&\n+\n+\t# and then repack, which will leave us with a nice\n+\t# big bitmap pack of the \"old\" history, and all of\n+\t# the new history will be loose, as if it had been pushed\n+\t# up incrementally and exploded via unpack-objects\n+\tgit repack -Ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\n+\t# and now restore our original tip, as if the pushes\n+\t# had happened\n+\tgit update-ref HEAD $orig_tip\n+'\n+\n+test_partial_bitmap\n+\n+test_done\n-- \n2.31.1.163.ga65ce7f831\n"},{"id":"433593","messageId":"6b03016c9937218071f1819dbbca988615b3b6a0.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 24/25] p5310: extract full and partial bitmap tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:51Z","receivedAt":"2021-08-24T16:17:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A new p5326 introduced by the next patch will want these same tests,\ninterjecting its own setup in between. Move them out so that both perf\ntests can reuse them.\n\nSigned-off-by: Taylor Blau \n---\n t/perf/lib-bitmap.sh | 69 ++++++++++++++++++++++++++++++++++++\n t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------\n 2 files changed, 72 insertions(+), 62 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n\ndiff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh\nnew file mode 100644\nindex 0000000000..63d3bc7cec\n--- /dev/null\n+++ b/t/perf/lib-bitmap.sh\n@@ -0,0 +1,69 @@\n+# Helper functions for testing bitmap performance; see p5310.\n+\n+test_full_bitmap () {\n+\ttest_perf 'simulated clone' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'simulated fetch' '\n+\t\thave=$(git rev-list HEAD~100 -1) &&\n+\t\t{\n+\t\t\techo HEAD &&\n+\t\t\techo ^$have\n+\t\t} | git pack-objects --revs --stdout >/dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (commits)' '\n+\t\tgit rev-list --all --use-bitmap-index >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (objects)' '\n+\t\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tag negated via --not --all (objects)' '\n+\t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with negative tag (objects)' '\n+\t\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:none' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:none >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:limit=1k' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:limit=1k >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with tree:0' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+\n+\ttest_perf 'simulated partial clone' '\n+\t\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n+\t'\n+}\n+\n+test_partial_bitmap () {\n+\ttest_perf 'clone (partial bitmap)' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (partial bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tree filter (partial bitmap)' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+}\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 452be01056..7ad4f237bc 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -2,6 +2,7 @@\n \n test_description='Tests pack performance using bitmaps'\n . ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n test_perf_large_repo\n \n@@ -25,56 +26,7 @@ test_perf 'repack to disk' '\n \tgit repack -ad\n '\n \n-test_perf 'simulated clone' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'simulated fetch' '\n-\thave=$(git rev-list HEAD~100 -1) &&\n-\t{\n-\t\techo HEAD &&\n-\t\techo ^$have\n-\t} | git pack-objects --revs --stdout >/dev/null\n-'\n-\n-test_perf 'pack to file (bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n-'\n-\n-test_perf 'rev-list (commits)' '\n-\tgit rev-list --all --use-bitmap-index >/dev/null\n-'\n-\n-test_perf 'rev-list (objects)' '\n-\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with tag negated via --not --all (objects)' '\n-\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with negative tag (objects)' '\n-\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:none' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:none >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:limit=1k' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:limit=1k >/dev/null\n-'\n-\n-test_perf 'rev-list count with tree:0' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n-\n-test_perf 'simulated partial clone' '\n-\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n-'\n+test_full_bitmap\n \n test_expect_success 'create partial bitmap state' '\n \t# pick a commit to represent the repo tip in the past\n@@ -97,17 +49,6 @@ test_expect_success 'create partial bitmap state' '\n \tgit update-ref HEAD $orig_tip\n '\n \n-test_perf 'clone (partial bitmap)' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'pack to file (partial bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n-'\n-\n-test_perf 'rev-list with tree filter (partial bitmap)' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n+test_partial_bitmap\n \n test_done\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433592","messageId":"3d78afa2ad35b96886af78a295f1fecc3d7e6170.1629821743.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"[PATCH v4 22/25] t7700: update to work with MIDX bitmap test knob","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T16:16:46Z","receivedAt":"2021-08-24T16:17:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A number of these tests are focused only on pack-based bitmaps and need\nto be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where\nnecessary.\n\nSigned-off-by: Taylor Blau \n---\n t/t7700-repack.sh | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 25b235c063..98eda3bfeb 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit repack -Adbl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '\n \n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n-\tgit -C bare.git repack -ad &&\n+\trm -f bare.git/objects/pack/*.bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \n2.31.1.163.ga65ce7f831\n\n"},{"id":"433618","messageId":"xmqqa6l6oafd.fsf@gitster.g","threadId":"55464","inReplyTo":"771741844be3570395abfda813ed5ef2fa78332e.1629821743.git.me@ttaylorr.com","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-24T20:27:34Z","receivedAt":"2021-08-24T20:27:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> When writing a new multi-pack index, write_midx_internal() attempts to\n> clean up any auxiliary files (currently just the MIDX's `.rev` file, but\n> soon to include a `.bitmap`, too) corresponding to the MIDX it's\n> replacing.\n>\n> This step should happen after the new MIDX is written into place, since\n> doing so beforehand means that the old MIDX could be read without its\n> corresponding .rev file.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> midx.c | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 321c6fdd2f..73b199ca49 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \n> \tif (flags & MIDX_WRITE_REV_INDEX)\n> \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n> -\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n> \n> \tcommit_lock_file(&lk);\n> \n> +\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n\nThis needs to take object_dir into account, no?\n\nThere are a few more calls to clear_midx_files_ext() added in 15/25\nand they use the_repository, too.\n\n> cleanup:\n> \tfor (i = 0; i < ctx.nr; i++) {\n> \t\tif (ctx.info[i].p) {\n"},{"id":"433619","messageId":"YSVX18UXh9vX+Zhp@nand.local","threadId":"55464","inReplyTo":"xmqqa6l6oafd.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T20:34:31Z","receivedAt":"2021-08-24T20:34:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 24, 2021 at 01:27:34PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > When writing a new multi-pack index, write_midx_internal() attempts to\n> > clean up any auxiliary files (currently just the MIDX's `.rev` file, but\n> > soon to include a `.bitmap`, too) corresponding to the MIDX it's\n> > replacing.\n> >\n> > This step should happen after the new MIDX is written into place, since\n> > doing so beforehand means that the old MIDX could be read without its\n> > corresponding .rev file.\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > midx.c | 3 ++-\n> > 1 file changed, 2 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/midx.c b/midx.c\n> > index 321c6fdd2f..73b199ca49 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> >\n> > \tif (flags & MIDX_WRITE_REV_INDEX)\n> > \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n> > -\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n> >\n> > \tcommit_lock_file(&lk);\n> >\n> > +\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n>\n> This needs to take object_dir into account, no?\n\nYes and no; clear_midx_files_ext() still takes a pointer to a 'struct\nrepository' until we pick up [1].\n\nI asked for some changes in the latest version that Johannes posted. So\nI'd be OK to live with this behavior for the time being, and then I can\nsend another patch on top that fixes the new and existing callers\n(incorporating [1] with some new tests).\n\nOr we can hold one up and expedite the other. I would suggest that we\npick up this series to next if you're otherwise happy with it and then I\ncan send the trivial fixes on top.\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/20210823171011.80588-1-johannes@sipsolutions.net/\n"},{"id":"433623","messageId":"xmqqr1eimtrp.fsf@gitster.g","threadId":"55464","inReplyTo":"YSVX18UXh9vX+Zhp@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-24T21:12:42Z","receivedAt":"2021-08-24T21:12:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n>> This needs to take object_dir into account, no?\n>\n> Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct\n> repository' until we pick up [1].\n\nI was hoping that [1] will become part of this series as a trivial\nclean-up and bugfix, perhaps in its early part.\n\n"},{"id":"433624","messageId":"YSVjnSDaBXgXvT9W@nand.local","threadId":"55464","inReplyTo":"xmqqr1eimtrp.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T21:24:45Z","receivedAt":"2021-08-24T21:24:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 24, 2021 at 02:12:42PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> >> This needs to take object_dir into account, no?\n> >\n> > Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct\n> > repository' until we pick up [1].\n>\n> I was hoping that [1] will become part of this series as a trivial\n> clean-up and bugfix, perhaps in its early part.\n\nSure, that works even better. I'll send a reroll incorporating it as\nsoon as I finish re-testing.\n\nThanks,\nTaylor\n"},{"id":"433667","messageId":"YSVsHo2wLhnraBnv@nand.local","threadId":"55464","inReplyTo":"YSVjnSDaBXgXvT9W@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T22:01:02Z","receivedAt":"2021-08-24T22:01:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 24, 2021 at 05:24:45PM -0400, Taylor Blau wrote:\n> On Tue, Aug 24, 2021 at 02:12:42PM -0700, Junio C Hamano wrote:\n> > Taylor Blau writes:\n> >\n> > >> This needs to take object_dir into account, no?\n> > >\n> > > Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct\n> > > repository' until we pick up [1].\n> >\n> > I was hoping that [1] will become part of this series as a trivial\n> > clean-up and bugfix, perhaps in its early part.\n>\n> Sure, that works even better. I'll send a reroll incorporating it as\n> soon as I finish re-testing.\n\nHmm, this got me wondering: what should be the behavior be when we run\nthe multi-pack-index command outside of a Git repository? For example,\nin patch 15 we do:\n\n for (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n if (!strcmp(object_dir, cur->object_dir)) {\n ctx.m = cur;\n break;\n }\n }\n\nbut obviously get_multi_pack_index(the_repository) will fail when there\nis no repository to begin with.\n\nThe real question is whether we should allow munging arbitrary MIDXs, or\nrestrict the ones we can modify to just our alternates. If we allow the\nformer, then that code needs to be tweaked. If not, and we only allow\ntouching alternates, then we need to require a repository for the\nmulti-pack-index builtin.\n\nThanks,\nTaylor\n"},{"id":"433668","messageId":"xmqq35qymrcn.fsf@gitster.g","threadId":"55464","inReplyTo":"YSVjnSDaBXgXvT9W@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-24T22:04:56Z","receivedAt":"2021-08-24T22:05:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Tue, Aug 24, 2021 at 02:12:42PM -0700, Junio C Hamano wrote:\n>> Taylor Blau writes:\n>>\n>> >> This needs to take object_dir into account, no?\n>> >\n>> > Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct\n>> > repository' until we pick up [1].\n>>\n>> I was hoping that [1] will become part of this series as a trivial\n>> clean-up and bugfix, perhaps in its early part.\n>\n> Sure, that works even better. I'll send a reroll incorporating it as\n> soon as I finish re-testing.\n\nFWIW, here is what I have somewhere in 'seen' where two topics meet.\n\ndiff --cc midx.c\nindex c0209751b5,4574e6d411..0000000000\n--- i/midx.c\n+++ w/midx.c\n@@@ -1090,6 -1351,9 +1351,9 @@@ static int write_midx_internal(const ch\n \n \tcommit_lock_file(&lk);\n \n -\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n -\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n++\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n++\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+ \n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n@@@ -1165,7 -1429,8 +1429,8 @@@ void clear_midx_file(struct repository \n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n -\tclear_midx_files_ext(r, \".bitmap\", NULL);\n -\tclear_midx_files_ext(r, \".rev\", NULL);\n++\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n +\tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n \n \tfree(midx);\n }\n"},{"id":"433669","messageId":"xmqqy28qlcow.fsf@gitster.g","threadId":"55464","inReplyTo":"xmqq35qymrcn.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-24T22:06:55Z","receivedAt":"2021-08-24T22:07:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> FWIW, here is what I have somewhere in 'seen' where two topics meet.\n\nOops, one change missed.\n\ndiff --cc midx.c\nindex c0209751b5,4574e6d411..0000000000\n--- i/midx.c\n+++ w/midx.c\n@@@ -947,11 -1136,29 +1136,29 @@@ static int write_midx_internal(const ch\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n- \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n- \t\tgoto cleanup;\n+ \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+ \t\tstruct bitmap_index *bitmap_git;\n+ \t\tint bitmap_exists;\n+ \t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+ \n+ \t\tbitmap_git = prepare_midx_bitmap_git(the_repository, ctx.m);\n+ \t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+ \t\tfree_bitmap_index(bitmap_git);\n+ \n+ \t\tif (bitmap_exists || !want_bitmap) {\n+ \t\t\t/*\n+ \t\t\t * The correct MIDX already exists, and so does a\n+ \t\t\t * corresponding bitmap (or one wasn't requested).\n+ \t\t\t */\n+ \t\t\tif (!want_bitmap)\n -\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n++\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n+ \t\t\t\t\t\t NULL);\n+ \t\t\tgoto cleanup;\n+ \t\t}\n+ \t}\n \n- \tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n+ \t\tint found = 0;\n \t\tfor (i = 0; i < ctx.nr; i++) {\n \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n@@@ -1090,6 -1351,9 +1351,9 @@@\n \n \tcommit_lock_file(&lk);\n \n -\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n -\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n++\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n++\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+ \n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n@@@ -1165,7 -1429,8 +1429,8 @@@ void clear_midx_file(struct repository \n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n -\tclear_midx_files_ext(r, \".bitmap\", NULL);\n -\tclear_midx_files_ext(r, \".rev\", NULL);\n++\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n +\tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n \n \tfree(midx);\n }\n"},{"id":"433670","messageId":"YSVuUYFh7lmhNlEy@nand.local","threadId":"55464","inReplyTo":"xmqqy28qlcow.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-24T22:10:25Z","receivedAt":"2021-08-24T22:10:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 24, 2021 at 03:06:55PM -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n> > FWIW, here is what I have somewhere in 'seen' where two topics meet.\n>\n> Oops, one change missed.\n\nThanks; that matches my own resolution. I noticed that it does fail the\nnew test in t5319, since writing a MIDX wants to make sure that we are\nonly touching an alternate's object directory (which will fail if we are\nrunning `git multi-pack-index` from outside of a repository).\n\nMy opinion is that we should require being inside of a repository to run\nthe MIDX builtin. Otherwise we're allowing that command to modify any\nold MIDX, which doesn't make sense.\n\nI think we probably need a single unifying topic, so I'm happy if you\nwant to discard one of our two topics from seen in the meantime.\n\nThanks,\nTaylor\n"},{"id":"433681","messageId":"YSWOtNoxirDdmBXG@coredump.intra.peff.net","threadId":"55464","inReplyTo":"cover.1629821743.git.me@ttaylorr.com","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-25T00:28:36Z","receivedAt":"2021-08-25T00:28:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 24, 2021 at 12:15:47PM -0400, Taylor Blau wrote:\n\n> Range-diff against v3:\n> [...]\n> 9: 40cff5beb5 ! 9: c9fea31fa8 midx: avoid opening multiple MIDXs when writing\n> @@ Commit message\n> one and should invalidate the object store's memory of any MIDX that\n> might have existed beforehand.\n> \n> + Note that this now forbids passing object directories that don't belong\n> + to alternate repositories over `--object-dir`, since before we would\n> + have happily opened a MIDX in any directory, but now restrict ourselves\n> + to only those reachable by `r->objects->multi_pack_index` (and alternate\n> + MIDXs that we can see by walking the `next` pointer).\n> +\n> + As far as I can tell, supporting arbitrary directories with\n> + `--object-dir` was a historical accident, since even the documentation\n> + says `` when referring to the value passed to this option.\n> +\n> + A future patch could clean this up and provide a warning() when a\n> + non-alternate directory was given, since we'll still write a new MIDX\n> + there, we just won't reuse any MIDX that might happen to already exist\n> + in that directory.\n> +\n\nSo this is definitely fixed as we discussed. But since that discussion,\nwe've had the thread over in:\n\n https://lore.kernel.org/git/20210820195558.44275-1-johannes@sipsolutions.net/\n\nand its siblings:\n\n https://lore.kernel.org/git/20210823094049.44136-1-johannes@sipsolutions.net/\n\n https://lore.kernel.org/git/20210823171011.80588-1-johannes@sipsolutions.net/\n\nIt's not clear to me that we have a resolution on whether calling \"cd ..\n&& git multi-pack-index write --object-dir repo.git\" is supposed to\nwork.\n\nIt has traditionally worked (at least for trivial cases, AFAICT), but I\nfind the behavior surprising and unlike most of the rest of Git, and I'm\nnot at all certain that there aren't subtle bugs lurking (basically\nanything that wants to do object lookup, like oh say, a bitmap\ngenerator).\n\nBut if we do want to support it, then we have to find a different\nsolution here, don't we? I think the least-painful version of that is\nprobably recording _whether_ we found ctx.m in the_repository's\nobject_store, and switching behavior based on that (e.g., calling\nclose_midx() versus close_object_store() depending).\n\n-Peff\n"},{"id":"433691","messageId":"YSWmhMID1hGs7Yp1@nand.local","threadId":"55464","inReplyTo":"YSWOtNoxirDdmBXG@coredump.intra.peff.net","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-25T02:10:12Z","receivedAt":"2021-08-25T02:10:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 24, 2021 at 08:28:36PM -0400, Jeff King wrote:\n> On Tue, Aug 24, 2021 at 12:15:47PM -0400, Taylor Blau wrote:\n>\n> > Range-diff against v3:\n> > [...]\n> > 9: 40cff5beb5 ! 9: c9fea31fa8 midx: avoid opening multiple MIDXs when writing\n> > @@ Commit message\n> > one and should invalidate the object store's memory of any MIDX that\n> > might have existed beforehand.\n> >\n> > + Note that this now forbids passing object directories that don't belong\n> > + to alternate repositories over `--object-dir`, since before we would\n> > + have happily opened a MIDX in any directory, but now restrict ourselves\n> > + to only those reachable by `r->objects->multi_pack_index` (and alternate\n> > + MIDXs that we can see by walking the `next` pointer).\n> > +\n> > + As far as I can tell, supporting arbitrary directories with\n> > + `--object-dir` was a historical accident, since even the documentation\n> > + says `` when referring to the value passed to this option.\n> > +\n> > + A future patch could clean this up and provide a warning() when a\n> > + non-alternate directory was given, since we'll still write a new MIDX\n> > + there, we just won't reuse any MIDX that might happen to already exist\n> > + in that directory.\n> > +\n>\n> So this is definitely fixed as we discussed. But since that discussion,\n> we've had the thread over in:\n>\n> https://lore.kernel.org/git/20210820195558.44275-1-johannes@sipsolutions.net/\n>\n> and its siblings:\n>\n> https://lore.kernel.org/git/20210823094049.44136-1-johannes@sipsolutions.net/\n>\n> https://lore.kernel.org/git/20210823171011.80588-1-johannes@sipsolutions.net/\n>\n> It's not clear to me that we have a resolution on whether calling \"cd ..\n> && git multi-pack-index write --object-dir repo.git\" is supposed to\n> work.\n\nMy recommendation would be to do the following things, all in a reroll\nof this series:\n\n - Fix the bug by which we would delete a .rev or .bitmap file out of a\n different object store than we were working in (when the caller\n passes `--object-dir`).\n\n - Disallow running `git multi-pack-index` outside of a Git repository.\n\n - Restrict `--object-dir` to only work with alternates of the\n repository in the current working directory.\n\nTo me, that seems like both the least-surprising behavior, and what\nwould lend itself to the easiest implementation. I would probably argue\nthat the existing behavior (where `--object-dir` would work against\narbitrary repositories) is a bug, and shouldn't continue to be\nsupported.\n\nSo my plan would be to do that, which would generate something like the\nfollowing range-diff. If nobody has any objections, I'd like to send\nwhat I currently have in ttaylorr/git on GitHub in the\ntb/multi-pack-bitmaps branch as a reroll of this series, and then merge\nthat early in the cycle to give it a chance to be tested before we cut\n2.34.\n\nThanks,\nTaylor\n"},{"id":"433692","messageId":"YSWnNP09G2wF0gd1@nand.local","threadId":"55464","inReplyTo":"YSWmhMID1hGs7Yp1@nand.local","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-25T02:13:08Z","receivedAt":"2021-08-25T02:13:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 24, 2021 at 10:10:12PM -0400, Taylor Blau wrote:\n> My recommendation would be to do the following things, all in a reroll\n> of this series:\n\nFor what it's worth, the substantive changes (which I have not figured\nout how to include in a range-diff since they are entirely new patches)\nare these:\n\n - Replacing Johannes' patch with:\n https://github.com/ttaylorr/git/commit/2b1afbd516a75bb43a8aae6ff1cac6a83ed7f589,\n\n - and then adding another patch immediately after it:\n https://github.com/git/git/commit/0a2d4d8dbf3c50eb3e2b659d1dcdf432d3b4d223\n\n...and otherwise keeping the remainder of the series unchanged.\n\nThanks,\nTaylor\n"},{"id":"433715","messageId":"YSXy73lWKteiuY6s@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YSWmhMID1hGs7Yp1@nand.local","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-25T07:36:15Z","receivedAt":"2021-08-25T07:36:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 24, 2021 at 10:10:12PM -0400, Taylor Blau wrote:\n\n> > It's not clear to me that we have a resolution on whether calling \"cd ..\n> > && git multi-pack-index write --object-dir repo.git\" is supposed to\n> > work.\n> \n> My recommendation would be to do the following things, all in a reroll\n> of this series:\n> \n> - Fix the bug by which we would delete a .rev or .bitmap file out of a\n> different object store than we were working in (when the caller\n> passes `--object-dir`).\n> \n> - Disallow running `git multi-pack-index` outside of a Git repository.\n> \n> - Restrict `--object-dir` to only work with alternates of the\n> repository in the current working directory.\n> \n> To me, that seems like both the least-surprising behavior, and what\n> would lend itself to the easiest implementation. I would probably argue\n> that the existing behavior (where `--object-dir` would work against\n> arbitrary repositories) is a bug, and shouldn't continue to be\n> supported.\n\nAll of those seem reasonable to me, and are what I would suggest if we\nwere starting from scratch. My only hesitation is whether people are\nusing the weird behavior of --object-dir in the wild (e.g., are bup\nfolks relying on it).\n\nJohannes, is this something you're using _now_, and it works, or\nsomething you hoped to use in the future?\n\nIn a sense, \"hope to use\" does not make you any less disappointed. ;)\nBut what I'm wondering is whether using --object-dir from outside a repo\nentirely is actually something that even works. I.e., would we be\ndisabling a behavior that was not intended, but does happen to work? Or\nare we closing off a possibly buggy and half-working part of the system?\n\n-Peff\n"},{"id":"433716","messageId":"d66f040e663f935c5407fdecad43a4c3d8e57105.camel@sipsolutions.net","threadId":"55464","inReplyTo":"YSXy73lWKteiuY6s@coredump.intra.peff.net","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Johannes Berg","fromEmail":"johannes@sipsolutions.net","sentAt":"2021-08-25T07:48:02Z","receivedAt":"2021-08-25T07:48:11Z","isPatch":true,"sender":{"key":"johannes@sipsolutions.net","avatar":"https://avatars.githubusercontent.com/u/5159728?v=4"},"body":"On Wed, 2021-08-25 at 03:36 -0400, Jeff King wrote:\n> On Tue, Aug 24, 2021 at 10:10:12PM -0400, Taylor Blau wrote:\n> \n> > > It's not clear to me that we have a resolution on whether calling \"cd ..\n> > > && git multi-pack-index write --object-dir repo.git\" is supposed to\n> > > work.\n> > \n> > My recommendation would be to do the following things, all in a reroll\n> > of this series:\n> > \n> >   - Fix the bug by which we would delete a .rev or .bitmap file out of a\n> >     different object store than we were working in (when the caller\n> >     passes `--object-dir`).\n\nThat was what my patch did, afaict.\n\n> >   - Disallow running `git multi-pack-index` outside of a Git repository.\n> > \n> >   - Restrict `--object-dir` to only work with alternates of the\n> >     repository in the current working directory.\n> > \n> > To me, that seems like both the least-surprising behavior, and what\n> > would lend itself to the easiest implementation. I would probably argue\n> > that the existing behavior (where `--object-dir` would work against\n> > arbitrary repositories) is a bug, and shouldn't continue to be\n> > supported.\n> \n> All of those seem reasonable to me, and are what I would suggest if we\n> were starting from scratch. My only hesitation is whether people are\n> using the weird behavior of --object-dir in the wild (e.g., are bup\n> folks relying on it).\n> \n> Johannes, is this something you're using _now_, and it works, or\n> something you hoped to use in the future?\n\nI was \"hoping\" to use\n\n\tgit multi-pack-index --object-dir=... write\n\nbut never\n\n\t$ git multi-pack-index write --object-dir=...\n\nwhich almost seems like it really is more like\n\n\t$ git -C ... multi-pack-index write\n\nanyway, because you specify a repo? At least per the above example, I\nnever tried.\n\n\nAs I started playing with that again (I had done before, and it worked)\nI noticed the segfault, hence my previous patch.\n\n\nHowever, what I was thinking of doing is more outlined in this thread:\nhttps://lore.kernel.org/git/20210820195558.44275-1-johannes@sipsolutions.net/\n\n\nAnd essentially, as I described later in\nhttps://lore.kernel.org/git/dbb24573efc3dd945acd8acdfd9fe627ad7cbcd2.camel@sipsolutions.net/\n\nI have two only vaguely overlapping use cases.\n\nOne of them doesn't need \"--object-dir\", and the other requires that\n[RFC PATCH] to be applied as well, which would basically let me use only\nthe small subset of git that is \"git multi-pack-index\" as machinery to\n*just* do indexing, *without* really ever having a real \"repository\"\nthat git could otherwise operate on and worry about the actual objects\netc.\n\nI might resend that with the code style issues fixed, but the objects\nseemed more fundamental.\n\n> But what I'm wondering is whether using --object-dir from outside a repo\n> entirely is actually something that even works. I.e., would we be\n> disabling a behavior that was not intended, but does happen to work? Or\n> are we closing off a possibly buggy and half-working part of the system?\n\nWell, it does work now, modulo the segfault, but that is actually a very\nrecent addition, I'd tried this before :)\n\njohannes\n\n"},{"id":"433859","messageId":"YSfiJmYMPPyEueUG@nand.local","threadId":"55464","inReplyTo":"YSXy73lWKteiuY6s@coredump.intra.peff.net","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-26T18:49:10Z","receivedAt":"2021-08-26T18:49:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 25, 2021 at 03:36:15AM -0400, Jeff King wrote:\n> On Tue, Aug 24, 2021 at 10:10:12PM -0400, Taylor Blau wrote:\n>\n> > > It's not clear to me that we have a resolution on whether calling \"cd ..\n> > > && git multi-pack-index write --object-dir repo.git\" is supposed to\n> > > work.\n> >\n> > My recommendation would be to do the following things, all in a reroll\n> > of this series:\n> >\n> > - Fix the bug by which we would delete a .rev or .bitmap file out of a\n> > different object store than we were working in (when the caller\n> > passes `--object-dir`).\n> >\n> > - Disallow running `git multi-pack-index` outside of a Git repository.\n> >\n> > - Restrict `--object-dir` to only work with alternates of the\n> > repository in the current working directory.\n> >\n> > To me, that seems like both the least-surprising behavior, and what\n> > would lend itself to the easiest implementation. I would probably argue\n> > that the existing behavior (where `--object-dir` would work against\n> > arbitrary repositories) is a bug, and shouldn't continue to be\n> > supported.\n>\n> All of those seem reasonable to me, and are what I would suggest if we\n> were starting from scratch. My only hesitation is whether people are\n> using the weird behavior of --object-dir in the wild (e.g., are bup\n> folks relying on it).\n>\n> Johannes, is this something you're using _now_, and it works, or\n> something you hoped to use in the future?\n\nI did some research[1] on what parts of `--object-dir` have worked (and not\nworked) in the past, and came to the conclusion that although this\nbehavior is surprising, we do bear the responsibility of continuing to\nmaintain it.\n\nAnd in that sense, I agree with your \"only call close_object_store() if\nthe MIDX we are using came from the object store, or otherwise call\nclose_midx() if it didn't\", so that's what I did in the\ntb/multi-pack-bitmaps branch of my fork[2].\n\nI think that this is the most reasonable path forward, since it resolves\nJohannes' concerns while also not breaking any existing functionality in\nthe meantime as we add new features on top. It has the added benefit of\nclosing some holes that were open in the past, so I think that it's\nworth doing.\n\nBefore I drop 27 patches onto the inboxes of list subscribers, would you\nmind taking a look at [1] (and the rest of the patches in [2]) to make\nsure that you're OK with the approach too?\n\nThanks,\nTaylor\n\n[1]: https://github.com/ttaylorr/git/commit/a24290489c2b30f3caed7e33fe8f85226a12778f\n[2]: https://github.com/ttaylorr/git/compare/tb/multi-pack-bitmaps\n"},{"id":"433881","messageId":"YSgGBxh24UAZR5X3@nand.local","threadId":"55464","inReplyTo":"YSfiJmYMPPyEueUG@nand.local","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-26T21:22:15Z","receivedAt":"2021-08-26T21:22:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 26, 2021 at 02:49:10PM -0400, Taylor Blau wrote:\n> On Wed, Aug 25, 2021 at 03:36:15AM -0400, Jeff King wrote:\n> > On Tue, Aug 24, 2021 at 10:10:12PM -0400, Taylor Blau wrote:\n> >\n> > > > It's not clear to me that we have a resolution on whether calling \"cd ..\n> > > > && git multi-pack-index write --object-dir repo.git\" is supposed to\n> > > > work.\n> > >\n> > > My recommendation would be to do the following things, all in a reroll\n> > > of this series:\n> > >\n> > > - Fix the bug by which we would delete a .rev or .bitmap file out of a\n> > > different object store than we were working in (when the caller\n> > > passes `--object-dir`).\n> > >\n> > > - Disallow running `git multi-pack-index` outside of a Git repository.\n> > >\n> > > - Restrict `--object-dir` to only work with alternates of the\n> > > repository in the current working directory.\n> > >\n> > > To me, that seems like both the least-surprising behavior, and what\n> > > would lend itself to the easiest implementation. I would probably argue\n> > > that the existing behavior (where `--object-dir` would work against\n> > > arbitrary repositories) is a bug, and shouldn't continue to be\n> > > supported.\n> >\n> > All of those seem reasonable to me, and are what I would suggest if we\n> > were starting from scratch. My only hesitation is whether people are\n> > using the weird behavior of --object-dir in the wild (e.g., are bup\n> > folks relying on it).\n> >\n> > Johannes, is this something you're using _now_, and it works, or\n> > something you hoped to use in the future?\n>\n> I did some research[1] on what parts of `--object-dir` have worked (and not\n> worked) in the past, and came to the conclusion that although this\n> behavior is surprising, we do bear the responsibility of continuing to\n> maintain it.\n\nHmm. Upon thinking on in more, here is some evidence to the contrary.\nThe new test, specifically this snippet:\n\n git init repo &&\n test_when_finished \"rm -fr repo\" &&\n (\n cd repo &&\n test_commit base &&\n git repack -d\n ) &&\n\n nongit git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write\n\nwill fail with GIT_TEST_DEFAULT_HASH=sha256, since the MIDX internals\nsettle on the hash size via `the_hash_algo` which doesn't respect the\nhash algorithm used by the target repository.\n\nAnd that seems like it never could have worked. Try this at your shell\nto observe the failure:\n\n git init --object-format=sha256 repo &&\n git -C repo commit --allow-empty -m initial &&\n git -C repo repack -d &&\n\n git multi-pack-index write --object-dir=$(pwd)/repo/.git/objects\n\nand get:\n\n error: wrong index v2 file size in\n /home/ttaylorr/repo/.git/objects/pack/pack-9f08dc78ae6f37407a5acad69e3fdf5a1887eb7da5c043a1ddedc56ea7160814.idx\n warning: failed to open pack-index\n '/home/ttaylorr/repo/.git/objects/pack/pack-9f08dc78ae6f37407a5acad69e3fdf5a1887eb7da5c043a1ddedc56ea7160814.idx'\n\nsince we're trying to open a sha256 index with the_hash_algo in\nsha1-mode.\n\nThe question is do we consider this to be a bug in the existing behavior\nthat we should patch, or an indication that the feature shouldn't exist\nin the first place?\n\nI think that I tend to agree more with the latter, so I'm inclined to\ndrop support for it (where \"it\" is running the midx command outside of a\nrepository) in this series (i.e., by making the midx builtin have the\nRUN_SETUP flag instead of RUN_SETUP_GENTLY).\n\nThoughts?\n\nThanks,\nTaylor\n"},{"id":"433903","messageId":"xmqqo89jbf49.fsf@gitster.g","threadId":"55464","inReplyTo":"YSVuUYFh7lmhNlEy@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-27T06:01:26Z","receivedAt":"2021-08-27T06:01:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Tue, Aug 24, 2021 at 03:06:55PM -0700, Junio C Hamano wrote:\n>> Junio C Hamano writes:\n>>\n>> > FWIW, here is what I have somewhere in 'seen' where two topics meet.\n>>\n>> Oops, one change missed.\n>\n> Thanks; that matches my own resolution. I noticed that it does fail the\n> new test in t5319, since writing a MIDX wants to make sure that we are\n> only touching an alternate's object directory (which will fail if we are\n> running `git multi-pack-index` from outside of a repository).\n>\n> My opinion is that we should require being inside of a repository to run\n> the MIDX builtin. Otherwise we're allowing that command to modify any\n> old MIDX, which doesn't make sense.\n>\n> I think we probably need a single unifying topic, so I'm happy if you\n> want to discard one of our two topics from seen in the meantime.\n\nIt seems that the *.rev test (probably added by the other topic that\nis a single patch fix) fails under sha256 hash. I am not going to\ndig it any further myself, but for the interested, CI breakage is\nhere:\n\n https://github.com/git/git/runs/3440068613?check_suite_focus=true#step:5:1219\n\nThanks.\n\n"},{"id":"433940","messageId":"YSko4OwwPb7MwEMa@nand.local","threadId":"55464","inReplyTo":"xmqqo89jbf49.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-27T18:03:12Z","receivedAt":"2021-08-27T18:03:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 26, 2021 at 11:01:26PM -0700, Junio C Hamano wrote:\n> It seems that the *.rev test (probably added by the other topic that\n> is a single patch fix) fails under sha256 hash. I am not going to\n> dig it any further myself, but for the interested, CI breakage is\n> here:\n>\n> https://github.com/git/git/runs/3440068613?check_suite_focus=true#step:5:1219\n>\n> Thanks.\n\nI saw the same error myself when integrating that patch into my series.\nI discussed it more in [1], but the failure is basically caused by the\nmidx code using the_hash_algo even when operating in a different\nrepository via --object-dir.\n\nIf the_hash_algo doesn't match (as is the case when using `--object-dir`\nto point at a SHA-256 repository when invoking the builtin from a\nrepository using SHA-1 or outside of a repository altogether), then\nwe'll fail when trying to open the pack indexes.\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/YSgGBxh24UAZR5X3@nand.local/\n"},{"id":"433967","messageId":"YSlZkMhD1vlc/48i@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YSgGBxh24UAZR5X3@nand.local","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-27T21:30:56Z","receivedAt":"2021-08-27T21:30:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 26, 2021 at 05:22:15PM -0400, Taylor Blau wrote:\n\n> > I did some research[1] on what parts of `--object-dir` have worked (and not\n> > worked) in the past, and came to the conclusion that although this\n> > behavior is surprising, we do bear the responsibility of continuing to\n> > maintain it.\n> \n> Hmm. Upon thinking on in more, here is some evidence to the contrary.\n> The new test, specifically this snippet:\n> \n> git init repo &&\n> test_when_finished \"rm -fr repo\" &&\n> (\n> cd repo &&\n> test_commit base &&\n> git repack -d\n> ) &&\n> \n> nongit git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write\n> \n> will fail with GIT_TEST_DEFAULT_HASH=sha256, since the MIDX internals\n> settle on the hash size via `the_hash_algo` which doesn't respect the\n> hash algorithm used by the target repository.\n\nYeah, I think this is a good example of the class of things that might\nfail: anything that requires the repo config to behave correctly.\n\nI do think the hash format is somewhat unusual here. Most of the changes\nto the on-disk files are reflected in the files themselves (e.g., pack\nindex v2 is chosen by config at _write_ time, but readers can interpret\nthe file stand-alone).\n\nThere may be other config that could influence the writing of the midx,\nand we'd skip it in this kind of non-repo setup. An example here is\nrepack.usedeltabaseoffset, which midx_repack() tries to respect.\nIgnoring that doesn't produce a nonsense result, but it doesn't follow\nwhat would happen if run from inside the repo.\n\nThe other class of problems I'd expect is where part of the midx\noperation needs to look at other parts of the repo. Bitmap generation is\nan obvious one there, since we'd want to look at refs to find the\nreachable tips. Now obviously that's a new feature we're trying to\nintroduce here, so it can't be an existing breakage. But it does make me\nwonder what other problems might be lurking.\n\nSo I dunno. Even if it mostly works now, I'm not sure it's something\nthat I'm all that happy about supporting going forward. It seems like a\nrecipe for subtle bugs where the midx code calls into other library code\nthat assumes that it can look at the repository struct.\n\n-Peff\n"},{"id":"434031","messageId":"xmqq8s0j98jz.fsf@gitster.g","threadId":"55464","inReplyTo":"YSlZkMhD1vlc/48i@coredump.intra.peff.net","subject":"Re: [PATCH v4 00/25] multi-pack reachability bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-29T22:42:56Z","receivedAt":"2021-08-29T22:43:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> The new test, specifically this snippet:\n>> \n>> git init repo &&\n>> test_when_finished \"rm -fr repo\" &&\n>> (\n>> cd repo &&\n>> test_commit base &&\n>> git repack -d\n>> ) &&\n>> \n>> nongit git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write\n>> \n>> will fail with GIT_TEST_DEFAULT_HASH=sha256, since the MIDX internals\n>> settle on the hash size via `the_hash_algo` which doesn't respect the\n>> hash algorithm used by the target repository.\n>\n> Yeah, I think this is a good example of the class of things that might\n> fail: anything that requires the repo config to behave correctly.\n>\n> I do think the hash format is somewhat unusual here. Most of the changes\n> to the on-disk files are reflected in the files themselves (e.g., pack\n> index v2 is chosen by config at _write_ time, but readers can interpret\n> the file stand-alone).\n>\n> There may be other config that could influence the writing of the midx,\n> and we'd skip it in this kind of non-repo setup. An example here is\n> repack.usedeltabaseoffset, which midx_repack() tries to respect.\n> Ignoring that doesn't produce a nonsense result, but it doesn't follow\n> what would happen if run from inside the repo.\n>\n> The other class of problems I'd expect is where part of the midx\n> operation needs to look at other parts of the repo. Bitmap generation is\n> an obvious one there, since we'd want to look at refs to find the\n> reachable tips. Now obviously that's a new feature we're trying to\n> introduce here, so it can't be an existing breakage. But it does make me\n> wonder what other problems might be lurking.\n>\n> So I dunno. Even if it mostly works now, I'm not sure it's something\n> that I'm all that happy about supporting going forward. It seems like a\n> recipe for subtle bugs where the midx code calls into other library code\n> that assumes that it can look at the repository struct.\n\nI tend to agree that we should first disallow things that are not\nwhat we know we definitely need, and the non-repo setup is something\nwe would want to punt on to make sure we have a solid support for\nthe mainstream usecase.\n\nThanks.\n"},{"id":"434032","messageId":"xmqq4kb797xc.fsf@gitster.g","threadId":"55464","inReplyTo":"YSko4OwwPb7MwEMa@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-29T22:56:31Z","receivedAt":"2021-08-29T22:56:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Thu, Aug 26, 2021 at 11:01:26PM -0700, Junio C Hamano wrote:\n>> It seems that the *.rev test (probably added by the other topic that\n>> is a single patch fix) fails under sha256 hash. I am not going to\n>> dig it any further myself, but for the interested, CI breakage is\n>> here:\n>>\n>> https://github.com/git/git/runs/3440068613?check_suite_focus=true#step:5:1219\n>>\n>> Thanks.\n>\n> I saw the same error myself when integrating that patch into my series.\n> I discussed it more in [1], but the failure is basically caused by the\n> midx code using the_hash_algo even when operating in a different\n> repository via --object-dir.\n>\n> If the_hash_algo doesn't match (as is the case when using `--object-dir`\n> to point at a SHA-256 repository when invoking the builtin from a\n> repository using SHA-1 or outside of a repository altogether), then\n> we'll fail when trying to open the pack indexes.\n\nMy recollection is that \"--object-dir\" is mostly about the alternate\nodb usecase---am I correct? It is unfortunate that we didn't start\nwith \"alternate repository\" and said \"we only care about the objects\nin the object store they have, and we do not have to care what refs\nthey point into their object database or what configuration they\nhave\" instead.\n\nI wonder if it is safe to assume that in practice a directory given\nto the \"--object-dir\" option is always the \"objects\" subdirectory in\na repository, and it is an error if there is no \"config\" file next\nto the directory. Then, we could check ../config relative to the\ngiven directory and error out if they use different hash.\n\nI do not recall offhand how careful link_alt_odb_entries() is, but I\nsuspect it isn't at all (back when I invented it, there weren't need\nfor configuration to switch between hashes, and since then I do not\nrecall seeing any heavy update to the alternate odb code). Perhaps\nwe should tighten it so that we check the accompanying \"config\" file\nfirst and ignore the entry with incompatible \"hash\" (and we may\nlater discover other trait on a repository that is incompatible with\nthe current one)?\n\nThanks.\n\n\n\n\n"},{"id":"434036","messageId":"YSwhNxqAS8JajA7p@nand.local","threadId":"55464","inReplyTo":"xmqq4kb797xc.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-30T00:07:19Z","receivedAt":"2021-08-30T00:08:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Aug 29, 2021 at 03:56:31PM -0700, Junio C Hamano wrote:\n> My recollection is that \"--object-dir\" is mostly about the alternate\n> odb usecase---am I correct?\n\nThat matches my understanding. The documentation refers to the value of\nthis flag as ``, making me think that supporting non-alternates is\na historical accident.\n\n> I wonder if it is safe to assume that in practice a directory given\n> to the \"--object-dir\" option is always the \"objects\" subdirectory in\n> a repository, and it is an error if there is no \"config\" file next\n> to the directory. Then, we could check ../config relative to the\n> given directory and error out if they use different hash.\n\nMaybe... although I have to admit to not being very excited about it. Is\nthe idea to read ../config to try and check for any incompatibilities\nbetween the in-core state and the target repository's settings? If so,\nthis seems like a recipe for catching bugs too late.\n\nFor e.g., catching the_hash_algo != target_repository->hash would\ndefinitely squash the bug you saw when integrating, but we would have to\nremember to update this spot later on if, say, the target repository\nstarted using a different reference storage backend (since bitmap\ngeneration necessarily iterates the references to figure out which\ncommits should receive coverage).\n\n> I do not recall offhand how careful link_alt_odb_entries() is, but I\n> suspect it isn't at all (back when I invented it, there weren't need\n> for configuration to switch between hashes, and since then I do not\n> recall seeing any heavy update to the alternate odb code). Perhaps\n> we should tighten it so that we check the accompanying \"config\" file\n> first and ignore the entry with incompatible \"hash\" (and we may\n> later discover other trait on a repository that is incompatible with\n> the current one)?\n\nOr are you saying you're concerned about an alternates chain which\ndon't all use the same object format?\n\nIf the former, then I would say:\n\n \"Supporting arbitrary --object-dir when invoked from outside a\n repository is a bug that happened to not cause any problems, but\n is surprising, error-prone, and should fall outside of the burden of\n backwards compatibility, so we should get rid of it.\"\n\nIf the latter, then I agree we could and should do better at detecting\nit and providing a helpful error message, but I don't see how doing so\nnow or later would affect this series. Even if we just disallow\n--object-dir pointing at a non-alternate repository, we would still have\nthe issue of having alternate chains which don't all have the same\nobject format.\n\nSo that makes me feel like the latter is a problem outside of this\nseries that can be dealt with later.\n\nI'm admittedly a little unsure of how to progress here. Given that this\nseries has received positive review over the complicated parts, it seems\nthat it is getting stuck on how to deal with `--object-dir`, especially\nwhen invoked outside of a Git repository. My inclination would be to\nsend a new version that simply requires the MIDX builtin to be run from\nwithin a repository (as well as the cleanups from Johannes).\n\nDoes that seem like a good direction forward to you? If not, let me know\nif there's another issue that we should deal with first and I'd be happy\nto start there.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"434042","messageId":"xmqqfsur7otx.fsf@gitster.g","threadId":"55464","inReplyTo":"YSwhNxqAS8JajA7p@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-30T00:34:18Z","receivedAt":"2021-08-30T00:34:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> now or later would affect this series. Even if we just disallow\n> --object-dir pointing at a non-alternate repository, we would still have\n> the issue of having alternate chains which don't all have the same\n> object format.\n\nExactly. That is why I feel that it probably needs to be dealt with\nbefore doing anything else. The alternate mechanism pulling in an\nobject store that uses incompatible hash algo would break not just\nthe multi-pack-index but probably the basic object access layer as\nwell, which would be more grave problem, no?\n\n> My inclination would be to\n> send a new version that simply requires the MIDX builtin to be run from\n> within a repository (as well as the cleanups from Johannes).\n\nSounds like a good first step.\n"},{"id":"434044","messageId":"YSwpsp/hQsPFnj+I@nand.local","threadId":"55464","inReplyTo":"xmqqfsur7otx.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-30T00:43:30Z","receivedAt":"2021-08-30T00:43:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Aug 29, 2021 at 05:34:18PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > now or later would affect this series. Even if we just disallow\n> > --object-dir pointing at a non-alternate repository, we would still have\n> > the issue of having alternate chains which don't all have the same\n> > object format.\n>\n> Exactly. That is why I feel that it probably needs to be dealt with\n> before doing anything else. The alternate mechanism pulling in an\n> object store that uses incompatible hash algo would break not just\n> the multi-pack-index but probably the basic object access layer as\n> well, which would be more grave problem, no?\n\nYeah; it does. Maybe I'm holding it wrong (and brian, cc'd, can help\nme), but this is an easy way to see the problem:\n\n git init repo\n git init alternate\n\n git -C repo commit --allow-empty -m foo\n ( cd repo/.git/objects && pwd ) >alternate/.git/objects/info/alternates\n git -C alternate rev-list --objects --alternate-refs\n\nwhich will produce:\n\n $ git rev-list --objects --alternate-refs\n warning: invalid line while parsing alternate refs: \n\nBut I don't know if I quite understand your \"probably needs to be dealt\nwith before doing anything else\". I think we can proceed with this\nseries and deal with the alternate object-format thing separately, no?\n\nThanks,\nTaylor\n"},{"id":"434199","messageId":"YS1XOMtj94BcI9HM@camp.crustytoothpaste.net","threadId":"55464","inReplyTo":"YSwpsp/hQsPFnj+I@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-08-30T22:10:00Z","receivedAt":"2021-08-30T22:11:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-08-30 at 00:43:30, Taylor Blau wrote:\n> On Sun, Aug 29, 2021 at 05:34:18PM -0700, Junio C Hamano wrote:\n> > Taylor Blau writes:\n> >\n> > > now or later would affect this series. Even if we just disallow\n> > > --object-dir pointing at a non-alternate repository, we would still have\n> > > the issue of having alternate chains which don't all have the same\n> > > object format.\n> >\n> > Exactly. That is why I feel that it probably needs to be dealt with\n> > before doing anything else. The alternate mechanism pulling in an\n> > object store that uses incompatible hash algo would break not just\n> > the multi-pack-index but probably the basic object access layer as\n> > well, which would be more grave problem, no?\n> \n> Yeah; it does. Maybe I'm holding it wrong (and brian, cc'd, can help\n> me), but this is an easy way to see the problem:\n> \n> git init repo\n> git init alternate\n> \n> git -C repo commit --allow-empty -m foo\n> ( cd repo/.git/objects && pwd ) >alternate/.git/objects/info/alternates\n> git -C alternate rev-list --objects --alternate-refs\n> \n> which will produce:\n> \n> $ git rev-list --objects --alternate-refs\n> warning: invalid line while parsing alternate refs: \n> \n> But I don't know if I quite understand your \"probably needs to be dealt\n> with before doing anything else\". I think we can proceed with this\n> series and deal with the alternate object-format thing separately, no?\n\nYeah, this is a possible problem. You can also see it when using git\nindex-pack outside of a repository with an incorrect --object-format\noption.\n\nI'm not sure how folks want to deal with that; I'm just fine saying,\n\"Well, don't do that,\" but other folks may have different opinions.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"434201","messageId":"xmqqmtoy1s9s.fsf@gitster.g","threadId":"55464","inReplyTo":"YS1XOMtj94BcI9HM@camp.crustytoothpaste.net","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-30T22:28:47Z","receivedAt":"2021-08-30T22:28:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> Yeah, this is a possible problem. You can also see it when using git\n> index-pack outside of a repository with an incorrect --object-format\n> option.\n>\n> I'm not sure how folks want to deal with that; I'm just fine saying,\n> \"Well, don't do that,\" but other folks may have different opinions.\n\nOK, so if we go back to the original breakage of the test script\nthat triggered this discussion, the right solution would be to make\nsure both test repositories/object stores are prepared with the\nalgorithm specified with GIT_TEST_DEFAULT_HASH?\n\nThanks.\n"},{"id":"434202","messageId":"YS1croR3etCfMQhR@nand.local","threadId":"55464","inReplyTo":"xmqqmtoy1s9s.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-30T22:33:18Z","receivedAt":"2021-08-30T22:33:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Aug 30, 2021 at 03:28:47PM -0700, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n>\n> > Yeah, this is a possible problem. You can also see it when using git\n> > index-pack outside of a repository with an incorrect --object-format\n> > option.\n> >\n> > I'm not sure how folks want to deal with that; I'm just fine saying,\n> > \"Well, don't do that,\" but other folks may have different opinions.\n>\n> OK, so if we go back to the original breakage of the test script\n> that triggered this discussion, the right solution would be to make\n> sure both test repositories/object stores are prepared with the\n> algorithm specified with GIT_TEST_DEFAULT_HASH?\n\nJust to make sure do you still see this as a separate issue from running\nthe midx builtin outside of a repository?\n\nI.e., if we require the midx builtin to be run in a repository, it\nside-steps this issue (but presumably not completely, and so we should\ndeal with both eventually). I want to make sure that I'm on the same\npage before I drop 25+ emails on the list.\n\nThanks,\nTaylor\n"},{"id":"434214","messageId":"22366f81-65a6-55d1-706c-59f877127be0@gmail.com","threadId":"55464","inReplyTo":"YSwhNxqAS8JajA7p@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-31T01:21:31Z","receivedAt":"2021-08-31T01:21:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/29/21 8:07 PM, Taylor Blau wrote:\n> On Sun, Aug 29, 2021 at 03:56:31PM -0700, Junio C Hamano wrote:\n>> My recollection is that \"--object-dir\" is mostly about the alternate\n>> odb usecase---am I correct?\n> \n> That matches my understanding. The documentation refers to the value of\n> this flag as ``, making me think that supporting non-alternates is\n> a historical accident.\n\nYes, supporting non-alternates is a historical accident. Supporting\nalternates that are not actually the core object database of a full\nrepository is on purpose.\n\nSo, hopefully the remaining discussion that I am seeing can be\nsolved by a decision such as:\n\n \"If we add the restriction that the builtin always runs with a\n repository and --object-dir always points to its objects dir\n or one of its registered alternates, then we have access to a\n local config file to learn how to interpret that object directory.\"\n\n>> I wonder if it is safe to assume that in practice a directory given\n>> to the \"--object-dir\" option is always the \"objects\" subdirectory in\n>> a repository, and it is an error if there is no \"config\" file next\n>> to the directory. Then, we could check ../config relative to the\n>> given directory and error out if they use different hash.\n\nI would say that is not always the case, and we should not error out.\n\nI think taking a look to see if ../config exists to use the data\nmight be helpful for some cases, but should not be a blocker for\ncompleting the requested operation. The config from the non-alternate\nrepo should be sufficient for this (somewhat strange) case.\n\n> I'm admittedly a little unsure of how to progress here. Given that this\n> series has received positive review over the complicated parts, it seems\n> that it is getting stuck on how to deal with `--object-dir`, especially\n> when invoked outside of a Git repository. My inclination would be to\n> send a new version that simply requires the MIDX builtin to be run from\n> within a repository (as well as the cleanups from Johannes).\n> \n> Does that seem like a good direction forward to you? If not, let me know\n> if there's another issue that we should deal with first and I'd be happy\n> to start there.\n\nI think it is sensible to restrict 'git multi-pack-index' to run\ninside a repository on its own merits. It happens to also solve\nsome tricky problems that have come up since its creation.\n\nSorry I'm so late to this thread. I gave most of the messages in this\nchain a quick read and this seemed like the best place to chime in.\nHopefully this isn't too much of a re-tread of things covered elsewhere.\n\nThanks,\n-Stolee\n"},{"id":"434231","messageId":"YS271RmGhwne0iTm@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YS1croR3etCfMQhR@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-31T05:19:17Z","receivedAt":"2021-08-31T05:19:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 30, 2021 at 06:33:18PM -0400, Taylor Blau wrote:\n\n> On Mon, Aug 30, 2021 at 03:28:47PM -0700, Junio C Hamano wrote:\n> > \"brian m. carlson\" writes:\n> >\n> > > Yeah, this is a possible problem. You can also see it when using git\n> > > index-pack outside of a repository with an incorrect --object-format\n> > > option.\n> > >\n> > > I'm not sure how folks want to deal with that; I'm just fine saying,\n> > > \"Well, don't do that,\" but other folks may have different opinions.\n> >\n> > OK, so if we go back to the original breakage of the test script\n> > that triggered this discussion, the right solution would be to make\n> > sure both test repositories/object stores are prepared with the\n> > algorithm specified with GIT_TEST_DEFAULT_HASH?\n> \n> Just to make sure do you still see this as a separate issue from running\n> the midx builtin outside of a repository?\n\nAdding my two cents: yes, I think it most definitely should be a\nseparate issue. As you demonstrated, differing config between alternates\nand repos that point to them is not specific to the midx code. I agree\nwith brian's \"well, don't do that\". But _if_ we want to try to behave\nbetter in such a case, whatever we changes we make would then naturally\napply to the midx code as well.\n\nThe two midx-specific things we have to care about are:\n\n - is it OK for the midx command to refuse to operate when we are not\n in a repository at all? I think yes; we can't even know which hash\n is being used, along with who knows what other lurking\n complications.\n\n - is it OK to restrict the midx command's --object-dir to only operate\n on a directory which is an alternate of the current repo? I think\n yes again. If it _isn't_ related, we have all the lurking problems\n from the first point, but even worse (because we use config, refs,\n and other information from our current repo with the _totally\n unrelated_ object dir).\n\nSo I'm all in favor of locking those down now before things get any more\ncomplicated. If we later want to make the object store more aware of of\ndifferences between alternates and the main store (like say, the object\nhash in use), then we could consider loosening using the same mechanism.\n\n-Peff\n"},{"id":"434232","messageId":"YS3AKhQJjMrFm1JO@coredump.intra.peff.net","threadId":"55464","inReplyTo":"22366f81-65a6-55d1-706c-59f877127be0@gmail.com","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-31T05:37:46Z","receivedAt":"2021-08-31T05:37:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 30, 2021 at 09:21:31PM -0400, Derrick Stolee wrote:\n\n> Yes, supporting non-alternates is a historical accident. Supporting\n> alternates that are not actually the core object database of a full\n> repository is on purpose.\n> \n> So, hopefully the remaining discussion that I am seeing can be\n> solved by a decision such as:\n> \n> \"If we add the restriction that the builtin always runs with a\n> repository and --object-dir always points to its objects dir\n> or one of its registered alternates, then we have access to a\n> local config file to learn how to interpret that object directory.\"\n\nI left a similar comment in the other part of the thread. :)\n\n> >> I wonder if it is safe to assume that in practice a directory given\n> >> to the \"--object-dir\" option is always the \"objects\" subdirectory in\n> >> a repository, and it is an error if there is no \"config\" file next\n> >> to the directory. Then, we could check ../config relative to the\n> >> given directory and error out if they use different hash.\n> \n> I would say that is not always the case, and we should not error out.\n> \n> I think taking a look to see if ../config exists to use the data\n> might be helpful for some cases, but should not be a blocker for\n> completing the requested operation. The config from the non-alternate\n> repo should be sufficient for this (somewhat strange) case.\n\nYes, agreed. We have long supported these kind of \"bare\" alternates, and\nI wouldn't be surprised if they are in wide use (though I do wonder how\nfolks actually modify them, since most commands that touch objects\nreally do want to be in a repository).\n\nIn other cases where we may benefit from their being a containing repo\n(e.g., accessing the ref tips of the alternate), we speculatively look\nat \"..\" and see if there are any refs. See refs_from_alternate_cb()[0].\n\nThe natural extension for the hash-format problem would probably be to\ncall check_repository_format_gently() on the parent directory of the\nalternate-objects dir. If it succeeds, then we can pull out the\nhash_algo parameter from its repository_format struct. And if not, then\nwe just assume it matches the main repo.\n\nBut I suspect all of this is moot for now, beyond being able to return a\nnicer error message. The rest of the code is not at all ready to handle\npacks with two different hashes in the same process. And I suspect it\nwould take a reasonable amount of refactoring to make it so. If somebody\nwants to work on that, I won't stop them, but I kind of doubt it is\nworth anybody's time.\n\n[0] Looking at refs_from_alternate_cb(), I did wonder if it would work\n at all with a reftable alternate, but I suspect it would. I think we\n ended up still having a \"refs/\" directory in that case, so we'd\n recognize it as a repo (though really, it ought to be using\n is_git_directory() instead of its hacky check). And then we farm out\n the actual ref iteration to a separate for-each-ref process, passing\n along --git-dir, which will read that alternate repo's config. So it\n should Just Work, even with a different ref backend. It's almost\n certainly broken if the hash algorithms don't match, though, because\n we'd get oddly sized results from for-each-ref's output.\n\n That's all just interesting tangent, though. :)\n\n-Peff\n"},{"id":"434299","messageId":"xmqqmtoxwpad.fsf@gitster.g","threadId":"55464","inReplyTo":"YS1croR3etCfMQhR@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-31T16:29:46Z","receivedAt":"2021-08-31T16:29:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Mon, Aug 30, 2021 at 03:28:47PM -0700, Junio C Hamano wrote:\n>> \"brian m. carlson\" writes:\n>>\n>> > Yeah, this is a possible problem. You can also see it when using git\n>> > index-pack outside of a repository with an incorrect --object-format\n>> > option.\n>> >\n>> > I'm not sure how folks want to deal with that; I'm just fine saying,\n>> > \"Well, don't do that,\" but other folks may have different opinions.\n>>\n>> OK, so if we go back to the original breakage of the test script\n>> that triggered this discussion, the right solution would be to make\n>> sure both test repositories/object stores are prepared with the\n>> algorithm specified with GIT_TEST_DEFAULT_HASH?\n>\n> Just to make sure do you still see this as a separate issue from running\n> the midx builtin outside of a repository?\n\nThey are separate issues, but the .midx issue has a small overlap\nwith the much bigger \"do not mix repositories and object stores with\ndifferent hashes\" issue.\n\nThe users of raw object stores (e.g. $GIT_OBJECT_DIRECTORIES,\n\"--object-dir\", there may be others) need to be updated so that the\ncode paths involved can reliably learn what hash algorithm is used\nand other traits that may not be available in the object store alone\n(e.g. refs might be relevant if the using code needs to learn which\nobjects are still reachable) for the latter. It would need a couple\nof things that are fairly isolated to solve, I would imagine:\n\n (1) convention to either tie a raw object store with its repository\n or declare a raw object store is unusable because \"other\n traits\" are not found for it.\n\n (2) given a repository, inspect it and decide if it is \"compatible\"\n with the current repository.\n\n (3) update code paths involved in prepare_alt_odb() to use (1) and\n (2) to inspect and reject incompatible object store as\n alternate.\n\nAnd once we have that, \"git multi-pack-index --object-dir=X\" can use\n(1) and (2) for the same \"Is this other object store compatible with\nthe current repository?\" check, no?\n\nThe other side of the coin is that midx needs to do equivalents of\n(1) and (2) anyway, and the required amount of the work for (3)\nsmells a lot smaller than work for (1) and (2). (3) may be just a\nmatter of \"add a call to is_odb_compatible(dir) for the directory\nbeing added as an alt odb\", and the same single validation call may\nbe all it needs on the --object-dir argument on the midx side.\n\nI think it makes sense for the midx command to require being in a\nrepository to run (to establish what \"the current repository\" is)\nand insist on the other object store given with --object-dir to be\n\"compatible\" with the current repository (i.e. the same hash\nalgorithm, there may be others). I am a bit fuzzy why we want it\nto be already our alternate.\n\nThanks.\n"},{"id":"434300","messageId":"xmqqk0k1wp3x.fsf@gitster.g","threadId":"55464","inReplyTo":"YS3AKhQJjMrFm1JO@coredump.intra.peff.net","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-31T16:33:38Z","receivedAt":"2021-08-31T16:33:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> I think taking a look to see if ../config exists to use the data\n>> might be helpful for some cases, but should not be a blocker for\n>> completing the requested operation. The config from the non-alternate\n>> repo should be sufficient for this (somewhat strange) case.\n>\n> Yes, agreed. We have long supported these kind of \"bare\" alternates, and\n> I wouldn't be surprised if they are in wide use (though I do wonder how\n> folks actually modify them, since most commands that touch objects\n> really do want to be in a repository).\n\nI kind of find the above two somewhat surprising, but I am willing\nto go with the less safer option if that is what people want.\n\nIt has been perfectly OK in the pre-alternative-hash-algorithms\nworld, but we no longer live in such a world, so we'd need to come\nup with a way to keep using alternates in a safer way.\n\nI do not see the reasoning behind \"should not be a blocker\" from\nDerrick substantiated. What's the reason why that raw object store\ncannot come from an existing repository, and what's the benefit we\nget from not having to have a repository there?\n\n> The natural extension for the hash-format problem would probably be to\n> call check_repository_format_gently() on the parent directory of the\n> alternate-objects dir. If it succeeds, then we can pull out the\n> hash_algo parameter from its repository_format struct. And if not, then\n> we just assume it matches the main repo.\n>\n> But I suspect all of this is moot for now, beyond being able to return a\n> nicer error message. The rest of the code is not at all ready to handle\n> packs with two different hashes in the same process.\n\nI do not think it is all that urgent to make it possible for packs\nwith different algorithms to be used. It is sufficient to _ignore_\n(or error out) configured odb that is incompatible with the current\nrepository.\n\nThanks.\n"},{"id":"434301","messageId":"YS5bWMcXbODi+KmS@nand.local","threadId":"55464","inReplyTo":"xmqqmtoxwpad.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T16:39:52Z","receivedAt":"2021-08-31T16:39:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 31, 2021 at 09:29:46AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > On Mon, Aug 30, 2021 at 03:28:47PM -0700, Junio C Hamano wrote:\n> >> \"brian m. carlson\" writes:\n> >>\n> >> > Yeah, this is a possible problem. You can also see it when using git\n> >> > index-pack outside of a repository with an incorrect --object-format\n> >> > option.\n> >> >\n> >> > I'm not sure how folks want to deal with that; I'm just fine saying,\n> >> > \"Well, don't do that,\" but other folks may have different opinions.\n> >>\n> >> OK, so if we go back to the original breakage of the test script\n> >> that triggered this discussion, the right solution would be to make\n> >> sure both test repositories/object stores are prepared with the\n> >> algorithm specified with GIT_TEST_DEFAULT_HASH?\n> >\n> > Just to make sure do you still see this as a separate issue from running\n> > the midx builtin outside of a repository?\n>\n> They are separate issues, but the .midx issue has a small overlap\n> with the much bigger \"do not mix repositories and object stores with\n> different hashes\" issue.\n\nOK, good. Everything you wrote below (which I snipped off in my reply)\nmakes sense to me, and seems like a worthwhile direction to pursue\noutside of this series, especially as more users start using sha256\nrepositories.\n\n> I think it makes sense for the midx command to require being in a\n> repository to run (to establish what \"the current repository\" is)\n> and insist on the other object store given with --object-dir to be\n> \"compatible\" with the current repository (i.e. the same hash\n> algorithm, there may be others). I am a bit fuzzy why we want it\n> to be already our alternate.\n\nI don't think there's any strict requirement to the other repository\nbeing our alternate, other than touching arbitrary repositories is a\nsurprising behavior that appears (to me, at least) to be inconsistent\nwith the rest of Git.\n\nAfter (the rerolled version of) this series, we'll be in a state where:\n\n - `git multi-pack-index` will not run when outside of a Git\n repository.\n - The `--object-dir` argument will only recognize object directories\n belonging to an alternate of the current repository.\n - Using `--object-dir` to point to a repository which uses a\n different hash than the repository in the current working directory\n will continue to not work (as was the case before this series).\n\nI think(?) that there is consensus for that approach, so patches\nincoming...\n\n> Thanks.\n\n(Thank you, by the way, for clarifying this all in so much detail. I\nwould much rather just have code to talk about, but it feels\nparticularly important to be on the same page beforehand in this\ninstance, since there is *so much* code, and this discussion is centered\naround so little of it).\n\nThanks,\nTaylor\n"},{"id":"434303","messageId":"YS5cJjlV0Rkpu49n@nand.local","threadId":"55464","inReplyTo":"xmqqk0k1wp3x.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T16:43:18Z","receivedAt":"2021-08-31T16:43:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 31, 2021 at 09:33:38AM -0700, Junio C Hamano wrote:\n> I do not see the reasoning behind \"should not be a blocker\" from\n> Derrick substantiated. What's the reason why that raw object store\n> cannot come from an existing repository, and what's the benefit we\n> get from not having to have a repository there?\n\nI also didn't find the reasoning spelled out in his response, but I have\ndefinitely had off-list discussions with Stolee where it was important to\nbe able to pass a value to `--object-dir` which does *not* belong to a\nGit repository (but is used as a dumping ground for packs, a MIDX, and\nloose objects).\n\nIt may be worthwhile to recapitulate that discussion here on the list.\n(I'm hoping that Stolee won't mind filling in the details, since I seem\nto have forgotten most of them).\n\nThanks,\nTaylor\n"},{"id":"434308","messageId":"d23bca9b-9da2-984f-065c-6cf60a80ddef@gmail.com","threadId":"55464","inReplyTo":"YS5cJjlV0Rkpu49n@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-31T17:17:33Z","receivedAt":"2021-08-31T17:17:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/31/2021 12:43 PM, Taylor Blau wrote:\n> On Tue, Aug 31, 2021 at 09:33:38AM -0700, Junio C Hamano wrote:\n>> I do not see the reasoning behind \"should not be a blocker\" from\n>> Derrick substantiated. What's the reason why that raw object store\n>> cannot come from an existing repository, and what's the benefit we\n>> get from not having to have a repository there?\n> \n> I also didn't find the reasoning spelled out in his response, but I have\n> definitely had off-list discussions with Stolee where it was important to\n> be able to pass a value to `--object-dir` which does *not* belong to a\n> Git repository (but is used as a dumping ground for packs, a MIDX, and\n> loose objects).\n> \n> It may be worthwhile to recapitulate that discussion here on the list.\n> (I'm hoping that Stolee won't mind filling in the details, since I seem\n> to have forgotten most of them).\n\nThe way we have been using alternates in VFS for Git and Scalar is as a\n\"shared object cache\" that is shared across multiple full Git repositories\nwith their own working trees. The shared object cache is located in a\nlocation that can be found during \"scalar clone\" such as\n\n\t~/.scalarCache/url_/\n\nThis directory contains the same data as a .git/objects directory would.\n\nData is added to that cache using hooks during 'git fetch' or other\nrequests for remote data. This means that the second \"scalar clone\"\ncommand is much faster than the first, because it already has most of\nthe commit and tree data required to satisfy the partial clone.\n\n(Note: this feature does not exist in the current Scalar CLI RFC, but\nwould be contributed later.)\n\nThese caches were designed before the multi-pack-index -- in fact,\nthey were an inspiration for them because now deleting a repo would not\nclean up old pack-files. The data would be added as a raw pack-file that\nis processed with 'git index-pack' or as loose objects. The --object-dir\noption was directly created as a way to target the creation and\nmaintenance of a multi-pack-index within one of these caches that don't\nexist as full repositories. Clearly, there were some gaps in that\nimplementation and I regret creating those gaps.\n\nIf I were to redesign the shared object cache, then I would have created\nthe cache directories as bare repos and then create the \"clone\" repo as\na worktree linked to that base. That would allow all objects and refs to\nbe shared, achieving the same goals and an even better user experience.\n\nI'm advocating for the position to continue allowing this feature to\nexist without a necessary on-upgrade conversion of these non-repos to\nfull repos. Maybe that is the best thing to do in the long-term, but\nwill take some time to do. Keeping compatibility for now seems like it\nwon't hurt too much.\n\nThanks,\n-Stolee\n"},{"id":"434311","messageId":"xmqq8s0hv795.fsf@gitster.g","threadId":"55464","inReplyTo":"YS5bWMcXbODi+KmS@nand.local","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-31T17:44:38Z","receivedAt":"2021-08-31T17:44:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> After (the rerolled version of) this series, we'll be in a state where:\n>\n> - `git multi-pack-index` will not run when outside of a Git\n> repository.\n> - The `--object-dir` argument will only recognize object directories\n> belonging to an alternate of the current repository.\n> - Using `--object-dir` to point to a repository which uses a\n> different hash than the repository in the current working directory\n> will continue to not work (as was the case before this series).\n\nHmph, re-reading the document for midx:\n\n --object-dir=::\n Use given directory for the location of Git objects. We check\n `/packs/multi-pack-index` for the current MIDX file, and\n `/packs` for the pack-files to index.\n\nwhy does it matter if we are in a repository in the first place?\nIt's not like we combine the objects from the specified object dir\nand our local object store (if that were the case, these two object\nstores must be compatible).\n\nHow old is --object-dir option and how widely is it used? Can we\njust remove it and have users go to the repository that uses it\nas its object store with \"git -C \" mechanism, or have we come\ntoo far with this (apparently broken) design to make such a fix\ninfeasible?\n\nThanks.\n"},{"id":"434319","messageId":"YS55m5touOZdzZ7g@nand.local","threadId":"55464","inReplyTo":"xmqq8s0hv795.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T18:48:59Z","receivedAt":"2021-08-31T18:49:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 31, 2021 at 10:44:38AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > After (the rerolled version of) this series, we'll be in a state where:\n> >\n> > - `git multi-pack-index` will not run when outside of a Git\n> > repository.\n> > - The `--object-dir` argument will only recognize object directories\n> > belonging to an alternate of the current repository.\n> > - Using `--object-dir` to point to a repository which uses a\n> > different hash than the repository in the current working directory\n> > will continue to not work (as was the case before this series).\n>\n> Hmph, re-reading the document for midx:\n>\n> --object-dir=::\n> Use given directory for the location of Git objects. We check\n> `/packs/multi-pack-index` for the current MIDX file, and\n> `/packs` for the pack-files to index.\n>\n> why does it matter if we are in a repository in the first place?\n> It's not like we combine the objects from the specified object dir\n> and our local object store (if that were the case, these two object\n> stores must be compatible).\n\nIt shouldn't matter, but the use-case is described in [1] by Stolee. He\nexplains it in detail, but I do think we have to live with\n`--object-dir` in one way or another. He does say it'd be OK to only\nbe able to invoke it from within a repository, and to only be able to\nreference alternates, though.\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/d23bca9b-9da2-984f-065c-6cf60a80ddef@gmail.com/\n"},{"id":"434336","messageId":"cover.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1617991824.git.me@ttaylorr.com","subject":"[PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:33Z","receivedAt":"2021-08-31T20:51:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another version of the multi-pack reachability bitmaps series. It is\nvirtually unchanged since last time.\n\nThe changes that did occur is that I integrated Johannes' patch from [1] to fix\ncleaning up MIDX .rev and .bitmap files when using `--object-dir`. That inspired\na lengthy discussion [2] about `--object-dir`, alternates, object-format and\nrunning the MIDX builtin outside of a Git repository.\n\nThis series resolves that discussion by leaving everything as-is, and only\nchanging the following:\n\n - `git multi-pack-index` will not run when outside of a Git\n repository.\n\n - The `--object-dir` argument will only recognize object directories\n belonging to an alternate of the current repository.\n\n - Using `--object-dir` to point to a repository which uses a\n different hash than the repository in the current working directory\n will continue to not work (as was the case before this series).\n\nAnd because this incorporates [1], we will also not accidentally clean `.rev`\nfiles from the wrong object directory.\n\nI think that this version is ready-to-go, and that we can turn our attention to\nsquashing some of these cross-alternate buglets, and integrating MIDX bitmaps\nwith `git repack`.\n\n[1]: https://lore.kernel.org/git/20210823171011.80588-1-johannes@sipsolutions.net/\n[2]: https://lore.kernel.org/git/YSVsHo2wLhnraBnv@nand.local/\n\nJeff King (2):\n t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n\nTaylor Blau (25):\n pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n pack-bitmap-write.c: free existing bitmaps\n Documentation: describe MIDX-based bitmaps\n midx: disallow running outside of a repository\n midx: fix `*.rev` cleanups with `--object-dir`\n midx: clear auxiliary .rev after replacing the MIDX\n midx: reject empty `--preferred-pack`'s\n midx: infer preferred pack when not given one\n midx: close linked MIDXs, avoid leaking memory\n midx: avoid opening multiple MIDXs when writing\n pack-bitmap.c: introduce 'bitmap_num_objects()'\n pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n pack-bitmap.c: avoid redundant calls to try_partial_reuse\n pack-bitmap: read multi-pack bitmaps\n pack-bitmap: write multi-pack bitmaps\n t5310: move some tests to lib-bitmap.sh\n t/helper/test-read-midx.c: add --checksum mode\n t5326: test multi-pack bitmap behavior\n t5319: don't write MIDX bitmaps in t5319\n t7700: update to work with MIDX bitmap test knob\n midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n p5310: extract full and partial bitmap tests\n p5326: perf tests for MIDX bitmaps\n\n Documentation/git-multi-pack-index.txt | 20 +-\n Documentation/technical/bitmap-format.txt | 71 ++-\n Documentation/technical/multi-pack-index.txt | 10 +-\n builtin/multi-pack-index.c | 2 +\n builtin/pack-objects.c | 8 +-\n builtin/repack.c | 12 +-\n ci/run-build-and-tests.sh | 1 +\n git.c | 2 +-\n midx.c | 328 ++++++++++--\n midx.h | 5 +\n pack-bitmap-write.c | 79 ++-\n pack-bitmap.c | 499 ++++++++++++++++---\n pack-bitmap.h | 9 +-\n packfile.c | 2 +-\n t/README | 4 +\n t/helper/test-read-midx.c | 16 +-\n t/lib-bitmap.sh | 240 +++++++++\n t/perf/lib-bitmap.sh | 69 +++\n t/perf/p5310-pack-bitmaps.sh | 65 +--\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++\n t/t0410-partial-clone.sh | 12 +-\n t/t5310-pack-bitmaps.sh | 231 +--------\n t/t5319-multi-pack-index.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 286 +++++++++++\n t/t7700-repack.sh | 18 +-\n 25 files changed, 1644 insertions(+), 441 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\nRange-diff against v4:\n 1: 92dc0bbc0d = 1: 7815d9929d pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps\n 2: 979276bc74 = 2: 629171115a pack-bitmap-write.c: gracefully fail to write non-closed bitmaps\n 3: 8f00493955 = 3: d469c1d8f6 pack-bitmap-write.c: free existing bitmaps\n 4: bc7db926d8 = 4: 158ff797c4 Documentation: describe MIDX-based bitmaps\n -: ---------- > 5: 5f24be8985 midx: disallow running outside of a repository\n -: ---------- > 6: 0aacaa9283 midx: fix `*.rev` cleanups with `--object-dir`\n 5: 771741844b ! 7: d30e6fe9a5 midx: clear auxiliary .rev after replacing the MIDX\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n --\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n +-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n -+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n ++\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n +\n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n 6: dab5dbf228 = 8: db2a24a8ae midx: reject empty `--preferred-pack`'s\n 7: 31f4517de0 = 9: 059c583e34 midx: infer preferred pack when not given one\n 8: aa3bd96d9b = 10: 6f5ca446f3 midx: close linked MIDXs, avoid leaking memory\n 9: c9fea31fa8 ! 11: 4656608f73 midx: avoid opening multiple MIDXs when writing\n @@ Commit message\n \n Signed-off-by: Taylor Blau \n \n + ## Documentation/git-multi-pack-index.txt ##\n +@@ Documentation/git-multi-pack-index.txt: OPTIONS\n + \tUse given directory for the location of Git objects. We check\n + \t`/packs/multi-pack-index` for the current MIDX file, and\n + \t`/packs` for the pack-files to index.\n +++\n ++`` must be an alternate of the current repository.\n + \n + --[no-]progress::\n + \tTurn progress on/off explicitly. If neither is specified, progress is\n +\n ## midx.c ##\n @@ midx.c: static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n10: ee72fb7e38 = 12: 4c793df9d1 pack-bitmap.c: introduce 'bitmap_num_objects()'\n11: ede0bf1ce1 = 13: 9f165037ce pack-bitmap.c: introduce 'nth_bitmap_object_oid()'\n12: df6844def0 = 14: ba5fd71fb3 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'\n13: 4e06f051a7 = 15: 06db8dbbc1 pack-bitmap.c: avoid redundant calls to try_partial_reuse\n14: a0d73eb3d3 = 16: 61798853b6 pack-bitmap: read multi-pack bitmaps\n15: 9d83ad77ab ! 17: 4968229663 pack-bitmap: write multi-pack bitmaps\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n +\t\t\t * corresponding bitmap (or one wasn't requested).\n +\t\t\t */\n +\t\t\tif (!want_bitmap)\n -+\t\t\t\tclear_midx_files_ext(the_repository, \".bitmap\",\n ++\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n +\t\t\t\t\t\t NULL);\n +\t\t\tgoto cleanup;\n +\t\t}\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n +\t\t}\n +\t}\n +\n -+\tclose_object_store(the_repository->objects);\n ++\tif (ctx.m)\n ++\t\tclose_object_store(the_repository->objects);\n \n \tcommit_lock_file(&lk);\n \n -+\tclear_midx_files_ext(the_repository, \".bitmap\", midx_hash);\n - \tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n ++\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n + \tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n \n cleanup:\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n @@ midx.c: void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n -+\tclear_midx_files_ext(r, \".bitmap\", NULL);\n - \tclear_midx_files_ext(r, \".rev\", NULL);\n ++\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n + \tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n \n \tfree(midx);\n \n16: a92af89884 = 18: 5d60b07e2e t5310: move some tests to lib-bitmap.sh\n17: d47aa4a919 = 19: 1a9c3538db t/helper/test-read-midx.c: add --checksum mode\n18: 9d9d9f28a6 = 20: 8895114ace t5326: test multi-pack bitmap behavior\n19: 3e0da7e5ed = 21: 94b1317e0c t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n20: 4e0d49a2dd = 22: a4f4d90bba t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\n21: 47eba8ecf9 = 23: 92a6370e77 t5319: don't write MIDX bitmaps in t5319\n22: 3d78afa2ad = 24: c49dc46fb2 t7700: update to work with MIDX bitmap test knob\n23: c2f94e033d = 25: 44a4800756 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'\n24: 6b03016c99 = 26: bf0981b606 p5310: extract full and partial bitmap tests\n25: d98faa4c2c = 27: 6888fe01aa p5326: perf tests for MIDX bitmaps\n-- \n2.33.0.96.g73915697e6\n"},{"id":"434337","messageId":"7815d9929d7b34f566b9748e03818c37d22a6808.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 01/27] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:42Z","receivedAt":"2021-08-31T20:51:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The special `--test-bitmap` mode of `git rev-list` is used to compare\nthe result of an object traversal with a bitmap to check its integrity.\nThis mode does not, however, assert that the types of reachable objects\nare stored correctly.\n\nHarden this mode by teaching it to also check that each time an object's\nbit is marked, the corresponding bit should be set in exactly one of the\ntype bitmaps (whose type matches the object's true type).\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d999616c9e..9b11af87aa 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1325,10 +1325,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,\n struct bitmap_test_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\n+\tstruct bitmap *commits;\n+\tstruct bitmap *trees;\n+\tstruct bitmap *blobs;\n+\tstruct bitmap *tags;\n \tstruct progress *prg;\n \tsize_t seen;\n };\n \n+static void test_bitmap_type(struct bitmap_test_data *tdata,\n+\t\t\t struct object *obj, int pos)\n+{\n+\tenum object_type bitmap_type = OBJ_NONE;\n+\tint bitmaps_nr = 0;\n+\n+\tif (bitmap_get(tdata->commits, pos)) {\n+\t\tbitmap_type = OBJ_COMMIT;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->trees, pos)) {\n+\t\tbitmap_type = OBJ_TREE;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->blobs, pos)) {\n+\t\tbitmap_type = OBJ_BLOB;\n+\t\tbitmaps_nr++;\n+\t}\n+\tif (bitmap_get(tdata->tags, pos)) {\n+\t\tbitmap_type = OBJ_TAG;\n+\t\tbitmaps_nr++;\n+\t}\n+\n+\tif (bitmap_type == OBJ_NONE)\n+\t\tdie(\"object %s not found in type bitmaps\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmaps_nr > 1)\n+\t\tdie(\"object %s does not have a unique type\",\n+\t\t oid_to_hex(&obj->oid));\n+\n+\tif (bitmap_type != obj->type)\n+\t\tdie(\"object %s: real type %s, expected: %s\",\n+\t\t oid_to_hex(&obj->oid),\n+\t\t type_name(obj->type),\n+\t\t type_name(bitmap_type));\n+}\n+\n static void test_show_object(struct object *object, const char *name,\n \t\t\t void *data)\n {\n@@ -1338,6 +1380,7 @@ static void test_show_object(struct object *object, const char *name,\n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&object->oid));\n+\ttest_bitmap_type(tdata, object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1352,6 +1395,7 @@ static void test_show_commit(struct commit *commit, void *data)\n \t\t\t\t &commit->object.oid);\n \tif (bitmap_pos < 0)\n \t\tdie(\"Object not in bitmap: %s\\n\", oid_to_hex(&commit->object.oid));\n+\ttest_bitmap_type(tdata, &commit->object, bitmap_pos);\n \n \tbitmap_set(tdata->base, bitmap_pos);\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n@@ -1399,6 +1443,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \ttdata.bitmap_git = bitmap_git;\n \ttdata.base = bitmap_new();\n+\ttdata.commits = ewah_to_bitmap(bitmap_git->commits);\n+\ttdata.trees = ewah_to_bitmap(bitmap_git->trees);\n+\ttdata.blobs = ewah_to_bitmap(bitmap_git->blobs);\n+\ttdata.tags = ewah_to_bitmap(bitmap_git->tags);\n \ttdata.prg = start_progress(\"Verifying bitmap entries\", result_popcnt);\n \ttdata.seen = 0;\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434339","messageId":"629171115a36071969ea56569e07c8b179affa29.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 02/27] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:44Z","receivedAt":"2021-08-31T20:51:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The set of objects covered by a bitmap must be closed under\nreachability, since it must be the case that there is a valid bit\nposition assigned for every possible reachable object (otherwise the\nbitmaps would be incomplete).\n\nPack bitmaps are never written from 'git repack' unless repacking\nall-into-one, and so we never write non-closed bitmaps (except in the\ncase of partial clones where we aren't guaranteed to have all objects).\n\nBut multi-pack bitmaps change this, since it isn't known whether the\nset of objects in the MIDX is closed under reachability until walking\nthem. Plumb through a bit that is set when a reachable object isn't\nfound.\n\nAs soon as a reachable object isn't found in the set of objects to\ninclude in the bitmap, bitmap_writer_build() knows that the set is not\nclosed, and so it now fails gracefully.\n\nA test is added in t0410 to trigger a bitmap write without full\nreachability closure by removing local copies of some reachable objects\nfrom a promisor remote.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 +-\n pack-bitmap-write.c | 76 ++++++++++++++++++++++++++++------------\n pack-bitmap.h | 2 +-\n t/t0410-partial-clone.sh | 9 ++++-\n 4 files changed, 64 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df49f656b9..b63e06e46c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1256,7 +1256,8 @@ static void write_pack_file(void)\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n-\t\t\t\tbitmap_writer_build(&to_pack);\n+\t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n+\t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..d374f7884b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit)\n \twriter.selected_nr++;\n }\n \n-static uint32_t find_object_pos(const struct object_id *oid)\n+static uint32_t find_object_pos(const struct object_id *oid, int *found)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n \n \tif (!entry) {\n-\t\tdie(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n+\t\tif (found)\n+\t\t\t*found = 0;\n+\t\twarning(\"Failed to write bitmap index. Packfile doesn't have full closure \"\n \t\t\t\"(object %s is missing)\", oid_to_hex(oid));\n+\t\treturn 0;\n \t}\n \n+\tif (found)\n+\t\t*found = 1;\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n@@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb)\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n \n-static void fill_bitmap_tree(struct bitmap *bitmap,\n-\t\t\t struct tree *tree)\n+static int fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n {\n+\tint found;\n \tuint32_t pos;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t * If our bit is already set, then there is nothing to do. Both this\n \t * tree and all of its children will be set.\n \t */\n-\tpos = find_object_pos(&tree->object.oid);\n+\tpos = find_object_pos(&tree->object.oid, &found);\n+\tif (!found)\n+\t\treturn -1;\n \tif (bitmap_get(bitmap, pos))\n-\t\treturn;\n+\t\treturn 0;\n \tbitmap_set(bitmap, pos);\n \n \tif (parse_tree(tree) < 0)\n@@ -355,11 +363,15 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \twhile (tree_entry(&desc, &entry)) {\n \t\tswitch (object_type(entry.mode)) {\n \t\tcase OBJ_TREE:\n-\t\t\tfill_bitmap_tree(bitmap,\n-\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tif (fill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid)) < 0)\n+\t\t\t\treturn -1;\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n-\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tpos = find_object_pos(&entry.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(bitmap, pos);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\t/* Gitlink, etc; not reachable */\n@@ -368,15 +380,18 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \t}\n \n \tfree_tree_buffer(tree);\n+\treturn 0;\n }\n \n-static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue,\n-\t\t\t struct prio_queue *tree_queue,\n-\t\t\t struct bitmap_index *old_bitmap,\n-\t\t\t const uint32_t *mapping)\n+static int fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n+\tint found;\n+\tuint32_t pos;\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n@@ -401,11 +416,16 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\tif (!found)\n+\t\t\treturn -1;\n+\t\tbitmap_set(ent->bitmap, pos);\n \t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n-\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n \t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n \t\t\t\tbitmap_set(ent->bitmap, pos);\n \t\t\t\tprio_queue_put(queue, p->item);\n@@ -413,8 +433,12 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t}\n \t}\n \n-\twhile (tree_queue->nr)\n-\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n+\twhile (tree_queue->nr) {\n+\t\tif (fill_bitmap_tree(ent->bitmap,\n+\t\t\t\t prio_queue_get(tree_queue)) < 0)\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -432,7 +456,7 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+int bitmap_writer_build(struct packing_data *to_pack)\n {\n \tstruct bitmap_builder bb;\n \tsize_t i;\n@@ -441,6 +465,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct prio_queue tree_queue = { NULL };\n \tstruct bitmap_index *old_bitmap;\n \tuint32_t *mapping;\n+\tint closed = 1; /* until proven otherwise */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n-\t\t\t\t old_bitmap, mapping);\n+\t\tif (fill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping) < 0) {\n+\t\t\tclosed = 0;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -499,7 +527,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \tstop_progress(&writer.progress);\n \n-\tcompute_xor_offsets();\n+\tif (closed)\n+\t\tcompute_xor_offsets();\n+\treturn closed ? 0 : -1;\n }\n \n /**\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 99d733eb26..020cd8d868 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -87,7 +87,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n-void bitmap_writer_build(struct packing_data *to_pack);\n+int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex a211a66c67..bbcc51ee8e 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '\n repack_and_check () {\n \trm -rf repo2 &&\n \tcp -r repo repo2 &&\n-\tgit -C repo2 repack $1 -d &&\n+\tif test x\"$1\" = \"x--must-fail\"\n+\tthen\n+\t\tshift\n+\t\ttest_must_fail git -C repo2 repack $1 -d\n+\telse\n+\t\tgit -C repo2 repack $1 -d\n+\tfi &&\n \tgit -C repo2 fsck &&\n \n \tgit -C repo2 cat-file -e $2 &&\n@@ -561,6 +567,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' '\n \tprintf \"$THREE\\n\" | pack_as_from_promisor &&\n \tdelete_object repo \"$ONE\" &&\n \n+\trepack_and_check --must-fail -ab \"$TWO\" \"$THREE\" &&\n \trepack_and_check -a \"$TWO\" \"$THREE\" &&\n \trepack_and_check -A \"$TWO\" \"$THREE\" &&\n \trepack_and_check -l \"$TWO\" \"$THREE\"\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434338","messageId":"d469c1d8f6ae521621c72fc15d8c4b3a0d7e57cb.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 03/27] pack-bitmap-write.c: free existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:47Z","receivedAt":"2021-08-31T20:51:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new bitmap, the bitmap writer code attempts to read the\nexisting bitmap (if one is present). This is done in order to quickly\npermute the bits of any bitmaps for commits which appear in the existing\nbitmap, and were also selected for the new bitmap.\n\nBut since this code was added in 341fa34887 (pack-bitmap-write: use\nexisting bitmaps, 2020-12-08), the resources associated with opening an\nexisting bitmap were never released.\n\nIt's fine to ignore this, but it's bad hygiene. It will also cause a\nproblem for the multi-pack-index builtin, which will be responsible not\nonly for writing bitmaps, but also for expiring any old multi-pack\nbitmaps.\n\nIf an existing bitmap was reused here, it will also be expired. That\nwill cause a problem on platforms which require file resources to be\nclosed before unlinking them, like Windows. Avoid this by ensuring we\nclose reused bitmaps with free_bitmap_index() before removing them.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d374f7884b..142fd0adb8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tclear_prio_queue(&queue);\n \tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree_bitmap_index(old_bitmap);\n \tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434340","messageId":"158ff797c4cf79ee50efe31c85c032f4d79af609.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 04/27] Documentation: describe MIDX-based bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:50Z","receivedAt":"2021-08-31T20:51:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Update the technical documentation to describe the multi-pack bitmap\nformat. This patch merely introduces the new format, and describes its\nhigh-level ideas. Git does not yet know how to read nor write these\nmulti-pack variants, and so the subsequent patches will:\n\n - Introduce code to interpret multi-pack bitmaps, according to this\n document.\n\n - Then, introduce code to write multi-pack bitmaps from the 'git\n multi-pack-index write' sub-command.\n\nFinally, the implementation will gain tests in subsequent patches (as\nopposed to inline with the patch teaching Git how to write multi-pack\nbitmaps) to avoid a cyclic dependency.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 71 ++++++++++++++++----\n Documentation/technical/multi-pack-index.txt | 10 +--\n 2 files changed, 60 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f8c18a0f7a..04b3ec2178 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -1,6 +1,44 @@\n GIT bitmap v1 format\n ====================\n \n+== Pack and multi-pack bitmaps\n+\n+Bitmaps store reachability information about the set of objects in a packfile,\n+or a multi-pack index (MIDX). The former is defined obviously, and the latter is\n+defined as the union of objects in packs contained in the MIDX.\n+\n+A bitmap may belong to either one pack, or the repository's multi-pack index (if\n+it exists). A repository may have at most one bitmap.\n+\n+An object is uniquely described by its bit position within a bitmap:\n+\n+\t- If the bitmap belongs to a packfile, the __n__th bit corresponds to\n+\tthe __n__th object in pack order. For a function `offset` which maps\n+\tobjects to their byte offset within a pack, pack order is defined as\n+\tfollows:\n+\n+\t\to1 <= o2 <==> offset(o1) <= offset(o2)\n+\n+\t- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the\n+\t__n__th object in MIDX order. With an additional function `pack` which\n+\tmaps objects to the pack they were selected from by the MIDX, MIDX order\n+\tis defined as follows:\n+\n+\t\to1 <= o2 <==> pack(o1) <= pack(o2) /\\ offset(o1) <= offset(o2)\n+\n+\tThe ordering between packs is done according to the MIDX's .rev file.\n+\tNotably, the preferred pack sorts ahead of all other packs.\n+\n+The on-disk representation (described below) of a bitmap is the same regardless\n+of whether or not that bitmap belongs to a packfile or a MIDX. The only\n+difference is the interpretation of the bits, which is described above.\n+\n+Certain bitmap extensions are supported (see: Appendix B). No extensions are\n+required for bitmaps corresponding to packfiles. For bitmaps that correspond to\n+MIDXs, both the bit-cache and rev-cache extensions are required.\n+\n+== On-disk format\n+\n \t- A header appears at the beginning:\n \n \t\t4-byte signature: {'B', 'I', 'T', 'M'}\n@@ -14,17 +52,19 @@ GIT bitmap v1 format\n \t\t\tThe following flags are supported:\n \n \t\t\t- BITMAP_OPT_FULL_DAG (0x1) REQUIRED\n-\t\t\tThis flag must always be present. It implies that the bitmap\n-\t\t\tindex has been generated for a packfile with full closure\n-\t\t\t(i.e. where every single object in the packfile can find\n-\t\t\t its parent links inside the same packfile). This is a\n-\t\t\trequirement for the bitmap index format, also present in JGit,\n-\t\t\tthat greatly reduces the complexity of the implementation.\n+\t\t\tThis flag must always be present. It implies that the\n+\t\t\tbitmap index has been generated for a packfile or\n+\t\t\tmulti-pack index (MIDX) with full closure (i.e. where\n+\t\t\tevery single object in the packfile/MIDX can find its\n+\t\t\tparent links inside the same packfile/MIDX). This is a\n+\t\t\trequirement for the bitmap index format, also present in\n+\t\t\tJGit, that greatly reduces the complexity of the\n+\t\t\timplementation.\n \n \t\t\t- BITMAP_OPT_HASH_CACHE (0x4)\n \t\t\tIf present, the end of the bitmap file contains\n \t\t\t`N` 32-bit name-hash values, one per object in the\n-\t\t\tpack. The format and meaning of the name-hash is\n+\t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n \t\t4-byte entry count (network byte order)\n@@ -33,7 +73,8 @@ GIT bitmap v1 format\n \n \t\t20-byte checksum\n \n-\t\t\tThe SHA1 checksum of the pack this bitmap index belongs to.\n+\t\t\tThe SHA1 checksum of the pack/MIDX this bitmap index\n+\t\t\tbelongs to.\n \n \t- 4 EWAH bitmaps that act as type indexes\n \n@@ -50,7 +91,7 @@ GIT bitmap v1 format\n \t\t\t- Tags\n \n \t\tIn each bitmap, the `n`th bit is set to true if the `n`th object\n-\t\tin the packfile is of that type.\n+\t\tin the packfile or multi-pack index is of that type.\n \n \t\tThe obvious consequence is that the OR of all 4 bitmaps will result\n \t\tin a full set (all bits set), and the AND of all 4 bitmaps will\n@@ -62,8 +103,9 @@ GIT bitmap v1 format\n \t\tEach entry contains the following:\n \n \t\t- 4-byte object position (network byte order)\n-\t\t\tThe position **in the index for the packfile** where the\n-\t\t\tbitmap for this commit is found.\n+\t\t\tThe position **in the index for the packfile or\n+\t\t\tmulti-pack index** where the bitmap for this commit is\n+\t\t\tfound.\n \n \t\t- 1-byte XOR-offset\n \t\t\tThe xor offset used to compress this bitmap. For an entry\n@@ -146,10 +188,11 @@ Name-hash cache\n ---------------\n \n If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains\n-a cache of 32-bit values, one per object in the pack. The value at\n+a cache of 32-bit values, one per object in the pack/MIDX. The value at\n position `i` is the hash of the pathname at which the `i`th object\n-(counting in index order) in the pack can be found. This can be fed\n-into the delta heuristics to compare objects with similar pathnames.\n+(counting in index or multi-pack index order) in the pack/MIDX can be found.\n+This can be fed into the delta heuristics to compare objects with similar\n+pathnames.\n \n The hash algorithm used is:\n \ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex fb688976c4..1a73c3ee20 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -71,14 +71,10 @@ Future Work\n still reducing the number of binary searches required for object\n lookups.\n \n-- The reachability bitmap is currently paired directly with a single\n- packfile, using the pack-order as the object order to hopefully\n- compress the bitmaps well using run-length encoding. This could be\n- extended to pair a reachability bitmap with a multi-pack-index. If\n- the multi-pack-index is extended to store a \"stable object order\"\n+- If the multi-pack-index is extended to store a \"stable object order\"\n (a function Order(hash) = integer that is constant for a given hash,\n- even as the multi-pack-index is updated) then a reachability bitmap\n- could point to a multi-pack-index and be updated independently.\n+ even as the multi-pack-index is updated) then MIDX bitmaps could be\n+ updated independently of the MIDX.\n \n - Packfiles can be marked as \"special\" using empty files that share\n the initial name but replace \".pack\" with \".keep\" or \".promisor\".\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434341","messageId":"5f24be8985fe0f20b3ea3001dca6c72b6fac17e1.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 05/27] midx: disallow running outside of a repository","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:53Z","receivedAt":"2021-08-31T20:51:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The multi-pack-index command supports working with arbitrary object\ndirectories via the `--object-dir` flag. Though this has historically\nworked in arbitrary repositories (including when the command itself was\nrun outside of a Git repository), this has been somewhat of an accident.\n\nFor example, running:\n\n git multi-pack-index write --object-dir=/path/to/repo/objects\n\noutside of a Git repository causes a BUG(). This is because the\ntop-level `cmd_multi_pack_index()` function stops parsing when it sees\n\"write\", and then fills in the default object directory (the result of\ncalling `get_object_directory()`) before handing off to\n`cmd_multi_pack_index_write()`. But there is no repository to\ninitialize, and so calling `get_object_directory()` results in a BUG()\n(indicating that the current repository is not initialized).\n\nAnother case where this doesn't quite work as expected is when operating\nin a SHA-256 repository. To see the failure, try this in your shell:\n\n git init --object-format=sha256 repo\n git -C repo commit --allow-empty base\n git -C repo repack -d\n\n git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write\n\nand observe that we cannot open the `.idx` file in \"repo\", because the\noutermost process assumes that any repository that it works in also uses\nthe default value of `the_hash_algo` (at the time of writing, SHA-1).\n\nThere may be compelling reasons for trying to work around these bugs,\nbut working in arbitrary `--object-dir`'s is non-standard enough (and\nlikewise, these bugs prevalent enough) that I don't think any workflows\nwould be broken by abandoning this behavior.\n\nAccordingly, restrict the `multi-pack-index` builtin to only work when\ninside of a Git repository (i.e., its main utility becomes selecting\nwhich alternate to operate in), which avoids both of the bugs above.\n\n(Note that you can still trigger a bug when writing a MIDX in an\nalternate which does not use the same object format as the repository\nwhich it is an alternate of, but that is an unrelated bug to this one).\n\nSigned-off-by: Taylor Blau \n---\n git.c | 2 +-\n t/t5319-multi-pack-index.sh | 5 +++++\n 2 files changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/git.c b/git.c\nindex 18bed9a996..60c2784be4 100644\n--- a/git.c\n+++ b/git.c\n@@ -561,7 +561,7 @@ static struct cmd_struct commands[] = {\n \t{ \"merge-tree\", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktag\", cmd_mktag, RUN_SETUP | NO_PARSEOPT },\n \t{ \"mktree\", cmd_mktree, RUN_SETUP },\n-\t{ \"multi-pack-index\", cmd_multi_pack_index, RUN_SETUP_GENTLY },\n+\t{ \"multi-pack-index\", cmd_multi_pack_index, RUN_SETUP },\n \t{ \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"name-rev\", cmd_name_rev, RUN_SETUP },\n \t{ \"notes\", cmd_notes, RUN_SETUP },\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 3d4d9f10c3..9034e94c0a 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -842,4 +842,9 @@ test_expect_success 'usage shown without sub-command' '\n \t! test_i18ngrep \"unrecognized subcommand\" err\n '\n \n+test_expect_success 'complains when run outside of a repository' '\n+\tnongit test_must_fail git multi-pack-index write 2>err &&\n+\tgrep \"not a git repository\" err\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434342","messageId":"0aacaa928395bdc041cc366a25047442ff7a33f1.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 06/27] midx: fix `*.rev` cleanups with `--object-dir`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:55Z","receivedAt":"2021-08-31T20:52:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"If using --object-dir to point into an object directory which belongs to\na different repository than the one in the current working directory,\nsuch as:\n\n git init repo\n git -C repo ... # add some objects\n cd alternate\n git multi-pack-index --object-dir ../repo/.git/objects write\n\nthe binary will segfault trying to access the object-dir via the repo it\nfound, but that's not fully initialized. Worse, if we later call\nclear_midx_files_ext(), we will use `the_repository` and remove files\nout of the wrong object directory.\n\nFix this by using the given object_dir (or the object directory of\n`the_repository` if `--object-dir` wasn't given) to properly to clean up\nthe *.rev files, avoiding the crash.\n\nOriginal-patch-by: Johannes Berg \nSigned-off-by: Taylor Blau \n---\n midx.c | 10 +++++-----\n t/t5319-multi-pack-index.sh | 28 ++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 5 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 321c6fdd2f..902e1a7a7d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -882,7 +882,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n \tstrbuf_release(&buf);\n }\n \n-static void clear_midx_files_ext(struct repository *r, const char *ext,\n+static void clear_midx_files_ext(const char *object_dir, const char *ext,\n \t\t\t\t unsigned char *keep_hash);\n \n static int midx_checksum_valid(struct multi_pack_index *m)\n@@ -1086,7 +1086,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n-\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n@@ -1135,7 +1135,7 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n \n-static void clear_midx_files_ext(struct repository *r, const char *ext,\n+static void clear_midx_files_ext(const char *object_dir, const char *ext,\n \t\t\t\t unsigned char *keep_hash)\n {\n \tstruct clear_midx_data data;\n@@ -1146,7 +1146,7 @@ static void clear_midx_files_ext(struct repository *r, const char *ext,\n \t\t\t\t hash_to_hex(keep_hash), ext);\n \tdata.ext = ext;\n \n-\tfor_each_file_in_pack_dir(r->objects->odb->path,\n+\tfor_each_file_in_pack_dir(object_dir,\n \t\t\t\t clear_midx_file_ext,\n \t\t\t\t &data);\n \n@@ -1165,7 +1165,7 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n-\tclear_midx_files_ext(r, \".rev\", NULL);\n+\tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n \n \tfree(midx);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 9034e94c0a..e953cdd6d1 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -201,6 +201,34 @@ test_expect_success 'write midx with twelve packs' '\n \n compare_results_with_midx \"twelve packs\"\n \n+test_expect_success 'multi-pack-index *.rev cleanup with --object-dir' '\n+\tgit init repo &&\n+\tgit clone -s repo alternate &&\n+\n+\ttest_when_finished \"rm -rf repo alternate\" &&\n+\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack -d\n+\t) &&\n+\n+\tours=\"alternate/.git/objects/pack/multi-pack-index-123.rev\" &&\n+\ttheirs=\"repo/.git/objects/pack/multi-pack-index-abc.rev\" &&\n+\ttouch \"$ours\" \"$theirs\" &&\n+\n+\t(\n+\t\tcd alternate &&\n+\t\tgit multi-pack-index --object-dir ../repo/.git/objects write\n+\t) &&\n+\n+\t# writing a midx in \"repo\" should not remove the .rev file in the\n+\t# alternate\n+\ttest_path_is_file repo/.git/objects/pack/multi-pack-index &&\n+\ttest_path_is_file $ours &&\n+\ttest_path_is_missing $theirs\n+'\n+\n test_expect_success 'warn on improper hash version' '\n \tgit init --object-format=sha1 sha1 &&\n \t(\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434343","messageId":"d30e6fe9a51be6ebf271b2871f6171ffcd8d7594.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 07/27] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:51:59Z","receivedAt":"2021-08-31T20:52:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing a new multi-pack index, write_midx_internal() attempts to\nclean up any auxiliary files (currently just the MIDX's `.rev` file, but\nsoon to include a `.bitmap`, too) corresponding to the MIDX it's\nreplacing.\n\nThis step should happen after the new MIDX is written into place, since\ndoing so beforehand means that the old MIDX could be read without its\ncorresponding .rev file.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 902e1a7a7d..0bcb403bae 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+\n cleanup:\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].p) {\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434344","messageId":"059c583e34f4fd90e7cc2a9aad20fc21ea2bbc8b.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 09/27] midx: infer preferred pack when not given one","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:04Z","receivedAt":"2021-08-31T20:52:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the\nmulti-pack index code learned how to select a pack which all duplicate\nobjects are selected from. That is, if an object appears in multiple\npacks, select the copy in the preferred pack before breaking ties\naccording to the other rules like pack mtime and readdir() order.\n\nNot specifying a preferred pack can cause serious problems with\nmulti-pack reachability bitmaps, because these bitmaps rely on having at\nleast one pack from which all duplicates are selected. Not having such a\npack causes problems with the code in pack-objects to reuse packs\nverbatim (e.g., that code assumes that a delta object in a chunk of pack\nsent verbatim will have its base object sent from the same pack).\n\nSo why does not marking a pack preferred cause problems here? The reason\nis roughly as follows:\n\n - Ties are broken (when handling duplicate objects) by sorting\n according to midx_oid_compare(), which sorts objects by OID,\n preferred-ness, pack mtime, and finally pack ID (more on that\n later).\n\n - The psuedo pack-order (described in\n Documentation/technical/pack-format.txt under the section\n \"multi-pack-index reverse indexes\") is computed by\n midx_pack_order(), and sorts by pack ID and pack offset, with\n preferred packs sorting first.\n\n - But! Pack IDs come from incrementing the pack count in\n add_pack_to_midx(), which is a callback to\n for_each_file_in_pack_dir(), meaning that pack IDs are assigned in\n readdir() order.\n\nWhen specifying a preferred pack, all of that works fine, because\nduplicate objects are correctly resolved in favor of the copy in the\npreferred pack, and the preferred pack sorts first in the object order.\n\n\"Sorting first\" is critical, because the bitmap code relies on finding\nout which pack holds the first object in the MIDX's pseudo pack-order to\ndetermine which pack is preferred.\n\nBut if we didn't specify a preferred pack, and the pack which comes\nfirst in readdir() order does not also have the lowest timestamp, then\nit's possible that that pack (the one that sorts first in pseudo-pack\norder, which the bitmap code will treat as the preferred one) did *not*\nhave all duplicate objects resolved in its favor, resulting in breakage.\n\nThe fix is simple: pick a (semi-arbitrary, non-empty) preferred pack\nwhen none was specified. This forces that pack to have duplicates\nresolved in its favor, and (critically) to sort first in pseudo-pack\norder. Unfortunately, testing this behavior portably isn't possible,\nsince it depends on readdir() order which isn't guaranteed by POSIX.\n\n(Note that multi-pack reachability bitmaps have yet to be implemented;\nso in that sense this patch is fixing a bug which does not yet exist.\nBut by having this patch beforehand, we can prevent the bug from ever\nmaterializing.)\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 44 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 26089ec9c7..67de1dbaeb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -969,15 +969,57 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n+\t\tint found = 0;\n \t\tfor (i = 0; i < ctx.nr; i++) {\n \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tfound = 1;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n+\n+\t\tif (!found)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tctx.preferred_pack_idx = 0;\n+\n+\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n+\n+\t\t/*\n+\t\t * set a preferred pack when writing a bitmap to ensure that\n+\t\t * the pack from which the first object is selected in pseudo\n+\t\t * pack-order has all of its objects selected from that pack\n+\t\t * (and not another pack containing a duplicate)\n+\t\t */\n+\t\tfor (i = 1; i < ctx.nr; i++) {\n+\t\t\tstruct packed_git *p = ctx.info[i].p;\n+\n+\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n+\t\t\t\toldest = p;\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!oldest->num_objects) {\n+\t\t\t/*\n+\t\t\t * If all packs are empty; unset the preferred index.\n+\t\t\t * This is acceptable since there will be no duplicate\n+\t\t\t * objects to resolve, so the preferred value doesn't\n+\t\t\t * matter.\n+\t\t\t */\n+\t\t\tctx.preferred_pack_idx = -1;\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * otherwise don't mark any pack as preferred to avoid\n+\t\t * interfering with expiration logic below\n+\t\t */\n+\t\tctx.preferred_pack_idx = -1;\n \t}\n \n \tif (ctx.preferred_pack_idx > -1) {\n@@ -1058,11 +1100,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n-\n-\t\tif (!preferred)\n-\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n-\t\telse {\n+\t\tif (preferred) {\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434346","messageId":"db2a24a8ae072aa520812a8b1dcac2a8fee209a1.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 08/27] midx: reject empty `--preferred-pack`'s","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:02Z","receivedAt":"2021-08-31T20:52:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The soon-to-be-implemented multi-pack bitmap treats object in the first\nbit position specially by assuming that all objects in the pack it was\nselected from are also represented from that pack in the MIDX. In other\nwords, the pack from which the first object was selected must also have\nall of its other objects selected from that same pack in the MIDX in\ncase of any duplicates.\n\nBut this assumption relies on the fact that there is at least one object\nin that pack to begin with; otherwise the object in the first bit\nposition isn't from a preferred pack, in which case we can no longer\nassume that all objects in that pack were also selected from the same\npack.\n\nGuard this assumption by checking the number of objects in the given\npreferred pack, and failing if the given pack is empty.\n\nTo make sure we can safely perform this check, open any packs which are\ncontained in an existing MIDX via prepare_midx_pack(). The same is done\nfor new packs via the add_pack_to_midx() callback, but packs picked up\nfrom a previous MIDX will not yet have these opened.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 6 +++---\n midx.c | 29 ++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh | 17 +++++++++++++++\n 3 files changed, 49 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex ffd601bc17..c9b063d31e 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -37,9 +37,9 @@ write::\n --\n \t--preferred-pack=::\n \t\tOptionally specify the tie-breaking pack used when\n-\t\tmultiple packs contain the same object. If not given,\n-\t\tties are broken in favor of the pack with the lowest\n-\t\tmtime.\n+\t\tmultiple packs contain the same object. `` must\n+\t\tcontain at least one object. If not given, ties are\n+\t\tbroken in favor of the pack with the lowest mtime.\n --\n \n verify::\ndiff --git a/midx.c b/midx.c\nindex 0bcb403bae..26089ec9c7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -934,6 +934,25 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n \t\t\tctx.info[ctx.nr].p = NULL;\n \t\t\tctx.info[ctx.nr].expired = 0;\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n+\t\t\t\t/*\n+\t\t\t\t * If generating a reverse index, need to have\n+\t\t\t\t * packed_git's loaded to compare their\n+\t\t\t\t * mtimes and object count.\n+\t\t\t\t */\n+\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\tresult = 1;\n+\t\t\t\t\tgoto cleanup;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t ctx.m->packs[i]->pack_name);\n+\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n+\t\t\t}\n+\n \t\t\tctx.nr++;\n \t\t}\n \t}\n@@ -961,6 +980,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t}\n \t}\n \n+\tif (ctx.preferred_pack_idx > -1) {\n+\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tif (!preferred->num_objects) {\n+\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n+\t\t\t preferred->pack_name);\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n \t\t\t\t\t ctx.preferred_pack_idx);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex e953cdd6d1..d7e4988f2b 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -305,6 +305,23 @@ test_expect_success 'midx picks objects from preferred pack' '\n \t)\n '\n \n+test_expect_success 'preferred packs must be non-empty' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ttest_commit base &&\n+\t\tgit repack -ad &&\n+\n+\t\tempty=\"$(git pack-objects $objdir/pack/pack err &&\n+\t\tgrep \"with no objects\" err\n+\t)\n+'\n+\n test_expect_success 'verify multi-pack-index success' '\n \tgit multi-pack-index verify --object-dir=$objdir\n '\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434345","messageId":"6f5ca446f3820ae433c9fcbd7439f92a44d0eb2c.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 10/27] midx: close linked MIDXs, avoid leaking memory","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:07Z","receivedAt":"2021-08-31T20:52:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a repository has at least one alternate, the MIDX belonging to each\nalternate is accessed through the `next` pointer on the main object\nstore's copy of the MIDX. close_midx() didn't bother to close any\nof the linked MIDXs. It likewise didn't free the memory pointed to by\n`m`, leaving uninitialized bytes with live pointers to them left around\nin the heap.\n\nClean this up by closing linked MIDXs, and freeing up the memory pointed\nto by each of them. When callers call close_midx(), then they can\ndiscard the entire linked list of MIDXs and set their pointer to the\nhead of that list to NULL.\n\nThis isn't strictly required for the upcoming patches, but it makes it\nmuch more difficult (though still possible, for e.g., by calling\n`close_midx(m->next)` which leaves `m->next` pointing at uninitialized\nbytes) to have pointers to uninitialized memory.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 67de1dbaeb..e83f22b5ee 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -195,6 +195,8 @@ void close_midx(struct multi_pack_index *m)\n \tif (!m)\n \t\treturn;\n \n+\tclose_midx(m->next);\n+\n \tmunmap((unsigned char *)m->data, m->data_len);\n \n \tfor (i = 0; i < m->num_packs; i++) {\n@@ -203,6 +205,7 @@ void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n \tFREE_AND_NULL(m->pack_names);\n+\tfree(m);\n }\n \n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434347","messageId":"4656608f7350a059748c75c5ed8731cb66a52d9b.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 11/27] midx: avoid opening multiple MIDXs when writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:09Z","receivedAt":"2021-08-31T20:52:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Opening multiple instance of the same MIDX can lead to problems like two\nseparate packed_git structures which represent the same pack being added\nto the repository's object store.\n\nThe above scenario can happen because prepare_midx_pack() checks if\n`m->packs[pack_int_id]` is NULL in order to determine if a pack has been\nopened and installed in the repository before. But a caller can\nconstruct two copies of the same MIDX by calling get_multi_pack_index()\nand load_multi_pack_index() since the former manipulates the\nobject store directly but the latter is a lower-level routine which\nallocates a new MIDX for each call.\n\nSo if prepare_midx_pack() is called on multiple MIDXs with the same\npack_int_id, then that pack will be installed twice in the object\nstore's packed_git pointer.\n\nThis can lead to problems in, for e.g., the pack-bitmap code, which does\nsomething like the following (in pack-bitmap.c:open_pack_bitmap()):\n\n struct bitmap_index *bitmap_git = ...;\n for (p = get_all_packs(r); p; p = p->next) {\n if (open_pack_bitmap_1(bitmap_git, p) == 0)\n ret = 0;\n }\n\nwhich is a problem if two copies of the same pack exist in the\npacked_git list because pack-bitmap.c:open_pack_bitmap_1() contains a\nconditional like the following:\n\n if (bitmap_git->pack || bitmap_git->midx) {\n /* ignore extra bitmap file; we can only handle one */\n warning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n close(fd);\n return -1;\n }\n\nAvoid this scenario by not letting write_midx_internal() open a MIDX\nthat isn't also pointed at by the object store. So long as this is the\ncase, other routines should prefer to open MIDXs with\nget_multi_pack_index() or reprepare_packed_git() instead of creating\ninstances on their own. Because get_multi_pack_index() returns\n`r->object_store->multi_pack_index` if it is non-NULL, we'll only have\none instance of a MIDX open at one time, avoiding these problems.\n\nTo encourage this, drop the `struct multi_pack_index *` parameter from\n`write_midx_internal()`, and rely instead on the `object_dir` to find\n(or initialize) the correct MIDX instance.\n\nLikewise, replace the call to `close_midx()` with\n`close_object_store()`, since we're about to replace the MIDX with a new\none and should invalidate the object store's memory of any MIDX that\nmight have existed beforehand.\n\nNote that this now forbids passing object directories that don't belong\nto alternate repositories over `--object-dir`, since before we would\nhave happily opened a MIDX in any directory, but now restrict ourselves\nto only those reachable by `r->objects->multi_pack_index` (and alternate\nMIDXs that we can see by walking the `next` pointer).\n\nAs far as I can tell, supporting arbitrary directories with\n`--object-dir` was a historical accident, since even the documentation\nsays `` when referring to the value passed to this option.\n\nA future patch could clean this up and provide a warning() when a\nnon-alternate directory was given, since we'll still write a new MIDX\nthere, we just won't reuse any MIDX that might happen to already exist\nin that directory.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 2 ++\n midx.c | 26 +++++++++++++++-----------\n 2 files changed, 17 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex c9b063d31e..0af6beb2dd 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -23,6 +23,8 @@ OPTIONS\n \tUse given directory for the location of Git objects. We check\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n++\n+`` must be an alternate of the current repository.\n \n --[no-]progress::\n \tTurn progress on/off explicitly. If neither is specified, progress is\ndiff --git a/midx.c b/midx.c\nindex e83f22b5ee..43510290dc 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n \t\t\t unsigned flags)\n@@ -904,6 +904,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n+\tstruct multi_pack_index *cur;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -914,10 +915,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name);\n \n-\tif (m)\n-\t\tctx.m = m;\n-\telse\n-\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\tctx.m = cur;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n \n \tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n \t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n@@ -1119,7 +1122,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n \tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n+\t\tclose_object_store(the_repository->objects);\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -1182,8 +1185,7 @@ int write_midx_file(const char *object_dir,\n \t\t const char *preferred_pack_name,\n \t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t flags);\n+\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n }\n \n struct clear_midx_data {\n@@ -1461,8 +1463,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \n \tfree(count);\n \n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n+\tif (packs_to_drop.nr) {\n+\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tm = NULL;\n+\t}\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1651,7 +1655,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434348","messageId":"4c793df9d13bfec35ca5dc10323a0a91c1ae7269.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 12/27] pack-bitmap.c: introduce 'bitmap_num_objects()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:12Z","receivedAt":"2021-08-31T20:52:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to return how many objects are\ncontained in a bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 37 +++++++++++++++++++++----------------\n 1 file changed, 21 insertions(+), 16 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9b11af87aa..65356f9657 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \treturn b;\n }\n \n+static uint32_t bitmap_num_objects(struct bitmap_index *index)\n+{\n+\treturn index->pack->num_objects;\n+}\n+\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n-\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tsize_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));\n \t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n@@ -404,7 +409,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \n \tif (pos < kh_end(positions)) {\n \t\tint bitmap_pos = kh_value(positions, pos);\n-\t\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\t\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n \t}\n \n \treturn -1;\n@@ -456,7 +461,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,\n \t\tbitmap_pos = kh_value(eindex->positions, hash_pos);\n \t}\n \n-\treturn bitmap_pos + bitmap_git->pack->num_objects;\n+\treturn bitmap_pos + bitmap_num_objects(bitmap_git);\n }\n \n struct bitmap_show_data {\n@@ -673,7 +678,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tstruct object *obj;\n \n-\t\tif (!bitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\tif (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n@@ -832,7 +837,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \t * them individually.\n \t */\n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == type &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos))\n@@ -859,7 +864,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \n \toi.sizep = &size;\n \n-\tif (pos < pack->num_objects) {\n+\tif (pos < bitmap_num_objects(bitmap_git)) {\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n@@ -869,7 +874,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t}\n \t} else {\n \t\tstruct eindex *eindex = &bitmap_git->ext_index;\n-\t\tstruct object *obj = eindex->objects[pos - pack->num_objects];\n+\t\tstruct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];\n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&obj->oid));\n \t}\n@@ -911,7 +916,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,\n \t}\n \n \tfor (i = 0; i < eindex->count; i++) {\n-\t\tuint32_t pos = i + bitmap_git->pack->num_objects;\n+\t\tuint32_t pos = i + bitmap_num_objects(bitmap_git);\n \t\tif (eindex->objects[i]->type == OBJ_BLOB &&\n \t\t bitmap_get(to_filter, pos) &&\n \t\t !bitmap_get(tips, pos) &&\n@@ -1137,8 +1142,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_git->pack->num_objects)\n-\t\treturn; /* not actually in the pack */\n+\tif (pos >= bitmap_num_objects(bitmap_git))\n+\t\treturn; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n@@ -1204,6 +1209,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n+\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n \n \tassert(result);\n \n@@ -1211,8 +1217,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\ti++;\n \n \t/* Don't mark objects not in the packfile */\n-\tif (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)\n-\t\ti = bitmap_git->pack->num_objects / BITS_IN_EWORD;\n+\tif (i > objects_nr / BITS_IN_EWORD)\n+\t\ti = objects_nr / BITS_IN_EWORD;\n \n \treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n@@ -1296,7 +1302,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < eindex->count; ++i) {\n \t\tif (eindex->objects[i]->type == type &&\n-\t\t\tbitmap_get(objects, bitmap_git->pack->num_objects + i))\n+\t\t\tbitmap_get(objects, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcount++;\n \t}\n \n@@ -1517,7 +1523,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n-\tnum_objects = bitmap_git->pack->num_objects;\n+\tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n \tfor (i = 0; i < num_objects; ++i) {\n@@ -1600,7 +1606,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tstruct eindex *eindex = &bitmap_git->ext_index;\n \toff_t total = 0;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n@@ -1612,7 +1617,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)\n \tfor (i = 0; i < eindex->count; i++) {\n \t\tstruct object *obj = eindex->objects[i];\n \n-\t\tif (!bitmap_get(result, pack->num_objects + i))\n+\t\tif (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))\n \t\t\tcontinue;\n \n \t\tif (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434349","messageId":"9f165037ce5e002fc7f4cc8d41969693f2376772.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 13/27] pack-bitmap.c: introduce 'nth_bitmap_object_oid()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:14Z","receivedAt":"2021-08-31T20:52:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch to support reading MIDX bitmaps will be less noisy\nafter extracting a generic function to fetch the nth OID contained in\nthe bitmap.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 65356f9657..612f62da97 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)\n \n #define MAX_XOR_OFFSET 160\n \n+static int nth_bitmap_object_oid(struct bitmap_index *index,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t uint32_t n)\n+{\n+\treturn nth_packed_object_id(oid, index->pack, n);\n+}\n+\n static int load_bitmap_entries_v1(struct bitmap_index *index)\n {\n \tuint32_t i;\n@@ -242,7 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\tif (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)\n \t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n \t\t\t\t (unsigned)commit_idx_pos);\n \n@@ -868,8 +875,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\toff_t ofs = pack_pos_to_offset(pack, pos);\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack,\n-\t\t\t\t\t pack_pos_to_index(pack, pos));\n+\t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n+\t\t\t\t\t pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434350","messageId":"ba5fd71fb3f4bb51b3cc086cbfe470e244b8e914.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 14/27] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:16Z","receivedAt":"2021-08-31T20:52:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a recent commit, pack-objects learned support for the\n'pack.preferBitmapTips' configuration. This patch prepares the\nmulti-pack bitmap code to respect this configuration, too.\n\nThe yet-to-be implemented code will find that it is more efficient to\ncheck whether each reference contains a prefix found in the configured\nset of values rather than doing an additional traversal.\n\nImplement a function 'bitmap_is_preferred_refname()' which will perform\nthat check. Its caller will be added in a subsequent patch.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 16 ++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 612f62da97..d5296750eb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1658,3 +1658,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\n }\n+\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname)\n+{\n+\tconst struct string_list *preferred_tips = bitmap_preferred_tips(r);\n+\tstruct string_list_item *item;\n+\n+\tif (!preferred_tips)\n+\t\treturn 0;\n+\n+\tfor_each_string_list_item(item, preferred_tips) {\n+\t\tif (starts_with(refname, item->string))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 020cd8d868..52ea10de51 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint16_t options);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n+int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n #endif\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434351","messageId":"61798853b6c5bec0ecc92d8aacc6c676441840d3.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 16/27] pack-bitmap: read multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:21Z","receivedAt":"2021-08-31T20:52:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This prepares the code in pack-bitmap to interpret the new multi-pack\nbitmaps described in Documentation/technical/bitmap-format.txt, which\nmostly involves converting bit positions to accommodate looking them up\nin a MIDX.\n\nNote that there are currently no writers who write multi-pack bitmaps,\nand that this will be implemented in the subsequent commit. Note also\nthat get_midx_checksum() and get_midx_filename() are made non-static so\nthey can be called from pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 5 +\n midx.c | 4 +-\n midx.h | 2 +\n pack-bitmap-write.c | 2 +-\n pack-bitmap.c | 357 ++++++++++++++++++++++++++++++++++++-----\n pack-bitmap.h | 6 +\n packfile.c | 2 +-\n 7 files changed, 336 insertions(+), 42 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b63e06e46c..c26dedfe5d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\t/*\n+\t\t\t * Can use bit positions directly, even for MIDX\n+\t\t\t * bitmaps. See comment in try_partial_reuse()\n+\t\t\t * for why.\n+\t\t\t */\n \t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\ndiff --git a/midx.c b/midx.c\nindex 43510290dc..6a10f7a042 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -48,12 +48,12 @@ static uint8_t oid_version(void)\n \t}\n }\n \n-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - the_hash_algo->rawsz;\n }\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex 8684cf0fef..1172df1a71 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -42,6 +42,8 @@ struct multi_pack_index {\n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n \n+const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+char *get_midx_filename(const char *object_dir);\n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 142fd0adb8..9c55c1531e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)\n }\n \n /**\n- * Build the initial type index for the packfile\n+ * Build the initial type index for the packfile or multi-pack-index\n */\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4e37f5d574..fa69ed7a6d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -13,6 +13,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"midx.h\"\n #include \"config.h\"\n \n /*\n@@ -35,8 +36,15 @@ struct stored_bitmap {\n * the active bitmap index is the largest one.\n */\n struct bitmap_index {\n-\t/* Packfile to which this bitmap index belongs to */\n+\t/*\n+\t * The pack or multi-pack index (MIDX) that this bitmap index belongs\n+\t * to.\n+\t *\n+\t * Exactly one of these must be non-NULL; this specifies the object\n+\t * order used to interpret this bitmap.\n+\t */\n \tstruct packed_git *pack;\n+\tstruct multi_pack_index *midx;\n \n \t/*\n \t * Mark the first `reuse_objects` in the packfile as reused:\n@@ -71,6 +79,9 @@ struct bitmap_index {\n \t/* If not NULL, this is a name-hash cache pointing into map. */\n \tuint32_t *hashes;\n \n+\t/* The checksum of the packfile or MIDX; points into map. */\n+\tconst unsigned char *checksum;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -138,6 +149,8 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n+\tif (index->midx)\n+\t\treturn index->midx->num_objects;\n \treturn index->pack->num_objects;\n }\n \n@@ -175,6 +188,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n+\tindex->checksum = header->checksum;\n \tindex->map_pos += header_size;\n \treturn 0;\n }\n@@ -227,6 +241,8 @@ static int nth_bitmap_object_oid(struct bitmap_index *index,\n \t\t\t\t struct object_id *oid,\n \t\t\t\t uint32_t n)\n {\n+\tif (index->midx)\n+\t\treturn nth_midxed_object_oid(oid, index->midx, n) ? 0 : -1;\n \treturn nth_packed_object_id(oid, index->pack, n);\n }\n \n@@ -274,7 +290,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \treturn 0;\n }\n \n-static char *pack_bitmap_filename(struct packed_git *p)\n+char *midx_bitmap_filename(struct multi_pack_index *midx)\n+{\n+\treturn xstrfmt(\"%s-%s.bitmap\",\n+\t\t get_midx_filename(midx->object_dir),\n+\t\t hash_to_hex(get_midx_checksum(midx)));\n+}\n+\n+char *pack_bitmap_filename(struct packed_git *p)\n {\n \tsize_t len;\n \n@@ -283,6 +306,57 @@ static char *pack_bitmap_filename(struct packed_git *p)\n \treturn xstrfmt(\"%.*s.bitmap\", (int)len, p->pack_name);\n }\n \n+static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct stat st;\n+\tchar *idx_name = midx_bitmap_filename(midx);\n+\tint fd = git_open(idx_name);\n+\n+\tfree(idx_name);\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n+\t\twarning(\"ignoring extra bitmap file: %s\",\n+\t\t\tget_midx_filename(midx->object_dir));\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tbitmap_git->midx = midx;\n+\tbitmap_git->map_size = xsize_t(st.st_size);\n+\tbitmap_git->map_pos = 0;\n+\tbitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ,\n+\t\t\t\tMAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\tif (load_bitmap_header(bitmap_git) < 0)\n+\t\tgoto cleanup;\n+\n+\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum))\n+\t\tgoto cleanup;\n+\n+\tif (load_midx_revindex(bitmap_git->midx) < 0) {\n+\t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n+\t\tgoto cleanup;\n+\t}\n+\treturn 0;\n+\n+cleanup:\n+\tmunmap(bitmap_git->map, bitmap_git->map_size);\n+\tbitmap_git->map_size = 0;\n+\tbitmap_git->map = NULL;\n+\treturn -1;\n+}\n+\n static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git *packfile)\n {\n \tint fd;\n@@ -304,7 +378,8 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\treturn -1;\n \t}\n \n-\tif (bitmap_git->pack) {\n+\tif (bitmap_git->pack || bitmap_git->midx) {\n+\t\t/* ignore extra bitmap file; we can only handle one */\n \t\twarning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n \t\tclose(fd);\n \t\treturn -1;\n@@ -331,13 +406,39 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \treturn 0;\n }\n \n-static int load_pack_bitmap(struct bitmap_index *bitmap_git)\n+static int load_reverse_index(struct bitmap_index *bitmap_git)\n+{\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tuint32_t i;\n+\t\tint ret;\n+\n+\t\t/*\n+\t\t * The multi-pack-index's .rev file is already loaded via\n+\t\t * open_pack_bitmap_1().\n+\t\t *\n+\t\t * But we still need to open the individual pack .rev files,\n+\t\t * since we will need to make use of them in pack-objects.\n+\t\t */\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n+\t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\treturn load_pack_revindex(bitmap_git->pack);\n+}\n+\n+static int load_bitmap(struct bitmap_index *bitmap_git)\n {\n \tassert(bitmap_git->map);\n \n \tbitmap_git->bitmaps = kh_init_oid_map();\n \tbitmap_git->ext_index.positions = kh_init_oid_pos();\n-\tif (load_pack_revindex(bitmap_git->pack))\n+\n+\tif (load_reverse_index(bitmap_git))\n \t\tgoto failed;\n \n \tif (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) ||\n@@ -381,11 +482,47 @@ static int open_pack_bitmap(struct repository *r,\n \treturn ret;\n }\n \n+static int open_midx_bitmap(struct repository *r,\n+\t\t\t struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *midx;\n+\n+\tassert(!bitmap_git->map);\n+\n+\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n+\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\t\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n+static int open_bitmap(struct repository *r,\n+\t\t struct bitmap_index *bitmap_git)\n+{\n+\tassert(!bitmap_git->map);\n+\n+\tif (!open_midx_bitmap(r, bitmap_git))\n+\t\treturn 0;\n+\treturn open_pack_bitmap(r, bitmap_git);\n+}\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n \n-\tif (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git))\n+\tif (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git))\n+\t\treturn bitmap_git;\n+\n+\tfree_bitmap_index(bitmap_git);\n+\treturn NULL;\n+}\n+\n+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *midx)\n+{\n+\tstruct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));\n+\n+\tif (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(bitmap_git))\n \t\treturn bitmap_git;\n \n \tfree_bitmap_index(bitmap_git);\n@@ -435,10 +572,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \treturn pos;\n }\n \n+static int bitmap_position_midx(struct bitmap_index *bitmap_git,\n+\t\t\t\tconst struct object_id *oid)\n+{\n+\tuint32_t want, got;\n+\tif (!bsearch_midx(oid, bitmap_git->midx, &want))\n+\t\treturn -1;\n+\n+\tif (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0)\n+\t\treturn -1;\n+\treturn got;\n+}\n+\n static int bitmap_position(struct bitmap_index *bitmap_git,\n \t\t\t const struct object_id *oid)\n {\n-\tint pos = bitmap_position_packfile(bitmap_git, oid);\n+\tint pos;\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpos = bitmap_position_midx(bitmap_git, oid);\n+\telse\n+\t\tpos = bitmap_position_packfile(bitmap_git, oid);\n \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\n }\n \n@@ -749,6 +902,7 @@ static void show_objects_for_type(\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n+\t\t\tstruct packed_git *pack;\n \t\t\tstruct object_id oid;\n \t\t\tuint32_t hash = 0, index_pos;\n \t\t\toff_t ofs;\n@@ -758,14 +912,28 @@ static void show_objects_for_type(\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n-\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tstruct multi_pack_index *m = bitmap_git->midx;\n+\t\t\t\tuint32_t pack_id;\n+\n+\t\t\t\tindex_pos = pack_pos_to_midx(m, pos + offset);\n+\t\t\t\tofs = nth_midxed_offset(m, index_pos);\n+\t\t\t\tnth_midxed_object_oid(&oid, m, index_pos);\n+\n+\t\t\t\tpack_id = nth_midxed_pack_int_id(m, index_pos);\n+\t\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\t} else {\n+\t\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\t\tnth_bitmap_object_oid(bitmap_git, &oid, index_pos);\n+\n+\t\t\t\tpack = bitmap_git->pack;\n+\t\t\t}\n \n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n \t\t}\n \t}\n }\n@@ -777,8 +945,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n-\t\t\treturn 1;\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tif (bsearch_midx(&object->oid, bitmap_git->midx, NULL))\n+\t\t\t\treturn 1;\n+\t\t} else {\n+\t\t\tif (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0)\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -865,14 +1038,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git,\n static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \t\t\t\t uint32_t pos)\n {\n-\tstruct packed_git *pack = bitmap_git->pack;\n \tunsigned long size;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = &size;\n \n \tif (pos < bitmap_num_objects(bitmap_git)) {\n-\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tstruct packed_git *pack;\n+\t\toff_t ofs;\n+\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);\n+\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\n+\t\t\tpack = bitmap_git->midx->packs[pack_id];\n+\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\t\t} else {\n+\t\t\tpack = bitmap_git->pack;\n+\t\t\tofs = pack_pos_to_offset(pack, pos);\n+\t\t}\n+\n \t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n \t\t\tnth_bitmap_object_oid(bitmap_git, &oid,\n@@ -1053,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t/* try to open a bitmapped pack, but don't parse it yet\n \t * because we may not need to use it */\n \tCALLOC_ARRAY(bitmap_git, 1);\n-\tif (open_pack_bitmap(revs->repo, bitmap_git) < 0)\n+\tif (open_bitmap(revs->repo, bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tfor (i = 0; i < revs->pending.nr; ++i) {\n@@ -1097,7 +1282,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t * from disk. this is the point of no return; after this the rev_list\n \t * becomes invalidated and we must perform the revwalk through bitmaps\n \t */\n-\tif (load_pack_bitmap(bitmap_git) < 0)\n+\tif (load_bitmap(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n \tobject_array_clear(&revs->pending);\n@@ -1145,19 +1330,43 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n * reused, but you can keep feeding bits.\n */\n static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t struct packed_git *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n-\toff_t offset, header;\n+\toff_t offset, delta_obj_offset;\n \tenum object_type type;\n \tunsigned long size;\n \n-\tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn -1; /* not actually in the pack or MIDX */\n+\t/*\n+\t * try_partial_reuse() is called either on (a) objects in the\n+\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n+\t * objects in the preferred pack of a multi-pack bitmap.\n+\t * Importantly, the latter can pretend as if only a single pack\n+\t * exists because:\n+\t *\n+\t * - The first pack->num_objects bits of a MIDX bitmap are\n+\t * reserved for the preferred pack, and\n+\t *\n+\t * - Ties due to duplicate objects are always resolved in\n+\t * favor of the preferred pack.\n+\t *\n+\t * Therefore we do not need to ever ask the MIDX for its copy of\n+\t * an object by OID, since it will always select it from the\n+\t * preferred pack. Likewise, the selected copy of the base\n+\t * object for any deltas will reside in the same pack.\n+\t *\n+\t * This means that we can reuse pos when looking up the bit in\n+\t * the reuse bitmap, too, since bits corresponding to the\n+\t * preferred pack precede all bits from other packs.\n+\t */\n \n-\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n-\ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n+\tif (pos >= pack->num_objects)\n+\t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n+\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n+\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn -1; /* broken packfile, punt */\n \n@@ -1173,11 +1382,11 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t &offset, type, header);\n+\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn 0;\n-\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n \t\t\treturn 0;\n \n \t\t/*\n@@ -1211,24 +1420,48 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,\n \treturn 0;\n }\n \n+static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+{\n+\tstruct multi_pack_index *m = bitmap_git->midx;\n+\tif (!m)\n+\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n+\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n+}\n+\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t struct packed_git **packfile_out,\n \t\t\t\t uint32_t *entries,\n \t\t\t\t struct bitmap **reuse_out)\n {\n+\tstruct packed_git *pack;\n \tstruct bitmap *result = bitmap_git->result;\n \tstruct bitmap *reuse;\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n \tuint32_t offset;\n-\tuint32_t objects_nr = bitmap_num_objects(bitmap_git);\n+\tuint32_t objects_nr;\n \n \tassert(result);\n \n+\tload_reverse_index(bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\telse\n+\t\tpack = bitmap_git->pack;\n+\tobjects_nr = pack->num_objects;\n+\n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n \n-\t/* Don't mark objects not in the packfile */\n+\t/*\n+\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n+\t * marks objects eligible for reuse, but the pack-reuse code only\n+\t * understands how to reuse a single pack. Since the preferred pack is\n+\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n+\t * we use it instead of another pack. In single-pack bitmaps, the choice\n+\t * is made for us.\n+\t */\n \tif (i > objects_nr / BITS_IN_EWORD)\n \t\ti = objects_nr / BITS_IN_EWORD;\n \n@@ -1244,8 +1477,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n-\t\t\t\t\t &w_curs) < 0) {\n+\t\t\tif (try_partial_reuse(bitmap_git, pack, pos + offset,\n+\t\t\t\t\t reuse, &w_curs) < 0) {\n \t\t\t\t/*\n \t\t\t\t * try_partial_reuse indicated we couldn't reuse\n \t\t\t\t * any bits, so there is no point in trying more\n@@ -1274,7 +1507,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = bitmap_git->pack;\n+\t*packfile_out = pack;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n@@ -1548,6 +1781,12 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n \n+\tif (!bitmap_is_midx(bitmap_git))\n+\t\tload_reverse_index(bitmap_git);\n+\telse if (load_midx_revindex(bitmap_git->midx) < 0)\n+\t\tBUG(\"rebuild_existing_bitmaps: missing required rev-cache \"\n+\t\t \"extension\");\n+\n \tnum_objects = bitmap_num_objects(bitmap_git);\n \tCALLOC_ARRAY(reposition, num_objects);\n \n@@ -1555,8 +1794,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\tstruct object_id oid;\n \t\tstruct object_entry *oe;\n \n-\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n-\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n+\t\tif (bitmap_is_midx(bitmap_git))\n+\t\t\tnth_midxed_object_oid(&oid,\n+\t\t\t\t\t bitmap_git->midx,\n+\t\t\t\t\t pack_pos_to_midx(bitmap_git->midx, i));\n+\t\telse\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t\t pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n@@ -1582,6 +1826,19 @@ void free_bitmap_index(struct bitmap_index *b)\n \tfree(b->ext_index.hashes);\n \tbitmap_free(b->result);\n \tbitmap_free(b->haves);\n+\tif (bitmap_is_midx(b)) {\n+\t\t/*\n+\t\t * Multi-pack bitmaps need to have resources associated with\n+\t\t * their on-disk reverse indexes unmapped so that stale .rev and\n+\t\t * .bitmap files can be removed.\n+\t\t *\n+\t\t * Unlike pack-based bitmaps, multi-pack bitmaps can be read and\n+\t\t * written in the same 'git multi-pack-index write --bitmap'\n+\t\t * process. Close resources so they can be removed safely on\n+\t\t * platforms like Windows.\n+\t\t */\n+\t\tclose_midx_revindex(b->midx);\n+\t}\n \tfree(b);\n }\n \n@@ -1596,7 +1853,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\t\t enum object_type object_type)\n {\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct packed_git *pack = bitmap_git->pack;\n \toff_t total = 0;\n \tstruct ewah_iterator it;\n \teword_t filter;\n@@ -1613,15 +1869,35 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n-\t\t\tsize_t pos;\n-\n \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tpos = base + offset;\n-\t\t\ttotal += pack_pos_to_offset(pack, pos + 1) -\n-\t\t\t\t pack_pos_to_offset(pack, pos);\n+\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tuint32_t pack_pos;\n+\t\t\t\tuint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset);\n+\t\t\t\toff_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\n+\t\t\t\tuint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);\n+\t\t\t\tstruct packed_git *pack = bitmap_git->midx->packs[pack_id];\n+\n+\t\t\t\tif (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {\n+\t\t\t\t\tstruct object_id oid;\n+\t\t\t\t\tnth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos);\n+\n+\t\t\t\t\tdie(_(\"could not find %s in pack %s at offset %\"PRIuMAX),\n+\t\t\t\t\t oid_to_hex(&oid),\n+\t\t\t\t\t pack->pack_name,\n+\t\t\t\t\t (uintmax_t)offset);\n+\t\t\t\t}\n+\n+\t\t\t\ttotal += pack_pos_to_offset(pack, pack_pos + 1) - offset;\n+\t\t\t} else {\n+\t\t\t\tsize_t pos = base + offset;\n+\t\t\t\ttotal += pack_pos_to_offset(bitmap_git->pack, pos + 1) -\n+\t\t\t\t\t pack_pos_to_offset(bitmap_git->pack, pos);\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1672,6 +1948,11 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git,\n \treturn total;\n }\n \n+int bitmap_is_midx(struct bitmap_index *bitmap_git)\n+{\n+\treturn !!bitmap_git->midx;\n+}\n+\n const struct string_list *bitmap_preferred_tips(struct repository *r)\n {\n \treturn repo_config_get_value_multi(r, \"pack.preferbitmaptips\");\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 52ea10de51..81664f933f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -44,6 +44,8 @@ typedef int (*show_reachable_fn)(\n struct bitmap_index;\n \n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *midx);\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n \t\t\t uint32_t *trees, uint32_t *blobs, uint32_t *tags);\n void traverse_bitmap_commit_list(struct bitmap_index *,\n@@ -92,6 +94,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n \t\t\t const char *filename,\n \t\t\t uint16_t options);\n+char *midx_bitmap_filename(struct multi_pack_index *midx);\n+char *pack_bitmap_filename(struct packed_git *p);\n+\n+int bitmap_is_midx(struct bitmap_index *bitmap_git);\n \n const struct string_list *bitmap_preferred_tips(struct repository *r);\n int bitmap_is_preferred_refname(struct repository *r, const char *refname);\ndiff --git a/packfile.c b/packfile.c\nindex 9ef6d98292..371f5488cf 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (starts_with(file_name, \"multi-pack-index\") &&\n-\t ends_with(file_name, \".rev\"))\n+\t (ends_with(file_name, \".bitmap\") || ends_with(file_name, \".rev\")))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434353","messageId":"06db8dbbc163ac41f0bd6e45b83e684660ca2746.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 15/27] pack-bitmap.c: avoid redundant calls to try_partial_reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:19Z","receivedAt":"2021-08-31T20:52:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"try_partial_reuse() is used to mark any bits in the beginning of a\nbitmap whose objects can be reused verbatim from the pack they came\nfrom.\n\nCurrently this function returns void, and signals nothing to the caller\nwhen bits could not be reused. But multi-pack bitmaps would benefit from\nhaving such a signal, because they may try to pass objects which are in\nbounds, but from a pack other than the preferred one.\n\nAny extra calls are noops because of a conditional in\nreuse_partial_packfile_from_bitmap(), but those loop iterations can be\navoided by letting try_partial_reuse() indicate when it can't accept any\nmore bits for reuse, and then listening to that signal.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 40 +++++++++++++++++++++++++++++-----------\n 1 file changed, 29 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d5296750eb..4e37f5d574 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1140,22 +1140,26 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \treturn NULL;\n }\n \n-static void try_partial_reuse(struct bitmap_index *bitmap_git,\n-\t\t\t size_t pos,\n-\t\t\t struct bitmap *reuse,\n-\t\t\t struct pack_window **w_curs)\n+/*\n+ * -1 means \"stop trying further objects\"; 0 means we may or may not have\n+ * reused, but you can keep feeding bits.\n+ */\n+static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t size_t pos,\n+\t\t\t struct bitmap *reuse,\n+\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, header;\n \tenum object_type type;\n \tunsigned long size;\n \n \tif (pos >= bitmap_num_objects(bitmap_git))\n-\t\treturn; /* not actually in the pack or MIDX */\n+\t\treturn -1; /* not actually in the pack or MIDX */\n \n \toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n \tif (type < 0)\n-\t\treturn; /* broken packfile, punt */\n+\t\treturn -1; /* broken packfile, punt */\n \n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n@@ -1172,9 +1176,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n \t\t\t\t\t &offset, type, header);\n \t\tif (!base_offset)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * We assume delta dependencies always point backwards. This\n@@ -1186,7 +1190,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * odd parameters.\n \t\t */\n \t\tif (base_pos >= pos)\n-\t\t\treturn;\n+\t\t\treturn 0;\n \n \t\t/*\n \t\t * And finally, if we're not sending the base as part of our\n@@ -1197,13 +1201,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * object_entry code path handle it.\n \t\t */\n \t\tif (!bitmap_get(reuse, base_pos))\n-\t\t\treturn;\n+\t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n \tbitmap_set(reuse, pos);\n+\treturn 0;\n }\n \n int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n@@ -1239,10 +1244,23 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\ttry_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);\n+\t\t\tif (try_partial_reuse(bitmap_git, pos + offset, reuse,\n+\t\t\t\t\t &w_curs) < 0) {\n+\t\t\t\t/*\n+\t\t\t\t * try_partial_reuse indicated we couldn't reuse\n+\t\t\t\t * any bits, so there is no point in trying more\n+\t\t\t\t * bits in the current word, or any other words\n+\t\t\t\t * in result.\n+\t\t\t\t *\n+\t\t\t\t * Jump out of both loops to avoid future\n+\t\t\t\t * unnecessary calls to try_partial_reuse.\n+\t\t\t\t */\n+\t\t\t\tgoto done;\n+\t\t\t}\n \t\t}\n \t}\n \n+done:\n \tunuse_pack(&w_curs);\n \n \t*entries = bitmap_popcount(reuse);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434352","messageId":"4968229663ae98588b70e46f43540cb346f47455.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 17/27] pack-bitmap: write multi-pack bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:24Z","receivedAt":"2021-08-31T20:52:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Write multi-pack bitmaps in the format described by\nDocumentation/technical/bitmap-format.txt, inferring their presence with\nthe absence of '--bitmap'.\n\nTo write a multi-pack bitmap, this patch attempts to reuse as much of\nthe existing machinery from pack-objects as possible. Specifically, the\nMIDX code prepares a packing_data struct that pretends as if a single\npackfile has been generated containing all of the objects contained\nwithin the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 12 +-\n builtin/multi-pack-index.c | 2 +\n midx.c | 209 ++++++++++++++++++++++++-\n midx.h | 1 +\n 4 files changed, 215 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 0af6beb2dd..a9df3dbd32 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n-\t[--preferred-pack=] \n+\t[--preferred-pack=] [--[no-]bitmap] \n \n DESCRIPTION\n -----------\n@@ -42,6 +42,9 @@ write::\n \t\tmultiple packs contain the same object. `` must\n \t\tcontain at least one object. If not given, ties are\n \t\tbroken in favor of the pack with the lowest mtime.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\n@@ -83,6 +86,13 @@ EXAMPLES\n $ git multi-pack-index write\n -----------------------------------------------\n \n+* Write a MIDX file for the packfiles in the current .git folder with a\n+corresponding bitmap.\n++\n+-------------------------------------------------------------\n+$ git multi-pack-index write --preferred-pack= --bitmap\n+-------------------------------------------------------------\n+\n * Write a MIDX file for the packfiles in an alternate object store.\n +\n -----------------------------------------------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 8ff0dee2ec..73c0113b48 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n \t\t\t N_(\"preferred-pack\"),\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/midx.c b/midx.c\nindex 6a10f7a042..284221ae62 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -13,6 +13,10 @@\n #include \"repository.h\"\n #include \"chunk-format.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -893,6 +897,166 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t\t data->ctx->entries_nr,\n+\t\t\t bitmap_oid_access);\n+\tif (pos < 0)\n+\t\treturn;\n+\n+\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n+\tdata->commits[data->commits_nr++] = commit;\n+}\n+\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb = {0};\n+\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tsetup_revisions(0, NULL, &revs, NULL);\n+\tfor_each_ref(add_ref_to_pending, &revs);\n+\n+\t/*\n+\t * Skipping promisor objects here is intentional, since it only excludes\n+\t * them from the list of reachable commits that we want to select from\n+\t * when computing the selection of MIDX'd commits to receive bitmaps.\n+\t *\n+\t * Reachability bitmaps do require that their objects be closed under\n+\t * reachability, but fetching any objects missing from promisors at this\n+\t * point is too late. But, if one of those objects can be reached from\n+\t * an another object that is included in the bitmap, then we will\n+\t * complain later that we don't have reachability closure (and fail\n+\t * appropriately).\n+\t */\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t unsigned flags)\n+{\n+\tstruct packing_data pdata;\n+\tstruct pack_idx_entry **index;\n+\tstruct commit **commits = NULL;\n+\tuint32_t i, commits_nr;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name, hash_to_hex(midx_hash));\n+\tint ret;\n+\n+\tprepare_midx_packing_data(&pdata, ctx);\n+\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata.nr_objects);\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[i] = &pdata.objects[i].idx;\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(&pdata, index, pdata.nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_finish expects objects in lex order, but pack_order\n+\t * gives us exactly that. use it directly instead of re-sorting the\n+\t * array.\n+\t *\n+\t * This changes the order of objects in 'index' between\n+\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n+\t *\n+\t * The same re-ordering takes place in the single-pack bitmap code via\n+\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n+\t * happens between bitmap_writer_build_type_index() and\n+\t * bitmap_writer_finish().\n+\t */\n+\tfor (i = 0; i < pdata.nr_objects; i++)\n+\t\tindex[ctx->pack_order[i]] = &pdata.objects[i].idx;\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(&pdata);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\treturn ret;\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -938,7 +1102,7 @@ static int write_midx_internal(const char *object_dir,\n \n \t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n \t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\tctx.info[ctx.nr].expired = 0;\n \n \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n@@ -972,8 +1136,26 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n-\t\tgoto cleanup;\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_midx_bitmap_git(the_repository, ctx.m);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n+\t\t\t\t\t\t NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n \n \tif (preferred_pack_name) {\n \t\tint found = 0;\n@@ -989,7 +1171,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tif (!found)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n \t\t\t\tpreferred_pack_name);\n-\t} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {\n+\t} else if (ctx.nr &&\n+\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1121,9 +1304,6 @@ static int write_midx_internal(const char *object_dir,\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n \n-\tif (ctx.m)\n-\t\tclose_object_store(the_repository->objects);\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n@@ -1154,14 +1334,25 @@ static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tif (flags & MIDX_WRITE_BITMAP) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\t\terror(_(\"could not write multi-pack bitmap\"));\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tif (ctx.m)\n+\t\tclose_object_store(the_repository->objects);\n \n \tcommit_lock_file(&lk);\n \n+\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n \tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n \n cleanup:\n@@ -1178,6 +1369,7 @@ static int write_midx_internal(const char *object_dir,\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tfree(midx_name);\n+\n \treturn result;\n }\n \n@@ -1238,6 +1430,7 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r->objects->odb->path, \".bitmap\", NULL);\n \tclear_midx_files_ext(r->objects->odb->path, \".rev\", NULL);\n \n \tfree(midx);\ndiff --git a/midx.h b/midx.h\nindex 1172df1a71..350f4d0a7b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -41,6 +41,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n+#define MIDX_WRITE_BITMAP (1 << 2)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n char *get_midx_filename(const char *object_dir);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434354","messageId":"5d60b07e2e0631eaaefe21cc94d3d1c7d9f7cfe3.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 18/27] t5310: move some tests to lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:26Z","receivedAt":"2021-08-31T20:52:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We'll soon be adding a test script that will cover many of the same\nbitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many\nof the applicable tests as we can so we don't have to rewrite them.\n\nThere should be no functional change to t5310; we still run the same\noperations in the same order.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 236 ++++++++++++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 227 +-------------------------------------\n 2 files changed, 240 insertions(+), 223 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex fe3f98be24..77464da6fd 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,3 +1,6 @@\n+# Helpers for scripts testing bitmap functionality; see t5310 for\n+# example usage.\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -24,3 +27,236 @@ test_bitmap_traversal () {\n \ttest_cmp \"$1.normalized\" \"$2.normalized\" &&\n \trm -f \"$1.normalized\" \"$2.normalized\"\n }\n+\n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other second\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-second * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for second (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n+#\n+# second: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n+setup_bitmap_history() {\n+\ttest_expect_success 'setup repo with moderate-sized history' '\n+\t\ttest_commit_bulk --id=file 10 &&\n+\t\tgit branch -M second &&\n+\t\tgit checkout -b other HEAD~5 &&\n+\t\ttest_commit_bulk --id=side 10 &&\n+\n+\t\t# add complicated history setup, including merges and\n+\t\t# ambiguous merge-bases\n+\n+\t\tgit checkout -b merge-left other~2 &&\n+\t\tgit merge second~2 -m \"merge-left\" &&\n+\n+\t\tgit checkout -b merge-right second~1 &&\n+\t\tgit merge other~1 -m \"merge-right\" &&\n+\n+\t\tgit checkout -b octo-second second &&\n+\t\tgit merge merge-left merge-right -m \"octopus-second\" &&\n+\n+\t\tgit checkout -b octo-other other &&\n+\t\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\t\tgit checkout other &&\n+\t\tgit merge octo-other -m \"pull octopus\" &&\n+\n+\t\tgit checkout second &&\n+\t\tgit merge octo-second -m \"pull octopus\" &&\n+\n+\t\t# Remove these branches so they are not selected\n+\t\t# as bitmap tips\n+\t\tgit branch -D merge-left &&\n+\t\tgit branch -D merge-right &&\n+\t\tgit branch -D octo-other &&\n+\t\tgit branch -D octo-second &&\n+\n+\t\t# add padding to make these merges less interesting\n+\t\t# and avoid having them selected for bitmaps\n+\t\ttest_commit_bulk --id=file 100 &&\n+\t\tgit checkout other &&\n+\t\ttest_commit_bulk --id=side 100 &&\n+\t\tgit checkout second &&\n+\n+\t\tbitmaptip=$(git rev-parse second) &&\n+\t\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n+\t\tgit tag tagged-blob $blob\n+\t'\n+}\n+\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n+\t\tgit rev-list --count other...second >expect &&\n+\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n+\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n+\t'\n+\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n+\t\ttest_bitmap_traversal expect actual\n+\t'\n+\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n+\t\tgrep $blob actual\n+\t'\n+}\n+\n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"second\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n+basic_bitmap_tests () {\n+\ttip=\"$1\"\n+\ttest_expect_success 'rev-list --test-bitmap verifies bitmaps' \"\n+\t\tgit rev-list --test-bitmap \"${tip:-HEAD}\"\n+\t\"\n+\n+\trev_list_tests 'full bitmap'\n+\n+\ttest_expect_success 'clone from bitmapped repository' '\n+\t\trm -fr clone.git &&\n+\t\tgit clone --no-local --bare . clone.git &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'partial clone from bitmapped repository' '\n+\t\ttest_config uploadpack.allowfilter true &&\n+\t\trm -fr partial-clone.git &&\n+\t\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n+\t\t(\n+\t\t\tcd partial-clone.git &&\n+\t\t\tpack=$(echo objects/pack/*.pack) &&\n+\t\t\tgit verify-pack -v \"$pack\" >have &&\n+\t\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n+\t\t\t# we expect this single blob because of the direct ref\n+\t\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n+\t\t\ttest_cmp expect blobs\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'setup further non-bitmapped commits' '\n+\t\ttest_commit_bulk --id=further 10\n+\t'\n+\n+\trev_list_tests 'partial bitmap'\n+\n+\ttest_expect_success 'fetch (partial bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'enumerating progress counts pack-reused objects' '\n+\t\tcount=$(git rev-list --objects --all --count) &&\n+\t\tgit repack -adb &&\n+\n+\t\t# check first with only reused objects; confirm that our\n+\t\t# progress showed the right number, and also that we did\n+\t\t# pack-reuse as expected. Check only the final \"done\"\n+\t\t# line of the meter (there may be an arbitrary number of\n+\t\t# intermediate lines ending with CR).\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $count, done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr &&\n+\n+\t\t# now the same but with one non-reused object\n+\t\tgit commit --allow-empty -m \"an extra commit object\" &&\n+\t\tGIT_PROGRESS_DELAY=0 \\\n+\t\t\tgit pack-objects --all --stdout --progress \\\n+\t\t\t/dev/null 2>stderr &&\n+\t\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n+\t\tgrep \"pack-reused $count\" stderr\n+\t'\n+}\n+\n+# have_delta \n+#\n+# Note that because this relies on cat-file, it might find _any_ copy of an\n+# object in the repository. The caller is responsible for making sure\n+# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n+have_delta () {\n+\techo $2 >expect &&\n+\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n+\ttest_cmp expect actual\n+}\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex b02838750e..4318f84d53 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -25,93 +25,10 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-# To ensure the logic for \"maximal commits\" is exercised, make\n-# the repository a bit more complicated.\n-#\n-# other second\n-# * *\n-# (99 commits) (99 commits)\n-# * *\n-# |\\ /|\n-# | * octo-other octo-second * |\n-# |/|\\_________ ____________/|\\|\n-# | \\ \\/ __________/ |\n-# | | ________/\\ / |\n-# * |/ * merge-right *\n-# | _|__________/ \\____________ |\n-# |/ | \\|\n-# (l1) * * merge-left * (r1)\n-# | / \\________________________ |\n-# |/ \\|\n-# (l2) * * (r2)\n-# \\___________________________ |\n-# \\|\n-# * (base)\n-#\n-# We only push bits down the first-parent history, which\n-# makes some of these commits unimportant!\n-#\n-# The important part for the maximal commit algorithm is how\n-# the bitmasks are extended. Assuming starting bit positions\n-# for second (bit 0) and other (bit 1), the bitmasks at the\n-# end should be:\n-#\n-# second: 1 (maximal, selected)\n-# other: 01 (maximal, selected)\n-# (base): 11 (maximal)\n-#\n-# This complicated history was important for a previous\n-# version of the walk that guarantees never walking a\n-# commit multiple times. That goal might be important\n-# again, so preserve this complicated case. For now, this\n-# test will guarantee that the bitmaps are computed\n-# correctly, even with the repeat calculations.\n+setup_bitmap_history\n \n-test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 10 &&\n-\tgit branch -M second &&\n-\tgit checkout -b other HEAD~5 &&\n-\ttest_commit_bulk --id=side 10 &&\n-\n-\t# add complicated history setup, including merges and\n-\t# ambiguous merge-bases\n-\n-\tgit checkout -b merge-left other~2 &&\n-\tgit merge second~2 -m \"merge-left\" &&\n-\n-\tgit checkout -b merge-right second~1 &&\n-\tgit merge other~1 -m \"merge-right\" &&\n-\n-\tgit checkout -b octo-second second &&\n-\tgit merge merge-left merge-right -m \"octopus-second\" &&\n-\n-\tgit checkout -b octo-other other &&\n-\tgit merge merge-left merge-right -m \"octopus-other\" &&\n-\n-\tgit checkout other &&\n-\tgit merge octo-other -m \"pull octopus\" &&\n-\n-\tgit checkout second &&\n-\tgit merge octo-second -m \"pull octopus\" &&\n-\n-\t# Remove these branches so they are not selected\n-\t# as bitmap tips\n-\tgit branch -D merge-left &&\n-\tgit branch -D merge-right &&\n-\tgit branch -D octo-other &&\n-\tgit branch -D octo-second &&\n-\n-\t# add padding to make these merges less interesting\n-\t# and avoid having them selected for bitmaps\n-\ttest_commit_bulk --id=file 100 &&\n-\tgit checkout other &&\n-\ttest_commit_bulk --id=side 100 &&\n-\tgit checkout second &&\n-\n-\tbitmaptip=$(git rev-parse second) &&\n-\tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n-\tgit tag tagged-blob $blob &&\n-\tgit config repack.writebitmaps true\n+test_expect_success 'setup writing bitmaps during repack' '\n+\tgit config repack.writeBitmaps true\n '\n \n test_expect_success 'full repack creates bitmaps' '\n@@ -123,109 +40,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n-test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-\tgit rev-list --test-bitmap HEAD\n-'\n-\n-rev_list_tests_head () {\n-\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count $branch~5..$branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n-\t\tgit rev-list --count -n 1 $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n-\t\tgit rev-list --count other...second >expect &&\n-\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n-\t\tgit rev-list --count $branch -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n-\t\tgit rev-list --count --objects $branch >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n-\t\ttest_cmp expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n-\t\tgit rev-list --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list $branch >expect &&\n-\t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n-\t'\n-\n-\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n-\t\tgit rev-list --objects $branch >expect &&\n-\t\ttest_bitmap_traversal expect actual\n-\t'\n-\n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n-\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n-\t\tgrep $blob actual\n-\t'\n-}\n-\n-rev_list_tests () {\n-\tstate=$1\n-\n-\tfor branch in \"second\" \"other\"\n-\tdo\n-\t\trev_list_tests_head\n-\tdone\n-}\n-\n-rev_list_tests 'full bitmap'\n-\n-test_expect_success 'clone from bitmapped repository' '\n-\tgit clone --no-local --bare . clone.git &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'partial clone from bitmapped repository' '\n-\ttest_config uploadpack.allowfilter true &&\n-\tgit clone --no-local --bare --filter=blob:none . partial-clone.git &&\n-\t(\n-\t\tcd partial-clone.git &&\n-\t\tpack=$(echo objects/pack/*.pack) &&\n-\t\tgit verify-pack -v \"$pack\" >have &&\n-\t\tawk \"/blob/ { print \\$1 }\" blobs &&\n-\t\t# we expect this single blob because of the direct ref\n-\t\tgit rev-parse refs/tags/tagged-blob >expect &&\n-\t\ttest_cmp expect blobs\n-\t)\n-'\n-\n-test_expect_success 'setup further non-bitmapped commits' '\n-\ttest_commit_bulk --id=further 10\n-'\n-\n-rev_list_tests 'partial bitmap'\n-\n-test_expect_success 'fetch (partial bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+basic_bitmap_tests\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -461,40 +276,6 @@ test_expect_success 'truncated bitmap fails gracefully (cache)' '\n \ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n-test_expect_success 'enumerating progress counts pack-reused objects' '\n-\tcount=$(git rev-list --objects --all --count) &&\n-\tgit repack -adb &&\n-\n-\t# check first with only reused objects; confirm that our progress\n-\t# showed the right number, and also that we did pack-reuse as expected.\n-\t# Check only the final \"done\" line of the meter (there may be an\n-\t# arbitrary number of intermediate lines ending with CR).\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $count, done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr &&\n-\n-\t# now the same but with one non-reused object\n-\tgit commit --allow-empty -m \"an extra commit object\" &&\n-\tGIT_PROGRESS_DELAY=0 \\\n-\t\tgit pack-objects --all --stdout --progress \\\n-\t\t/dev/null 2>stderr &&\n-\tgrep \"Enumerating objects: $((count+1)), done\" stderr &&\n-\tgrep \"pack-reused $count\" stderr\n-'\n-\n-# have_delta \n-#\n-# Note that because this relies on cat-file, it might find _any_ copy of an\n-# object in the repository. The caller is responsible for making sure\n-# there's only one (e.g., via \"repack -ad\", or having just fetched a copy).\n-have_delta () {\n-\techo $2 >expect &&\n-\techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n-\ttest_cmp expect actual\n-}\n-\n # Create a state of history with these properties:\n #\n # - refs that allow a client to fetch some new history, while sharing some old\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434355","messageId":"1a9c3538dbb7b7077e4a34b1d40628a770e44af8.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 19/27] t/helper/test-read-midx.c: add --checksum mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:28Z","receivedAt":"2021-08-31T20:52:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subsequent tests will want to check for the existence of a multi-pack\nbitmap which matches the multi-pack-index stored in the pack directory.\n\nThe multi-pack bitmap includes the hex checksum of the MIDX it\ncorresponds to in its filename (for example,\n'$packdir/multi-pack-index-.bitmap'). As a result, some tests\nwant a way to learn what '' is.\n\nThis helper addresses that need by printing the checksum of the\nrepository's multi-pack-index.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 16 +++++++++++++++-\n t/lib-bitmap.sh | 4 ++++\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 7c2eb11a8e..cb0d27049a 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)\n \treturn 0;\n }\n \n+static int read_midx_checksum(const char *object_dir)\n+{\n+\tstruct multi_pack_index *m;\n+\n+\tsetup_git_directory();\n+\tm = load_multi_pack_index(object_dir, 1);\n+\tif (!m)\n+\t\treturn 1;\n+\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n+\telse if (!strcmp(argv[1], \"--checksum\"))\n+\t\treturn read_midx_checksum(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 77464da6fd..21d0392dda 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -260,3 +260,7 @@ have_delta () {\n \techo $1 | git cat-file --batch-check=\"%(deltabase)\" >actual &&\n \ttest_cmp expect actual\n }\n+\n+midx_checksum () {\n+\ttest-tool read-midx --checksum \"$1\"\n+}\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434356","messageId":"8895114ace2e9d5392f5c71b7ea4190b90c01db8.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 20/27] t5326: test multi-pack bitmap behavior","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:31Z","receivedAt":"2021-08-31T20:52:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch introduces a new test, t5326, which tests the basic\nfunctionality of multi-pack bitmaps.\n\nSome trivial behavior is tested, such as:\n\n - Whether bitmaps can be generated with more than one pack.\n - Whether clones can be served with all objects in the bitmap.\n - Whether follow-up fetches can be served with some objects outside of\n the server's bitmap\n\nThese use lib-bitmap's tests (which in turn were pulled from t5310), and\nwe cover cases where the MIDX represents both a single pack and multiple\npacks.\n\nIn addition, some non-trivial and MIDX-specific behavior is tested, too,\nincluding:\n\n - Whether multi-pack bitmaps behave correctly with respect to the\n pack-reuse machinery when the base for some object is selected from\n a different pack than the delta.\n - Whether multi-pack bitmaps correctly respect the\n pack.preferBitmapTips configuration.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 286 ++++++++++++++++++++++++++++++++++\n 1 file changed, 286 insertions(+)\n create mode 100755 t/t5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..4ad7c2c969\n--- /dev/null\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,286 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality'\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+setup_bitmap_history\n+\n+test_expect_success 'enable core.multiPackIndex' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_expect_success 'create single-pack midx with bitmaps' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success 'create new additional packs' '\n+\tfor i in $(test_seq 1 16)\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\n+\tgit checkout -b other2 HEAD~8 &&\n+\tfor i in $(test_seq 1 8)\n+\tdo\n+\t\ttest_commit \"side-$i\" &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\tgit checkout second\n+'\n+\n+test_expect_success 'create multi-pack midx with bitmaps' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\tls $objdir/pack/pack-*.pack >packs &&\n+\ttest_line_count = 25 packs &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+'\n+\n+basic_bitmap_tests\n+\n+test_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\tgit multi-pack-index write --bitmap &&\n+\n+\ttest_commit respect--no-bitmap &&\n+\tgit repack -d &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\n+\tgit multi-pack-index write --no-bitmap &&\n+\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n+'\n+\n+test_expect_success 'setup midx with base from later pack' '\n+\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t# prefers to delete contents out of a base rather than add to a shorter\n+\t# object.\n+\ttest_seq 1 128 >a &&\n+\ttest_seq 1 130 >b &&\n+\n+\tgit add a b &&\n+\tgit commit -m \"initial commit\" &&\n+\n+\ta=$(git rev-parse HEAD:a) &&\n+\tb=$(git rev-parse HEAD:b) &&\n+\n+\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$a\n+\t$b\n+\tEOF\n+\t) &&\n+\n+\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t# any other object.\n+\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t$b\n+\t$(git rev-parse HEAD)\n+\t$(git rev-parse HEAD^{tree})\n+\tEOF\n+\t) &&\n+\n+\tgit prune-packed &&\n+\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t# reuse.\n+\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\thave_delta $a $b &&\n+\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+'\n+\n+rev_list_tests 'full bitmap with backwards delta'\n+\n+test_expect_success 'clone with bitmaps enabled' '\n+\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\tgit rev-parse HEAD >expect &&\n+\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+bitmap_reuse_tests() {\n+\tfrom=$1\n+\tto=$2\n+\n+\ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk 16 &&\n+\t\t\tgit tag old-tip &&\n+\n+\t\t\tgit config core.multiPackIndex true &&\n+\t\t\tif test \"MIDX\" = \"$from\"\n+\t\t\tthen\n+\t\t\t\tgit repack -Ad &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tgit repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit_bulk --id=further 16 &&\n+\t\t\tgit tag new-tip &&\n+\n+\t\t\tif test \"MIDX\" = \"$to\"\n+\t\t\tthen\n+\t\t\t\tgit repack -d &&\n+\t\t\t\tgit multi-pack-index write --bitmap\n+\t\t\telse\n+\t\t\t\tgit repack -Adb\n+\t\t\tfi\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit for-each-ref &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n+\t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n+\t\t)\n+\t'\n+}\n+\n+bitmap_reuse_tests 'pack' 'MIDX'\n+bitmap_reuse_tests 'MIDX' 'pack'\n+bitmap_reuse_tests 'MIDX' 'MIDX'\n+\n+test_expect_success 'missing object closure fails gracefully' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit loose &&\n+\t\ttest_commit packed &&\n+\n+\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t# commit.\n+\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t$(git rev-parse packed)\n+\t\tEOF\n+\n+\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\tgrep \"doesn.t have full closure\" err &&\n+\t\ttest_path_is_missing $midx\n+\t)\n+'\n+\n+test_expect_success 'setup partial bitmaps' '\n+\ttest_commit packed &&\n+\tgit repack &&\n+\ttest_commit loose &&\n+\tgit multi-pack-index write --bitmap 2>err &&\n+\ttest_path_is_file $midx &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+'\n+\n+basic_bitmap_tests HEAD~\n+\n+test_expect_success 'removing a MIDX clears stale bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\tstale_rev=$midx-$(midx_checksum $objdir).rev &&\n+\t\trm $midx &&\n+\n+\t\t# Then write a new MIDX.\n+\t\ttest_commit new &&\n+\t\tgit repack &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\ttest_path_is_missing $stale_bitmap &&\n+\t\ttest_path_is_missing $stale_rev\n+\t)\n+'\n+\n+test_expect_success 'pack.preferBitmapTips' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\tbitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n+test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434357","messageId":"a4f4d90bba48bbf9cc6f150cdb61310b7678c3c8.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 22/27] t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:36Z","receivedAt":"2021-08-31T20:52:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap confuses many of the tests in t5310, which\nexpect to control whether and how bitmaps are written. Since the\nrelevant MIDX-bitmap tests here are covered already in t5326, let's just\ndisable the flag for the whole t5310 script.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 4318f84d53..673baa5c3c 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n . \"$TEST_DIRECTORY\"/lib-bundle.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in\n+# their place.\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434359","messageId":"94b1317e0c42bf5fd7492eb6e51e8ff6079595cc.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 21/27] t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:33Z","receivedAt":"2021-08-31T20:52:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nGenerating a MIDX bitmap causes tests which repack in a partial clone to\nfail because they are missing objects. Missing objects is an expected\ncomponent of tests in t0410, so disable this knob altogether. Graceful\ndegradation when writing a bitmap with missing objects is tested in\nt5326.\n\nSigned-off-by: Taylor Blau \n---\n t/t0410-partial-clone.sh | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex bbcc51ee8e..bba679685f 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -4,6 +4,9 @@ test_description='partial clone'\n \n . ./test-lib.sh\n \n+# missing promisor objects cause repacks which write bitmaps to fail\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n delete_object () {\n \trm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')\n }\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434358","messageId":"92a6370e7715c34a1ac7ccc090ef501c1b5dcc58.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 23/27] t5319: don't write MIDX bitmaps in t5319","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:38Z","receivedAt":"2021-08-31T20:52:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This test is specifically about generating a midx still respecting a\npack-based bitmap file. Generating a MIDX bitmap would confuse the test.\nLet's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to\nmake sure we don't do so.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex d7e4988f2b..b3f9f3969d 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -532,7 +532,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '\n compare_results_with_midx \"after repack\"\n \n test_expect_success 'multi-pack-index and pack-bitmap' '\n-\tgit -c repack.writeBitmaps=true repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=true repack -ad &&\n \tgit multi-pack-index write &&\n \tgit rev-list --test-bitmap HEAD\n '\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434360","messageId":"c49dc46fb2d123090918869cd1307f8b3f8ffd06.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 24/27] t7700: update to work with MIDX bitmap test knob","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:41Z","receivedAt":"2021-08-31T20:52:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A number of these tests are focused only on pack-based bitmaps and need\nto be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where\nnecessary.\n\nSigned-off-by: Taylor Blau \n---\n t/t7700-repack.sh | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 25b235c063..98eda3bfeb 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '\n \n test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit repack -Adbl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&\n \ttest_has_duplicate_object true\n '\n \n test_expect_success 'writing bitmaps via config can duplicate .keep objects' '\n \t# build on $oid, $packid, and .keep state from previous\n-\tgit -c repack.writebitmaps=true repack -Adl &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writebitmaps=true repack -Adl &&\n \ttest_has_duplicate_object true\n '\n \n@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '\n \n test_expect_success 'bitmaps are created by default in bare repos' '\n \tgit clone --bare .git bare.git &&\n-\tgit -C bare.git repack -ad &&\n+\trm -f bare.git/objects/pack/*.bitmap &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap) &&\n \ttest_path_is_file \"$bitmap\"\n '\n@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '\n '\n \n test_expect_success 'bitmaps can be disabled on bare repos' '\n-\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -c repack.writeBitmaps=false -C bare.git repack -ad &&\n \tbitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&\n \ttest -z \"$bitmap\"\n '\n@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\tGIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434361","messageId":"44a4800756de7749e17cea0db5790a876db96d28.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 25/27] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:43Z","receivedAt":"2021-08-31T20:52:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment\nvariable to also write a multi-pack bitmap when\n'GIT_TEST_MULTI_PACK_INDEX' is set.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 12 ++++++++++--\n ci/run-build-and-tests.sh | 1 +\n midx.h | 2 ++\n t/README | 4 ++++\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..82ab668272 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (!(pack_everything & ALL_INTO_ONE) ||\n \t\t !is_bare_repository())\n \t\t\twrite_bitmaps = 0;\n+\t} else if (write_bitmaps &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n+\t\t git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {\n+\t\twrite_bitmaps = 0;\n \t}\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tupdate_server_info(0);\n \tremove_temporary_files();\n \n-\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), NULL, 0);\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {\n+\t\tunsigned flags = 0;\n+\t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n+\t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n+\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t}\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..7ee9ba9325 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -23,6 +23,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n+\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n \texport GIT_TEST_WRITE_REV_INDEX=1\ndiff --git a/midx.h b/midx.h\nindex 350f4d0a7b..aa3da557bb 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,8 @@ struct pack_entry;\n struct repository;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n+\t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n \n struct multi_pack_index {\n \tstruct multi_pack_index *next;\ndiff --git a/t/README b/t/README\nindex 9e70122302..12014aa988 100644\n--- a/t/README\n+++ b/t/README\n@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the\n+'--bitmap' option on all invocations of 'git multi-pack-index write',\n+and ignores pack-objects' '--write-bitmap-index'.\n+\n GIT_TEST_SIDEBAND_ALL=, when true, overrides the\n 'uploadpack.allowSidebandAll' setting to true, and when false, forces\n fetch-pack to not request sideband-all (even if the server advertises\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434362","messageId":"bf0981b606ebf2620752e1aeb8dcf7b6ce9222cb.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 26/27] p5310: extract full and partial bitmap tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:46Z","receivedAt":"2021-08-31T20:53:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A new p5326 introduced by the next patch will want these same tests,\ninterjecting its own setup in between. Move them out so that both perf\ntests can reuse them.\n\nSigned-off-by: Taylor Blau \n---\n t/perf/lib-bitmap.sh | 69 ++++++++++++++++++++++++++++++++++++\n t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------\n 2 files changed, 72 insertions(+), 62 deletions(-)\n create mode 100644 t/perf/lib-bitmap.sh\n\ndiff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh\nnew file mode 100644\nindex 0000000000..63d3bc7cec\n--- /dev/null\n+++ b/t/perf/lib-bitmap.sh\n@@ -0,0 +1,69 @@\n+# Helper functions for testing bitmap performance; see p5310.\n+\n+test_full_bitmap () {\n+\ttest_perf 'simulated clone' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'simulated fetch' '\n+\t\thave=$(git rev-list HEAD~100 -1) &&\n+\t\t{\n+\t\t\techo HEAD &&\n+\t\t\techo ^$have\n+\t\t} | git pack-objects --revs --stdout >/dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (commits)' '\n+\t\tgit rev-list --all --use-bitmap-index >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list (objects)' '\n+\t\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tag negated via --not --all (objects)' '\n+\t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with negative tag (objects)' '\n+\t\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:none' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:none >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with blob:limit=1k' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=blob:limit=1k >/dev/null\n+\t'\n+\n+\ttest_perf 'rev-list count with tree:0' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+\n+\ttest_perf 'simulated partial clone' '\n+\t\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n+\t'\n+}\n+\n+test_partial_bitmap () {\n+\ttest_perf 'clone (partial bitmap)' '\n+\t\tgit pack-objects --stdout --all /dev/null\n+\t'\n+\n+\ttest_perf 'pack to file (partial bitmap)' '\n+\t\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n+\t'\n+\n+\ttest_perf 'rev-list with tree filter (partial bitmap)' '\n+\t\tgit rev-list --use-bitmap-index --count --objects --all \\\n+\t\t\t--filter=tree:0 >/dev/null\n+\t'\n+}\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 452be01056..7ad4f237bc 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -2,6 +2,7 @@\n \n test_description='Tests pack performance using bitmaps'\n . ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n test_perf_large_repo\n \n@@ -25,56 +26,7 @@ test_perf 'repack to disk' '\n \tgit repack -ad\n '\n \n-test_perf 'simulated clone' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'simulated fetch' '\n-\thave=$(git rev-list HEAD~100 -1) &&\n-\t{\n-\t\techo HEAD &&\n-\t\techo ^$have\n-\t} | git pack-objects --revs --stdout >/dev/null\n-'\n-\n-test_perf 'pack to file (bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n-'\n-\n-test_perf 'rev-list (commits)' '\n-\tgit rev-list --all --use-bitmap-index >/dev/null\n-'\n-\n-test_perf 'rev-list (objects)' '\n-\tgit rev-list --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with tag negated via --not --all (objects)' '\n-\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list with negative tag (objects)' '\n-\tgit rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:none' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:none >/dev/null\n-'\n-\n-test_perf 'rev-list count with blob:limit=1k' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=blob:limit=1k >/dev/null\n-'\n-\n-test_perf 'rev-list count with tree:0' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n-\n-test_perf 'simulated partial clone' '\n-\tgit pack-objects --stdout --all --filter=blob:none /dev/null\n-'\n+test_full_bitmap\n \n test_expect_success 'create partial bitmap state' '\n \t# pick a commit to represent the repo tip in the past\n@@ -97,17 +49,6 @@ test_expect_success 'create partial bitmap state' '\n \tgit update-ref HEAD $orig_tip\n '\n \n-test_perf 'clone (partial bitmap)' '\n-\tgit pack-objects --stdout --all /dev/null\n-'\n-\n-test_perf 'pack to file (partial bitmap)' '\n-\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n-'\n-\n-test_perf 'rev-list with tree filter (partial bitmap)' '\n-\tgit rev-list --use-bitmap-index --count --objects --all \\\n-\t\t--filter=tree:0 >/dev/null\n-'\n+test_partial_bitmap\n \n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"434363","messageId":"6888fe01aae6710959f52f15a862f14adba02d22.1630443072.git.me@ttaylorr.com","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"[PATCH v5 27/27] p5326: perf tests for MIDX bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-08-31T20:52:48Z","receivedAt":"2021-08-31T20:53:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"These new performance tests demonstrate effectively the same behavior as\np5310, but use a multi-pack bitmap instead of a single-pack one.\n\nNotably, p5326 does not create a MIDX bitmap with multiple packs. This\nis so we can measure a direct comparison between it and p5310. Any\ndifference between the two is measuring just the overhead of using MIDX\nbitmaps.\n\nHere are the results of p5310 and p5326 together, measured at the same\ntime and on the same machine (using a Xenon W-2255 CPU):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5310.2: repack to disk 96.78(93.39+11.33)\n 5310.3: simulated clone 9.98(9.79+0.19)\n 5310.4: simulated fetch 1.75(4.26+0.19)\n 5310.5: pack to file (bitmap) 28.20(27.87+8.70)\n 5310.6: rev-list (commits) 0.41(0.36+0.05)\n 5310.7: rev-list (objects) 1.61(1.54+0.07)\n 5310.8: rev-list count with blob:none 0.25(0.21+0.04)\n 5310.9: rev-list count with blob:limit=1k 2.65(2.54+0.10)\n 5310.10: rev-list count with tree:0 0.23(0.19+0.04)\n 5310.11: simulated partial clone 4.34(4.21+0.12)\n 5310.13: clone (partial bitmap) 11.05(12.21+0.48)\n 5310.14: pack to file (partial bitmap) 31.25(34.22+3.70)\n 5310.15: rev-list with tree filter (partial bitmap) 0.26(0.22+0.04)\n\nversus the same tests (this time using a multi-pack index):\n\n Test HEAD\n ------------------------------------------------------------------------\n 5326.2: setup multi-pack index 78.99(75.29+11.58)\n 5326.3: simulated clone 11.78(11.56+0.22)\n 5326.4: simulated fetch 1.70(4.49+0.13)\n 5326.5: pack to file (bitmap) 28.02(27.72+8.76)\n 5326.6: rev-list (commits) 0.42(0.36+0.06)\n 5326.7: rev-list (objects) 1.65(1.58+0.06)\n 5326.8: rev-list count with blob:none 0.26(0.21+0.05)\n 5326.9: rev-list count with blob:limit=1k 2.97(2.86+0.10)\n 5326.10: rev-list count with tree:0 0.25(0.20+0.04)\n 5326.11: simulated partial clone 5.65(5.49+0.16)\n 5326.13: clone (partial bitmap) 12.22(13.43+0.38)\n 5326.14: pack to file (partial bitmap) 30.05(31.57+7.25)\n 5326.15: rev-list with tree filter (partial bitmap) 0.24(0.20+0.04)\n\nThere is slight overhead in \"simulated clone\", \"simulated partial\nclone\", and \"clone (partial bitmap)\". Unsurprisingly, that overhead is\ndue to using the MIDX's reverse index to map between bit positions and\nMIDX positions.\n\nThis can be reproduced by running \"git repack -adb\" along with \"git\nmulti-pack-index write --bitmap\" in a large-ish repository. Then run:\n\n $ perf record -o pack.perf git -c core.multiPackIndex=false \\\n pack-objects --all --stdout >/dev/null /dev/null \n---\n t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh\n\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nnew file mode 100755\nindex 0000000000..5845109ac7\n--- /dev/null\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -0,0 +1,43 @@\n+#!/bin/sh\n+\n+test_description='Tests performance using midx bitmaps'\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n+\n+test_perf_large_repo\n+\n+test_expect_success 'enable multi-pack index' '\n+\tgit config core.multiPackIndex true\n+'\n+\n+test_perf 'setup multi-pack index' '\n+\tgit repack -ad &&\n+\tgit multi-pack-index write --bitmap\n+'\n+\n+test_full_bitmap\n+\n+test_expect_success 'create partial bitmap state' '\n+\t# pick a commit to represent the repo tip in the past\n+\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t# now pretend we have just one tip\n+\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\tgit update-ref HEAD $cutoff &&\n+\n+\t# and then repack, which will leave us with a nice\n+\t# big bitmap pack of the \"old\" history, and all of\n+\t# the new history will be loose, as if it had been pushed\n+\t# up incrementally and exploded via unpack-objects\n+\tgit repack -Ad &&\n+\tgit multi-pack-index write --bitmap &&\n+\n+\t# and now restore our original tip, as if the pushes\n+\t# had happened\n+\tgit update-ref HEAD $orig_tip\n+'\n+\n+test_partial_bitmap\n+\n+test_done\n-- \n2.33.0.96.g73915697e6\n"},{"id":"434406","messageId":"YS9P2j1FoI5Q+ZLX@coredump.intra.peff.net","threadId":"55464","inReplyTo":"xmqqk0k1wp3x.fsf@gitster.g","subject":"Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T10:03:06Z","receivedAt":"2021-09-01T10:03:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 09:33:38AM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> >> I think taking a look to see if ../config exists to use the data\n> >> might be helpful for some cases, but should not be a blocker for\n> >> completing the requested operation. The config from the non-alternate\n> >> repo should be sufficient for this (somewhat strange) case.\n> >\n> > Yes, agreed. We have long supported these kind of \"bare\" alternates, and\n> > I wouldn't be surprised if they are in wide use (though I do wonder how\n> > folks actually modify them, since most commands that touch objects\n> > really do want to be in a repository).\n> \n> I kind of find the above two somewhat surprising, but I am willing\n> to go with the less safer option if that is what people want.\n> \n> It has been perfectly OK in the pre-alternative-hash-algorithms\n> world, but we no longer live in such a world, so we'd need to come\n> up with a way to keep using alternates in a safer way.\n\nI think the point is that most people _do_ still live in that world.\nThey have not started using the new hash algorithm yet, and what they\nhave been doing for years will continue to work. Likewise, once they\nswitch, things will continue to work as long as each repo's alternates\nuse the same hash.\n\nSo my reasoning was less \"this is useful, and a good idea\" and more \"it\nworks now, and will probably continue to work OK in practice, so taking\nit away will probably bother people\".\n\nNow if somebody wants to make an argument that they are not actually\nworkable now, I could buy that. ;) You cannot even run \"pack-objects\"\nwithout a repository, though it is not too hard to copy the result\naround.\n\n> > But I suspect all of this is moot for now, beyond being able to return a\n> > nicer error message. The rest of the code is not at all ready to handle\n> > packs with two different hashes in the same process.\n> \n> I do not think it is all that urgent to make it possible for packs\n> with different algorithms to be used. It is sufficient to _ignore_\n> (or error out) configured odb that is incompatible with the current\n> repository.\n\nYes, I think that would be an improvement. I just don't find it all that\nurgent, since they're likely to get an error anyway (just probably one\nthat is more mysterious). Given the work involved to even detect the\nsituation, it doesn't seem like that high a priority to me.\n\n-Peff\n"},{"id":"434463","messageId":"xmqq5yvkqidc.fsf@gitster.g","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-01T18:07:59Z","receivedAt":"2021-09-01T18:08:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here is another version of the multi-pack reachability bitmaps series. It is\n> virtually unchanged since last time.\n>\n> The changes that did occur is that I integrated Johannes' patch from [1] to fix\n> cleaning up MIDX .rev and .bitmap files when using `--object-dir`. That inspired\n> a lengthy discussion [2] about `--object-dir`, alternates, object-format and\n> running the MIDX builtin outside of a Git repository.\n>\n> This series resolves that discussion by leaving everything as-is, and only\n> changing the following:\n>\n> - `git multi-pack-index` will not run when outside of a Git\n> repository.\n>\n> - The `--object-dir` argument will only recognize object directories\n> belonging to an alternate of the current repository.\n>\n> - Using `--object-dir` to point to a repository which uses a\n> different hash than the repository in the current working directory\n> will continue to not work (as was the case before this series).\n>\n> And because this incorporates [1], we will also not accidentally clean `.rev`\n> files from the wrong object directory.\n>\n> I think that this version is ready-to-go, and that we can turn our attention to\n> squashing some of these cross-alternate buglets, and integrating MIDX bitmaps\n> with `git repack`.\n\nThanks.\n\n> +@@ Documentation/git-multi-pack-index.txt: OPTIONS\n> + \tUse given directory for the location of Git objects. We check\n> + \t`/packs/multi-pack-index` for the current MIDX file, and\n> + \t`/packs` for the pack-files to index.\n> +++\n> ++`` must be an alternate of the current repository.\n\nAfter replacing the previous round with this round and running \"git\ndiff @{1}\" on the branch, I noticed this documentation update, but\ndid't find any new code that tries to ensure that the requirement is\nmet. It's a bit curious omission.\n\nI think it is OK to allow running this command on and then add\nit as a new alternate (iow, the being an alternate is not a\nstrict requirement for correct computation and writing of the midx,\neven though it may be a requirement for correct use of the resulting\nmidx), so perhaps that is where the lack of validation comes from?\n\nTHanks.\n\n"},{"id":"434464","messageId":"YS/Pqc7lkMlnlBYR@nand.local","threadId":"55464","inReplyTo":"xmqq5yvkqidc.fsf@gitster.g","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T19:08:25Z","receivedAt":"2021-09-01T19:09:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 01, 2021 at 11:07:59AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n> > +@@ Documentation/git-multi-pack-index.txt: OPTIONS\n> > + \tUse given directory for the location of Git objects. We check\n> > + \t`/packs/multi-pack-index` for the current MIDX file, and\n> > + \t`/packs` for the pack-files to index.\n> > +++\n> > ++`` must be an alternate of the current repository.\n>\n> After replacing the previous round with this round and running \"git\n> diff @{1}\" on the branch, I noticed this documentation update, but\n> did't find any new code that tries to ensure that the requirement is\n> met. It's a bit curious omission.\n>\n> I think it is OK to allow running this command on and then add\n> it as a new alternate (iow, the being an alternate is not a\n> strict requirement for correct computation and writing of the midx,\n> even though it may be a requirement for correct use of the resulting\n> midx), so perhaps that is where the lack of validation comes from?\n\nI wasn't sure whether to include it or not, since we technically will\nstill write a MIDX in that object directory (alternate or not), but we\nwon't load up an existing MIDX that is already there to reference. So\nwe'll get the same result, just slower.\n\nI'm comfortable with saying what's written in the documentation, since\neven though it happens to work today, we should leave ourselves open to\nnot supporting directories which aren't alternates.\n\nBut I'm equally OK if you would rather drop this hunk from the\ndocumentation when staging.\n\nThanks,\nTaylor\n"},{"id":"434465","messageId":"xmqq1r68qevl.fsf@gitster.g","threadId":"55464","inReplyTo":"YS/Pqc7lkMlnlBYR@nand.local","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-01T19:23:26Z","receivedAt":"2021-09-01T19:23:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> I'm comfortable with saying what's written in the documentation, since\n> even though it happens to work today, we should leave ourselves open to\n> not supporting directories which aren't alternates.\n>\n> But I'm equally OK if you would rather drop this hunk from the\n> documentation when staging.\n\nOh, no, don't get me wrong. I am comfortable with the documented\nlimitation, as that is what the area experts have agreed that is\nreasonable given the expected use case.\n\nI however am much less comfortable with a documented limitation that\nwe make no attempt to enforce, and that is why the first thing I\nlooked for after seeing the documentation update was new code to\nmake sure we reject a random directory that is not our alternate\nobject store.\n\nThanks.\n"},{"id":"434469","messageId":"YS/juRg9N/cCoR0d@nand.local","threadId":"55464","inReplyTo":"xmqq1r68qevl.fsf@gitster.g","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T20:34:01Z","receivedAt":"2021-09-01T20:34:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 01, 2021 at 12:23:26PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > I'm comfortable with saying what's written in the documentation, since\n> > even though it happens to work today, we should leave ourselves open to\n> > not supporting directories which aren't alternates.\n> >\n> > But I'm equally OK if you would rather drop this hunk from the\n> > documentation when staging.\n>\n> Oh, no, don't get me wrong. I am comfortable with the documented\n> limitation, as that is what the area experts have agreed that is\n> reasonable given the expected use case.\n>\n> I however am much less comfortable with a documented limitation that\n> we make no attempt to enforce, and that is why the first thing I\n> looked for after seeing the documentation update was new code to\n> make sure we reject a random directory that is not our alternate\n> object store.\n\nSure, I don't mind getting more strict here in this series. If you want,\nthe below could be queued instead of the original 11/27:\n\n--- 8< ---\n\nSubject: [PATCH] midx: avoid opening multiple MIDXs when writing\n\nOpening multiple instance of the same MIDX can lead to problems like two\nseparate packed_git structures which represent the same pack being added\nto the repository's object store.\n\nThe above scenario can happen because prepare_midx_pack() checks if\n`m->packs[pack_int_id]` is NULL in order to determine if a pack has been\nopened and installed in the repository before. But a caller can\nconstruct two copies of the same MIDX by calling get_multi_pack_index()\nand load_multi_pack_index() since the former manipulates the\nobject store directly but the latter is a lower-level routine which\nallocates a new MIDX for each call.\n\nSo if prepare_midx_pack() is called on multiple MIDXs with the same\npack_int_id, then that pack will be installed twice in the object\nstore's packed_git pointer.\n\nThis can lead to problems in, for e.g., the pack-bitmap code, which does\nsomething like the following (in pack-bitmap.c:open_pack_bitmap()):\n\n struct bitmap_index *bitmap_git = ...;\n for (p = get_all_packs(r); p; p = p->next) {\n if (open_pack_bitmap_1(bitmap_git, p) == 0)\n ret = 0;\n }\n\nwhich is a problem if two copies of the same pack exist in the\npacked_git list because pack-bitmap.c:open_pack_bitmap_1() contains a\nconditional like the following:\n\n if (bitmap_git->pack || bitmap_git->midx) {\n /* ignore extra bitmap file; we can only handle one */\n warning(\"ignoring extra bitmap file: %s\", packfile->pack_name);\n close(fd);\n return -1;\n }\n\nAvoid this scenario by not letting write_midx_internal() open a MIDX\nthat isn't also pointed at by the object store. So long as this is the\ncase, other routines should prefer to open MIDXs with\nget_multi_pack_index() or reprepare_packed_git() instead of creating\ninstances on their own. Because get_multi_pack_index() returns\n`r->object_store->multi_pack_index` if it is non-NULL, we'll only have\none instance of a MIDX open at one time, avoiding these problems.\n\nTo encourage this, drop the `struct multi_pack_index *` parameter from\n`write_midx_internal()`, and rely instead on the `object_dir` to find\n(or initialize) the correct MIDX instance.\n\nLikewise, replace the call to `close_midx()` with\n`close_object_store()`, since we're about to replace the MIDX with a new\none and should invalidate the object store's memory of any MIDX that\nmight have existed beforehand.\n\nNote that this now forbids passing object directories that don't belong\nto alternate repositories over `--object-dir`, since before we would\nhave happily opened a MIDX in any directory, but now restrict ourselves\nto only those reachable by `r->objects->multi_pack_index` (and alternate\nMIDXs that we can see by walking the `next` pointer).\n\nAs far as I can tell, supporting arbitrary directories with\n`--object-dir` was a historical accident, since even the documentation\nsays `` when referring to the value passed to this option.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 2 ++\n builtin/commit-graph.c | 22 -------------------\n midx.c | 29 ++++++++++++++++----------\n object-file.c | 21 +++++++++++++++++++\n object-store.h | 1 +\n t/t5319-multi-pack-index.sh | 10 ++++++++-\n 6 files changed, 51 insertions(+), 34 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex c9b063d31e..0af6beb2dd 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -23,6 +23,8 @@ OPTIONS\n \tUse given directory for the location of Git objects. We check\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n++\n+`` must be an alternate of the current repository.\n\n --[no-]progress::\n \tTurn progress on/off explicitly. If neither is specified, progress is\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex cd86315221..003eaaac5c 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -43,28 +43,6 @@ static struct opts_commit_graph {\n \tint enable_changed_paths;\n } opts;\n\n-static struct object_directory *find_odb(struct repository *r,\n-\t\t\t\t\t const char *obj_dir)\n-{\n-\tstruct object_directory *odb;\n-\tchar *obj_dir_real = real_pathdup(obj_dir, 1);\n-\tstruct strbuf odb_path_real = STRBUF_INIT;\n-\n-\tprepare_alt_odb(r);\n-\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\tstrbuf_realpath(&odb_path_real, odb->path, 1);\n-\t\tif (!strcmp(obj_dir_real, odb_path_real.buf))\n-\t\t\tbreak;\n-\t}\n-\n-\tfree(obj_dir_real);\n-\tstrbuf_release(&odb_path_real);\n-\n-\tif (!odb)\n-\t\tdie(_(\"could not find object directory matching %s\"), obj_dir);\n-\treturn odb;\n-}\n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\ndiff --git a/midx.c b/midx.c\nindex e83f22b5ee..25906044ff 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n\n-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n \t\t\t unsigned flags)\n@@ -904,20 +904,26 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n+\tstruct multi_pack_index *cur;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n \tstruct chunkfile *cf;\n\n+\t/* Ensure the given object_dir is local, or a known alternate. */\n+\tfind_odb(the_repository, object_dir);\n+\n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name);\n\n-\tif (m)\n-\t\tctx.m = m;\n-\telse\n-\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\tctx.m = cur;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n\n \tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n \t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n@@ -1119,7 +1125,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n\n \tif (ctx.m)\n-\t\tclose_midx(ctx.m);\n+\t\tclose_object_store(the_repository->objects);\n\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -1182,8 +1188,7 @@ int write_midx_file(const char *object_dir,\n \t\t const char *preferred_pack_name,\n \t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t flags);\n+\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n }\n\n struct clear_midx_data {\n@@ -1461,8 +1466,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n\n \tfree(count);\n\n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n+\tif (packs_to_drop.nr) {\n+\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tm = NULL;\n+\t}\n\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1651,7 +1658,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n\n-\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n \tm = NULL;\n\n cleanup:\ndiff --git a/object-file.c b/object-file.c\nindex a8be899481..a4d720b4f5 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -820,6 +820,27 @@ char *compute_alternate_path(const char *path, struct strbuf *err)\n \treturn ref_git;\n }\n\n+struct object_directory *find_odb(struct repository *r, const char *obj_dir)\n+{\n+\tstruct object_directory *odb;\n+\tchar *obj_dir_real = real_pathdup(obj_dir, 1);\n+\tstruct strbuf odb_path_real = STRBUF_INIT;\n+\n+\tprepare_alt_odb(r);\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\tstrbuf_realpath(&odb_path_real, odb->path, 1);\n+\t\tif (!strcmp(obj_dir_real, odb_path_real.buf))\n+\t\t\tbreak;\n+\t}\n+\n+\tfree(obj_dir_real);\n+\tstrbuf_release(&odb_path_real);\n+\n+\tif (!odb)\n+\t\tdie(_(\"could not find object directory matching %s\"), obj_dir);\n+\treturn odb;\n+}\n+\n static void fill_alternate_refs_command(struct child_process *cmd,\n \t\t\t\t\tconst char *repo_path)\n {\ndiff --git a/object-store.h b/object-store.h\nindex d24915ced1..250aa5f33c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -38,6 +38,7 @@ KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n+struct object_directory *find_odb(struct repository *r, const char *obj_dir);\n typedef int alt_odb_fn(struct object_directory *, void *);\n int foreach_alt_odb(alt_odb_fn, void*);\n typedef void alternate_ref_fn(const struct object_id *oid, void *);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex d7e4988f2b..bd09c3194b 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -582,7 +582,15 @@ test_expect_success 'force some 64-bit offsets with pack-objects' '\n \tidx64=objects64/pack/test-64-$pack64.idx &&\n \tchmod u+w $idx64 &&\n \tcorrupt_data $idx64 $(test_oid idxoff) \"\\02\" &&\n-\tmidx64=$(git multi-pack-index --object-dir=objects64 write) &&\n+\t# objects64 is not a real repository, but can serve as an alternate\n+\t# anyway so we can write a MIDX into it\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\t( cd ../objects64 && pwd ) >.git/objects/info/alternates &&\n+\t\tmidx64=$(git multi-pack-index --object-dir=../objects64 write)\n+\t) &&\n \tmidx_read_expect 1 63 5 objects64 \" large-offsets\"\n '\n\n--\n2.33.0.96.g73915697e6\n\n"},{"id":"434470","messageId":"xmqq35qoowb8.fsf@gitster.g","threadId":"55464","inReplyTo":"YS/juRg9N/cCoR0d@nand.local","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-01T20:49:47Z","receivedAt":"2021-09-01T20:49:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Sure, I don't mind getting more strict here in this series. If you want,\n> the below could be queued instead of the original 11/27:\n\nThat may make the documentation and the code more consistent.\n\n> As far as I can tell, supporting arbitrary directories with\n> `--object-dir` was a historical accident, since even the documentation\n> says `` when referring to the value passed to this option.\n\nThe synopsis has [--object-dir=], which wants to be cleaned up\nfor consistency (or updated to , but I tend to agree with\nyou that unifying to may make our intention more clear).\n\nIt is unfortunate that \"git multi-pack-index -h\" says , which\nis probably doubly wrong. It seems this is the only instance that\nabuses OPT_FILENAME() for a non-file, so perhaps it is not too bad\nto fix it using the lower-level OPTION_FILENAME (instead of adding\na one-off OPT_DIRECTORY_NAME() helper).\n\nNeither is something that would block this step, of course.\n"},{"id":"434471","messageId":"YS/om1SbCZ1cOxZ2@nand.local","threadId":"55464","inReplyTo":"xmqq35qoowb8.fsf@gitster.g","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T20:54:51Z","receivedAt":"2021-09-01T20:54:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 01, 2021 at 01:49:47PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Sure, I don't mind getting more strict here in this series. If you want,\n> > the below could be queued instead of the original 11/27:\n>\n> That may make the documentation and the code more consistent.\n>\n> > As far as I can tell, supporting arbitrary directories with\n> > `--object-dir` was a historical accident, since even the documentation\n> > says `` when referring to the value passed to this option.\n>\n> The synopsis has [--object-dir=], which wants to be cleaned up\n> for consistency (or updated to , but I tend to agree with\n> you that unifying to may make our intention more clear).\n>\n> It is unfortunate that \"git multi-pack-index -h\" says , which\n> is probably doubly wrong. It seems this is the only instance that\n> abuses OPT_FILENAME() for a non-file, so perhaps it is not too bad\n> to fix it using the lower-level OPTION_FILENAME (instead of adding\n> a one-off OPT_DIRECTORY_NAME() helper).\n>\n> Neither is something that would block this step, of course.\n\nI think there is definitely plenty of opportunity to clean all of this\nup even more. But I don't think this already-long series is the place to\ndo it necessarily, since we don't want to let these last-minute (mostly)\ncosmetic issues get in the way of this series as a whole.\n\nHopefully this v5 is at a point where we could start merging it down to\n'next' and then address things like the helptext, `s/dir/alt` and so on.\n\nThanks,\nTaylor\n"},{"id":"434508","messageId":"YTCblArLMMupKDkU@coredump.intra.peff.net","threadId":"55464","inReplyTo":"YS/juRg9N/cCoR0d@nand.local","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-02T09:38:28Z","receivedAt":"2021-09-02T09:39:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 04:34:01PM -0400, Taylor Blau wrote:\n\n> > Oh, no, don't get me wrong. I am comfortable with the documented\n> > limitation, as that is what the area experts have agreed that is\n> > reasonable given the expected use case.\n> >\n> > I however am much less comfortable with a documented limitation that\n> > we make no attempt to enforce, and that is why the first thing I\n> > looked for after seeing the documentation update was new code to\n> > make sure we reject a random directory that is not our alternate\n> > object store.\n> \n> Sure, I don't mind getting more strict here in this series. If you want,\n> the below could be queued instead of the original 11/27:\n> \n> --- 8< ---\n> \n> Subject: [PATCH] midx: avoid opening multiple MIDXs when writing\n\nI think this is worth doing here, as part of this series.\n\nTwo observations (neither of which would lead to changing the patch):\n\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index c9b063d31e..0af6beb2dd 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -23,6 +23,8 @@ OPTIONS\n> \tUse given directory for the location of Git objects. We check\n> \t`/packs/multi-pack-index` for the current MIDX file, and\n> \t`/packs` for the pack-files to index.\n> ++\n> +`` must be an alternate of the current repository.\n\nI wondered if this needed to say \"must be the main object directory of\nor an alternate of the current repository\". But if you are intending to\noperate in the main object directory, you would simply omit --object-dir\nentirely. It is good that it will still work if you specified it\nexplicitly, but I don't think we need to clutter the documentation with\nit.\n\n> index cd86315221..003eaaac5c 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -43,28 +43,6 @@ static struct opts_commit_graph {\n> \tint enable_changed_paths;\n> } opts;\n> \n> -static struct object_directory *find_odb(struct repository *r,\n> -\t\t\t\t\t const char *obj_dir)\n> -{\n> -\tstruct object_directory *odb;\n> -\tchar *obj_dir_real = real_pathdup(obj_dir, 1);\n> -\tstruct strbuf odb_path_real = STRBUF_INIT;\n> -\n> -\tprepare_alt_odb(r);\n> -\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> -\t\tstrbuf_realpath(&odb_path_real, odb->path, 1);\n> -\t\tif (!strcmp(obj_dir_real, odb_path_real.buf))\n> -\t\t\tbreak;\n> -\t}\n> -\n> -\tfree(obj_dir_real);\n> -\tstrbuf_release(&odb_path_real);\n> -\n> -\tif (!odb)\n> -\t\tdie(_(\"could not find object directory matching %s\"), obj_dir);\n> -\treturn odb;\n> -}\n\nAh, right, commit-graph faces this same conundrum we've been discussing.\nAnd it behaves in the way that we concluded:\n\n $ git init one\n $ git commit-graph write --object-dir $PWD/one/.git/objects\n fatal: not a git repository (or any of the parent directories): .git\n\n $ git init two\n $ git -C two commit-graph write --object-dir $PWD/one/.git/objects\n fatal: could not find object directory matching /home/peff/tmp/one/.git/objects\n\nThat gives me more confidence in the direction we decided on.\n\n(Apologies if this was obvious to others, but I didn't see any mention\nof commit-graph's similar option in the recent discussion).\n\n-Peff\n"},{"id":"434509","messageId":"YTCcCXkUpnnHOzyK@coredump.intra.peff.net","threadId":"55464","inReplyTo":"xmqq35qoowb8.fsf@gitster.g","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-02T09:40:25Z","receivedAt":"2021-09-02T09:40:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 01:49:47PM -0700, Junio C Hamano wrote:\n\n> > As far as I can tell, supporting arbitrary directories with\n> > `--object-dir` was a historical accident, since even the documentation\n> > says `` when referring to the value passed to this option.\n> \n> The synopsis has [--object-dir=], which wants to be cleaned up\n> for consistency (or updated to , but I tend to agree with\n> you that unifying to may make our intention more clear).\n> \n> It is unfortunate that \"git multi-pack-index -h\" says , which\n> is probably doubly wrong. It seems this is the only instance that\n> abuses OPT_FILENAME() for a non-file, so perhaps it is not too bad\n> to fix it using the lower-level OPTION_FILENAME (instead of adding\n> a one-off OPT_DIRECTORY_NAME() helper).\n\nThat made me wonder what \"git commit-graph -h\" says. It says \"\"\n(even though it already must be an alternate), because it uses\nOPT_STRING().\n\nI think using OPTION_FILENAME or similar is better there, too, though,\nbecause it reinterprets the name after the repo-setup chdir() step. But\nnow you'd have to for an OPT_DIRNAME() helper. :)\n\n> Neither is something that would block this step, of course.\n\nYeah, very much agreed that this can come later on top.\n\n-Peff\n"},{"id":"434510","messageId":"YTCdHHRW/i3HVN0h@coredump.intra.peff.net","threadId":"55464","inReplyTo":"cover.1630443072.git.me@ttaylorr.com","subject":"Re: [PATCH v5 00/27] multi-pack reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-02T09:45:00Z","receivedAt":"2021-09-02T09:45:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 04:51:33PM -0400, Taylor Blau wrote:\n\n> This series resolves that discussion by leaving everything as-is, and only\n> changing the following:\n> \n> - `git multi-pack-index` will not run when outside of a Git\n> repository.\n> \n> - The `--object-dir` argument will only recognize object directories\n> belonging to an alternate of the current repository.\n> \n> - Using `--object-dir` to point to a repository which uses a\n> different hash than the repository in the current working directory\n> will continue to not work (as was the case before this series).\n> \n> And because this incorporates [1], we will also not accidentally clean `.rev`\n> files from the wrong object directory.\n\nThanks. I read over the new patches, and all looks good to me (using the\nrevised patch 11 you already sent, and which I commented on separately).\n\n-Peff\n"}]}