{"thread":{"id":"58038","subject":"[PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","startedAt":"2022-06-20T12:33:43Z","lastAt":"2022-08-26T16:02:40Z","messageCount":162,"participants":["Abhradeep Chakraborty via GitGitGadget","Taylor Blau via GitGitGadget","Derrick Stolee","Taylor Blau","Abhradeep Chakraborty","Junio C Hamano","Kaartic Sivaraam","Philip Oakley","Martin Ågren","Ævar Arnfjörð Bjarmason","Eric Sunshine","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"457549","messageId":"2e22ca5069af617fe23072d78efb08b26d6130be.1655728395.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:09Z","receivedAt":"2022-06-20T12:33:43Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nWhen reading bitmap file, git loads each and every bitmap one by one\neven if all the bitmaps are not required. A \"bitmap lookup table\"\nextension to the bitmap format can reduce the overhead of loading\nbitmaps which stores a list of bitmapped commit oids, along with their\noffset and xor offset. This way git can load only the neccesary bitmaps\nwithout loading the previous bitmaps.\n\nAdd some information for the new \"bitmap lookup table\" extension in the\nbitmap-format documentation.\n\nCo-Authored-by: Taylor Blau \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/technical/bitmap-format.txt | 31 +++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 04b3ec21785..34e98787b78 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -67,6 +67,14 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n \t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n+\t\t\t** {empty}\n+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0xf) : :::\n+\t\t\tIf present, the end of the bitmap file contains a table\n+\t\t\tcontaining a list of `N` object ids, a list of pairs of\n+\t\t\toffset and xor offset of respective objects, and 4-byte\n+\t\t\tinteger denoting the flags (currently none). The format\n+\t\t\tand meaning of the table is described below.\n+\n \t\t4-byte entry count (network byte order)\n \n \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n@@ -205,3 +213,26 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n hashes made under two different schemes would be pointless).\n+\n+Commit lookup table\n+-------------------\n+\n+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the end of the `.bitmap`\n+contains a lookup table specifying the positions of commits which have a\n+bitmap.\n+\n+For a `.bitmap` containing `nr_entries` reachability bitmaps, the format\n+is as follows:\n+\n+\t- `nr_entries` object names.\n+\n+\t- `nr_entries` pairs of 4-byte integers, each in network order.\n+\t The first holds the offset from which that commit's bitmap can\n+\t be read. The second number holds the position of the commit\n+\t whose bitmap the current bitmap is xor'd with in lexicographic\n+\t order, or 0xffffffff if the current commit is not xor'd with\n+\t anything.\n+\n+\t- One 4-byte network byte order integer specifying\n+\t table-specific flags. None exist currently, so this is always\n+\t \"0\".\n-- \ngitgitgadget\n\n"},{"id":"457550","messageId":"pull.1266.git.1655728395.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":null,"subject":"[PATCH 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:08Z","receivedAt":"2022-06-20T12:33:45Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"When parsing the .bitmap file, git loads all the bitmaps one by one even if\nsome of the bitmaps are not necessary. We can remove this overhead by\nloading only the necessary bitmaps. A look up table extension can solve this\nissue.\n\nThe proposed table has:\n\n * a list of nr_entries object ids. These objects are commits that has\n bitmaps. Ids are stored in lexicographic order (for better searching).\n * a list of pairs (4-byte integers, network-byte\n order). The i'th pair denotes the offset and xor-offset(respectively) of\n the bitmap of i'th commit in the previous list. These two informations\n are necessary because only in this way bitmaps can be found without\n parsing all the bitmap.\n * a 4-byte integer for table specific flags (none exists currently).\n\nWhenever git want to parse the bitmap for a specific commit, it will first\nrefer to the table and will look for the offset and xor-offset for that\ncommit. Git will then try to parse the bitmap located at the offset\nposition. The xor-offset can be used to find the xor-bitmap for the\nbitmap(if any). This process is recursive and will end if xor-offset is null\n(i.e. there is no xor-bitmap left).\n\nAbhradeep Chakraborty (5):\n Documentation/technical: describe bitmap lookup table extension\n pack-bitmap: prepare to read lookup table extension\n pack-bitmap-write.c: write lookup table extension\n bitmap-commit-table: add tests for the bitmap lookup table\n bitmap-lookup-table: add performance tests\n\nTaylor Blau (1):\n builtin/pack-objects.c: learn pack.writeBitmapLookupTable\n\n Documentation/config/pack.txt | 7 +\n Documentation/technical/bitmap-format.txt | 31 ++++\n builtin/pack-objects.c | 8 +\n pack-bitmap-write.c | 59 +++++++-\n pack-bitmap.c | 172 +++++++++++++++++++++-\n pack-bitmap.h | 1 +\n t/perf/p5310-pack-bitmaps.sh | 60 +++++---\n t/perf/p5326-multi-pack-bitmaps.sh | 55 ++++---\n t/t5310-pack-bitmaps.sh | 14 ++\n t/t5326-multi-pack-bitmaps.sh | 19 +++\n 10 files changed, 375 insertions(+), 51 deletions(-)\n\n\nbase-commit: 5699ec1b0aec51b9e9ba5a2785f65970c5a95d84\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1266%2FAbhra303%2Fbitmap-commit-table-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1266/Abhra303/bitmap-commit-table-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1266\n-- \ngitgitgadget\n"},{"id":"457551","messageId":"d139a4c48aa058b142c4860721b10344c5498031.1655728395.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:10Z","receivedAt":"2022-06-20T12:33:46Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nBitmap lookup table extension can let git to parse only the necessary\nbitmaps without loading the previous bitmaps one by one.\n\nTeach git to read and use the bitmap lookup table extension.\n\nCo-Authored-by: Taylor Blau \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap.c | 172 ++++++++++++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 1 +\n 2 files changed, 166 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 36134222d7a..d5e5973a79f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -15,6 +15,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"midx.h\"\n #include \"config.h\"\n+#include \"hash-lookup.h\"\n \n /*\n * An entry on the bitmap index, representing the bitmap for a given\n@@ -82,6 +83,13 @@ struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n+\t/*\n+\t * If not NULL, these point into the various commit table sections\n+\t * (within map).\n+\t */\n+\tunsigned char *table_lookup;\n+\tunsigned char *table_offsets;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -185,6 +193,24 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex->hashes = (void *)(index_end - cache_size);\n \t\t\tindex_end -= cache_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n+\t\t git_env_bool(\"GIT_READ_COMMIT_TABLE\", 1)) {\n+\t\t\tuint32_t entry_count = ntohl(header->entry_count);\n+\t\t\tuint32_t table_size =\n+\t\t\t\t(entry_count * the_hash_algo->rawsz) /* oids */ +\n+\t\t\t\t(entry_count * sizeof(uint32_t)) /* offsets */ +\n+\t\t\t\t(entry_count * sizeof(uint32_t)) /* xor offsets */ +\n+\t\t\t\t(sizeof(uint32_t)) /* flags */;\n+\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit commit table)\");\n+\n+\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n+\t\t\tindex->table_offsets = index->table_lookup + the_hash_algo->rawsz * entry_count;\n+\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n@@ -470,7 +496,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n \n-\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n \treturn 0;\n@@ -557,14 +583,145 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n-struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n-\t\t\t\t struct commit *commit)\n+static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t\t uint32_t *pos_hint);\n+\n+static inline const unsigned char *bitmap_oid_pos(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t uint32_t pos)\n+{\n+\treturn bitmap_git->table_lookup + (pos * the_hash_algo->rawsz);\n+}\n+\n+static inline const void *bitmap_offset_pos(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t uint32_t pos)\n+{\n+\treturn bitmap_git->table_offsets + (pos * 2 * sizeof(uint32_t));\n+}\n+\n+static inline const void *xor_position_pos(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t uint32_t pos)\n+{\n+\treturn (unsigned char*) bitmap_offset_pos(bitmap_git, pos) + sizeof(uint32_t);\n+}\n+\n+static int bitmap_lookup_cmp(const void *_va, const void *_vb)\n+{\n+\treturn hashcmp(_va, _vb);\n+}\n+\n+static int bitmap_table_lookup(struct bitmap_index *bitmap_git,\n+\t\t\t struct object_id *oid,\n+\t\t\t uint32_t *commit_pos)\n+{\n+\tunsigned char *found = bsearch(oid->hash, bitmap_git->table_lookup,\n+\t\t\t\t bitmap_git->entry_count,\n+\t\t\t\t the_hash_algo->rawsz, bitmap_lookup_cmp);\n+\tif (found)\n+\t\t*commit_pos = (found - bitmap_git->table_lookup) / the_hash_algo->rawsz;\n+\treturn !!found;\n+}\n+\n+static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t\t uint32_t commit_pos)\n+{\n+\tuint32_t xor_pos;\n+\toff_t bitmap_ofs;\n+\n+\tint flags;\n+\tstruct ewah_bitmap *bitmap;\n+\tstruct stored_bitmap *xor_bitmap;\n+\n+\tbitmap_ofs = get_be32(bitmap_offset_pos(bitmap_git, commit_pos));\n+\txor_pos = get_be32(xor_position_pos(bitmap_git, commit_pos));\n+\n+\t/*\n+\t * Lazily load the xor'd bitmap if required (and we haven't done so\n+\t * already). Make sure to pass the xor'd bitmap's position along as a\n+\t * hint to avoid an unnecessary binary search in\n+\t * stored_bitmap_for_commit().\n+\t */\n+\tif (xor_pos == 0xffffffff) {\n+\t\txor_bitmap = NULL;\n+\t} else {\n+\t\tstruct commit *xor_commit;\n+\t\tstruct object_id xor_oid;\n+\n+\t\toidread(&xor_oid, bitmap_oid_pos(bitmap_git, xor_pos));\n+\n+\t\txor_commit = lookup_commit(the_repository, &xor_oid);\n+\t\tif (!xor_commit)\n+\t\t\treturn NULL;\n+\n+\t\txor_bitmap = stored_bitmap_for_commit(bitmap_git, xor_commit,\n+\t\t\t\t\t\t &xor_pos);\n+\t}\n+\n+\t/*\n+\t * Don't bother reading the commit's index position or its xor\n+\t * offset:\n+\t *\n+\t * - The commit's index position is irrelevant to us, since\n+\t * load_bitmap_entries_v1 only uses it to learn the object\n+\t * id which is used to compute the hashmap's key. We already\n+\t * have an object id, so no need to look it up again.\n+\t *\n+\t * - The xor_offset is unusable for us, since it specifies how\n+\t * many entries previous to ours we should look at. This\n+\t * makes sense when reading the bitmaps sequentially (as in\n+\t * load_bitmap_entries_v1()), since we can keep track of\n+\t * each bitmap as we read them.\n+\t *\n+\t * But it can't work for us, since the bitmap's don't have a\n+\t * fixed size. So we learn the position of the xor'd bitmap\n+\t * from the commit table (and resolve it to a bitmap in the\n+\t * above if-statement).\n+\t *\n+\t * Instead, we can skip ahead and immediately read the flags and\n+\t * ewah bitmap.\n+\t */\n+\tbitmap_git->map_pos = bitmap_ofs + sizeof(uint32_t) + sizeof(uint8_t);\n+\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\tbitmap = read_bitmap_1(bitmap_git);\n+\tif (!bitmap)\n+\t\treturn NULL;\n+\n+\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n+}\n+\n+static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t\t uint32_t *pos_hint)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n-\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n+\t\tuint32_t commit_pos;\n+\t\tif (!bitmap_git->table_lookup)\n+\t\t\treturn NULL;\n+\n+\t\t/* NEEDSWORK: cache misses aren't recorded. */\n+\t\tif (pos_hint)\n+\t\t\tcommit_pos = *pos_hint;\n+\t\telse if (!bitmap_table_lookup(bitmap_git,\n+\t\t\t\t\t &commit->object.oid,\n+\t\t\t\t\t &commit_pos))\n+\t\t\treturn NULL;\n+\t\treturn lazy_bitmap_for_commit(bitmap_git, &commit->object.oid,\n+\t\t\t\t\t commit_pos);\n+\t}\n+\treturn kh_value(bitmap_git->bitmaps, hash_pos);\n+}\n+\n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit)\n+{\n+\tstruct stored_bitmap *sb = stored_bitmap_for_commit(bitmap_git, commit,\n+\t\t\t\t\t\t\t NULL);\n+\tif (!sb)\n \t\treturn NULL;\n-\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n+\treturn lookup_stored_bitmap(sb);\n }\n \n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n@@ -1699,8 +1856,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n-\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n-\t\tbitmap_git->version, bitmap_git->entry_count);\n+\tif (!bitmap_git->table_lookup)\n+\t\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n+\t\t\tbitmap_git->version, bitmap_git->entry_count);\n \n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3d3ddd77345..37f86787a4d 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -26,6 +26,7 @@ struct bitmap_disk_header {\n enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 1,\n \tBITMAP_OPT_HASH_CACHE = 4,\n+\tBITMAP_OPT_LOOKUP_TABLE = 16,\n };\n \n enum pack_bitmap_flags {\n-- \ngitgitgadget\n\n"},{"id":"457552","messageId":"ed91ebf69a84405f16a4390e6fd208251b8ec53e.1655728395.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH 3/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:11Z","receivedAt":"2022-06-20T12:33:48Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nTeach git to write bitmap lookup table extension. The table has the\nfollowing information:\n\n - `N` no of Object ids of each bitmapped commits\n\n - A list of offset, xor-offset pair; the i'th pair denotes the\n offsets and xor-offsets of i'th commit in the previous list.\n\n - 4-byte integer denoting the flags\n\nCo-authored-by: Taylor Blau \nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 59 +++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 57 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c43375bd344..9e88a64dd65 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -650,7 +650,8 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t off_t *offsets)\n {\n \tint i;\n \n@@ -663,6 +664,9 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t\tif (commit_pos < 0)\n \t\t\tBUG(\"trying to write commit not in index\");\n \n+\t\tif (offsets)\n+\t\t\toffsets[i] = hashfile_total(f);\n+\n \t\thashwrite_be32(f, commit_pos);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n@@ -671,6 +675,49 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static int table_cmp(const void *_va, const void *_vb)\n+{\n+\treturn oidcmp(&writer.selected[*(uint32_t*)_va].commit->object.oid,\n+\t\t &writer.selected[*(uint32_t*)_vb].commit->object.oid);\n+}\n+\n+static void write_lookup_table(struct hashfile *f,\n+\t\t\t off_t *offsets)\n+{\n+\tuint32_t i;\n+\tuint32_t flags = 0;\n+\tuint32_t *table, *table_inv;\n+\n+\tALLOC_ARRAY(table, writer.selected_nr);\n+\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable[i] = i;\n+\tQSORT(table, writer.selected_nr, table_cmp);\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable_inv[table[i]] = i;\n+\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\t\tstruct object_id *oid = &selected->commit->object.oid;\n+\n+\t\thashwrite(f, oid->hash, the_hash_algo->rawsz);\n+\t}\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\n+\t\thashwrite_be32(f, offsets[table[i]]);\n+\t\thashwrite_be32(f, selected->xor_offset\n+\t\t\t ? table_inv[table[i] - selected->xor_offset]\n+\t\t\t : 0xffffffff);\n+\t}\n+\n+\thashwrite_be32(f, flags);\n+\n+\tfree(table);\n+\tfree(table_inv);\n+}\n+\n static void write_hash_cache(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n@@ -695,6 +742,7 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n+\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \n@@ -715,8 +763,14 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n-\twrite_selected_commits_v1(f, index, index_nr);\n \n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\tCALLOC_ARRAY(offsets, index_nr);\n+\n+\twrite_selected_commits_v1(f, index, index_nr, offsets);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\twrite_lookup_table(f, offsets);\n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n \n@@ -730,4 +784,5 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n+\tfree(offsets);\n }\n-- \ngitgitgadget\n\n"},{"id":"457553","messageId":"a404779a30f767f7a05926fb99f354ea7958ab4b.1655728395.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH 5/6] bitmap-commit-table: add tests for the bitmap lookup table","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:13Z","receivedAt":"2022-06-20T12:33:53Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd tests to check the working of the newly implemented lookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 19 +++++++++++++++++++\n 2 files changed, 33 insertions(+)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f775fc1ce69..f05d3e6ace7 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -43,6 +43,20 @@ test_expect_success 'full repack creates bitmaps' '\n \n basic_bitmap_tests\n \n+test_expect_success 'using lookup table does not affect basic bitmap tests' '\n+\ttest_config pack.writeBitmapLookupTable true &&\n+\tgit repack -adb\n+'\n+basic_bitmap_tests\n+\n+test_expect_success 'using lookup table does not let each entries to be parsed one by one' '\n+\ttest_config pack.writeBitmapLookupTable true &&\n+\tgit repack -adb &&\n+\tgit rev-list --test-bitmap HEAD 2>out &&\n+\tgrep \"Found bitmap for\" out &&\n+\t! grep \"Bitmap v1 test \"\n+'\n+\n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n \ttest_must_fail git repack -d 2>err &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4fe57414c13..85fbdf5e4bb 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -306,5 +306,24 @@ test_expect_success 'graceful fallback when missing reverse index' '\n \t\t! grep \"ignoring extra bitmap file\" err\n \t)\n '\n+test_expect_success 'multi-pack-index write --bitmap writes lookup table if enabled' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit_bulk 106 &&\n+\n+\t\tgit repack -d &&\n+\n+\t\tgit config pack.writeBitmapLookupTable true &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\tgit rev-list --test-bitmap HEAD 2>out &&\n+\t\tgrep \"Found bitmap for\" out &&\n+\t\t! grep \"Bitmap v1 test \"\n+\n+\t)\n+'\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"457554","messageId":"661c1137e1c918619f6624d2e331bafd9c3281dc.1655728395.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH 4/6] builtin/pack-objects.c: learn pack.writeBitmapLookupTable","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:12Z","receivedAt":"2022-06-20T12:33:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nTeach git to provide a way for users to enable/disable bitmap lookup\ntable extension by providing a config option named 'writeBitmapLookupTable'.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/config/pack.txt | 7 +++++++\n builtin/pack-objects.c | 8 ++++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex ad7f73a1ead..e12008d2415 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n computed; instead, any namehashes stored in an existing bitmap are\n permuted into their appropriate location when writing a new bitmap.\n \n+pack.writeBitmapLookupTable::\n+\tWhen true, git will include a \"lookup table\" section in the\n+\tbitmap index (if one is written). This table is used to defer\n+\tloading individual bitmaps as late as possible. This can be\n+\tbeneficial in repositories which have relatively large bitmap\n+\tindexes. Defaults to false.\n+\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex cc5f41086da..3ba20301980 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n+\n+\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n+\t\telse\n+\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n+\t}\n+\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"457555","messageId":"f5f725a3fe2ac0c93088c48ac520303a3df2c83d.1655728395.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH 6/6] bitmap-lookup-table: add performance tests","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-20T12:33:14Z","receivedAt":"2022-06-20T12:35:18Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd performance tests for bitmap lookup table extension.\n\nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 60 +++++++++++++++++++-----------\n t/perf/p5326-multi-pack-bitmaps.sh | 55 +++++++++++++++++----------\n 2 files changed, 73 insertions(+), 42 deletions(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 7ad4f237bc3..a8d9414de92 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -10,10 +10,11 @@ test_perf_large_repo\n # since we want to be able to compare bitmap-aware\n # git versus non-bitmap git\n #\n-# We intentionally use the deprecated pack.writebitmaps\n+# We intentionally use the deprecated pack.writeBitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true\n+\tgit config pack.writeBitmaps true &&\n+\tgit config pack.writeReverseIndex true\n '\n \n # we need to create the tag up front such that it is covered by the repack and\n@@ -28,27 +29,42 @@ test_perf 'repack to disk' '\n \n test_full_bitmap\n \n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now kill off all of the refs and pretend we had\n-\t# just the one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n+test_perf 'use lookup table' '\n+ git config pack.writeBitmapLookupTable true\n '\n \n-test_partial_bitmap\n+test_perf 'repack to disk (lookup table)' '\n+ git repack -adb\n+'\n+\n+test_full_bitmap\n+\n+for i in false true\n+do\n+\t$i && lookup=\" (lookup table)\"\n+\ttest_expect_success \"create partial bitmap state$lookup\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$i\"' &&\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now kill off all of the refs and pretend we had\n+\t\t# just the one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+done\n \n test_done\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nindex f2fa228f16a..9001eb4533e 100755\n--- a/t/perf/p5326-multi-pack-bitmaps.sh\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -26,27 +26,42 @@ test_expect_success 'drop pack bitmap' '\n \n test_full_bitmap\n \n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now pretend we have just one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n+test_expect_success 'use lookup table' '\n+\tgit config pack.writeBitmapLookupTable true\n '\n \n-test_partial_bitmap\n+test_perf 'setup multi-pack-index (lookup table)' '\n+\tgit multi-pack-index write --bitmap\n+'\n+\n+test_full_bitmap\n+\n+for i in false true\n+do\n+\t$i && lookup=\" (lookup table)\"\n+\ttest_expect_success \"create partial bitmap state$lookup\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$i\"' &&\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now pretend we have just one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+done\n \n test_done\n-- \ngitgitgadget\n"},{"id":"457564","messageId":"b21af0bc-3234-3aa2-e4a0-82874e9a670e@github.com","threadId":"58038","inReplyTo":"2e22ca5069af617fe23072d78efb08b26d6130be.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-20T16:56:27Z","receivedAt":"2022-06-20T16:56:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/20/2022 8:33 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n> \n> When reading bitmap file, git loads each and every bitmap one by one\n> even if all the bitmaps are not required. A \"bitmap lookup table\"\n> extension to the bitmap format can reduce the overhead of loading\n> bitmaps which stores a list of bitmapped commit oids, along with their\n> offset and xor offset. This way git can load only the neccesary bitmaps\n> without loading the previous bitmaps.\n> \n> Add some information for the new \"bitmap lookup table\" extension in the\n> bitmap-format documentation.\n\n\n> @@ -67,6 +67,14 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n> \t\t\tpack/MIDX. The format and meaning of the name-hash is\n> \t\t\tdescribed below.\n> \n> +\t\t\t** {empty}\n> +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0xf) : :::\n> +\t\t\tIf present, the end of the bitmap file contains a table\n> +\t\t\tcontaining a list of `N` object ids, a list of pairs of\n> +\t\t\toffset and xor offset of respective objects, and 4-byte\n> +\t\t\tinteger denoting the flags (currently none). The format\n> +\t\t\tand meaning of the table is described below.\n> +\n\nHere, you are adding a new flag that indicates that the end of the file\ncontains this extra extension. This works because the size of the\nextension is predictable. As long as any future extensions are also of\na predictable size, then we can continue adding them via flags in this\nway.\n\nThis is better than updating the full file format to do something like\nlike use the chunk format API, especially because this format is shared\nacross other tools (JGit being mentioned frequently).\n\nIt might be worth mentioning in your commit message what happens when an\nolder version of Git (or JGit) notices this flag. Does it refuse to\noperate on the .bitmap file? Does it give a warning or die? It would be\nnice if this extension could be ignored (it seems like adding the extra\ndata at the end does not stop the bitmap data from being understood).\n\n> +\n> +Commit lookup table\n> +-------------------\n> +\n> +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the end of the `.bitmap`\n> +contains a lookup table specifying the positions of commits which have a\n> +bitmap.\n\nPerhaps it would be better to say \"the last N * (HASH_LEN + 8) + 4 bytes\npreceding the trailing hash\" or something? This gives us a concrete way\nto compute the start of the table, while also being clear that the table\nis included in the trailing hash.\n\n> +For a `.bitmap` containing `nr_entries` reachability bitmaps, the format\n> +is as follows:\n> +\n> +\t- `nr_entries` object names.\n\nCould you expand that these objects are commit OIDs, one for each bitmap\nin the file. Are they sorted in lexicographical order for binary search,\nor are we expecting to read the entire table into a hashtable in-memory?\n\n> +\t- `nr_entries` pairs of 4-byte integers, each in network order.\n> +\t The first holds the offset from which that commit's bitmap can\n> +\t be read. The second number holds the position of the commit\n> +\t whose bitmap the current bitmap is xor'd with in lexicographic\n> +\t order, or 0xffffffff if the current commit is not xor'd with\n> +\t anything.\n\nInteresting to give the xor chains directions here. You say \"position\"\nhere for the second commit: do you mean within the list of object names\nas opposed to the offset? That would make the most sense so we can trace\nthe full list of XORs we need to make all at once.\n\nAre .bitmap files already constrained to 4GB, so these 32-bit offsets\nmake sense? Using 64-bit offsets would be a small cost here, I think,\nwithout needing to do any fancy \"overflow\" tables that could introduce\na variable-length extension.\n\n> +\t- One 4-byte network byte order integer specifying\n> +\t table-specific flags. None exist currently, so this is always\n> +\t \"0\".\n\nI'm guessing this is at the end of the extension because a future flag\ncould modify the length of the extension, so we need the flags to be\nin a predictable location. Could we make that clear somewhere?\n\nHow does Git react to seeing flags here that it does not recognize?\nIt seems that Git should ignore the lookup table but continue using the\nrest of the .bitmap file as it did before, yes?\n\nThanks,\n-Stolee\n"},{"id":"457565","messageId":"YrCpv3XEoB6lOlY4@nand.local","threadId":"58038","inReplyTo":"b21af0bc-3234-3aa2-e4a0-82874e9a670e@github.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-20T17:09:19Z","receivedAt":"2022-06-20T17:09:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:56:27PM -0400, Derrick Stolee wrote:\n> On 6/20/2022 8:33 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> > From: Abhradeep Chakraborty \n> >\n> > When reading bitmap file, git loads each and every bitmap one by one\n> > even if all the bitmaps are not required. A \"bitmap lookup table\"\n> > extension to the bitmap format can reduce the overhead of loading\n> > bitmaps which stores a list of bitmapped commit oids, along with their\n> > offset and xor offset. This way git can load only the neccesary bitmaps\n> > without loading the previous bitmaps.\n> >\n> > Add some information for the new \"bitmap lookup table\" extension in the\n> > bitmap-format documentation.\n>\n>\n> > @@ -67,6 +67,14 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n> > \t\t\tpack/MIDX. The format and meaning of the name-hash is\n> > \t\t\tdescribed below.\n> >\n> > +\t\t\t** {empty}\n> > +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0xf) : :::\n> > +\t\t\tIf present, the end of the bitmap file contains a table\n> > +\t\t\tcontaining a list of `N` object ids, a list of pairs of\n> > +\t\t\toffset and xor offset of respective objects, and 4-byte\n> > +\t\t\tinteger denoting the flags (currently none). The format\n> > +\t\t\tand meaning of the table is described below.\n> > +\n>\n> Here, you are adding a new flag that indicates that the end of the file\n> contains this extra extension. This works because the size of the\n> extension is predictable. As long as any future extensions are also of\n> a predictable size, then we can continue adding them via flags in this\n> way.\n\nRight; any extensions that are added to the existing .bitmap format must\nhave a size that is predictable in order for readers to locate the next\nextension, if any.\n\n> This is better than updating the full file format to do something like\n> like use the chunk format API, especially because this format is shared\n> across other tools (JGit being mentioned frequently).\n\nAgreed. Abhradeep and I discussed whether or not it was worth exploring\na new .bitmap format, and the consensus we reached was that it may be\nrequired in the future (if we explored a compression scheme other than\nEWAH or made some other backwards-incompatible change), but as of yet it\nisn't necessary. So we avoided it to eliminate unnecessary churn,\nespecially of on-disk formats.\n\n> It might be worth mentioning in your commit message what happens when an\n> older version of Git (or JGit) notices this flag. Does it refuse to\n> operate on the .bitmap file? Does it give a warning or die? It would be\n> nice if this extension could be ignored (it seems like adding the extra\n> data at the end does not stop the bitmap data from being understood).\n\nI agree. The bitmap reader does not warn or die when it sees\nunrecognized extensions, that way new extensions can be added without\nrendering all previously-written bitmaps useless. But in order to\nunderstand an extension on bit N, the reader must also understand\nextensions N-1, N-2, and so on (in order to locate the end of\nextension N).\n\n> > +\t- `nr_entries` pairs of 4-byte integers, each in network order.\n> > +\t The first holds the offset from which that commit's bitmap can\n> > +\t be read. The second number holds the position of the commit\n> > +\t whose bitmap the current bitmap is xor'd with in lexicographic\n> > +\t order, or 0xffffffff if the current commit is not xor'd with\n> > +\t anything.\n>\n> Interesting to give the xor chains directions here. You say \"position\"\n> here for the second commit: do you mean within the list of object names\n> as opposed to the offset? That would make the most sense so we can trace\n> the full list of XORs we need to make all at once.\n>\n> Are .bitmap files already constrained to 4GB, so these 32-bit offsets\n> make sense? Using 64-bit offsets would be a small cost here, I think,\n> without needing to do any fancy \"overflow\" tables that could introduce\n> a variable-length extension.\n\nYeah, we should support >4GB bitmaps here. An overflow table could work,\nbut I agree with Stolee that in practice it won't matter. Most .bitmap\nfiles that I've looked at in the wild have around ~500 entries at most,\nand are usually small. So the cost of widening this section isn't a big\ndeal.\n\nBut note that the entry count is only one component of the bitmap size:\nthe individual entry lengths obviously matter too. And in repositories\nwhose bitmaps exceed 500 entries, the entries themselves are often\nseveral million bits long (before compression) already. So it is\ncertainly possible to exceed 4GB without having an astronomical entry\ncount.\n\nSo doubling the width of this extension might add an extra 250 KiB or\nso, which is negligible.\n\nI would much rather see us do that in cases where it makes sense (small\nnumber of entries, minimal cost to wider records, etc.) than adding\nunnecessary complexity via an extra lookup table for >4GB offsets.\n\n> > +\t- One 4-byte network byte order integer specifying\n> > +\t table-specific flags. None exist currently, so this is always\n> > +\t \"0\".\n>\n> I'm guessing this is at the end of the extension because a future flag\n> could modify the length of the extension, so we need the flags to be\n> in a predictable location. Could we make that clear somewhere?\n\nI can't remember what I had on my mind when I wrote this ;-).\n\nAbhradeep -- do you have any thoughts about what this might be used for?\nI'll try to remember it myself, but I imagine that we could just as\neasily remove this altogether and avoid the confusion.\n\n> How does Git react to seeing flags here that it does not recognize?\n> It seems that Git should ignore the lookup table but continue using the\n> rest of the .bitmap file as it did before, yes?\n\n(See above).\n\nThanks,\nTaylor\n"},{"id":"457566","messageId":"YrCsricF+2rQXiBk@nand.local","threadId":"58038","inReplyTo":"2e22ca5069af617fe23072d78efb08b26d6130be.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-20T17:21:50Z","receivedAt":"2022-06-20T17:21:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:33:09PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> When reading bitmap file, git loads each and every bitmap one by one\n> even if all the bitmaps are not required. A \"bitmap lookup table\"\n> extension to the bitmap format can reduce the overhead of loading\n> bitmaps which stores a list of bitmapped commit oids, along with their\n> offset and xor offset. This way git can load only the neccesary bitmaps\n> without loading the previous bitmaps.\n\nWell put. It might help to have a concrete example of where we expect\nthis to help and not help. I suspect that some of this will show up in\nyour work updating the perf suite to use this new table, but I imagine\nthat we'll find something like:\n\n In cases where the result can be read or computed without\n significant additional traversal (e.g., all commits of interest\n already have bitmaps computed), we can save some time loading and\n parsing a majority of the bitmap file that we will never read.\n\n But in cases where the bitmaps are out-of-date, or there is\n significant traversal required to go from the reference tips to\n what's contained in the .bitmap file, this table provides minimal\n benefit (or something).\n\nOf course, you should verify that that is actually true before we insert\nit into the commit message as such ;-). But that sort of information may\nhelp readers understand what the purpose of this change is towards the\nbeinning of the series.\n\n> Add some information for the new \"bitmap lookup table\" extension in the\n> bitmap-format documentation.\n>\n> Co-Authored-by: Taylor Blau \n> Mentored-by: Taylor Blau \n\nHere and elsewhere: I typically use my address when\ncontributing to Git. So any trailers that mention my email or commits\nthat you send on my behalf should use that address, too.\n\n> Co-Mentored-by: Kaartic Sivaraam \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> Documentation/technical/bitmap-format.txt | 31 +++++++++++++++++++++++\n> 1 file changed, 31 insertions(+)\n>\n> diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\n> index 04b3ec21785..34e98787b78 100644\n> --- a/Documentation/technical/bitmap-format.txt\n> +++ b/Documentation/technical/bitmap-format.txt\n> @@ -67,6 +67,14 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n> \t\t\tpack/MIDX. The format and meaning of the name-hash is\n> \t\t\tdescribed below.\n>\n> +\t\t\t** {empty}\n> +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0xf) : :::\n\nIt the space between \"(0xf)\" and the first \":\" intentional? Similarly,\nshould there be two or three colons at the end (either \"::\" or \":::\")?\n\n> +\t\t\tIf present, the end of the bitmap file contains a table\n> +\t\t\tcontaining a list of `N` object ids, a list of pairs of\n> +\t\t\toffset and xor offset of respective objects, and 4-byte\n> +\t\t\tinteger denoting the flags (currently none). The format\n> +\t\t\tand meaning of the table is described below.\n> +\n\nI remember we had a brief off-list discussion about whether we should\nstore the full object IDs in the offset table, or whether we could store\ntheir pack- or index-relative ordering. Is there a reason to prefer one\nor the other?\n\nI don't think we need to explain the choice fully in the documentation\nin this patch, but it may be worth thinking about separately\nnonetheless. We can store either order and convert it to an object ID in\nconstant time.\n\nTo figure out which is best, I would recommend trying a few different\nchoices here and seeing how they do or don't impact your performance\ntesting.\n\n> \t\t4-byte entry count (network byte order)\n>\n> \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n> @@ -205,3 +213,26 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n> If implementations want to choose a different hashing scheme, they are\n> free to do so, but MUST allocate a new header flag (because comparing\n> hashes made under two different schemes would be pointless).\n> +\n> +Commit lookup table\n> +-------------------\n> +\n> +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the end of the `.bitmap`\n> +contains a lookup table specifying the positions of commits which have a\n> +bitmap.\n> +\n> +For a `.bitmap` containing `nr_entries` reachability bitmaps, the format\n> +is as follows:\n> +\n> +\t- `nr_entries` object names.\n> +\n> +\t- `nr_entries` pairs of 4-byte integers, each in network order.\n> +\t The first holds the offset from which that commit's bitmap can\n> +\t be read. The second number holds the position of the commit\n> +\t whose bitmap the current bitmap is xor'd with in lexicographic\n> +\t order, or 0xffffffff if the current commit is not xor'd with\n> +\t anything.\n\nA couple of small thoughts here. I wonder if we'd get better locality if\nwe made each record look something like:\n\n (object_id, offset, xor_pos)\n\nWhere object_id is either 20- or 4-bytes long (depending if we store the\nfull object ID, or some 4-byte identifier that allows us to discover\nit), offset is 8 bytes long, and xor_pos is 4-bytes (since in practice\nwe don't support packs or MIDXs which have more than 2^32-1 objects).\n\nIn the event that this table doesn't fit into a single cache line, I\nthink we'll get better performance out of reading it by not forcing the\ncache to evict itself whenever we need to refer back to the object_id.\n\n> +\t- One 4-byte network byte order integer specifying\n> +\t table-specific flags. None exist currently, so this is always\n> +\t \"0\".\n\nI mentioned in my reply to Stolee earlier, but I think that we should\neither (a) try to remember what this is for and document it, or (b)\nremove it.\n\nThanks,\nTaylor\n"},{"id":"457572","messageId":"25e03c86-5a47-2100-2da7-a635673a8e38@github.com","threadId":"58038","inReplyTo":"2e22ca5069af617fe23072d78efb08b26d6130be.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-20T20:21:30Z","receivedAt":"2022-06-20T20:21:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/20/2022 8:33 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n\n> +\t\t\t** {empty}\n> +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0xf) : :::\n\nI think you mean 0x10 (b_1_0000) instead of 0xf (b_1111).\n\nI noticed when looking at the constant in patch 2.\n\nThanks,\n-Stolee\n"},{"id":"457576","messageId":"92dc6860-ff35-0989-5114-fe1e220ca10c@github.com","threadId":"58038","inReplyTo":"d139a4c48aa058b142c4860721b10344c5498031.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-20T20:49:26Z","receivedAt":"2022-06-20T20:49:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/20/2022 8:33 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n> \n> Bitmap lookup table extension can let git to parse only the necessary\n> bitmaps without loading the previous bitmaps one by one.\n\nHere is an attempt to reword this a bit:\n\n The bitmap lookup table extension was documented by an earlier\n change, but Git does not yet know how to parse that information.\n The extension allows parsing a smaller portion of the bitmap\n file in order to find bitmaps for specific commits.\n\n \n> Teach git to read and use the bitmap lookup table extension.\n\nNormally, I don't mind doing the read portion after the write\nportion, but it would be nice to have them in the opposite\norder so we can test writing the extension _and Git ignoring\nthe extension_ before implementing the parsing. As it stands,\nmost of the code in this patch is untested until patch 5.\n\nGeneral outline attempt:\n\n1. Document the format.\n2. Write the extension if the flag is given.\n3. Add pack.writeBitmapLookupTable and add tests that write\n the lookup table (and do other bitmap reads on that data).\n4. Read the lookup table. The tests from step 3 already cover\n acting upon the lookup table. (Perhaps we add a mode here\n that disables GIT_READ_COMMIT_TABLE since that is not used\n anywhere else.)\n5. Performance tests.\n\n> +\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n> +\t\t git_env_bool(\"GIT_READ_COMMIT_TABLE\", 1)) {\n\nThis environment variable does not appear to be used or\ndocumented anywhere. Do we really want to use it as a way\nto disable reading the lookup table in general? Or would it be\nbetter to have a GIT_TEST_* variable for disabling the read\nduring testing?\n\n> +\t\t\tuint32_t entry_count = ntohl(header->entry_count);\n> +\t\t\tuint32_t table_size =\n> +\t\t\t\t(entry_count * the_hash_algo->rawsz) /* oids */ +\n> +\t\t\t\t(entry_count * sizeof(uint32_t)) /* offsets */ +\n> +\t\t\t\t(entry_count * sizeof(uint32_t)) /* xor offsets */ +\n> +\t\t\t\t(sizeof(uint32_t)) /* flags */;\n\nHere, uint32_T is probably fine, but maybe we should just use\nsize_t instead? Should we use st_mult() and st_add() everywhere?\n\nNote: you're using the_hash_algo->rawsz here, which makes sense\nbecause the bitmap format doesn't specify which hash algorithm is\nused. Just making this note to say that we should include the hash\nalgorithm as a value in the bitmap format when we increment the\nformat version (in the future).\n\n> +\t\t\tif (table_size > index_end - index->map - header_size)\n> +\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit commit table)\");\n> +\n> +\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n> +\t\t\tindex->table_offsets = index->table_lookup + the_hash_algo->rawsz * entry_count;\n\nst_mult(), st_add()? Or, should we assume safety now?\n\n> +\t\t\tindex_end -= table_size;\n> +\t\t}\n\n> -\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n> +\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n> \t\tgoto failed;\n\nOk, don't load these entries pre-emptively if we have the lookup table.\n\n> +static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t struct commit *commit,\n> +\t\t\t\t\t\t uint32_t *pos_hint);\n\nI see that we have a two-method recursion loop. Please move this\ndeclaration to immediately before lazy_bitmap_for_commit() so it\nis declared as late as possible.\n\n> +static inline const unsigned char *bitmap_oid_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t uint32_t pos)\n> +{\n> +\treturn bitmap_git->table_lookup + (pos * the_hash_algo->rawsz);\n> +}\n\nI would call this \"bitmap_hash_pos()\" because we are getting a raw\nhash and not a 'struct object_id'. Do you want a helper that fills\na 'struct object_id', perhaps passed-by-reference?\n\n> +static inline const void *bitmap_offset_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t uint32_t pos)\n\n> +static inline const void *xor_position_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t uint32_t pos)\n\nThese two helpers should probably return a size_t and uint32_t\ninstead of a pointer. Let these do get_be[32|64]() on the computed\npointer.\n\n> +static int bitmap_table_lookup(struct bitmap_index *bitmap_git,\n> +\t\t\t struct object_id *oid,\n> +\t\t\t uint32_t *commit_pos)\n> +{\n> +\tunsigned char *found = bsearch(oid->hash, bitmap_git->table_lookup,\n> +\t\t\t\t bitmap_git->entry_count,\n> +\t\t\t\t the_hash_algo->rawsz, bitmap_lookup_cmp);\n> +\tif (found)\n> +\t\t*commit_pos = (found - bitmap_git->table_lookup) / the_hash_algo->rawsz;\n> +\treturn !!found;\n\nOk, we are running binary search and converting the pointer into a\nposition.\n\nFrequently, these kind of searches return an int, but use a negative\nvalue to indicate that the value was not found. Using an int in this\nway would restrict us to 2^31 bitmaps instead of 2^32, so maybe it\nis not worth matching that practice.\n\n> +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t struct object_id *oid,\n> +\t\t\t\t\t\t uint32_t commit_pos)\n> +{\n> +\tuint32_t xor_pos;\n> +\toff_t bitmap_ofs;\n> +\n> +\tint flags;\n> +\tstruct ewah_bitmap *bitmap;\n> +\tstruct stored_bitmap *xor_bitmap;\n> +\n> +\tbitmap_ofs = get_be32(bitmap_offset_pos(bitmap_git, commit_pos));\n> +\txor_pos = get_be32(xor_position_pos(bitmap_git, commit_pos));\n\nThese lines become simpler with a change in the helper methods'\nprototypes, as I recommended higher up.\n\n> +\t/*\n> +\t * Lazily load the xor'd bitmap if required (and we haven't done so\n> +\t * already). Make sure to pass the xor'd bitmap's position along as a\n> +\t * hint to avoid an unnecessary binary search in\n> +\t * stored_bitmap_for_commit().\n> +\t */\n> +\tif (xor_pos == 0xffffffff) {\n> +\t\txor_bitmap = NULL;\n> +\t} else {\n> +\t\tstruct commit *xor_commit;\n> +\t\tstruct object_id xor_oid;\n> +\n> +\t\toidread(&xor_oid, bitmap_oid_pos(bitmap_git, xor_pos));\n> +\n> +\t\txor_commit = lookup_commit(the_repository, &xor_oid);\n> +\t\tif (!xor_commit)\n> +\t\t\treturn NULL;\n> +\n> +\t\txor_bitmap = stored_bitmap_for_commit(bitmap_git, xor_commit,\n> +\t\t\t\t\t\t &xor_pos);\n> +\t}\n\nThis is using an interesting type of tail-recursion. We might be\nbetter off using a loop with a stack: push to the stack the commit\npositions of the XOR bitmaps. At the very bottom, we get a bitmap\nwithout an XOR base. Then, pop off the stack, modifying the bitmap\nwith XOR operations as we go. (Perhaps we also store these bitmaps\nin-memory along the way?) Finally, we have the necessary bitmap.\n\nThis iterative approach avoids possible stack exhaustion if there\nare long XOR chains in the file.\n\n> +\n> +\t/*\n> +\t * Don't bother reading the commit's index position or its xor\n> +\t * offset:\n> +\t *\n> +\t * - The commit's index position is irrelevant to us, since\n> +\t * load_bitmap_entries_v1 only uses it to learn the object\n> +\t * id which is used to compute the hashmap's key. We already\n> +\t * have an object id, so no need to look it up again.\n> +\t *\n> +\t * - The xor_offset is unusable for us, since it specifies how\n> +\t * many entries previous to ours we should look at. This\n> +\t * makes sense when reading the bitmaps sequentially (as in\n> +\t * load_bitmap_entries_v1()), since we can keep track of\n> +\t * each bitmap as we read them.\n> +\t *\n> +\t * But it can't work for us, since the bitmap's don't have a\n> +\t * fixed size. So we learn the position of the xor'd bitmap\n> +\t * from the commit table (and resolve it to a bitmap in the\n> +\t * above if-statement).\n> +\t *\n> +\t * Instead, we can skip ahead and immediately read the flags and\n> +\t * ewah bitmap.\n> +\t */\n> +\tbitmap_git->map_pos = bitmap_ofs + sizeof(uint32_t) + sizeof(uint8_t);\n> +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\tbitmap = read_bitmap_1(bitmap_git);\n> +\tif (!bitmap)\n> +\t\treturn NULL;\n> +\n> +\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n\nLooks like we'd want to call store_bitmap() while popping the stack\nin the loop I recommended above.\n\n> +}\n> +\n> +static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t struct commit *commit,\n> +\t\t\t\t\t\t uint32_t *pos_hint)\n> {\n> \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n> \t\t\t\t\t commit->object.oid);\n> -\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n> +\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n> +\t\tuint32_t commit_pos;\n> +\t\tif (!bitmap_git->table_lookup)\n> +\t\t\treturn NULL;\n> +\n> +\t\t/* NEEDSWORK: cache misses aren't recorded. */\n> +\t\tif (pos_hint)\n> +\t\t\tcommit_pos = *pos_hint;\n> +\t\telse if (!bitmap_table_lookup(bitmap_git,\n> +\t\t\t\t\t &commit->object.oid,\n> +\t\t\t\t\t &commit_pos))\n> +\t\t\treturn NULL;\n> +\t\treturn lazy_bitmap_for_commit(bitmap_git, &commit->object.oid,\n> +\t\t\t\t\t commit_pos);\n\nThe extra bonus of going incremental is that we don't have recursion\nacross two methods, which I always find difficult to reason about.\n\n> +\t}\n> +\treturn kh_value(bitmap_git->bitmaps, hash_pos);\n> +}\n\n> @@ -26,6 +26,7 @@ struct bitmap_disk_header {\n> enum pack_bitmap_opts {\n> \tBITMAP_OPT_FULL_DAG = 1,\n> \tBITMAP_OPT_HASH_CACHE = 4,\n> +\tBITMAP_OPT_LOOKUP_TABLE = 16,\n\nPerhaps it is time to use hexadecimal representation here to match the\nfile format document?\n\nThanks,\n-Stolee\n"},{"id":"457579","messageId":"YrDvaMHz9DnjBqLs@nand.local","threadId":"58038","inReplyTo":"d139a4c48aa058b142c4860721b10344c5498031.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-20T22:06:32Z","receivedAt":"2022-06-20T22:06:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:33:10PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Bitmap lookup table extension can let git to parse only the necessary\n> bitmaps without loading the previous bitmaps one by one.\n>\n> Teach git to read and use the bitmap lookup table extension.\n>\n> Co-Authored-by: Taylor Blau \n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> pack-bitmap.c | 172 ++++++++++++++++++++++++++++++++++++++++++++++++--\n> pack-bitmap.h | 1 +\n> 2 files changed, 166 insertions(+), 7 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 36134222d7a..d5e5973a79f 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -15,6 +15,7 @@\n> #include \"list-objects-filter-options.h\"\n> #include \"midx.h\"\n> #include \"config.h\"\n> +#include \"hash-lookup.h\"\n>\n> /*\n> * An entry on the bitmap index, representing the bitmap for a given\n> @@ -82,6 +83,13 @@ struct bitmap_index {\n> \t/* The checksum of the packfile or MIDX; points into map. */\n> \tconst unsigned char *checksum;\n>\n> +\t/*\n> +\t * If not NULL, these point into the various commit table sections\n> +\t * (within map).\n> +\t */\n> +\tunsigned char *table_lookup;\n> +\tunsigned char *table_offsets;\n> +\n\nIf table_offsets ends up being a list of just offsets, we could assign\nthis to the appropriate type, e.g., 'uint64_t *'. We would want to\navoid using a type whose width is platform dependent, like off_t.\n\nBut if you end up taking my suggestion from a previous response (of\nmaking each entry in the offset table a triple of commit, offset, and\nxor position), make sure to _not_ get tempted to define a struct and\nassign table_lookup to be a pointer of that structure type.\n\nThat's because even though the struct *should* be packed as you expect,\nthe packing is mostly up to the compiler, so you can't guarantee its\nmembers won't have padding between them or at the end of the struct for\nalignment purposes.\n\n> \t/*\n> \t * Extended index.\n> \t *\n> @@ -185,6 +193,24 @@ static int load_bitmap_header(struct bitmap_index *index)\n> \t\t\tindex->hashes = (void *)(index_end - cache_size);\n> \t\t\tindex_end -= cache_size;\n> \t\t}\n> +\n> +\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n> +\t\t git_env_bool(\"GIT_READ_COMMIT_TABLE\", 1)) {\n\nWhat is the purpose of the GIT_READ_COMMIT_TABLE environment variable? I\nassume that it's to make it easier to run tests (especially performance\nones) with and without access to the lookup table. If so, we should\ndocument that (lightly) in the commit message, and rename this to be\nGIT_TEST_READ_COMMIT_TABLE to indicate that it shouldn't be used outside\nof tests.\n\n> +\t\t\tuint32_t entry_count = ntohl(header->entry_count);\n> +\t\t\tuint32_t table_size =\n> +\t\t\t\t(entry_count * the_hash_algo->rawsz) /* oids */ +\n> +\t\t\t\t(entry_count * sizeof(uint32_t)) /* offsets */ +\n> +\t\t\t\t(entry_count * sizeof(uint32_t)) /* xor offsets */ +\n> +\t\t\t\t(sizeof(uint32_t)) /* flags */;\n\nentry_count is definitely a 4-byte integer, so uint32_t is the right\ntype. But I think table_size should be a size_t, and computations on it\nshould be more strictly checked. Perhaps something like;\n\n size_t table_size = sizeof(uint32_t); /* flags */\n table_size = st_add(table_size, st_mult(entry_count, the_hash_algo->rawsz)); /* oids */\n table_size = st_add(table_size, st_mult(entry_count, sizeof(uint32_t))); /* offsets */\n table_size = st_add(table_size, st_mult(entry_count, sizeof(uint32_t))); /* xor offsets */\n\nor even:\n\n size_t table_size = sizeof(uint32_t); /* flags */\n table_size = st_add(table_size,\n st_mult(entry_count,\n the_hash_algo->rawsz + /* oids */\n sizeof(uint32_t) + /* offsets*/\n sizeof(uint32_t) /* xor offsets */\n ));\n\n> +\t\t\tif (table_size > index_end - index->map - header_size)\n> +\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit commit table)\");\n> +\n> +\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n> +\t\t\tindex->table_offsets = index->table_lookup + the_hash_algo->rawsz * entry_count;\n> +\n> +\t\t\tindex_end -= table_size;\n> +\t\t}\n\nLooks good.\n\n> @@ -470,7 +496,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n> \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n> \t\tgoto failed;\n>\n> -\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n> +\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n> \t\tgoto failed;\n\nNo need to load each of the bitmaps individually via\nload_bitmap_entries_v1() if we have a lookup table. That function\ndoesn't do any other initialization that we depend on, so it's OK to\njust avoid calling it altogether.\n\n> \treturn 0;\n> @@ -557,14 +583,145 @@ struct include_data {\n> \tstruct bitmap *seen;\n> };\n>\n> -struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n> -\t\t\t\t struct commit *commit)\n> +static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t struct commit *commit,\n> +\t\t\t\t\t\t uint32_t *pos_hint);\n> +\n> +static inline const unsigned char *bitmap_oid_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t uint32_t pos)\n> +{\n> +\treturn bitmap_git->table_lookup + (pos * the_hash_algo->rawsz);\n> +}\n> +\n> +static inline const void *bitmap_offset_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t uint32_t pos)\n> +{\n> +\treturn bitmap_git->table_offsets + (pos * 2 * sizeof(uint32_t));\n> +}\n> +\n> +static inline const void *xor_position_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t uint32_t pos)\n> +{\n> +\treturn (unsigned char*) bitmap_offset_pos(bitmap_git, pos) + sizeof(uint32_t);\n> +}\n> +\n> +static int bitmap_lookup_cmp(const void *_va, const void *_vb)\n> +{\n> +\treturn hashcmp(_va, _vb);\n> +}\n\nAll makes sense. Some light documentation might help explain what this\ncomparator function is used for (the bsearch() call below in\nbitmap_table_lookup()), although I suspect that this function will get\nslightly more complicated if you pack the table contents as I suggest,\nin which case more documentation will definitely help.\n\n> +\n> +static int bitmap_table_lookup(struct bitmap_index *bitmap_git,\n> +\t\t\t struct object_id *oid,\n> +\t\t\t uint32_t *commit_pos)\n> +{\n> +\tunsigned char *found = bsearch(oid->hash, bitmap_git->table_lookup,\n> +\t\t\t\t bitmap_git->entry_count,\n> +\t\t\t\t the_hash_algo->rawsz, bitmap_lookup_cmp);\n> +\tif (found)\n> +\t\t*commit_pos = (found - bitmap_git->table_lookup) / the_hash_algo->rawsz;\n\nIf you end up chaning the type of bitmap_git->table_lookup, make sure\nthat you scale the result of the pointer arithmetic accordingly, or cast\ndown to an 'unsigned char *' before you do any math.\n\n> +\treturn !!found;\n> +}\n> +\n> +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t struct object_id *oid,\n> +\t\t\t\t\t\t uint32_t commit_pos)\n> +{\n> +\tuint32_t xor_pos;\n> +\toff_t bitmap_ofs;\n> +\n> +\tint flags;\n> +\tstruct ewah_bitmap *bitmap;\n> +\tstruct stored_bitmap *xor_bitmap;\n> +\n> +\tbitmap_ofs = get_be32(bitmap_offset_pos(bitmap_git, commit_pos));\n> +\txor_pos = get_be32(xor_position_pos(bitmap_git, commit_pos));\n> +\n> +\t/*\n> +\t * Lazily load the xor'd bitmap if required (and we haven't done so\n> +\t * already). Make sure to pass the xor'd bitmap's position along as a\n> +\t * hint to avoid an unnecessary binary search in\n> +\t * stored_bitmap_for_commit().\n> +\t */\n> +\tif (xor_pos == 0xffffffff) {\n> +\t\txor_bitmap = NULL;\n> +\t} else {\n> +\t\tstruct commit *xor_commit;\n> +\t\tstruct object_id xor_oid;\n> +\n> +\t\toidread(&xor_oid, bitmap_oid_pos(bitmap_git, xor_pos));\n\nInteresting; this is a point that I forgot about from the original\npatch. xor_pos is an index (not an offset) into the list of commits in\nthe table of contents in the order appear in that table. We should be\nclear about (a) what that order is, and (b) that xor_pos is an index\ninto that order.\n\nThe rest of this function looks good to me.\n\n> +static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t\t struct commit *commit,\n> +\t\t\t\t\t\t uint32_t *pos_hint)\n> {\n> \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n> \t\t\t\t\t commit->object.oid);\n> -\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n> +\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n> +\t\tuint32_t commit_pos;\n> +\t\tif (!bitmap_git->table_lookup)\n> +\t\t\treturn NULL;\n\nI was going to suggest moving this check into the caller\nbitmap_for_commit() and making it a BUG() to call\nstored_bitmap_for_commit() with a NULL bitmap_git->table_lookup pointer.\n\nAnd I think this makes sense... if we return NULL here, then we know\nthat we definitely don't have a stored bitmap, since there's no table to\nlook it up in and we have already loaded everything else. So we\npropagate that NULL to the return value of bitmap_for_commit(), and that\nmakes sense. Good.\n\n> +\t\t/* NEEDSWORK: cache misses aren't recorded. */\n\nYeah. The problem here is that we can't record every commit that\n_doesn't_ have a bitmap every time we return NULL from one of these\nqueries, since there are arbitrarily many such commits that don't have\nbitmaps.\n\nWe could approximate it using a Bloom filter or something, and much of\nthat code is already written and could be interesting to try and reuse.\n\nBut I wonder if we could get by with something simpler, though, which\nwould cause us to load all bitmaps from the lookup table after a fixed\nnumber of cache misses (at which point we should force ourselves to load\neverything and just read everything out of a single O(1) lookup in the\nstored bitmap table).\n\nThat may or may not be a good idea, and the threshold will probably be\nhighly dependent on the system. So it may not even be worth it, but I\nthink it's an interesting area to experiemnt in and think a little more\nabout.\n\n> +\t\tif (pos_hint)\n> +\t\t\tcommit_pos = *pos_hint;\n\nHow does this commit_pos work again? I confess I have forgetten since I\nwrote some of this code a while ago... :-).\n\n> @@ -1699,8 +1856,9 @@ void test_bitmap_walk(struct rev_info *revs)\n> \tif (revs->pending.nr != 1)\n> \t\tdie(\"you must specify exactly one commit to test\");\n>\n> -\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n> -\t\tbitmap_git->version, bitmap_git->entry_count);\n> +\tif (!bitmap_git->table_lookup)\n> +\t\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n> +\t\t\tbitmap_git->version, bitmap_git->entry_count);\n\nShould we print this regardless of whether or not there is a lookup\ntable? We should be able to learn the entry count either way.\n\nThanks,\nTaylor\n"},{"id":"457580","messageId":"YrDxt1MkQKdNJL1F@nand.local","threadId":"58038","inReplyTo":"ed91ebf69a84405f16a4390e6fd208251b8ec53e.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-20T22:16:23Z","receivedAt":"2022-06-20T22:17:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:33:11PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Teach git to write bitmap lookup table extension. The table has the\n> following information:\n>\n> - `N` no of Object ids of each bitmapped commits\n\ns/no/number, s/Object/object, s/ids/IDs, and s/commits/commit\n\n> - A list of offset, xor-offset pair; the i'th pair denotes the\n> offsets and xor-offsets of i'th commit in the previous list.\n\ns/pair/pairs\n\n> - 4-byte integer denoting the flags\n>\n> Co-authored-by: Taylor Blau \n> Mentored-by: Taylor Blau \n> Co-mentored-by: Kaartic Sivaraam \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> pack-bitmap-write.c | 59 +++++++++++++++++++++++++++++++++++++++++++--\n> 1 file changed, 57 insertions(+), 2 deletions(-)\n>\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index c43375bd344..9e88a64dd65 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -650,7 +650,8 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n>\n> static void write_selected_commits_v1(struct hashfile *f,\n> \t\t\t\t struct pack_idx_entry **index,\n> -\t\t\t\t uint32_t index_nr)\n> +\t\t\t\t uint32_t index_nr,\n> +\t\t\t\t off_t *offsets)\n> {\n> \tint i;\n>\n> @@ -663,6 +664,9 @@ static void write_selected_commits_v1(struct hashfile *f,\n> \t\tif (commit_pos < 0)\n> \t\t\tBUG(\"trying to write commit not in index\");\n>\n> +\t\tif (offsets)\n> +\t\t\toffsets[i] = hashfile_total(f);\n> +\n\nMakes sense; we record the offset for the ith commit as however many\nbytes we've already written into the hashfile up to this point, since\nthe subsequent byte will begin the bitmap (well, the preceding few\nbytes of it, anyways) itself.\n\n> \t\thashwrite_be32(f, commit_pos);\n> \t\thashwrite_u8(f, stored->xor_offset);\n> \t\thashwrite_u8(f, stored->flags);\n> @@ -671,6 +675,49 @@ static void write_selected_commits_v1(struct hashfile *f,\n> \t}\n> }\n>\n> +static int table_cmp(const void *_va, const void *_vb)\n> +{\n> +\treturn oidcmp(&writer.selected[*(uint32_t*)_va].commit->object.oid,\n> +\t\t &writer.selected[*(uint32_t*)_vb].commit->object.oid);\n\nThis implementation looks right to me, but perhaps we should expand it\nout from the one-liner here to make it more readable. Perhaps something\nlike:\n\n static int table_cmp(const void *_va, const void *_vb)\n {\n struct commit *c1 = &writer.selected[*(uint32_t*)_va];\n struct commit *c2 = &writer.selected[*(uint32_t*)_vb];\n\n return oidcmp(&c1->object.oid, &c2->object.oid);\n }\n\nwhich is arguably slightly more readable than the one-liner (but I don't\nfeel that strongly about it.)\n\n> +static void write_lookup_table(struct hashfile *f,\n> +\t\t\t off_t *offsets)\n> +{\n> +\tuint32_t i;\n> +\tuint32_t flags = 0;\n> +\tuint32_t *table, *table_inv;\n> +\n> +\tALLOC_ARRAY(table, writer.selected_nr);\n> +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n> +\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable[i] = i;\n> +\tQSORT(table, writer.selected_nr, table_cmp);\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable_inv[table[i]] = i;\n\nRight... so table[0] will give us the index into writer.selected of the\ncommit with the earliest OID in lexicographic order. And table_inv goes\nthe other way around: table_inv[i] will tell us the lexicographic\nposition of the commit at writer.selected[i].\n\n> +\tfor (i = 0; i < writer.selected_nr; i++) {\n> +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> +\t\tstruct object_id *oid = &selected->commit->object.oid;\n> +\n> +\t\thashwrite(f, oid->hash, the_hash_algo->rawsz);\n> +\t}\n> +\tfor (i = 0; i < writer.selected_nr; i++) {\n> +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> +\n> +\t\thashwrite_be32(f, offsets[table[i]]);\n> +\t\thashwrite_be32(f, selected->xor_offset\n> +\t\t\t ? table_inv[table[i] - selected->xor_offset]\n\n...which we need to discover the position of the XOR'd bitmap. Though\nI'm not sure if I remember why `table[i] - selected->xor_offset` is\nright and not `i - selected->xor_offset`.\n\nThanks,\nTaylor\n"},{"id":"457582","messageId":"YrDyOivHcnA1qirF@nand.local","threadId":"58038","inReplyTo":"661c1137e1c918619f6624d2e331bafd9c3281dc.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/6] builtin/pack-objects.c: learn pack.writeBitmapLookupTable","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-20T22:18:34Z","receivedAt":"2022-06-20T22:18:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:33:12PM +0000, Taylor Blau via GitGitGadget wrote:\n> From: Taylor Blau \n>\n> Teach git to provide a way for users to enable/disable bitmap lookup\n> table extension by providing a config option named 'writeBitmapLookupTable'.\n>\n> Signed-off-by: Taylor Blau \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> Documentation/config/pack.txt | 7 +++++++\n> builtin/pack-objects.c | 8 ++++++++\n> 2 files changed, 15 insertions(+)\n>\n> diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\n> index ad7f73a1ead..e12008d2415 100644\n> --- a/Documentation/config/pack.txt\n> +++ b/Documentation/config/pack.txt\n> @@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n> computed; instead, any namehashes stored in an existing bitmap are\n> permuted into their appropriate location when writing a new bitmap.\n>\n> +pack.writeBitmapLookupTable::\n> +\tWhen true, git will include a \"lookup table\" section in the\n\ns/git/Git (I typically use \"git\" when talking about the command-line\ntool, and Git when talking about the project as a proper noun).\n\n> +\tbitmap index (if one is written). This table is used to defer\n> +\tloading individual bitmaps as late as possible. This can be\n> +\tbeneficial in repositories which have relatively large bitmap\n> +\tindexes. Defaults to false.\n\nIs there a reason that we would want to default to \"false\" here? Perhaps\nin the first version of two we would want this to be an opt-in (since\nthere is no publicly documented way to opt-out of reading the extension\nonce it is written).\n\nWe should make sure to enable this by default at some point in the\nfuture.\n\n> pack.writeReverseIndex::\n\n...since it's easy to forget ;-).\n\nThanks,\nTaylor\n"},{"id":"457594","messageId":"20220621082308.21376-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"b21af0bc-3234-3aa2-e4a0-82874e9a670e@github.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T08:23:08Z","receivedAt":"2022-06-21T08:23:35Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stole wrote:\n\n> It might be worth mentioning in your commit message what happens when an\n> older version of Git (or JGit) notices this flag. Does it refuse to\n> operate on the .bitmap file? Does it give a warning or die? It would be\n> nice if this extension could be ignored (it seems like adding the extra\n> data at the end does not stop the bitmap data from being understood).\n\nNo, it doesn't refuse to operate on the .bitmap file. It just ignores the\nextension. Will update the commit message.\n\n> Perhaps it would be better to say \"the last N * (HASH_LEN + 8) + 4 bytes\n> preceding the trailing hash\" or something? This gives us a concrete way\n> to compute the start of the table, while also being clear that the table\n> is included in the trailing hash.\n\nHmm, well said. Will update it.\n\n> Could you expand that these objects are commit OIDs, one for each bitmap\n> in the file. Are they sorted in lexicographical order for binary search,\n> or are we expecting to read the entire table into a hashtable in-memory?\n\nYeah, of course! They are sorted in lexicographical order for binary search.\n\n\n> Interesting to give the xor chains directions here. You say \"position\"\n> here for the second commit: do you mean within the list of object names\n> as opposed to the offset? That would make the most sense so we can trace\n> the full list of XORs we need to make all at once.\n\nI think I blundered here. I forgot that the xor-offset is relative to the\ncurrent bitmap. The current proposed code takes it as ABSOLUTE value and\ntries to find the commit on that position (in the list of commit ids). So,\nthere are two faults in my code - (1) As the xor-offset have an upper limit\n(which is 10 probably; not sure), any of the first 10 commits is always\nselected. (2) As xor-offsets are relative to the current bitmap, it depends\nOn the order of the bitmaps. These bitmaps are ordered by the date of their\ncorresponding commit and commit ids in the lookup table are ordered\nlexicographically. So, we can't use that xor-offset to find the xor'd\ncommit position.\n\nWill fix it.\n\n> Are .bitmap files already constrained to 4GB, so these 32-bit offsets\n> make sense? Using 64-bit offsets would be a small cost here, I think,\n> without needing to do any fancy \"overflow\" tables that could introduce\n> a variable-length extension.\n\nI think you're right. I should use 64-bit types here.\n\n> I'm guessing this is at the end of the extension because a future flag\n> could modify the length of the extension, so we need the flags to be\n> in a predictable location. Could we make that clear somewhere?\n\nFlags are at the end of this extension.\n\nThanks :)\n"},{"id":"457595","messageId":"20220621083114.21429-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrCpv3XEoB6lOlY4@nand.local","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T08:31:14Z","receivedAt":"2022-06-21T08:31:42Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> Abhradeep -- do you have any thoughts about what this might be used for?\n> I'll try to remember it myself, but I imagine that we could just as\n> easily remove this altogether and avoid the confusion.\n\nHonestly, I never understood the logic behind adding this flag option.\nI thought you have a reason to do that. Even I was thinking of curving\nit to 1 byte. I will remove it then.\n\nThanks :)\n"},{"id":"457596","messageId":"20220621092253.21667-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrCsricF+2rQXiBk@nand.local","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T09:22:53Z","receivedAt":"2022-06-21T09:23:15Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> In cases where the result can be read or computed without\n> significant additional traversal (e.g., all commits of interest\n> already have bitmaps computed), we can save some time loading and\n> parsing a majority of the bitmap file that we will never read.\n>\n> But in cases where the bitmaps are out-of-date, or there is\n> significant traversal required to go from the reference tips to\n> what's contained in the .bitmap file, this table provides minimal\n> benefit (or something).\n>\n> Of course, you should verify that that is actually true before we insert\n> it into the commit message as such ;-). But that sort of information may\n> help readers understand what the purpose of this change is towards the\n> beinning of the series.\n\nThe performance tests cover tests for command like \"git rev-list --count\n--objects --all\", \"simulated clone\", \"simulated fetch\" etc. And I tested\nit with both the Git and Linux. In both cases, the average cost of\n\"Without lookup table\" is bigger than \"with lookup table\". The margin of\ndifference is bigger for linux. Though, I need to fix the calculation\nof xor-offset (see my reply to derrick), the fix will not affect the\nperformance too much. So, what you're saying is true. I think I didn't\nwrite the bitmap out-of-date test though.\n\n> Here and elsewhere: I typically use my address when\n> contributing to Git. So any trailers that mention my email or commits\n> that you send on my behalf should use that address, too.\n\nOhh, sorry! Will fix it.\n\n> It the space between \"(0xf)\" and the first \":\" intentional? Similarly,\n> should there be two or three colons at the end (either \"::\" or \":::\")?\n\nYes, it is intentional. My previous patch (formatting the bitmap-format.txt)\nuses nested description lists. \":::\" means it is the level 3 description list.\nThe space is required else asciidoc will assume that it is level 4 description\nlist.\n\n> I remember we had a brief off-list discussion about whether we should\n> store the full object IDs in the offset table, or whether we could store\n> their pack- or index-relative ordering. Is there a reason to prefer one\n> or the other?\n>\n> I don't think we need to explain the choice fully in the documentation\n> in this patch, but it may be worth thinking about separately\n> nonetheless. We can store either order and convert it to an object ID in\n> constant time.\n>\n> To figure out which is best, I would recommend trying a few different\n> choices here and seeing how they do or don't impact your performance\n> testing.\n\nI think at that time I thought it would add extra cost of computing\nthe actual commit ids from those index position. So, I didn't go \nfurther here.\n\nI still have a feeling that there is some way to get rid of this\nlist of commit ids. But at the same time, I do not want to add\nextra computation to the code.\n\n> A couple of small thoughts here. I wonder if we'd get better locality if\n> we made each record look something like:\n>\n> (object_id, offset, xor_pos)\n>\n> Where object_id is either 20- or 4-bytes long (depending if we store the\n> full object ID, or some 4-byte identifier that allows us to discover\n> it), offset is 8 bytes long, and xor_pos is 4-bytes (since in practice\n> we don't support packs or MIDXs which have more than 2^32-1 objects).\n>\n> In the event that this table doesn't fit into a single cache line, I\n> think we'll get better performance out of reading it by not forcing the\n> cache to evict itself whenever we need to refer back to the object_id.\n\nOk, will look into it.\n\n> I mentioned in my reply to Stolee earlier, but I think that we should\n> either (a) try to remember what this is for and document it, or (b)\n> remove it.\n\nLet us for now remove it.\n\nThanks :)\n"},{"id":"457599","messageId":"20220621100800.21767-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"25e03c86-5a47-2100-2da7-a635673a8e38@github.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T10:08:00Z","receivedAt":"2022-06-21T10:09:00Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stolee wrote:\n\n> I think you mean 0x10 (b_1_0000) instead of 0xf (b_1111).\n>\n> I noticed when looking at the constant in patch 2.\n\nYes, you're right. It's kind of embarrassment for me :)\n\nIf the flag was Oxf it would enable all the extensions.\n\n"},{"id":"457600","messageId":"20220621102832.21837-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"92dc6860-ff35-0989-5114-fe1e220ca10c@github.com","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T10:28:32Z","receivedAt":"2022-06-21T10:29:09Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stolee wrote:\n\n> Here is an attempt to reword this a bit:\n>\n> The bitmap lookup table extension was documented by an earlier\n> change, but Git does not yet know how to parse that information.\n> The extension allows parsing a smaller portion of the bitmap\n> file in order to find bitmaps for specific commits.\n\nGot it. Thanks.\n\n> This environment variable does not appear to be used or\n> documented anywhere. Do we really want to use it as a way\n> to disable reading the lookup table in general? Or would it be\n> better to have a GIT_TEST_* variable for disabling the read\n> during testing?\n\nGIT_TEST_* is perfect. This was mainly for testing purpose.\n\n> Here, uint32_T is probably fine, but maybe we should just use\n size_t instead? Should we use st_mult() and st_add() everywhere?\n\nYeah, it would be better to use st_*().\n\n> I see that we have a two-method recursion loop. Please move this\n> declaration to immediately before lazy_bitmap_for_commit() so it\n> is declared as late as possible.\n\nOk.\n\n> These two helpers should probably return a size_t and uint32_t\n> instead of a pointer. Let these do get_be[32|64]() on the computed\n> pointer.\n\nOk.\n\n> This is using an interesting type of tail-recursion. We might be\n> better off using a loop with a stack: push to the stack the commit\n> positions of the XOR bitmaps. At the very bottom, we get a bitmap\n> without an XOR base. Then, pop off the stack, modifying the bitmap\n> with XOR operations as we go. (Perhaps we also store these bitmaps\n> in-memory along the way?) Finally, we have the necessary bitmap.\n\nHmm, got the point. I need to fix the xor-offset related issue first\n(That I said earlier) before doing this.\n\n> Perhaps it is time to use hexadecimal representation here to match the\n> file format document?\n\nYeah, of course!\n\nThanks :)\n"},{"id":"457606","messageId":"20220621115212.22383-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrDvaMHz9DnjBqLs@nand.local","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T11:52:12Z","receivedAt":"2022-06-21T11:52:42Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> What is the purpose of the GIT_READ_COMMIT_TABLE environment variable? I\n> assume that it's to make it easier to run tests (especially performance\n> ones) with and without access to the lookup table. If so, we should\n> document that (lightly) in the commit message, and rename this to be\n> GIT_TEST_READ_COMMIT_TABLE to indicate that it shouldn't be used outside\n> of tests.\n\nThis is mainly for testing, GIT_TEST_READ_COMMIT_TABLE is perfect.\n\n\n> All makes sense. Some light documentation might help explain what this\n> comparator function is used for (the bsearch() call below in\n> bitmap_table_lookup()), although I suspect that this function will get\n> slightly more complicated if you pack the table contents as I suggest,\n> in which case more documentation will definitely help.\n\nOk.\n\n> Interesting; this is a point that I forgot about from the original\n> patch. xor_pos is an index (not an offset) into the list of commits in\n> the table of contents in the order appear in that table. We should be\n> clear about (a) what that order is, and (b) that xor_pos is an index\n> into that order.\n\nThis is exactly what I said in my first reply. I made a mistake here.\n(1) As xor_pos is relative to the current bitmap, it depends on the bitmap\nentry order. These two order are not same. One is ordered by date, another\nis lexicographically ordered. I will fix it.\n\n> Yeah. The problem here is that we can't record every commit that\n> _doesn't_ have a bitmap every time we return NULL from one of these\n> queries, since there are arbitrarily many such commits that don't have\n> bitmaps.\n>\n> We could approximate it using a Bloom filter or something, and much of\n> that code is already written and could be interesting to try and reuse.\n> But I wonder if we could get by with something simpler, though, which\n> would cause us to load all bitmaps from the lookup table after a fixed\n> number of cache misses (at which point we should force ourselves to load\n> everything and just read everything out of a single O(1) lookup in the\n> stored bitmap table).\n>\n> That may or may not be a good idea, and the threshold will probably be\n> highly dependent on the system. So it may not even be worth it, but I\n> think it's an interesting area to experiemnt in and think a little more\n> about.\n\nNow I got the point. I wonder what if we leave it as it is. How much will\nit affect the code?\n\n> How does this commit_pos work again? I confess I have forgetten since I\n> wrote some of this code a while ago... :-).\n\nIt is using recursive strategy. The first call to `stored_bitmap_for_commit`\nfunction do not have `pos_hint`. So, it uses `bitmap_table_lookup` to find\nthe commit position in the list and makes a call to `lazy_bitmap_for_commit`\nfunction. This function gets the offset and xor-offset using the commit id's\nposition in the list. If xor-offset exists, it is using this xor-offset to\nget the xor-bitmap by calling `stored_bitmap_for_commit` again. But this time\n`pos_hint` is xor-offset. This goes on till the last non-xor bitmap has found.\n\nAs I said before, xor-offset should be an absolute value to make it work\ncorrectly.\n\n> Should we print this regardless of whether or not there is a lookup\n> table? We should be able to learn the entry count either way.\n\nNo, this is necessary. \"Bitmap v1 test (%d entries loaded)\" means\nall the bitmap entries has been loaded. It is basically for \n`load_bitmap_entries_bitmap_v1` function which loads all the bitmaps\nOne by one. But if there is a lookup table, `prepare_bitmap_git`\nfunction will not load every entries and thus printing the above\nline is wrong.\n\nThanks :)\n"},{"id":"457607","messageId":"20220621125054.23035-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrDxt1MkQKdNJL1F@nand.local","subject":"Re: [PATCH 3/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-21T12:50:54Z","receivedAt":"2022-06-21T12:51:27Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> I'm not sure if I remember why `table[i] - selected->xor_offset` is\n> right and not `i - selected->xor_offset`.\n\nEven I myself got confused! Before sending the patch to the mailing\nlist, I was clear about that. That's why I didn't catch the so called\nmistake I have been notifying till now. Thanks Taylor for asking\nthe question!\n\nI should add a comment before the line so that people can understand it.\nLet us parse `table_inv[table[i] - selected->xor_offset]` -\n\nSuppose bitmap entries be like - \n\nBitmap 0 (for commit 0)\nBitmap 1 (for commit 1)\nBitmap 2 (for commit 2)\nBitmap 3 (for commit 3)\n.\n.\n.\nBitmap 20 (for commit 20)\n\nThese bitmaps are ordered by the date of their corresponding commit.\n`table` array maps commit's lexicographic order to its bitmap order.\n`table_inv` stores the reverse (i.e. it maps bitmap order to lexicographic\norder). Say for example, if commit 4 is lexicographically first among all the\nCommits then `table[0]` is 4. Similarly `table[1]`=2, table[2]=1 etc.\n`table_inv[4]` is 0, table_inv[2]=1 etc.\n\nNow suppose commit 4's bitmap has xor-relation with commit 2's bitmap.\nSo, xor-offset for bitmap 4 is 2. And `table[0] - selected->xor_offset`\nis equal to 4-2 = 2. It is pointing to the commit 2. Now, 2 is in bitmap\nOrder. We need to convert it into lexicographic order. So, table_inv[2]\ngives us the lexicographic order position of commit 2 I.e. 1.\n\nLong story short, there is no issue regarding xor_offset. This xor_offset\nis not relative to the current commit. It is absolute.\n\nSorry for the initial claim :)\n"},{"id":"457719","messageId":"YrNCqcU4onE43Vl7@nand.local","threadId":"58038","inReplyTo":"20220621083114.21429-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T16:26:17Z","receivedAt":"2022-06-22T16:26:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 21, 2022 at 02:01:14PM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n>\n> > Abhradeep -- do you have any thoughts about what this might be used for?\n> > I'll try to remember it myself, but I imagine that we could just as\n> > easily remove this altogether and avoid the confusion.\n>\n> Honestly, I never understood the logic behind adding this flag option.\n> I thought you have a reason to do that. Even I was thinking of curving\n> it to 1 byte. I will remove it then.\n\nI think removing it makes more sense. Since many of the other fields are\n4-bytes wide, it's important for alignment purposes that those fields\nhave addresses which are a multiple of four (relative to the start of\nthe region, hence the 4-byte wide flags field).\n\nBut I'd just as soon get rid of it, so I think that makes sense to me.\n\nThanks,\nTaylor\n"},{"id":"457720","messageId":"YrNDZMixyt8xNAvS@nand.local","threadId":"58038","inReplyTo":"20220621092253.21667-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T16:29:24Z","receivedAt":"2022-06-22T16:29:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 21, 2022 at 02:52:53PM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n> > I remember we had a brief off-list discussion about whether we should\n> > store the full object IDs in the offset table, or whether we could store\n> > their pack- or index-relative ordering. Is there a reason to prefer one\n> > or the other?\n> >\n> > I don't think we need to explain the choice fully in the documentation\n> > in this patch, but it may be worth thinking about separately\n> > nonetheless. We can store either order and convert it to an object ID in\n> > constant time.\n> >\n> > To figure out which is best, I would recommend trying a few different\n> > choices here and seeing how they do or don't impact your performance\n> > testing.\n>\n> I think at that time I thought it would add extra cost of computing\n> the actual commit ids from those index position. So, I didn't go\n> further here.\n\nIt should be negligible relative to everything else, I would imagine.\nThe function that converts an index position into an object ID is\n`nth_packed_object_id()`.\n\n> I still have a feeling that there is some way to get rid of this\n> list of commit ids. But at the same time, I do not want to add\n> extra computation to the code.\n\nI'm hoping that the additional complexity is minor. And if we can save\nsome extra bytes that aren't necessary in the first place without\ncompromising on performance, I think that's worthwhile to do.\n\nThanks,\nTaylor\n"},{"id":"457721","messageId":"YrNDmHrvVlI5cFJC@nand.local","threadId":"58038","inReplyTo":"20220621100800.21767-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T16:30:16Z","receivedAt":"2022-06-22T16:30:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 21, 2022 at 03:38:00PM +0530, Abhradeep Chakraborty wrote:\n> Derrick Stolee wrote:\n>\n> > I think you mean 0x10 (b_1_0000) instead of 0xf (b_1111).\n> >\n> > I noticed when looking at the constant in patch 2.\n>\n> Yes, you're right. It's kind of embarrassment for me :)\n\nIt happens ;). Let's use 0x10 instead.\n\nThanks,\nTaylor\n"},{"id":"457722","messageId":"20220622164523.46256-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrNDZMixyt8xNAvS@nand.local","subject":"Re: [PATCH 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-22T16:45:23Z","receivedAt":"2022-06-22T16:47:43Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> It should be negligible relative to everything else, I would imagine.\n> The function that converts an index position into an object ID is\n> `nth_packed_object_id()`.\n>\n> > I still have a feeling that there is some way to get rid of this\n> > list of commit ids. But at the same time, I do not want to add\n> > extra computation to the code.\n>\n> I'm hoping that the additional complexity is minor. And if we can save\n> some extra bytes that aren't necessary in the first place without\n> compromising on performance, I think that's worthwhile to do.\n\nOk. I will look into it then.\n\nMost of the reviews has been addressed. Hope I will be able to submit\nit soon.\n\nThanks :)\n"},{"id":"457723","messageId":"YrNIJ6z+a4++MGQ8@nand.local","threadId":"58038","inReplyTo":"20220621115212.22383-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T16:49:43Z","receivedAt":"2022-06-22T16:52:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 21, 2022 at 05:22:12PM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n> > Yeah. The problem here is that we can't record every commit that\n> > _doesn't_ have a bitmap every time we return NULL from one of these\n> > queries, since there are arbitrarily many such commits that don't have\n> > bitmaps.\n> >\n> > We could approximate it using a Bloom filter or something, and much of\n> > that code is already written and could be interesting to try and reuse.\n> > But I wonder if we could get by with something simpler, though, which\n> > would cause us to load all bitmaps from the lookup table after a fixed\n> > number of cache misses (at which point we should force ourselves to load\n> > everything and just read everything out of a single O(1) lookup in the\n> > stored bitmap table).\n> >\n> > That may or may not be a good idea, and the threshold will probably be\n> > highly dependent on the system. So it may not even be worth it, but I\n> > think it's an interesting area to experiemnt in and think a little more\n> > about.\n>\n> Now I got the point. I wonder what if we leave it as it is. How much will\n> it affect the code?\n\nI'm not sure, and I think that it depends a lot on the repository and\nquery that we're running.\n\nI'd imagine that the effect is probably measurable, but small. Each hash\nlookup is cheap, but if there are many such lookups (a large proportion\nof which end up resulting in \"no, we haven't loaded this bitmap yet\" and\nthen \"...because no such bitmap exists for that commit\") at some point\nit is worth it to fault all of the commits that _do_ have bitmaps in and\nanswer authoritatively.\n\nIn other words, right now we have to do two queries when an commit\ndoesn't have a bitmap stored:\n\n - first, a lookup to see whether we have already loaded a bitmap for\n that commit\n\n - then, a subsequent lookup to see whether the .bitmap file itself has\n a bitmap for that commit, but we just haven't loaded it yet\n\nIf we knew that we had loaded all of the bitmaps in the file, then we\ncould simplify the above two queries into one, since whatever the first\none returns is enough to know whether or not a bitmap exists at all.\n\n> > How does this commit_pos work again? I confess I have forgetten since I\n> > wrote some of this code a while ago... :-).\n>\n> It is using recursive strategy. The first call to `stored_bitmap_for_commit`\n> function do not have `pos_hint`. So, it uses `bitmap_table_lookup` to find\n> the commit position in the list and makes a call to `lazy_bitmap_for_commit`\n> function. This function gets the offset and xor-offset using the commit id's\n> position in the list. If xor-offset exists, it is using this xor-offset to\n> get the xor-bitmap by calling `stored_bitmap_for_commit` again. But this time\n> `pos_hint` is xor-offset. This goes on till the last non-xor bitmap has found.\n\nAhhh. Thanks for refreshing my memory. I wonder if you think there is a\nconvenient way to work some of this into a short comment to help other\nreaders in the future, too.\n\n> As I said before, xor-offset should be an absolute value to make it work\n> correctly.\n\nYep, makes sense.\n\n> > Should we print this regardless of whether or not there is a lookup\n> > table? We should be able to learn the entry count either way.\n>\n> No, this is necessary. \"Bitmap v1 test (%d entries loaded)\" means\n> all the bitmap entries has been loaded. It is basically for\n> `load_bitmap_entries_bitmap_v1` function which loads all the bitmaps\n> One by one. But if there is a lookup table, `prepare_bitmap_git`\n> function will not load every entries and thus printing the above\n> line is wrong.\n\nRight, that part makes sense to me. But I wonder if we should still\nprint something, perhaps just \"Bitmap v1 test\" or \"Bitmap v1 test (%d\nentries)\" omitting the \"loaded\" part.\n\nThanks,\nTaylor\n"},{"id":"457724","messageId":"YrNIhGt8mlKMO40D@nand.local","threadId":"58038","inReplyTo":"20220621125054.23035-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH 3/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T16:51:16Z","receivedAt":"2022-06-22T16:54:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 21, 2022 at 06:20:54PM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n>\n> > I'm not sure if I remember why `table[i] - selected->xor_offset` is\n> > right and not `i - selected->xor_offset`.\n>\n> Even I myself got confused! Before sending the patch to the mailing\n> list, I was clear about that. That's why I didn't catch the so called\n> mistake I have been notifying till now. Thanks Taylor for asking\n> the question!\n>\n> I should add a comment before the line so that people can understand it.\n> Let us parse `table_inv[table[i] - selected->xor_offset]` -\n>\n> Suppose bitmap entries be like -\n>\n> Bitmap 0 (for commit 0)\n> Bitmap 1 (for commit 1)\n> Bitmap 2 (for commit 2)\n> Bitmap 3 (for commit 3)\n> .\n> .\n> .\n> Bitmap 20 (for commit 20)\n>\n> These bitmaps are ordered by the date of their corresponding commit.\n> `table` array maps commit's lexicographic order to its bitmap order.\n> `table_inv` stores the reverse (i.e. it maps bitmap order to lexicographic\n> order). Say for example, if commit 4 is lexicographically first among all the\n> Commits then `table[0]` is 4. Similarly `table[1]`=2, table[2]=1 etc.\n> `table_inv[4]` is 0, table_inv[2]=1 etc.\n>\n> Now suppose commit 4's bitmap has xor-relation with commit 2's bitmap.\n> So, xor-offset for bitmap 4 is 2. And `table[0] - selected->xor_offset`\n> is equal to 4-2 = 2. It is pointing to the commit 2. Now, 2 is in bitmap\n> Order. We need to convert it into lexicographic order. So, table_inv[2]\n> gives us the lexicographic order position of commit 2 I.e. 1.\n>\n> Long story short, there is no issue regarding xor_offset. This xor_offset\n> is not relative to the current commit. It is absolute.\n>\n> Sorry for the initial claim :)\n\nAhhhhh. Makes perfect sense. Thanks!\n\nThanks,\nTaylor\n"},{"id":"457725","messageId":"YrNJW00QuHbfi090@nand.local","threadId":"58038","inReplyTo":"a404779a30f767f7a05926fb99f354ea7958ab4b.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] bitmap-commit-table: add tests for the bitmap lookup table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T16:54:51Z","receivedAt":"2022-06-22T16:55:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:33:13PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Add tests to check the working of the newly implemented lookup table.\n>\n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n> t/t5326-multi-pack-bitmaps.sh | 19 +++++++++++++++++++\n> 2 files changed, 33 insertions(+)\n>\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index f775fc1ce69..f05d3e6ace7 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -43,6 +43,20 @@ test_expect_success 'full repack creates bitmaps' '\n>\n> basic_bitmap_tests\n>\n> +test_expect_success 'using lookup table does not affect basic bitmap tests' '\n> +\ttest_config pack.writeBitmapLookupTable true &&\n> +\tgit repack -adb\n> +'\n\nWhether or not we end up making pack.writeBitmapLookupTable be \"true\" by\ndefault, I wonder if we should just set it to \"true\" whenever we write a\nbitmap in this file, and then adjust whether or not we *read* the lookup\ntable with the GIT_TEST_ environment variable you introduced a few\ncommits back.\n\nThinking on it more, though, I don't think it makes a huge practical\ndifference for the code here in \"t\", since these repositories are tiny\nand repacking them or rewriting their bitmaps is cheap.\n\nBut in the performance tests it probably makes a bigger difference.\n\n> +basic_bitmap_tests\n> +\n> +test_expect_success 'using lookup table does not let each entries to be parsed one by one' '\n> +\ttest_config pack.writeBitmapLookupTable true &&\n> +\tgit repack -adb &&\n> +\tgit rev-list --test-bitmap HEAD 2>out &&\n> +\tgrep \"Found bitmap for\" out &&\n> +\t! grep \"Bitmap v1 test \"\n> +'\n> +\n> test_expect_success 'incremental repack fails when bitmaps are requested' '\n> \ttest_commit more-1 &&\n> \ttest_must_fail git repack -d 2>err &&\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> index 4fe57414c13..85fbdf5e4bb 100755\n> --- a/t/t5326-multi-pack-bitmaps.sh\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -306,5 +306,24 @@ test_expect_success 'graceful fallback when missing reverse index' '\n> \t\t! grep \"ignoring extra bitmap file\" err\n> \t)\n> '\n> +test_expect_success 'multi-pack-index write --bitmap writes lookup table if enabled' '\n> +\trm -fr repo &&\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\ttest_commit_bulk 106 &&\n\nIs there a reason we need to write this many commits? I think this is\ncopied from a test further up which deals explicitly with a case where\nthere are too many commits to write bitmaps for all of them (hence we\nneed to write more commits than 100 or so).\n\nBut I think for our purposes here we just need a single commit, written\ninto a single pack, which is covered with a MIDX.\n\nSo it should suffice to do something like:\n\n test_commit base &&\n git repack -ad &&\n git config pack.writeBitmapLookupTable true &&\n git multi-pack-index write --bitmap &&\n [...]\n\ninstead of what's written here.\n\nThanks,\nTaylor\n"},{"id":"457730","messageId":"YrNODmtvF6/Vlwv2@nand.local","threadId":"58038","inReplyTo":"f5f725a3fe2ac0c93088c48ac520303a3df2c83d.1655728395.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/6] bitmap-lookup-table: add performance tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T17:14:54Z","receivedAt":"2022-06-22T17:15:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 20, 2022 at 12:33:14PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Add performance tests for bitmap lookup table extension.\n\nThese tests look good, though I left a few notes below which boil down\nto recommending a separate commit to set pack.writeReverseIndex=true,\nand some suggestions for how to clean up the diff in the two performance\nscripts you modified.\n\nI would be interested to see the relevant results from running these\nperf scripts on a reasonably large-sized repository, e.g. the kernel or\nsimilar.\n\nFor the next version of this series, would you mind running these\nscripts and including the results in this commit message?\n\n> Mentored-by: Taylor Blau \n> Co-mentored-by: Kaartic Sivaraam \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> t/perf/p5310-pack-bitmaps.sh | 60 +++++++++++++++++++-----------\n> t/perf/p5326-multi-pack-bitmaps.sh | 55 +++++++++++++++++----------\n> 2 files changed, 73 insertions(+), 42 deletions(-)\n>\n> diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\n> index 7ad4f237bc3..a8d9414de92 100755\n> --- a/t/perf/p5310-pack-bitmaps.sh\n> +++ b/t/perf/p5310-pack-bitmaps.sh\n> @@ -10,10 +10,11 @@ test_perf_large_repo\n> # since we want to be able to compare bitmap-aware\n> # git versus non-bitmap git\n> #\n> -# We intentionally use the deprecated pack.writebitmaps\n> +# We intentionally use the deprecated pack.writeBitmaps\n> # config so that we can test against older versions of git.\n> test_expect_success 'setup bitmap config' '\n> -\tgit config pack.writebitmaps true\n> +\tgit config pack.writeBitmaps true &&\n> +\tgit config pack.writeReverseIndex true\n\nI suspect that eliminating the overhead of generating the reverse index\nin memory is important to see the effect of this test. We should make\nsure that this is done in a separate step so when we compare two commits\nthat both have a reverse index written.\n\nThat being said, we should probably make reverse indexes be the default\nanyways, since they help significantly with all kinds of things (really,\nany operation which has to generate a reverse index in memory, like\npreparing a pack to push, the '%(objectsize:disk)' cat-file formatting\natom, and so on.\n\nSo at a minimum I would suggest extracting a separate commit here which\nsets pack.writeReverseIndex to true for this test. That way the commit\nprior to this has reverse indexes written, and comparing \"this commit\"\nto \"the previous one\" is isolating the effect of just the lookup table.\n\nBut as a useful sideproject, it would be worthwhile to investigate\nsetting this to true by default everywhere, perhaps after this series\nhas settled a little more (or if you are blocked / want something else\nto do).\n\n> '\n>\n> # we need to create the tag up front such that it is covered by the repack and\n> @@ -28,27 +29,42 @@ test_perf 'repack to disk' '\n>\n> test_full_bitmap\n>\n> -test_expect_success 'create partial bitmap state' '\n> -\t# pick a commit to represent the repo tip in the past\n> -\tcutoff=$(git rev-list HEAD~100 -1) &&\n> -\torig_tip=$(git rev-parse HEAD) &&\n> -\n> -\t# now kill off all of the refs and pretend we had\n> -\t# just the one tip\n> -\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n> -\tgit update-ref HEAD $cutoff &&\n> -\n> -\t# and then repack, which will leave us with a nice\n> -\t# big bitmap pack of the \"old\" history, and all of\n> -\t# the new history will be loose, as if it had been pushed\n> -\t# up incrementally and exploded via unpack-objects\n> -\tgit repack -Ad &&\n> -\n> -\t# and now restore our original tip, as if the pushes\n> -\t# had happened\n> -\tgit update-ref HEAD $orig_tip\n> +test_perf 'use lookup table' '\n> + git config pack.writeBitmapLookupTable true\n> '\n\nThis part doesn't need to use 'test_perf', since we don't care about the\nperformance of running \"git config\". Instead, using\n`test_expect_success` is more appropriate here.\n\n> -test_partial_bitmap\n> +test_perf 'repack to disk (lookup table)' '\n> + git repack -adb\n> +'\n> +\n> +test_full_bitmap\n> +\n> +for i in false true\n> +do\n> +\t$i && lookup=\" (lookup table)\"\n> +\ttest_expect_success \"create partial bitmap state$lookup\" '\n> +\t\tgit config pack.writeBitmapLookupTable '\"$i\"' &&\n> +\t\t# pick a commit to represent the repo tip in the past\n> +\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n> +\t\torig_tip=$(git rev-parse HEAD) &&\n> +\n> +\t\t# now kill off all of the refs and pretend we had\n> +\t\t# just the one tip\n> +\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n> +\t\tgit update-ref HEAD $cutoff &&\n> +\n> +\t\t# and then repack, which will leave us with a nice\n> +\t\t# big bitmap pack of the \"old\" history, and all of\n> +\t\t# the new history will be loose, as if it had been pushed\n> +\t\t# up incrementally and exploded via unpack-objects\n> +\t\tgit repack -Ad &&\n> +\n> +\t\t# and now restore our original tip, as if the pushes\n> +\t\t# had happened\n> +\t\tgit update-ref HEAD $orig_tip\n> +\t'\n> +\n> +\ttest_partial_bitmap\n> +done\n\nCould we extract the body of this loop into a function whose first\nargument is either true/false? I think that would improve readability\nhere, and potentially clean up the diff a little bit.\n\nFor what it's worth, I don't think we need to do anything fancier for\nthe test name other than:\n\n\n test_partial_bitmap () {\n local enabled=\"$1\"\n test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n git config pack.writeBitmapLookupTable \"$enabled\" &&\n [...]\n '\n }\n\n test_partial_bitmap false\n test_partial_bitmap true\n\nor something.\n\n> +for i in false true\n> +do\n> +\t$i && lookup=\" (lookup table)\"\n> +\ttest_expect_success \"create partial bitmap state$lookup\" '\n> +\t\tgit config pack.writeBitmapLookupTable '\"$i\"' &&\n> +\t\t# pick a commit to represent the repo tip in the past\n> +\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n> +\t\torig_tip=$(git rev-parse HEAD) &&\n> +\n> +\t\t# now pretend we have just one tip\n> +\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n> +\t\tgit update-ref HEAD $cutoff &&\n> +\n> +\t\t# and then repack, which will leave us with a nice\n> +\t\t# big bitmap pack of the \"old\" history, and all of\n> +\t\t# the new history will be loose, as if it had been pushed\n> +\t\t# up incrementally and exploded via unpack-objects\n> +\t\tgit repack -Ad &&\n> +\t\tgit multi-pack-index write --bitmap &&\n> +\n> +\t\t# and now restore our original tip, as if the pushes\n> +\t\t# had happened\n> +\t\tgit update-ref HEAD $orig_tip\n> +\t'\n> +\n> +\ttest_partial_bitmap\n> +done\n\nSame note here.\n\nThanks,\nTaylor\n"},{"id":"457731","messageId":"20220622171814.46313-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrNIJ6z+a4++MGQ8@nand.local","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-22T17:18:14Z","receivedAt":"2022-06-22T17:18:43Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> In other words, right now we have to do two queries when an commit\n> doesn't have a bitmap stored:\n>\n> - first, a lookup to see whether we have already loaded a bitmap for\n> that commit\n>\n> - then, a subsequent lookup to see whether the .bitmap file itself has\n> a bitmap for that commit, but we just haven't loaded it yet\n>\n> If we knew that we had loaded all of the bitmaps in the file, then we\n> could simplify the above two queries into one, since whatever the first\n> one returns is enough to know whether or not a bitmap exists at all.\n\nHmm, agreed.\n\n> Ahhh. Thanks for refreshing my memory. I wonder if you think there is a\n> convenient way to work some of this into a short comment to help other\n> readers in the future, too.\n\nActually, Derrick has suggested to go with iterative approach[1] instead of\nRecursive approach. What's your view on it?\n\n> Right, that part makes sense to me. But I wonder if we should still\n> print something, perhaps just \"Bitmap v1 test\" or \"Bitmap v1 test (%d\n> entries)\" omitting the \"loaded\" part.\n\nYeah, of course we can!\n\nThanks :)\n\n[1] https://lore.kernel.org/git/92dc6860-ff35-0989-5114-fe1e220ca10c@github.com/\n"},{"id":"457749","messageId":"YrOK5wf/tvGSFbSH@nand.local","threadId":"58038","inReplyTo":"20220622171814.46313-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH 2/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-22T21:34:31Z","receivedAt":"2022-06-22T21:34:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 22, 2022 at 10:48:14PM +0530, Abhradeep Chakraborty wrote:\n> > Ahhh. Thanks for refreshing my memory. I wonder if you think there is a\n> > convenient way to work some of this into a short comment to help other\n> > readers in the future, too.\n>\n> Actually, Derrick has suggested to go with iterative approach[1] instead of\n> Recursive approach. What's your view on it?\n\nI don't have a strong feeling about it. In practice, we seem to top out\nat ~500 bitmaps or so for large-ish repositories, so I would be\nsurprised to see this result in stack exhaustion even in the worst case\n(every bitmap xor'd with the previous one, forming a long chain).\n\nBut it doesn't hurt to be defensive, so I think it's worth it as long as\nyou don't find the implementation too complex.\n\n> [1] https://lore.kernel.org/git/92dc6860-ff35-0989-5114-fe1e220ca10c@github.com/\n\nThanks,\nTaylor\n"},{"id":"457877","messageId":"4d11be66cfa2cd667df56ab9260903a37900bd4c.1656249017.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:12Z","receivedAt":"2022-06-26T13:10:25Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nWhen reading bitmap file, git loads each and every bitmap one by one\neven if all the bitmaps are not required. A \"bitmap lookup table\"\nextension to the bitmap format can reduce the overhead of loading\nbitmaps which stores a list of bitmapped commit id pos (in the midx\nor pack, along with their offset and xor offset. This way git can\nload only the neccesary bitmaps without loading the previous bitmaps.\n\nThe older version of Git ignores the lookup table extension and doesn't\nthrow any kind of warning or error while parsing the bitmap file.\n\nAdd some information for the new \"bitmap lookup table\" extension in the\nbitmap-format documentation.\n\nCo-Authored-by: Taylor Blau \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/technical/bitmap-format.txt | 41 +++++++++++++++++++++++\n 1 file changed, 41 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 04b3ec21785..7d4e450d3d8 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -67,6 +67,19 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n \t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n+\t\t\t** {empty}\n+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n+\t\t\tIf present, the end of the bitmap file contains a table\n+\t\t\tcontaining a list of `N` \n+\t\t\ttriplets. The format and meaning of the table is described\n+\t\t\tbelow.\n++\n+NOTE: This xor_offset is different from the bitmap's xor_offset.\n+Bitmap's xor_offset is relative i.e. it tells how many bitmaps we have\n+to go back from the current bitmap. Lookup table's xor_offset tells the\n+position of the triplet in the list whose bitmap the current commit's\n+bitmap have to xor with.\n+\n \t\t4-byte entry count (network byte order)\n \n \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n@@ -205,3 +218,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n hashes made under two different schemes would be pointless).\n+\n+Commit lookup table\n+-------------------\n+\n+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n+(preceding the name-hash cache and trailing hash) of the `.bitmap` file\n+contains a lookup table specifying the information needed to get the\n+desired bitmap from the entries without parsing previous unnecessary\n+bitmaps.\n+\n+For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n+contains a list of `nr_entries` triplets.\n+The content of i'th triplet is -\n+\n+\t* {empty}\n+\tcommit pos (4 byte integer, network byte order): ::\n+\tIt stores the object position of the commit (in the midx or pack index)\n+\tto which the i'th bitmap in the bitmap entries belongs.\n+\n+\t* {empty}\n+\toffset (8 byte integer, network byte order): ::\n+\tThe offset from which that commit's bitmap can be read.\n+\n+\t* {empty}\n+\txor offset (4 byte integer, network byte order): ::\n+\tIt holds the position of the triplet with whose bitmap the\n+\tcurrent bitmap need to xor. If the current triplet's bitmap\n+\tdo not have any xor bitmap, it defaults to 0xffffffff.\n-- \ngitgitgadget\n\n"},{"id":"457878","messageId":"d118f1d45e6202925d4efd5435acdd08545bf132.1656249017.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:13Z","receivedAt":"2022-06-26T13:10:28Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe bitmap lookup table extension was documentated by an earlier\nchange, but Git does not yet knowhow to write that extension.\n\nTeach git to write bitmap lookup table extension. The table contains\nthe list of `N` ` triplets. These\ntriplets are sorted according to their commit pos (ascending order).\nThe meaning of each data in the i'th triplet is given below:\n\n - Commit pos is the position of the commit in the pack-index\n (or midx) to which the i'th bitmap belongs. It is a 4 byte\n network byte order integer.\n\n - offset is the position of the i'th bitmap.\n\n - xor offset denotes the position of the triplet with whose\n bitmap the current triplet's bitmap need to xor with.\n\nCo-authored-by: Taylor Blau \nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 5 ++--\n 2 files changed, 73 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c43375bd344..899a4a941e1 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -650,7 +650,9 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t uint64_t *offsets,\n+\t\t\t\t uint32_t *commit_positions)\n {\n \tint i;\n \n@@ -663,6 +665,11 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t\tif (commit_pos < 0)\n \t\t\tBUG(\"trying to write commit not in index\");\n \n+\t\tif (offsets)\n+\t\t\toffsets[i] = hashfile_total(f);\n+\t\tif (commit_positions)\n+\t\t\tcommit_positions[i] = commit_pos;\n+\n \t\thashwrite_be32(f, commit_pos);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n@@ -671,6 +678,55 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static int table_cmp(const void *_va, const void *_vb, void *commit_positions)\n+{\n+\tint8_t result = 0;\n+\tuint32_t *positions = (uint32_t *) commit_positions;\n+\tuint32_t a = positions[*(uint32_t *)_va];\n+\tuint32_t b = positions[*(uint32_t *)_vb];\n+\n+\tif (a > b)\n+\t\tresult = 1;\n+\telse if (a < b)\n+\t\tresult = -1;\n+\telse\n+\t\tresult = 0;\n+\n+\treturn result;\n+}\n+\n+static void write_lookup_table(struct hashfile *f,\n+\t\t\t uint64_t *offsets,\n+\t\t\t uint32_t *commit_positions)\n+{\n+\tuint32_t i;\n+\tuint32_t *table, *table_inv;\n+\n+\tALLOC_ARRAY(table, writer.selected_nr);\n+\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable[i] = i;\n+\n+\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable_inv[table[i]] = i;\n+\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\t\tuint32_t xor_offset = selected->xor_offset;\n+\n+\t\thashwrite_be32(f, commit_positions[table[i]]);\n+\t\thashwrite_be64(f, offsets[table[i]]);\n+\t\thashwrite_be32(f, xor_offset ?\n+\t\t\t\ttable_inv[table[i] - xor_offset]: 0xffffffff);\n+\t}\n+\n+\tfree(table);\n+\tfree(table_inv);\n+}\n+\n static void write_hash_cache(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n@@ -695,6 +751,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n+\tuint64_t *offsets = NULL;\n+\tuint32_t *commit_positions = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \n@@ -715,8 +773,16 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n-\twrite_selected_commits_v1(f, index, index_nr);\n \n+\tif (options & BITMAP_OPT_LOOKUP_TABLE) {\n+\t\tCALLOC_ARRAY(offsets, index_nr);\n+\t\tCALLOC_ARRAY(commit_positions, index_nr);\n+\t}\n+\n+\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\twrite_lookup_table(f, offsets, commit_positions);\n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n \n@@ -730,4 +796,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n+\tfree(offsets);\n+\tfree(commit_positions);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3d3ddd77345..67a9d0fc303 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -24,8 +24,9 @@ struct bitmap_disk_header {\n #define NEEDS_BITMAP (1u<<22)\n \n enum pack_bitmap_opts {\n-\tBITMAP_OPT_FULL_DAG = 1,\n-\tBITMAP_OPT_HASH_CACHE = 4,\n+\tBITMAP_OPT_FULL_DAG = 0x1,\n+\tBITMAP_OPT_HASH_CACHE = 0x4,\n+\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n };\n \n enum pack_bitmap_flags {\n-- \ngitgitgadget\n\n"},{"id":"457879","messageId":"7786dc879f006c8316c33dd70e98888ceb50a014.1656249017.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:14Z","receivedAt":"2022-06-26T13:10:29Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nTeach git to provide a way for users to enable/disable bitmap lookup\ntable extension by providing a config option named 'writeBitmapLookupTable'.\nDefault is true.\n\nAlso add test to verify writting of lookup table.\n\nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \n---\n Documentation/config/pack.txt | 7 +++++++\n builtin/multi-pack-index.c | 8 ++++++++\n builtin/pack-objects.c | 10 +++++++++-\n midx.c | 3 +++\n midx.h | 1 +\n pack-bitmap-write.c | 2 ++\n t/t5310-pack-bitmaps.sh | 3 ++-\n t/t5326-multi-pack-bitmaps.sh | 13 +++++++++++++\n 8 files changed, 45 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex ad7f73a1ead..6e1f454c4d6 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n computed; instead, any namehashes stored in an existing bitmap are\n permuted into their appropriate location when writing a new bitmap.\n \n+pack.writeBitmapLookupTable::\n+\tWhen true, git will include a \"lookup table\" section in the\n+\tbitmap index (if one is written). This table is used to defer\n+\tloading individual bitmaps as late as possible. This can be\n+\tbeneficial in repositories which have relatively large bitmap\n+\tindexes. Defaults to true.\n+\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5edbb7fe86e..3757616f09c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n \t}\n \n+\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(var, value))\n+\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t\telse\n+\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t}\n+\n \t/*\n \t * We should never make a fall-back call to 'git_default_config', since\n \t * this was already called in 'cmd_multi_pack_index()'.\n@@ -123,6 +130,7 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t};\n \n \topts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE;\n+\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n \n \tgit_config(git_multi_pack_index_write_config, NULL);\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39e28cfcafc..d6a33fd486c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -228,7 +228,7 @@ static enum {\n \tWRITE_BITMAP_QUIET,\n \tWRITE_BITMAP_TRUE,\n } write_bitmap_index;\n-static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE;\n+static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE | BITMAP_OPT_LOOKUP_TABLE;\n \n static int exclude_promisor_objects;\n \n@@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n+\n+\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n+\t\telse\n+\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n+\t}\n+\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\ndiff --git a/midx.c b/midx.c\nindex 5f0dd386b02..9c26d04bfde 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1072,6 +1072,9 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n \n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n \tprepare_midx_packing_data(&pdata, ctx);\n \n \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\ndiff --git a/midx.h b/midx.h\nindex 22e8e53288e..5578cd7b835 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,6 +47,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n+#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 899a4a941e1..79be0cf80e6 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -713,6 +713,7 @@ static void write_lookup_table(struct hashfile *f,\n \tfor (i = 0; i < writer.selected_nr; i++)\n \t\ttable_inv[table[i]] = i;\n \n+\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n \tfor (i = 0; i < writer.selected_nr; i++) {\n \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n \t\tuint32_t xor_offset = selected->xor_offset;\n@@ -725,6 +726,7 @@ static void write_lookup_table(struct hashfile *f,\n \n \tfree(table);\n \tfree(table_inv);\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n }\n \n static void write_hash_cache(struct hashfile *f,\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f775fc1ce69..c669ed959e9 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -38,7 +38,8 @@ test_expect_success 'full repack creates bitmaps' '\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace &&\n+\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n '\n \n basic_bitmap_tests\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4fe57414c13..43be49617b8 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -307,4 +307,17 @@ test_expect_success 'graceful fallback when missing reverse index' '\n \t)\n '\n \n+test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit base &&\n+\t\tgit repack -ad &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n+\t)\n+'\n test_done\n-- \ngitgitgadget\n\n"},{"id":"457880","messageId":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.git.1655728395.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:11Z","receivedAt":"2022-06-26T13:10:34Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"When parsing the .bitmap file, git loads all the bitmaps one by one even if\nsome of the bitmaps are not necessary. We can remove this overhead by\nloading only the necessary bitmaps. A look up table extension can solve this\nissue.\n\nChanges since v1:\n\nThis is the second version which addressed all (I think) the reviews. Please\nnotify me if some reviews are not addressed :)\n\n * The table size is decreased and the format has also changed. It now\n contains nr_entries triplets of size 4+8+4 bytes. Each triplet contains\n the following things - (1) 4 byte commit position (in the pack-index or\n midx) (2) 8 byte offset and (3) 4 byte xor triplet (i.e. with whose\n bitmap the current triplet's bitmap has to xor) position.\n * Performance tests are splitted into two commits. First contains the\n actual performance tests and second enables the pack.writeReverseIndex\n (as suggested by Taylor).\n * st_*() functions are used.\n * commit order is changed according to Derrick's suggestion.\n * Iterative approach is used instead of recursive approach to parse xor\n bitmaps. (As suggested by Derrick).\n * Some minor bug fixes of previous version.\n\nInitial version:\n\nThe proposed table has:\n\n * a list of nr_entries object ids. These objects are commits that has\n bitmaps. Ids are stored in lexicographic order (for better searching).\n * a list of pairs (4-byte integers, network-byte\n order). The i'th pair denotes the offset and xor-offset(respectively) of\n the bitmap of i'th commit in the previous list. These two informations\n are necessary because only in this way bitmaps can be found without\n parsing all the bitmap.\n * a 4-byte integer for table specific flags (none exists currently).\n\nWhenever git want to parse the bitmap for a specific commit, it will first\nrefer to the table and will look for the offset and xor-offset for that\ncommit. Git will then try to parse the bitmap located at the offset\nposition. The xor-offset can be used to find the xor-bitmap for the\nbitmap(if any).\n\nAbhradeep Chakraborty (6):\n Documentation/technical: describe bitmap lookup table extension\n pack-bitmap-write.c: write lookup table extension\n pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n pack-bitmap: prepare to read lookup table extension\n bitmap-lookup-table: add performance tests for lookup table\n p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing\n\n Documentation/config/pack.txt | 7 +\n Documentation/technical/bitmap-format.txt | 41 +++++\n builtin/multi-pack-index.c | 8 +\n builtin/pack-objects.c | 10 +-\n midx.c | 3 +\n midx.h | 1 +\n pack-bitmap-write.c | 74 ++++++++-\n pack-bitmap.c | 193 ++++++++++++++++++++--\n pack-bitmap.h | 5 +-\n t/perf/p5310-pack-bitmaps.sh | 66 ++++----\n t/perf/p5326-multi-pack-bitmaps.sh | 93 ++++++-----\n t/t5310-pack-bitmaps.sh | 10 +-\n t/t5326-multi-pack-bitmaps.sh | 14 ++\n 13 files changed, 439 insertions(+), 86 deletions(-)\n\n\nbase-commit: 39c15e485575089eb77c769f6da02f98a55905e0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1266%2FAbhra303%2Fbitmap-commit-table-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1266/Abhra303/bitmap-commit-table-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1266\n\nRange-diff vs v1:\n\n 1: 2e22ca5069a ! 1: 4d11be66cfa Documentation/technical: describe bitmap lookup table extension\n @@ Commit message\n When reading bitmap file, git loads each and every bitmap one by one\n even if all the bitmaps are not required. A \"bitmap lookup table\"\n extension to the bitmap format can reduce the overhead of loading\n - bitmaps which stores a list of bitmapped commit oids, along with their\n - offset and xor offset. This way git can load only the neccesary bitmaps\n - without loading the previous bitmaps.\n + bitmaps which stores a list of bitmapped commit id pos (in the midx\n + or pack, along with their offset and xor offset. This way git can\n + load only the neccesary bitmaps without loading the previous bitmaps.\n +\n + The older version of Git ignores the lookup table extension and doesn't\n + throw any kind of warning or error while parsing the bitmap file.\n \n Add some information for the new \"bitmap lookup table\" extension in the\n bitmap-format documentation.\n \n - Co-Authored-by: Taylor Blau \n - Mentored-by: Taylor Blau \n + Co-Authored-by: Taylor Blau \n + Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n Signed-off-by: Abhradeep Chakraborty \n \n @@ Documentation/technical/bitmap-format.txt: MIDXs, both the bit-cache and rev-cac\n \t\t\tdescribed below.\n \n +\t\t\t** {empty}\n -+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0xf) : :::\n ++\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n +\t\t\tIf present, the end of the bitmap file contains a table\n -+\t\t\tcontaining a list of `N` object ids, a list of pairs of\n -+\t\t\toffset and xor offset of respective objects, and 4-byte\n -+\t\t\tinteger denoting the flags (currently none). The format\n -+\t\t\tand meaning of the table is described below.\n ++\t\t\tcontaining a list of `N` \n ++\t\t\ttriplets. The format and meaning of the table is described\n ++\t\t\tbelow.\n +++\n ++NOTE: This xor_offset is different from the bitmap's xor_offset.\n ++Bitmap's xor_offset is relative i.e. it tells how many bitmaps we have\n ++to go back from the current bitmap. Lookup table's xor_offset tells the\n ++position of the triplet in the list whose bitmap the current commit's\n ++bitmap have to xor with.\n +\n \t\t4-byte entry count (network byte order)\n \n @@ Documentation/technical/bitmap-format.txt: Note that this hashing scheme is tied\n +Commit lookup table\n +-------------------\n +\n -+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the end of the `.bitmap`\n -+contains a lookup table specifying the positions of commits which have a\n -+bitmap.\n ++If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n ++(preceding the name-hash cache and trailing hash) of the `.bitmap` file\n ++contains a lookup table specifying the information needed to get the\n ++desired bitmap from the entries without parsing previous unnecessary\n ++bitmaps.\n +\n -+For a `.bitmap` containing `nr_entries` reachability bitmaps, the format\n -+is as follows:\n ++For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n ++contains a list of `nr_entries` triplets.\n ++The content of i'th triplet is -\n +\n -+\t- `nr_entries` object names.\n ++\t* {empty}\n ++\tcommit pos (4 byte integer, network byte order): ::\n ++\tIt stores the object position of the commit (in the midx or pack index)\n ++\tto which the i'th bitmap in the bitmap entries belongs.\n +\n -+\t- `nr_entries` pairs of 4-byte integers, each in network order.\n -+\t The first holds the offset from which that commit's bitmap can\n -+\t be read. The second number holds the position of the commit\n -+\t whose bitmap the current bitmap is xor'd with in lexicographic\n -+\t order, or 0xffffffff if the current commit is not xor'd with\n -+\t anything.\n ++\t* {empty}\n ++\toffset (8 byte integer, network byte order): ::\n ++\tThe offset from which that commit's bitmap can be read.\n +\n -+\t- One 4-byte network byte order integer specifying\n -+\t table-specific flags. None exist currently, so this is always\n -+\t \"0\".\n ++\t* {empty}\n ++\txor offset (4 byte integer, network byte order): ::\n ++\tIt holds the position of the triplet with whose bitmap the\n ++\tcurrent bitmap need to xor. If the current triplet's bitmap\n ++\tdo not have any xor bitmap, it defaults to 0xffffffff.\n 3: ed91ebf69a8 ! 2: d118f1d45e6 pack-bitmap-write.c: write lookup table extension\n @@ Metadata\n ## Commit message ##\n pack-bitmap-write.c: write lookup table extension\n \n - Teach git to write bitmap lookup table extension. The table has the\n - following information:\n + The bitmap lookup table extension was documentated by an earlier\n + change, but Git does not yet knowhow to write that extension.\n \n - - `N` no of Object ids of each bitmapped commits\n + Teach git to write bitmap lookup table extension. The table contains\n + the list of `N` ` triplets. These\n + triplets are sorted according to their commit pos (ascending order).\n + The meaning of each data in the i'th triplet is given below:\n \n - - A list of offset, xor-offset pair; the i'th pair denotes the\n - offsets and xor-offsets of i'th commit in the previous list.\n + - Commit pos is the position of the commit in the pack-index\n + (or midx) to which the i'th bitmap belongs. It is a 4 byte\n + network byte order integer.\n \n - - 4-byte integer denoting the flags\n + - offset is the position of the i'th bitmap.\n \n - Co-authored-by: Taylor Blau \n - Mentored-by: Taylor Blau \n + - xor offset denotes the position of the triplet with whose\n + bitmap the current triplet's bitmap need to xor with.\n +\n + Co-authored-by: Taylor Blau \n + Mentored-by: Taylor Blau \n Co-mentored-by: Kaartic Sivaraam \n Signed-off-by: Abhradeep Chakraborty \n \n @@ pack-bitmap-write.c: static const struct object_id *oid_access(size_t pos, const\n \t\t\t\t struct pack_idx_entry **index,\n -\t\t\t\t uint32_t index_nr)\n +\t\t\t\t uint32_t index_nr,\n -+\t\t\t\t off_t *offsets)\n ++\t\t\t\t uint64_t *offsets,\n ++\t\t\t\t uint32_t *commit_positions)\n {\n \tint i;\n \n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n \n +\t\tif (offsets)\n +\t\t\toffsets[i] = hashfile_total(f);\n ++\t\tif (commit_positions)\n ++\t\t\tcommit_positions[i] = commit_pos;\n +\n \t\thashwrite_be32(f, commit_pos);\n \t\thashwrite_u8(f, stored->xor_offset);\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n -+static int table_cmp(const void *_va, const void *_vb)\n ++static int table_cmp(const void *_va, const void *_vb, void *commit_positions)\n +{\n -+\treturn oidcmp(&writer.selected[*(uint32_t*)_va].commit->object.oid,\n -+\t\t &writer.selected[*(uint32_t*)_vb].commit->object.oid);\n ++\tint8_t result = 0;\n ++\tuint32_t *positions = (uint32_t *) commit_positions;\n ++\tuint32_t a = positions[*(uint32_t *)_va];\n ++\tuint32_t b = positions[*(uint32_t *)_vb];\n ++\n ++\tif (a > b)\n ++\t\tresult = 1;\n ++\telse if (a < b)\n ++\t\tresult = -1;\n ++\telse\n ++\t\tresult = 0;\n ++\n ++\treturn result;\n +}\n +\n +static void write_lookup_table(struct hashfile *f,\n -+\t\t\t off_t *offsets)\n ++\t\t\t uint64_t *offsets,\n ++\t\t\t uint32_t *commit_positions)\n +{\n +\tuint32_t i;\n -+\tuint32_t flags = 0;\n +\tuint32_t *table, *table_inv;\n +\n +\tALLOC_ARRAY(table, writer.selected_nr);\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\n +\tfor (i = 0; i < writer.selected_nr; i++)\n +\t\ttable[i] = i;\n -+\tQSORT(table, writer.selected_nr, table_cmp);\n ++\n ++\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n ++\n +\tfor (i = 0; i < writer.selected_nr; i++)\n +\t\ttable_inv[table[i]] = i;\n +\n +\tfor (i = 0; i < writer.selected_nr; i++) {\n +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n -+\t\tstruct object_id *oid = &selected->commit->object.oid;\n ++\t\tuint32_t xor_offset = selected->xor_offset;\n +\n -+\t\thashwrite(f, oid->hash, the_hash_algo->rawsz);\n ++\t\thashwrite_be32(f, commit_positions[table[i]]);\n ++\t\thashwrite_be64(f, offsets[table[i]]);\n ++\t\thashwrite_be32(f, xor_offset ?\n ++\t\t\t\ttable_inv[table[i] - xor_offset]: 0xffffffff);\n +\t}\n -+\tfor (i = 0; i < writer.selected_nr; i++) {\n -+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n -+\n -+\t\thashwrite_be32(f, offsets[table[i]]);\n -+\t\thashwrite_be32(f, selected->xor_offset\n -+\t\t\t ? table_inv[table[i] - selected->xor_offset]\n -+\t\t\t : 0xffffffff);\n -+\t}\n -+\n -+\thashwrite_be32(f, flags);\n +\n +\tfree(table);\n +\tfree(table_inv);\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n -+\toff_t *offsets = NULL;\n ++\tuint64_t *offsets = NULL;\n ++\tuint32_t *commit_positions = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.tags);\n -\twrite_selected_commits_v1(f, index, index_nr);\n \n -+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n ++\tif (options & BITMAP_OPT_LOOKUP_TABLE) {\n +\t\tCALLOC_ARRAY(offsets, index_nr);\n ++\t\tCALLOC_ARRAY(commit_positions, index_nr);\n ++\t}\n +\n -+\twrite_selected_commits_v1(f, index, index_nr, offsets);\n ++\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n +\n +\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n -+\t\twrite_lookup_table(f, offsets);\n ++\t\twrite_lookup_table(f, offsets, commit_positions);\n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n \n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \tstrbuf_release(&tmp_file);\n +\tfree(offsets);\n ++\tfree(commit_positions);\n }\n +\n + ## pack-bitmap.h ##\n +@@ pack-bitmap.h: struct bitmap_disk_header {\n + #define NEEDS_BITMAP (1u<<22)\n + \n + enum pack_bitmap_opts {\n +-\tBITMAP_OPT_FULL_DAG = 1,\n +-\tBITMAP_OPT_HASH_CACHE = 4,\n ++\tBITMAP_OPT_FULL_DAG = 0x1,\n ++\tBITMAP_OPT_HASH_CACHE = 0x4,\n ++\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n + };\n + \n + enum pack_bitmap_flags {\n 4: 661c1137e1c ! 3: 7786dc879f0 builtin/pack-objects.c: learn pack.writeBitmapLookupTable\n @@\n ## Metadata ##\n -Author: Taylor Blau \n +Author: Abhradeep Chakraborty \n \n ## Commit message ##\n - builtin/pack-objects.c: learn pack.writeBitmapLookupTable\n + pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n \n Teach git to provide a way for users to enable/disable bitmap lookup\n table extension by providing a config option named 'writeBitmapLookupTable'.\n + Default is true.\n \n - Signed-off-by: Taylor Blau \n + Also add test to verify writting of lookup table.\n +\n + Co-Authored-by: Taylor Blau \n Signed-off-by: Abhradeep Chakraborty \n + Mentored-by: Taylor Blau \n + Co-Mentored-by: Kaartic Sivaraam \n \n ## Documentation/config/pack.txt ##\n @@ Documentation/config/pack.txt: When writing a multi-pack reachability bitmap, no new namehashes are\n @@ Documentation/config/pack.txt: When writing a multi-pack reachability bitmap, no\n +\tbitmap index (if one is written). This table is used to defer\n +\tloading individual bitmaps as late as possible. This can be\n +\tbeneficial in repositories which have relatively large bitmap\n -+\tindexes. Defaults to false.\n ++\tindexes. Defaults to true.\n +\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n \n + ## builtin/multi-pack-index.c ##\n +@@ builtin/multi-pack-index.c: static int git_multi_pack_index_write_config(const char *var, const char *value,\n + \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n + \t}\n + \n ++\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n ++\t\tif (git_config_bool(var, value))\n ++\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n ++\t\telse\n ++\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n ++\t}\n ++\n + \t/*\n + \t * We should never make a fall-back call to 'git_default_config', since\n + \t * this was already called in 'cmd_multi_pack_index()'.\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv)\n + \t};\n + \n + \topts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE;\n ++\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n + \n + \tgit_config(git_multi_pack_index_write_config, NULL);\n + \n +\n ## builtin/pack-objects.c ##\n +@@ builtin/pack-objects.c: static enum {\n + \tWRITE_BITMAP_QUIET,\n + \tWRITE_BITMAP_TRUE,\n + } write_bitmap_index;\n +-static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE;\n ++static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE | BITMAP_OPT_LOOKUP_TABLE;\n + \n + static int exclude_promisor_objects;\n + \n @@ builtin/pack-objects.c: static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n @@ builtin/pack-objects.c: static int git_pack_config(const char *k, const char *v,\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n +\n + ## midx.c ##\n +@@ midx.c: static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n + \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n + \t\toptions |= BITMAP_OPT_HASH_CACHE;\n + \n ++\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n ++\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n ++\n + \tprepare_midx_packing_data(&pdata, ctx);\n + \n + \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\n +\n + ## midx.h ##\n +@@ midx.h: struct multi_pack_index {\n + #define MIDX_WRITE_REV_INDEX (1 << 1)\n + #define MIDX_WRITE_BITMAP (1 << 2)\n + #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n ++#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n + \n + const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n + void get_midx_filename(struct strbuf *out, const char *object_dir);\n +\n + ## pack-bitmap-write.c ##\n +@@ pack-bitmap-write.c: static void write_lookup_table(struct hashfile *f,\n + \tfor (i = 0; i < writer.selected_nr; i++)\n + \t\ttable_inv[table[i]] = i;\n + \n ++\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n + \tfor (i = 0; i < writer.selected_nr; i++) {\n + \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n + \t\tuint32_t xor_offset = selected->xor_offset;\n +@@ pack-bitmap-write.c: static void write_lookup_table(struct hashfile *f,\n + \n + \tfree(table);\n + \tfree(table_inv);\n ++\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n + }\n + \n + static void write_hash_cache(struct hashfile *f,\n +\n + ## t/t5310-pack-bitmaps.sh ##\n +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'full repack creates bitmaps' '\n + \tls .git/objects/pack/ | grep bitmap >output &&\n + \ttest_line_count = 1 output &&\n + \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n +-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n ++\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace &&\n ++\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n + '\n + \n + basic_bitmap_tests\n +\n + ## t/t5326-multi-pack-bitmaps.sh ##\n +@@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'graceful fallback when missing reverse index' '\n + \t)\n + '\n + \n ++test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n ++\trm -fr repo &&\n ++\tgit init repo &&\n ++\ttest_when_finished \"rm -fr repo\" &&\n ++\t(\n ++\t\tcd repo &&\n ++\t\ttest_commit base &&\n ++\t\tgit repack -ad &&\n ++\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n ++\t\t\tgit multi-pack-index write --bitmap &&\n ++\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n ++\t)\n ++'\n + test_done\n 2: d139a4c48aa ! 4: 4fbfcff8a20 pack-bitmap: prepare to read lookup table extension\n @@ Metadata\n ## Commit message ##\n pack-bitmap: prepare to read lookup table extension\n \n - Bitmap lookup table extension can let git to parse only the necessary\n - bitmaps without loading the previous bitmaps one by one.\n + Earlier change teaches Git to write bitmap lookup table. But Git\n + does not know how to parse them.\n \n - Teach git to read and use the bitmap lookup table extension.\n + Teach Git to parse the existing bitmap lookup table. The older\n + versions of git are not affected by it. Those versions ignore the\n + lookup table.\n \n - Co-Authored-by: Taylor Blau \n - Mentored-by: Taylor Blau \n - Co-Mentored-by: Kaartic Sivaraam \n Signed-off-by: Abhradeep Chakraborty \n + Mentored-by: Taylor Blau \n + Co-Mentored-by: Kaartic Sivaraam \n \n ## pack-bitmap.c ##\n -@@\n - #include \"list-objects-filter-options.h\"\n - #include \"midx.h\"\n - #include \"config.h\"\n -+#include \"hash-lookup.h\"\n - \n - /*\n - * An entry on the bitmap index, representing the bitmap for a given\n @@ pack-bitmap.c: struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n +\t/*\n -+\t * If not NULL, these point into the various commit table sections\n ++\t * If not NULL, this point into the commit table extension\n +\t * (within map).\n +\t */\n +\tunsigned char *table_lookup;\n -+\tunsigned char *table_offsets;\n +\n \t/*\n \t * Extended index.\n @@ pack-bitmap.c: static int load_bitmap_header(struct bitmap_index *index)\n \t\t}\n +\n +\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n -+\t\t git_env_bool(\"GIT_READ_COMMIT_TABLE\", 1)) {\n -+\t\t\tuint32_t entry_count = ntohl(header->entry_count);\n -+\t\t\tuint32_t table_size =\n -+\t\t\t\t(entry_count * the_hash_algo->rawsz) /* oids */ +\n -+\t\t\t\t(entry_count * sizeof(uint32_t)) /* offsets */ +\n -+\t\t\t\t(entry_count * sizeof(uint32_t)) /* xor offsets */ +\n -+\t\t\t\t(sizeof(uint32_t)) /* flags */;\n ++\t\t\tgit_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n ++\t\t\tsize_t table_size = 0;\n ++\t\t\tsize_t triplet_sz = st_add3(sizeof(uint32_t), /* commit position */\n ++\t\t\t\t\t\t\tsizeof(uint64_t), /* offset */\n ++\t\t\t\t\t\t\tsizeof(uint32_t)); /* xor offset */\n +\n ++\t\t\ttable_size = st_add(table_size,\n ++\t\t\t\t\tst_mult(ntohl(header->entry_count),\n ++\t\t\t\t\t\ttriplet_sz));\n +\t\t\tif (table_size > index_end - index->map - header_size)\n -+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit commit table)\");\n -+\n ++\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit lookup table)\");\n +\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n -+\t\t\tindex->table_offsets = index->table_lookup + the_hash_algo->rawsz * entry_count;\n -+\n +\t\t\tindex_end -= table_size;\n +\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n +@@ pack-bitmap.c: static struct stored_bitmap *store_bitmap(struct bitmap_index *index,\n + \n + \thash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret);\n + \n +-\t/* a 0 return code means the insertion succeeded with no changes,\n +-\t * because the SHA1 already existed on the map. this is bad, there\n +-\t * shouldn't be duplicated commits in the index */\n ++\t/* A 0 return code means the insertion succeeded with no changes,\n ++\t * because the SHA1 already existed on the map. If lookup table\n ++\t * is NULL, this is bad, there shouldn't be duplicated commits\n ++\t * in the index.\n ++\t *\n ++\t * If table_lookup exists, that means the desired bitmap is already\n ++\t * loaded. Either this bitmap has been stored directly or another\n ++\t * bitmap has a direct or indirect xor relation with it. */\n + \tif (ret == 0) {\n +-\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n +-\t\treturn NULL;\n ++\t\tif (!index->table_lookup) {\n ++\t\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n ++\t\t\treturn NULL;\n ++\t\t}\n ++\t\treturn kh_value(index->bitmaps, hash_pos);\n + \t}\n + \n + \tkh_value(index->bitmaps, hash_pos) = stored;\n @@ pack-bitmap.c: static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n @@ pack-bitmap.c: struct include_data {\n \tstruct bitmap *seen;\n };\n \n --struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n --\t\t\t\t struct commit *commit)\n -+static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t\t struct commit *commit,\n -+\t\t\t\t\t\t uint32_t *pos_hint);\n -+\n -+static inline const unsigned char *bitmap_oid_pos(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t\t uint32_t pos)\n ++static inline const void *bitmap_get_triplet(struct bitmap_index *bitmap_git, uint32_t xor_pos)\n +{\n -+\treturn bitmap_git->table_lookup + (pos * the_hash_algo->rawsz);\n ++\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n ++\tconst void *p = bitmap_git->table_lookup + st_mult(xor_pos, triplet_sz);\n ++\treturn p;\n +}\n +\n -+static inline const void *bitmap_offset_pos(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t uint32_t pos)\n ++static uint64_t triplet_get_offset(const void *triplet)\n +{\n -+\treturn bitmap_git->table_offsets + (pos * 2 * sizeof(uint32_t));\n ++\tconst void *p = (unsigned char*) triplet + sizeof(uint32_t);\n ++\treturn get_be64(p);\n +}\n +\n -+static inline const void *xor_position_pos(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t uint32_t pos)\n ++static uint32_t triplet_get_xor_pos(const void *triplet)\n +{\n -+\treturn (unsigned char*) bitmap_offset_pos(bitmap_git, pos) + sizeof(uint32_t);\n ++\tconst void *p = (unsigned char*) triplet + st_add(sizeof(uint32_t), sizeof(uint64_t));\n ++\treturn get_be32(p);\n +}\n +\n -+static int bitmap_lookup_cmp(const void *_va, const void *_vb)\n ++static int triplet_cmp(const void *va, const void *vb)\n +{\n -+\treturn hashcmp(_va, _vb);\n ++\tint result = 0;\n ++\tuint32_t *a = (uint32_t *) va;\n ++\tuint32_t b = get_be32(vb);\n ++\tif (*a > b)\n ++\t\tresult = 1;\n ++\telse if (*a < b)\n ++\t\tresult = -1;\n ++\telse\n ++\t\tresult = 0;\n ++\n ++\treturn result;\n +}\n +\n -+static int bitmap_table_lookup(struct bitmap_index *bitmap_git,\n -+\t\t\t struct object_id *oid,\n -+\t\t\t uint32_t *commit_pos)\n ++static uint32_t bsearch_pos(struct bitmap_index *bitmap_git, struct object_id *oid,\n ++\t\t\t\t\t\tuint32_t *result)\n +{\n -+\tunsigned char *found = bsearch(oid->hash, bitmap_git->table_lookup,\n -+\t\t\t\t bitmap_git->entry_count,\n -+\t\t\t\t the_hash_algo->rawsz, bitmap_lookup_cmp);\n -+\tif (found)\n -+\t\t*commit_pos = (found - bitmap_git->table_lookup) / the_hash_algo->rawsz;\n -+\treturn !!found;\n ++\tint found;\n ++\n ++\tif (bitmap_git->midx)\n ++\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n ++\telse\n ++\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n ++\n ++\treturn found;\n +}\n +\n +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t\t struct object_id *oid,\n -+\t\t\t\t\t\t uint32_t commit_pos)\n ++\t\t\t\t\t struct commit *commit)\n +{\n -+\tuint32_t xor_pos;\n -+\toff_t bitmap_ofs;\n -+\n ++\tuint32_t commit_pos, xor_pos;\n ++\tuint64_t offset;\n +\tint flags;\n ++\tconst void *triplet = NULL;\n ++\tstruct object_id *oid = &commit->object.oid;\n +\tstruct ewah_bitmap *bitmap;\n -+\tstruct stored_bitmap *xor_bitmap;\n ++\tstruct stored_bitmap *xor_bitmap = NULL;\n ++\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n +\n -+\tbitmap_ofs = get_be32(bitmap_offset_pos(bitmap_git, commit_pos));\n -+\txor_pos = get_be32(xor_position_pos(bitmap_git, commit_pos));\n ++\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n +\n -+\t/*\n -+\t * Lazily load the xor'd bitmap if required (and we haven't done so\n -+\t * already). Make sure to pass the xor'd bitmap's position along as a\n -+\t * hint to avoid an unnecessary binary search in\n -+\t * stored_bitmap_for_commit().\n -+\t */\n -+\tif (xor_pos == 0xffffffff) {\n -+\t\txor_bitmap = NULL;\n -+\t} else {\n -+\t\tstruct commit *xor_commit;\n ++\tif (!found)\n ++\t\treturn NULL;\n ++\n ++\ttriplet = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n ++\t\t\t\t\t\ttriplet_sz, triplet_cmp);\n ++\tif (!triplet)\n ++\t\treturn NULL;\n ++\n ++\toffset = triplet_get_offset(triplet);\n ++\txor_pos = triplet_get_xor_pos(triplet);\n ++\n ++\tif (xor_pos != 0xffffffff) {\n ++\t\tint xor_flags;\n ++\t\tuint64_t offset_xor;\n ++\t\tuint32_t *xor_positions;\n +\t\tstruct object_id xor_oid;\n ++\t\tsize_t size = 0;\n +\n -+\t\toidread(&xor_oid, bitmap_oid_pos(bitmap_git, xor_pos));\n ++\t\tALLOC_ARRAY(xor_positions, bitmap_git->entry_count);\n ++\t\twhile (xor_pos != 0xffffffff) {\n ++\t\t\txor_positions[size++] = xor_pos;\n ++\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n ++\t\t\txor_pos = triplet_get_xor_pos(triplet);\n ++\t\t}\n +\n -+\t\txor_commit = lookup_commit(the_repository, &xor_oid);\n -+\t\tif (!xor_commit)\n -+\t\t\treturn NULL;\n ++\t\twhile (size){\n ++\t\t\txor_pos = xor_positions[size - 1];\n ++\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n ++\t\t\tcommit_pos = get_be32(triplet);\n ++\t\t\toffset_xor = triplet_get_offset(triplet);\n ++\n ++\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n ++\t\t\t\tfree(xor_positions);\n ++\t\t\t\treturn NULL;\n ++\t\t\t}\n ++\n ++\t\t\tbitmap_git->map_pos = offset_xor + sizeof(uint32_t) + sizeof(uint8_t);\n ++\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n ++\t\t\tbitmap = read_bitmap_1(bitmap_git);\n ++\n ++\t\t\tif (!bitmap){\n ++\t\t\t\tfree(xor_positions);\n ++\t\t\t\treturn NULL;\n ++\t\t\t}\n ++\n ++\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_oid, xor_bitmap, xor_flags);\n ++\t\t\tsize--;\n ++\t\t}\n +\n -+\t\txor_bitmap = stored_bitmap_for_commit(bitmap_git, xor_commit,\n -+\t\t\t\t\t\t &xor_pos);\n ++\t\tfree(xor_positions);\n +\t}\n +\n -+\t/*\n -+\t * Don't bother reading the commit's index position or its xor\n -+\t * offset:\n -+\t *\n -+\t * - The commit's index position is irrelevant to us, since\n -+\t * load_bitmap_entries_v1 only uses it to learn the object\n -+\t * id which is used to compute the hashmap's key. We already\n -+\t * have an object id, so no need to look it up again.\n -+\t *\n -+\t * - The xor_offset is unusable for us, since it specifies how\n -+\t * many entries previous to ours we should look at. This\n -+\t * makes sense when reading the bitmaps sequentially (as in\n -+\t * load_bitmap_entries_v1()), since we can keep track of\n -+\t * each bitmap as we read them.\n -+\t *\n -+\t * But it can't work for us, since the bitmap's don't have a\n -+\t * fixed size. So we learn the position of the xor'd bitmap\n -+\t * from the commit table (and resolve it to a bitmap in the\n -+\t * above if-statement).\n -+\t *\n -+\t * Instead, we can skip ahead and immediately read the flags and\n -+\t * ewah bitmap.\n -+\t */\n -+\tbitmap_git->map_pos = bitmap_ofs + sizeof(uint32_t) + sizeof(uint8_t);\n ++\tbitmap_git->map_pos = offset + sizeof(uint32_t) + sizeof(uint8_t);\n +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n +\tbitmap = read_bitmap_1(bitmap_git);\n ++\n +\tif (!bitmap)\n +\t\treturn NULL;\n +\n +\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n +}\n +\n -+static struct stored_bitmap *stored_bitmap_for_commit(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t\t struct commit *commit,\n -+\t\t\t\t\t\t uint32_t *pos_hint)\n + struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n + \t\t\t\t struct commit *commit)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n -\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n +-\t\treturn NULL;\n +\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n -+\t\tuint32_t commit_pos;\n ++\t\tstruct stored_bitmap *bitmap = NULL;\n +\t\tif (!bitmap_git->table_lookup)\n +\t\t\treturn NULL;\n +\n -+\t\t/* NEEDSWORK: cache misses aren't recorded. */\n -+\t\tif (pos_hint)\n -+\t\t\tcommit_pos = *pos_hint;\n -+\t\telse if (!bitmap_table_lookup(bitmap_git,\n -+\t\t\t\t\t &commit->object.oid,\n -+\t\t\t\t\t &commit_pos))\n ++\t\t/* NEEDSWORK: cache misses aren't recorded */\n ++\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n ++\t\tif(!bitmap)\n +\t\t\treturn NULL;\n -+\t\treturn lazy_bitmap_for_commit(bitmap_git, &commit->object.oid,\n -+\t\t\t\t\t commit_pos);\n ++\t\treturn lookup_stored_bitmap(bitmap);\n +\t}\n -+\treturn kh_value(bitmap_git->bitmaps, hash_pos);\n -+}\n -+\n -+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n -+\t\t\t\t struct commit *commit)\n -+{\n -+\tstruct stored_bitmap *sb = stored_bitmap_for_commit(bitmap_git, commit,\n -+\t\t\t\t\t\t\t NULL);\n -+\tif (!sb)\n - \t\treturn NULL;\n --\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n -+\treturn lookup_stored_bitmap(sb);\n + \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n - static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n -\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n --\t\tbitmap_git->version, bitmap_git->entry_count);\n ++\tfprintf(stderr, \"Bitmap v%d test (%d entries)\\n\",\n + \t\tbitmap_git->version, bitmap_git->entry_count);\n + \n +\tif (!bitmap_git->table_lookup)\n +\t\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n +\t\t\tbitmap_git->version, bitmap_git->entry_count);\n - \n ++\n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n + \n +@@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n + \n + int test_bitmap_commits(struct repository *r)\n + {\n +-\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n ++\tstruct bitmap_index *bitmap_git = NULL;\n + \tstruct object_id oid;\n + \tMAYBE_UNUSED void *value;\n + \n ++\t/* As this function is only used to print bitmap selected\n ++\t * commits, we don't have to read the commit table.\n ++\t */\n ++\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"0\", 1);\n ++\n ++\tbitmap_git = prepare_bitmap_git(r);\n + \tif (!bitmap_git)\n + \t\tdie(\"failed to load bitmap indexes\");\n + \n +@@ pack-bitmap.c: int test_bitmap_commits(struct repository *r)\n + \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n + \t});\n + \n ++\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"1\", 1);\n + \tfree_bitmap_index(bitmap_git);\n + \n + \treturn 0;\n \n - ## pack-bitmap.h ##\n -@@ pack-bitmap.h: struct bitmap_disk_header {\n - enum pack_bitmap_opts {\n - \tBITMAP_OPT_FULL_DAG = 1,\n - \tBITMAP_OPT_HASH_CACHE = 4,\n -+\tBITMAP_OPT_LOOKUP_TABLE = 16,\n - };\n + ## t/t5310-pack-bitmaps.sh ##\n +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'full repack creates bitmaps' '\n + \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n + '\n + \n ++test_expect_success 'using lookup table loads only necessary bitmaps' '\n ++\tgit rev-list --test-bitmap HEAD 2>out &&\n ++\t! grep \"Bitmap v1 test (106 entries loaded)\" out &&\n ++\tgrep \"Found bitmap for\" out\n ++'\n ++\n + basic_bitmap_tests\n \n - enum pack_bitmap_flags {\n + test_expect_success 'incremental repack fails when bitmaps are requested' '\n +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'pack reuse respects --incremental' '\n + \n + test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n + \ttest_config pack.writebitmaphashcache false &&\n ++\ttest_config pack.writebitmaplookuptable false &&\n + \tgit repack -ad &&\n + \tgit rev-list --use-bitmap-index --count --all >expect &&\n + \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n +\n + ## t/t5326-multi-pack-bitmaps.sh ##\n +@@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n + \t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n + \t)\n + '\n ++\n + test_done\n 5: a404779a30f < -: ----------- bitmap-commit-table: add tests for the bitmap lookup table\n 6: f5f725a3fe2 ! 5: 96c0041688f bitmap-lookup-table: add performance tests\n @@ Metadata\n Author: Abhradeep Chakraborty \n \n ## Commit message ##\n - bitmap-lookup-table: add performance tests\n + bitmap-lookup-table: add performance tests for lookup table\n \n - Add performance tests for bitmap lookup table extension.\n + Add performance tests to verify the performance of lookup table.\n +\n + Lookup table makes Git run faster in most of the cases. Below is the\n + result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\n + gives similar result. The repository used in the test is linux kernel.\n +\n + Test this tree\n + --------------------------------------------------------------------------\n + 5310.4: repack to disk (lookup=false) 295.94(250.45+15.24)\n + 5310.5: simulated clone 12.52(5.07+1.40)\n + 5310.6: simulated fetch 1.89(2.94+0.24)\n + 5310.7: pack to file (bitmap) 41.39(20.33+7.20)\n + 5310.8: rev-list (commits) 0.98(0.59+0.12)\n + 5310.9: rev-list (objects) 3.40(3.27+0.10)\n + 5310.10: rev-list with tag negated via --not 0.07(0.02+0.04)\n + --all (objects)\n + 5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n + 5310.12: rev-list count with blob:none 0.26(0.18+0.07)\n + 5310.13: rev-list count with blob:limit=1k 6.45(5.94+0.37)\n + 5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n + 5310.15: simulated partial clone 4.99(3.19+0.45)\n + 5310.19: repack to disk (lookup=true) 269.67(174.70+21.33)\n + 5310.20: simulated clone 11.03(5.07+1.11)\n + 5310.21: simulated fetch 0.79(0.79+0.17)\n + 5310.22: pack to file (bitmap) 43.03(20.28+7.43)\n + 5310.23: rev-list (commits) 0.86(0.54+0.09)\n + 5310.24: rev-list (objects) 3.35(3.26+0.07)\n + 5310.25: rev-list with tag negated via --not 0.05(0.00+0.03)\n + --all (objects)\n + 5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n + 5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n + 5310.28: rev-list count with blob:limit=1k 6.45(5.87+0.31)\n + 5310.29: rev-list count with tree:0 0.22(0.16+0.05)\n + 5310.30: simulated partial clone 5.17(3.12+0.48)\n +\n + Test 4-15 are tested without using lookup table. Same tests are\n + repeated in 16-30 (using lookup table).\n \n - Mentored-by: Taylor Blau \n - Co-mentored-by: Kaartic Sivaraam \n Signed-off-by: Abhradeep Chakraborty \n + Mentored-by: Taylor Blau \n + Co-Mentored-by: Kaartic Sivaraam \n \n ## t/perf/p5310-pack-bitmaps.sh ##\n -@@ t/perf/p5310-pack-bitmaps.sh: test_perf_large_repo\n - # since we want to be able to compare bitmap-aware\n - # git versus non-bitmap git\n - #\n --# We intentionally use the deprecated pack.writebitmaps\n -+# We intentionally use the deprecated pack.writeBitmaps\n - # config so that we can test against older versions of git.\n - test_expect_success 'setup bitmap config' '\n --\tgit config pack.writebitmaps true\n -+\tgit config pack.writeBitmaps true &&\n -+\tgit config pack.writeReverseIndex true\n +@@ t/perf/p5310-pack-bitmaps.sh: test_expect_success 'setup bitmap config' '\n + \tgit config pack.writebitmaps true\n '\n \n - # we need to create the tag up front such that it is covered by the repack and\n -@@ t/perf/p5310-pack-bitmaps.sh: test_perf 'repack to disk' '\n - \n - test_full_bitmap\n - \n +-# we need to create the tag up front such that it is covered by the repack and\n +-# thus by generated bitmaps.\n +-test_expect_success 'create tags' '\n +-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n +-'\n +-\n +-test_perf 'repack to disk' '\n +-\tgit repack -ad\n +-'\n +-\n +-test_full_bitmap\n +-\n -test_expect_success 'create partial bitmap state' '\n -\t# pick a commit to represent the repo tip in the past\n -\tcutoff=$(git rev-list HEAD~100 -1) &&\n @@ t/perf/p5310-pack-bitmaps.sh: test_perf 'repack to disk' '\n -\t# and now restore our original tip, as if the pushes\n -\t# had happened\n -\tgit update-ref HEAD $orig_tip\n -+test_perf 'use lookup table' '\n -+ git config pack.writeBitmapLookupTable true\n - '\n - \n +-'\n +-\n -test_partial_bitmap\n -+test_perf 'repack to disk (lookup table)' '\n -+ git repack -adb\n -+'\n ++test_bitmap () {\n ++ local enabled=\"$1\"\n +\n -+test_full_bitmap\n ++\t# we need to create the tag up front such that it is covered by the repack and\n ++\t# thus by generated bitmaps.\n ++\ttest_expect_success 'create tags' '\n ++\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n ++\t'\n +\n -+for i in false true\n -+do\n -+\t$i && lookup=\" (lookup table)\"\n -+\ttest_expect_success \"create partial bitmap state$lookup\" '\n -+\t\tgit config pack.writeBitmapLookupTable '\"$i\"' &&\n ++\ttest_expect_success \"use lookup table: $enabled\" '\n ++\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n ++\t'\n ++\n ++\ttest_perf \"repack to disk (lookup=$enabled)\" '\n ++\t\tgit repack -ad\n ++\t'\n ++\n ++\ttest_full_bitmap\n ++\n ++ test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n +\t\t# pick a commit to represent the repo tip in the past\n +\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n +\t\torig_tip=$(git rev-parse HEAD) &&\n @@ t/perf/p5310-pack-bitmaps.sh: test_perf 'repack to disk' '\n +\t\t# and now restore our original tip, as if the pushes\n +\t\t# had happened\n +\t\tgit update-ref HEAD $orig_tip\n -+\t'\n ++ '\n ++}\n +\n -+\ttest_partial_bitmap\n -+done\n ++test_bitmap false\n ++test_bitmap true\n \n test_done\n \n ## t/perf/p5326-multi-pack-bitmaps.sh ##\n -@@ t/perf/p5326-multi-pack-bitmaps.sh: test_expect_success 'drop pack bitmap' '\n +@@ t/perf/p5326-multi-pack-bitmaps.sh: test_description='Tests performance using midx bitmaps'\n \n - test_full_bitmap\n + test_perf_large_repo\n \n +-# we need to create the tag up front such that it is covered by the repack and\n +-# thus by generated bitmaps.\n +-test_expect_success 'create tags' '\n +-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n +-'\n +-\n +-test_expect_success 'start with bitmapped pack' '\n +-\tgit repack -adb\n +-'\n +-\n +-test_perf 'setup multi-pack index' '\n +-\tgit multi-pack-index write --bitmap\n +-'\n +-\n +-test_expect_success 'drop pack bitmap' '\n +-\trm -f .git/objects/pack/pack-*.bitmap\n +-'\n +-\n +-test_full_bitmap\n +-\n -test_expect_success 'create partial bitmap state' '\n -\t# pick a commit to represent the repo tip in the past\n -\tcutoff=$(git rev-list HEAD~100 -1) &&\n @@ t/perf/p5326-multi-pack-bitmaps.sh: test_expect_success 'drop pack bitmap' '\n -\t# and now restore our original tip, as if the pushes\n -\t# had happened\n -\tgit update-ref HEAD $orig_tip\n -+test_expect_success 'use lookup table' '\n -+\tgit config pack.writeBitmapLookupTable true\n - '\n - \n +-'\n +-\n -test_partial_bitmap\n -+test_perf 'setup multi-pack-index (lookup table)' '\n -+\tgit multi-pack-index write --bitmap\n -+'\n ++test_bitmap () {\n ++ local enabled=\"$1\"\n ++\n ++\t# we need to create the tag up front such that it is covered by the repack and\n ++\t# thus by generated bitmaps.\n ++\ttest_expect_success 'create tags' '\n ++\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n ++\t'\n +\n -+test_full_bitmap\n ++\ttest_expect_success \"use lookup table: $enabled\" '\n ++\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n ++\t'\n ++\n ++\ttest_expect_success \"start with bitmapped pack (lookup=$enabled)\" '\n ++\t\tgit repack -adb\n ++\t'\n ++\n ++\ttest_perf \"setup multi-pack index (lookup=$enabled)\" '\n ++\t\tgit multi-pack-index write --bitmap\n ++\t'\n +\n -+for i in false true\n -+do\n -+\t$i && lookup=\" (lookup table)\"\n -+\ttest_expect_success \"create partial bitmap state$lookup\" '\n -+\t\tgit config pack.writeBitmapLookupTable '\"$i\"' &&\n ++\ttest_expect_success \"drop pack bitmap (lookup=$enabled)\" '\n ++\t\trm -f .git/objects/pack/pack-*.bitmap\n ++\t'\n ++\n ++\ttest_full_bitmap\n ++\n ++ test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n +\t\t# pick a commit to represent the repo tip in the past\n +\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n +\t\torig_tip=$(git rev-parse HEAD) &&\n @@ t/perf/p5326-multi-pack-bitmaps.sh: test_expect_success 'drop pack bitmap' '\n +\t\t# and now restore our original tip, as if the pushes\n +\t\t# had happened\n +\t\tgit update-ref HEAD $orig_tip\n -+\t'\n ++ '\n ++}\n +\n -+\ttest_partial_bitmap\n -+done\n ++test_bitmap false\n ++test_bitmap true\n \n test_done\n -: ----------- > 6: fe556b58814 p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing\n\n-- \ngitgitgadget\n"},{"id":"457881","messageId":"96c0041688f6139c17611203f98274988ced25ab.1656249018.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:16Z","receivedAt":"2022-06-26T13:10:36Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd performance tests to verify the performance of lookup table.\n\nLookup table makes Git run faster in most of the cases. Below is the\nresult of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\ngives similar result. The repository used in the test is linux kernel.\n\nTest this tree\n--------------------------------------------------------------------------\n5310.4: repack to disk (lookup=false) 295.94(250.45+15.24)\n5310.5: simulated clone 12.52(5.07+1.40)\n5310.6: simulated fetch 1.89(2.94+0.24)\n5310.7: pack to file (bitmap) 41.39(20.33+7.20)\n5310.8: rev-list (commits) 0.98(0.59+0.12)\n5310.9: rev-list (objects) 3.40(3.27+0.10)\n5310.10: rev-list with tag negated via --not\t\t0.07(0.02+0.04)\n --all (objects)\n5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n5310.12: rev-list count with blob:none 0.26(0.18+0.07)\n5310.13: rev-list count with blob:limit=1k 6.45(5.94+0.37)\n5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n5310.15: simulated partial clone 4.99(3.19+0.45)\n5310.19: repack to disk (lookup=true) 269.67(174.70+21.33)\n5310.20: simulated clone 11.03(5.07+1.11)\n5310.21: simulated fetch 0.79(0.79+0.17)\n5310.22: pack to file (bitmap) 43.03(20.28+7.43)\n5310.23: rev-list (commits) 0.86(0.54+0.09)\n5310.24: rev-list (objects) 3.35(3.26+0.07)\n5310.25: rev-list with tag negated via --not\t\t0.05(0.00+0.03)\n\t --all (objects)\n5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n5310.28: rev-list count with blob:limit=1k 6.45(5.87+0.31)\n5310.29: rev-list count with tree:0 0.22(0.16+0.05)\n5310.30: simulated partial clone 5.17(3.12+0.48)\n\nTest 4-15 are tested without using lookup table. Same tests are\nrepeated in 16-30 (using lookup table).\n\nSigned-off-by: Abhradeep Chakraborty \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \n---\n t/perf/p5310-pack-bitmaps.sh | 77 ++++++++++++++-----------\n t/perf/p5326-multi-pack-bitmaps.sh | 93 ++++++++++++++++--------------\n 2 files changed, 94 insertions(+), 76 deletions(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 7ad4f237bc3..6ff42bdd391 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -16,39 +16,48 @@ test_expect_success 'setup bitmap config' '\n \tgit config pack.writebitmaps true\n '\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n-\n-test_perf 'repack to disk' '\n-\tgit repack -ad\n-'\n-\n-test_full_bitmap\n-\n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now kill off all of the refs and pretend we had\n-\t# just the one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n-\n-test_partial_bitmap\n+test_bitmap () {\n+ local enabled=\"$1\"\n+\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n+\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n+\n+\ttest_perf \"repack to disk (lookup=$enabled)\" '\n+\t\tgit repack -ad\n+\t'\n+\n+\ttest_full_bitmap\n+\n+ test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now kill off all of the refs and pretend we had\n+\t\t# just the one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+ '\n+}\n+\n+test_bitmap false\n+test_bitmap true\n \n test_done\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nindex f2fa228f16a..d67e7437493 100755\n--- a/t/perf/p5326-multi-pack-bitmaps.sh\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -6,47 +6,56 @@ test_description='Tests performance using midx bitmaps'\n \n test_perf_large_repo\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n-\n-test_expect_success 'start with bitmapped pack' '\n-\tgit repack -adb\n-'\n-\n-test_perf 'setup multi-pack index' '\n-\tgit multi-pack-index write --bitmap\n-'\n-\n-test_expect_success 'drop pack bitmap' '\n-\trm -f .git/objects/pack/pack-*.bitmap\n-'\n-\n-test_full_bitmap\n-\n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now pretend we have just one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n-\n-test_partial_bitmap\n+test_bitmap () {\n+ local enabled=\"$1\"\n+\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n+\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n+\n+\ttest_expect_success \"start with bitmapped pack (lookup=$enabled)\" '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_perf \"setup multi-pack index (lookup=$enabled)\" '\n+\t\tgit multi-pack-index write --bitmap\n+\t'\n+\n+\ttest_expect_success \"drop pack bitmap (lookup=$enabled)\" '\n+\t\trm -f .git/objects/pack/pack-*.bitmap\n+\t'\n+\n+\ttest_full_bitmap\n+\n+ test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now pretend we have just one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+ '\n+}\n+\n+test_bitmap false\n+test_bitmap true\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"457882","messageId":"fe556b58814405baf5f19f4dd3e89883d08edb8e.1656249018.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:17Z","receivedAt":"2022-06-26T13:10:38Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEnable pack.writeReverseIndex to true to see the effect of writing\nthe reverse index in the existing bitmap tests (with and without\nlookup table).\n\nBelow is the result of performance test. Output format is in\nseconds.\n\nTest this tree\n-------------------------------------------------------------------\n5310.4: repack to disk (lookup=false) 294.92(257.60+14.29)\n5310.5: simulated clone 14.97(8.95+1.31)\n5310.6: simulated fetch 1.64(2.77+0.20)\n5310.7: pack to file (bitmap) 41.76(29.33+6.77)\n5310.8: rev-list (commits) 0.71(0.49+0.09)\n5310.9: rev-list (objects) 4.65(4.55+0.09)\n5310.10: rev-list with tag negated via --not\t0.08(0.02+0.05)\n\t --all (objects)\n5310.11: rev-list with negative tag (objects) 0.06(0.01+0.04)\n5310.12: rev-list count with blob:none 0.09(0.03+0.05)\n5310.13: rev-list count with blob:limit=1k 7.58(7.06+0.33)\n5310.14: rev-list count with tree:0 0.09(0.03+0.06)\n5310.15: simulated partial clone 8.64(8.04+0.35)\n5310.19: repack to disk (lookup=true) 249.86(191.57+19.50)\n5310.20: simulated clone 13.67(8.83+1.06)\n5310.21: simulated fetch 0.50(0.63+0.13)\n5310.22: pack to file (bitmap) 41.24(28.99+6.67)\n5310.23: rev-list (commits) 0.67(0.50+0.07)\n5310.24: rev-list (objects) 4.88(4.79+0.08)\n5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n\t --all (objects)\n5310.26: rev-list with negative tag (objects) 0.05(0.00+0.04)\n5310.27: rev-list count with blob:none 0.05(0.01+0.03)\n5310.28: rev-list count with blob:limit=1k 8.02(7.16+0.34)\n5310.29: rev-list count with tree:0 0.05(0.01+0.04)\n5310.30: simulated partial clone 8.57(8.16+0.32)\n\nTests 4-15 are without the use of lookup table. The rests are\nrepeatation of the previous tests but using lookup table.\n\nSigned-off-by: Abhradeep Chakraborty \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \n---\n t/perf/p5310-pack-bitmaps.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 6ff42bdd391..9848c5d5040 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -13,7 +13,8 @@ test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true\n+\tgit config pack.writebitmaps true &&\n+\tgit config pack.writeReverseIndex true\n '\n \n test_bitmap () {\n-- \ngitgitgadget\n"},{"id":"457883","messageId":"4fbfcff8a208798146cd561b0185e094a116cf0e.1656249017.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-26T13:10:15Z","receivedAt":"2022-06-26T13:10:40Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEarlier change teaches Git to write bitmap lookup table. But Git\ndoes not know how to parse them.\n\nTeach Git to parse the existing bitmap lookup table. The older\nversions of git are not affected by it. Those versions ignore the\nlookup table.\n\nSigned-off-by: Abhradeep Chakraborty \nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \n---\n pack-bitmap.c | 193 ++++++++++++++++++++++++++++++++--\n t/t5310-pack-bitmaps.sh | 7 ++\n t/t5326-multi-pack-bitmaps.sh | 1 +\n 3 files changed, 191 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 36134222d7a..9e09c5824fc 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -82,6 +82,12 @@ struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n+\t/*\n+\t * If not NULL, this point into the commit table extension\n+\t * (within map).\n+\t */\n+\tunsigned char *table_lookup;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -185,6 +191,22 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex->hashes = (void *)(index_end - cache_size);\n \t\t\tindex_end -= cache_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n+\t\t\tgit_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n+\t\t\tsize_t table_size = 0;\n+\t\t\tsize_t triplet_sz = st_add3(sizeof(uint32_t), /* commit position */\n+\t\t\t\t\t\t\tsizeof(uint64_t), /* offset */\n+\t\t\t\t\t\t\tsizeof(uint32_t)); /* xor offset */\n+\n+\t\t\ttable_size = st_add(table_size,\n+\t\t\t\t\tst_mult(ntohl(header->entry_count),\n+\t\t\t\t\t\ttriplet_sz));\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit lookup table)\");\n+\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n@@ -211,12 +233,20 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,\n \n \thash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret);\n \n-\t/* a 0 return code means the insertion succeeded with no changes,\n-\t * because the SHA1 already existed on the map. this is bad, there\n-\t * shouldn't be duplicated commits in the index */\n+\t/* A 0 return code means the insertion succeeded with no changes,\n+\t * because the SHA1 already existed on the map. If lookup table\n+\t * is NULL, this is bad, there shouldn't be duplicated commits\n+\t * in the index.\n+\t *\n+\t * If table_lookup exists, that means the desired bitmap is already\n+\t * loaded. Either this bitmap has been stored directly or another\n+\t * bitmap has a direct or indirect xor relation with it. */\n \tif (ret == 0) {\n-\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n-\t\treturn NULL;\n+\t\tif (!index->table_lookup) {\n+\t\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n+\t\t\treturn NULL;\n+\t\t}\n+\t\treturn kh_value(index->bitmaps, hash_pos);\n \t}\n \n \tkh_value(index->bitmaps, hash_pos) = stored;\n@@ -470,7 +500,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n \n-\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n \treturn 0;\n@@ -557,13 +587,145 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+static inline const void *bitmap_get_triplet(struct bitmap_index *bitmap_git, uint32_t xor_pos)\n+{\n+\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n+\tconst void *p = bitmap_git->table_lookup + st_mult(xor_pos, triplet_sz);\n+\treturn p;\n+}\n+\n+static uint64_t triplet_get_offset(const void *triplet)\n+{\n+\tconst void *p = (unsigned char*) triplet + sizeof(uint32_t);\n+\treturn get_be64(p);\n+}\n+\n+static uint32_t triplet_get_xor_pos(const void *triplet)\n+{\n+\tconst void *p = (unsigned char*) triplet + st_add(sizeof(uint32_t), sizeof(uint64_t));\n+\treturn get_be32(p);\n+}\n+\n+static int triplet_cmp(const void *va, const void *vb)\n+{\n+\tint result = 0;\n+\tuint32_t *a = (uint32_t *) va;\n+\tuint32_t b = get_be32(vb);\n+\tif (*a > b)\n+\t\tresult = 1;\n+\telse if (*a < b)\n+\t\tresult = -1;\n+\telse\n+\t\tresult = 0;\n+\n+\treturn result;\n+}\n+\n+static uint32_t bsearch_pos(struct bitmap_index *bitmap_git, struct object_id *oid,\n+\t\t\t\t\t\tuint32_t *result)\n+{\n+\tint found;\n+\n+\tif (bitmap_git->midx)\n+\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n+\telse\n+\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n+\n+\treturn found;\n+}\n+\n+static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t struct commit *commit)\n+{\n+\tuint32_t commit_pos, xor_pos;\n+\tuint64_t offset;\n+\tint flags;\n+\tconst void *triplet = NULL;\n+\tstruct object_id *oid = &commit->object.oid;\n+\tstruct ewah_bitmap *bitmap;\n+\tstruct stored_bitmap *xor_bitmap = NULL;\n+\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n+\n+\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n+\n+\tif (!found)\n+\t\treturn NULL;\n+\n+\ttriplet = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n+\t\t\t\t\t\ttriplet_sz, triplet_cmp);\n+\tif (!triplet)\n+\t\treturn NULL;\n+\n+\toffset = triplet_get_offset(triplet);\n+\txor_pos = triplet_get_xor_pos(triplet);\n+\n+\tif (xor_pos != 0xffffffff) {\n+\t\tint xor_flags;\n+\t\tuint64_t offset_xor;\n+\t\tuint32_t *xor_positions;\n+\t\tstruct object_id xor_oid;\n+\t\tsize_t size = 0;\n+\n+\t\tALLOC_ARRAY(xor_positions, bitmap_git->entry_count);\n+\t\twhile (xor_pos != 0xffffffff) {\n+\t\t\txor_positions[size++] = xor_pos;\n+\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n+\t\t\txor_pos = triplet_get_xor_pos(triplet);\n+\t\t}\n+\n+\t\twhile (size){\n+\t\t\txor_pos = xor_positions[size - 1];\n+\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n+\t\t\tcommit_pos = get_be32(triplet);\n+\t\t\toffset_xor = triplet_get_offset(triplet);\n+\n+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n+\t\t\t\tfree(xor_positions);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\tbitmap_git->map_pos = offset_xor + sizeof(uint32_t) + sizeof(uint8_t);\n+\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\t\t\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\t\t\tif (!bitmap){\n+\t\t\t\tfree(xor_positions);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_oid, xor_bitmap, xor_flags);\n+\t\t\tsize--;\n+\t\t}\n+\n+\t\tfree(xor_positions);\n+\t}\n+\n+\tbitmap_git->map_pos = offset + sizeof(uint32_t) + sizeof(uint8_t);\n+\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\tif (!bitmap)\n+\t\treturn NULL;\n+\n+\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n+}\n+\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n-\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n-\t\treturn NULL;\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n+\t\tstruct stored_bitmap *bitmap = NULL;\n+\t\tif (!bitmap_git->table_lookup)\n+\t\t\treturn NULL;\n+\n+\t\t/* NEEDSWORK: cache misses aren't recorded */\n+\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n+\t\tif(!bitmap)\n+\t\t\treturn NULL;\n+\t\treturn lookup_stored_bitmap(bitmap);\n+\t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n@@ -1699,9 +1861,13 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n-\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n+\tfprintf(stderr, \"Bitmap v%d test (%d entries)\\n\",\n \t\tbitmap_git->version, bitmap_git->entry_count);\n \n+\tif (!bitmap_git->table_lookup)\n+\t\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n+\t\t\tbitmap_git->version, bitmap_git->entry_count);\n+\n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n \n@@ -1753,10 +1919,16 @@ void test_bitmap_walk(struct rev_info *revs)\n \n int test_bitmap_commits(struct repository *r)\n {\n-\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n+\tstruct bitmap_index *bitmap_git = NULL;\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n \n+\t/* As this function is only used to print bitmap selected\n+\t * commits, we don't have to read the commit table.\n+\t */\n+\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"0\", 1);\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n \tif (!bitmap_git)\n \t\tdie(\"failed to load bitmap indexes\");\n \n@@ -1764,6 +1936,7 @@ int test_bitmap_commits(struct repository *r)\n \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n \t});\n \n+\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"1\", 1);\n \tfree_bitmap_index(bitmap_git);\n \n \treturn 0;\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c669ed959e9..10d7691d973 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -42,6 +42,12 @@ test_expect_success 'full repack creates bitmaps' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n '\n \n+test_expect_success 'using lookup table loads only necessary bitmaps' '\n+\tgit rev-list --test-bitmap HEAD 2>out &&\n+\t! grep \"Bitmap v1 test (106 entries loaded)\" out &&\n+\tgrep \"Found bitmap for\" out\n+'\n+\n basic_bitmap_tests\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n@@ -255,6 +261,7 @@ test_expect_success 'pack reuse respects --incremental' '\n \n test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \ttest_config pack.writebitmaphashcache false &&\n+\ttest_config pack.writebitmaplookuptable false &&\n \tgit repack -ad &&\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 43be49617b8..7d36dbcf722 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -320,4 +320,5 @@ test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n \t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n \t)\n '\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"457906","messageId":"b2c947c6-b33b-066e-a578-65f769ef4f75@github.com","threadId":"58038","inReplyTo":"4d11be66cfa2cd667df56ab9260903a37900bd4c.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-27T14:18:51Z","receivedAt":"2022-06-27T14:18:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n> \n> When reading bitmap file, git loads each and every bitmap one by one\n> even if all the bitmaps are not required. A \"bitmap lookup table\"\n> extension to the bitmap format can reduce the overhead of loading\n> bitmaps which stores a list of bitmapped commit id pos (in the midx\n> or pack, along with their offset and xor offset. This way git can\n> load only the neccesary bitmaps without loading the previous bitmaps.\n\ns/neccesary/necessary/\n\n> +\t\t\t** {empty}\n> +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n> +\t\t\tIf present, the end of the bitmap file contains a table\n> +\t\t\tcontaining a list of `N` \n\n(Note that \"commit pos\" and \"xor offset\" here don't have underscores, but\nyour discussion below does use \"xor_offset\" with underscores.)\n\n> +\t\t\ttriplets. The format and meaning of the table is described\n> +\t\t\tbelow.\n> ++\n> +NOTE: This xor_offset is different from the bitmap's xor_offset.\n> +Bitmap's xor_offset is relative i.e. it tells how many bitmaps we have\n> +to go back from the current bitmap. Lookup table's xor_offset tells the\n> +position of the triplet in the list whose bitmap the current commit's\n> +bitmap have to xor with.\n\nI found this difficult to parse. Here is an attempt at a rewording. Please\nlet me know if I misunderstood something when reading your version:\n\n NOTE: The xor_offset stored in the BITMAP_OPT_LOOKUP_TABLE is different\n from the xor_offset used in the bitmap data table. The xor_offset in this\n table indicates the row number within this table of the commit whose\n bitmap is used for the XOR computation with the current commit's stored\n bitmap to create the proper logical reachability bitmap.\n\nThis does make me think that \"xor_offset\" should really be \"xor_row\" or\nsomething like that.\n\n> \t\t4-byte entry count (network byte order)\n> \n> \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n> @@ -205,3 +218,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n> If implementations want to choose a different hashing scheme, they are\n> free to do so, but MUST allocate a new header flag (because comparing\n> hashes made under two different schemes would be pointless).\n> +\n> +Commit lookup table\n> +-------------------\n> +\n> +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n> +(preceding the name-hash cache and trailing hash) of the `.bitmap` file\n> +contains a lookup table specifying the information needed to get the\n> +desired bitmap from the entries without parsing previous unnecessary\n> +bitmaps.\n> +\n> +For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n> +contains a list of `nr_entries` triplets.\n> +The content of i'th triplet is -\n> +\n> +\t* {empty}\n> +\tcommit pos (4 byte integer, network byte order): ::\n> +\tIt stores the object position of the commit (in the midx or pack index)\n> +\tto which the i'th bitmap in the bitmap entries belongs.\n\nOk, we are saving some space here, but relying on looking into the pack-index\nor multi-pack-index to get the actual commit OID.\n\nSince this is sorted by the order that stores the bitmaps, binary search will\nno longer work on this list (unless we enforce that on the rest of the bitmap\nfile). I am going to expect that you parse this table into a hashmap in order\nto allow fast commit lookups. I'll keep an eye out for that implementation.\n\n> +\t* {empty}\n> +\toffset (8 byte integer, network byte order): ::\n> +\tThe offset from which that commit's bitmap can be read.\n> +\n> +\t* {empty}\n> +\txor offset (4 byte integer, network byte order): ::\n> +\tIt holds the position of the triplet with whose bitmap the\n> +\tcurrent bitmap need to xor. If the current triplet's bitmap\n> +\tdo not have any xor bitmap, it defaults to 0xffffffff.\n\nThis last sentence seems backward. Perhaps:\n\n If the value is 0xffffffff, then the current bitmap has no xor bitmap.\n\nThanks,\n-Stolee\n"},{"id":"457907","messageId":"560d1802-8565-04eb-9faf-0f821861d321@github.com","threadId":"58038","inReplyTo":"d118f1d45e6202925d4efd5435acdd08545bf132.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-27T14:35:25Z","receivedAt":"2022-06-27T14:35:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n> \n> The bitmap lookup table extension was documentated by an earlier\n\ns/documentated/documented/\n\n> change, but Git does not yet knowhow to write that extension.\n\ns/knowhow/know how/\n\n> +static int table_cmp(const void *_va, const void *_vb, void *commit_positions)\n> +{\n> +\tint8_t result = 0;\n> +\tuint32_t *positions = (uint32_t *) commit_positions;\n\nnit: drop the space between the cast and commit_positions.\n\n> +\tuint32_t a = positions[*(uint32_t *)_va];\n> +\tuint32_t b = positions[*(uint32_t *)_vb];\n> +\n> +\tif (a > b)\n> +\t\tresult = 1;\n> +\telse if (a < b)\n> +\t\tresult = -1;\n> +\telse\n> +\t\tresult = 0;\n> +\n> +\treturn result;\n> +}\n\nOk, here you are sorting by commit OID (indirectly by the order in the\n[multi-]pack-index). I suppose that I misunderstood in the previous\npatch, so that could use some more specific language, maybe.\n\n> +static void write_lookup_table(struct hashfile *f,\n> +\t\t\t uint64_t *offsets,\n> +\t\t\t uint32_t *commit_positions)\n> +{\n> +\tuint32_t i;\n> +\tuint32_t *table, *table_inv;\n> +\n> +\tALLOC_ARRAY(table, writer.selected_nr);\n> +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n> +\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable[i] = i;\n> +\n> +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n\nAt the end of this sort, table[j] = i means that the ith bitmap corresponds\nto the jth bitmapped commit in lex order of OIDs.\n\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable_inv[table[i]] = i;\n\nAnd table_inv helps us discover that relationship (ith bitmap to jth commit\nby j = table_inv[i]).\n\n> +\tfor (i = 0; i < writer.selected_nr; i++) {\n> +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> +\t\tuint32_t xor_offset = selected->xor_offset;\n\nHere, xor_offset is \"number of bitmaps in relationship to the current bitmap\"\n\n> +\t\thashwrite_be32(f, commit_positions[table[i]]);\n> +\t\thashwrite_be64(f, offsets[table[i]]);\n> +\t\thashwrite_be32(f, xor_offset ?\n> +\t\t\t\ttable_inv[table[i] - xor_offset]: 0xffffffff);\n\nWhich means that if \"k = table[i] - xor_offset\" that the xor base is the kth\nbitmap. table_inv[k] gets us the position in this table of that bitmap's\ncommit.\n\n(It's also strange to me that the offset is being _subtracted_, but I guess\nthe bitmap format requires the xor base to appear first so the offset does\nnot need to be a negative number ever.)\n\nThis last line is a bit complex.\n\n\tuint32_t xor_offset = selected->xor_offset;\n\tuint32_t xor_row = 0xffffffff;\n\n\tif (xor_offset) {\n\t\tuint32_t xor_order = table[i] - xor_offset;\n\t\txor_row = table_inf[xor_order];\n\t}\n\n...then we can \"hashwrite_be32(f, xor_row);\" when necessary. I'm not sure\nthat we need the \"uint32_t xor_order\" inside the \"if (xor_offset)\" block,\nbut splitting it helps add clarity to the multi-step computation.\n\n> enum pack_bitmap_opts {\n> -\tBITMAP_OPT_FULL_DAG = 1,\n> -\tBITMAP_OPT_HASH_CACHE = 4,\n> +\tBITMAP_OPT_FULL_DAG = 0x1,\n> +\tBITMAP_OPT_HASH_CACHE = 0x4,\n> +\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n> };\n\nExcellent.\n\nThanks,\n-Stolee\n"},{"id":"457908","messageId":"05edd01a-8b6f-b25d-0cd1-b1a46ca7c219@github.com","threadId":"58038","inReplyTo":"7786dc879f006c8316c33dd70e98888ceb50a014.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-27T14:43:24Z","receivedAt":"2022-06-27T14:43:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n> \n> Teach git to provide a way for users to enable/disable bitmap lookup\n> table extension by providing a config option named 'writeBitmapLookupTable'.\n> Default is true.\n\nI wonder if it makes sense to have it default to 'false' for now, but to\nchange that default after the feature has been shipped and running in\nproduction for a while.\n\n> Also add test to verify writting of lookup table.\n\ns/writting/writing/\n\n> +pack.writeBitmapLookupTable::\n> +\tWhen true, git will include a \"lookup table\" section in the\n\nI think you should either use \"Git\" when talking about the software\ngenerally, OR use \"`git repack --write-bitmap-index` will include...\"\n\n> +\tbitmap index (if one is written). This table is used to defer\n> +\tloading individual bitmaps as late as possible. This can be\n> +\tbeneficial in repositories which have relatively large bitmap\n\ns/which/that/\n\n(I'm pretty sure that \"that\" is better. We're trying to restrict the set\nof repositories we are talking about, not implying that all repositories\nhave this property.)\n\n> +\tindexes. Defaults to true.\n> +\n\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -713,6 +713,7 @@ static void write_lookup_table(struct hashfile *f,\n> \tfor (i = 0; i < writer.selected_nr; i++)\n> \t\ttable_inv[table[i]] = i;\n> \n> +\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n> \tfor (i = 0; i < writer.selected_nr; i++) {\n> \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> \t\tuint32_t xor_offset = selected->xor_offset;\n> @@ -725,6 +726,7 @@ static void write_lookup_table(struct hashfile *f,\n> \n> \tfree(table);\n> \tfree(table_inv);\n> +\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n> }\n\nThese lines seem misplaced. Maybe they were meant for the previous\npatch?\n\nThanks,\n-Stolee\n"},{"id":"457909","messageId":"cebad3da-5779-5908-15d5-63d4c590c20b@github.com","threadId":"58038","inReplyTo":"4fbfcff8a208798146cd561b0185e094a116cf0e.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-27T15:12:09Z","receivedAt":"2022-06-27T15:12:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n> \n> Earlier change teaches Git to write bitmap lookup table. But Git\n> does not know how to parse them.\n> \n> Teach Git to parse the existing bitmap lookup table. The older\n> versions of git are not affected by it. Those versions ignore the\n> lookup table.\n> \n> Signed-off-by: Abhradeep Chakraborty \n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n\nI didn't check the previous patches, but your sign-off should be the\nlast line of the message. (You are singing off on all previous content,\nand any later content is not covered by your sign-off.)\n\n> +\n> +\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n> +\t\t\tgit_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n\nnit: This alignment should use four spaces at the end so the second phrase\nmatches the start of the previous phrase. Like this:\n\n\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n\t\t git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n\nPerhaps it looked right in your editor because it renders tabs as 4 spaces\ninstead of 8 spaces.\n\n> +\t\t\tsize_t table_size = 0;\n> +\t\t\tsize_t triplet_sz = st_add3(sizeof(uint32_t), /* commit position */\n> +\t\t\t\t\t\t\tsizeof(uint64_t), /* offset */\n> +\t\t\t\t\t\t\tsizeof(uint32_t)); /* xor offset */\n\nThe 4- vs 8-space tab view would also explain the alignment here:\n\n\t\t\tsize_t triplet_sz = st_add3(sizeof(uint32_t), /* commit position */\n\t\t\t\t\t\t sizeof(uint64_t), /* offset */\n\t\t\t\t\t\t sizeof(uint32_t)); /* xor offset */\n\n(I also modified the comment alignment.)\n\nOf course, since these values are constants and have no risk of overflowing,\nperhaps we can drop st_add3() here:\n\n\n\t\t\tsize_t triplet_sz = sizeof(uint32_t) + /* commit position */\n\t\t\t\t\t sizeof(uint64_t) + /* offset */\n\t\t\t\t\t sizeof(uint32_t); /* xor offset */\n\n> +\t\t\ttable_size = st_add(table_size,\n> +\t\t\t\t\tst_mult(ntohl(header->entry_count),\n> +\t\t\t\t\t\ttriplet_sz));\n\nHere, we _do_ want to keep the st_mult(). Is the st_add() still necessary? It\nseems this is a leftover from the previous version that had the 4-byte flag\ndata.\n\nWe set table_size to zero above. We could drop that initialization and instead\nhave this after the \"size_t triplet_sz\" definition:\n\n\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n\t\t\t\t\t\t triplet_sz));\n\n> +\t\t\tif (table_size > index_end - index->map - header_size)\n> +\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit lookup table)\");\n\nPlease add \"_(...)\" around the error message so it can be translated.\n\n> +\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n> +\t\t\tindex_end -= table_size;\n> +\t\t}\n\n> -\t/* a 0 return code means the insertion succeeded with no changes,\n> -\t * because the SHA1 already existed on the map. this is bad, there\n> -\t * shouldn't be duplicated commits in the index */\n> +\t/* A 0 return code means the insertion succeeded with no changes,\n> +\t * because the SHA1 already existed on the map. If lookup table\n> +\t * is NULL, this is bad, there shouldn't be duplicated commits\n> +\t * in the index.\n> +\t *\n> +\t * If table_lookup exists, that means the desired bitmap is already\n> +\t * loaded. Either this bitmap has been stored directly or another\n> +\t * bitmap has a direct or indirect xor relation with it. */\n\nIf we are modifying this multi-line comment, then we should reformat it to\nmatch convention:\n\n\t/*\n\t * The first sentence starts after the comment start\n\t * so it has symmetry with the comment end which is on\n\t * its own line.\n\t */\n\n> \tif (ret == 0) {\n> -\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n> -\t\treturn NULL;\n> +\t\tif (!index->table_lookup) {\n> +\t\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n\nErrors start with lowercase letters. Please add translation markers \"_(...)\"\n\n> +static uint32_t triplet_get_xor_pos(const void *triplet)\n> +{\n> +\tconst void *p = (unsigned char*) triplet + st_add(sizeof(uint32_t), sizeof(uint64_t));\n\nThis st_add() is not necessary since the constants will not overflow.\n\n> +\treturn get_be32(p);\n> +}\n> +\n> +static int triplet_cmp(const void *va, const void *vb)\n> +{\n> +\tint result = 0;\n> +\tuint32_t *a = (uint32_t *) va;\n> +\tuint32_t b = get_be32(vb);\n> +\tif (*a > b)\n> +\t\tresult = 1;\n> +\telse if (*a < b)\n> +\t\tresult = -1;\n> +\telse\n> +\t\tresult = 0;\n> +\n> +\treturn result;\n> +}\n> +\n> +static uint32_t bsearch_pos(struct bitmap_index *bitmap_git, struct object_id *oid,\n> +\t\t\t\t\t\tuint32_t *result)\n\nStrange wrapping. Perhaps\n\nstatic uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n\t\t\t struct object_id *oid,\n\t\t\t uint32_t *result)\n\n> +{\n> +\tint found;\n> +\n> +\tif (bitmap_git->midx)\n> +\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n> +\telse\n> +\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n> +\n> +\treturn found;\n\nHere, we are doing a binary search on the entire list of packed objects, which could\nuse quite a few more hops than a binary search on the bitmapped commits.\n\n> +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t struct commit *commit)\n...\n> +\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n> +\n> +\tif (!found)\n> +\t\treturn NULL;\n> +\n> +\ttriplet = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n> +\t\t\t\t\t\ttriplet_sz, triplet_cmp);\n\nBut I see, you are searching the pack-index for the position in the index, and _then_\nsearching the bitmap lookup table based on that position value.\n\nI expected something different: binary search on the triplets where the comparison is\nmade by looking up the OID from the [multi-]pack-index and comparing that OID to the\ncommit OID we are looking for.\n\nI'm not convinced that the binary search I had in mind is meaningfully faster than\nwhat you've implemented here, so I'm happy to leave it as you have it. We can investigate\nif that full search on the pack-index matters at all (it probably doesn't).\n\n> +\tif (!triplet)\n> +\t\treturn NULL;\n> +\n> +\toffset = triplet_get_offset(triplet);\n> +\txor_pos = triplet_get_xor_pos(triplet);\n> +\n> +\tif (xor_pos != 0xffffffff) {\n> +\t\tint xor_flags;\n> +\t\tuint64_t offset_xor;\n> +\t\tuint32_t *xor_positions;\n> +\t\tstruct object_id xor_oid;\n> +\t\tsize_t size = 0;\n> +\n> +\t\tALLOC_ARRAY(xor_positions, bitmap_git->entry_count);\n\nWhile there is potential that this is wasteful, it's probably not that huge,\nso we can start with the \"maximum XOR depth\" and then reconsider a smaller\nallocation in the future.\n\n> +\t\twhile (xor_pos != 0xffffffff) {\n\nWe should consider ensuring that also \"size < bitmap_git->entry_count\".\nBetter yet, create an xor_positions_alloc variable that is initialized\nto the entry_count value.\n\n\"size\" should probably be xor_positions_nr.\n\n> +\t\t\txor_positions[size++] = xor_pos;\n> +\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n> +\t\t\txor_pos = triplet_get_xor_pos(triplet);\n> +\t\t}\n\n(at this point, \"if (xor_positions_nr >= xor_positions_alloc)\", then error\nout since the file must be malformed with an XOR loop.)\n\n> +\t\twhile (size){\n\nnit: \") {\"\n\n> +\t\t\txor_pos = xor_positions[size - 1];\n> +\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n> +\t\t\tcommit_pos = get_be32(triplet);\n> +\t\t\toffset_xor = triplet_get_offset(triplet);\n> +\n> +\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n> +\t\t\t\tfree(xor_positions);\n> +\t\t\t\treturn NULL;\n> +\t\t\t}\n> +\n> +\t\t\tbitmap_git->map_pos = offset_xor + sizeof(uint32_t) + sizeof(uint8_t);\n> +\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\t\t\tbitmap = read_bitmap_1(bitmap_git);\n> +\n> +\t\t\tif (!bitmap){\n\nnit: \") {\"\n\n> +\t\t\t\tfree(xor_positions);\n> +\t\t\t\treturn NULL;\n> +\t\t\t}\n> +\n> +\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_oid, xor_bitmap, xor_flags);\n\nSince we are storing the bitmap here as we \"pop\" the stack, should we be\nlooking for a stored bitmap while pushing to the stack in the previous loop?\nThat would save time when using multiple bitmaps with common XOR bases.\n\n(Of course, we want to be careful that we do not create a recursive loop,\nbut instead _only_ look at the in-memory bitmaps that already exist.)\n\n> +\t\t\tsize--;\n> +\t\t}\n> +\n> +\t\tfree(xor_positions);\n> +\t}\n> +\n> +\tbitmap_git->map_pos = offset + sizeof(uint32_t) + sizeof(uint8_t);\n> +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\tbitmap = read_bitmap_1(bitmap_git);\n> +\n> +\tif (!bitmap)\n> +\t\treturn NULL;\n> +\n> +\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n> +}\n> +\n\nI'm happy with the structure of this iterative algorithm!\n\nI'll pause my review here for now.\n\nThanks,\n-Stolee\n"},{"id":"457912","messageId":"YrnRSv64nppGldF2@nand.local","threadId":"58038","inReplyTo":"b2c947c6-b33b-066e-a578-65f769ef4f75@github.com","subject":"Re: [PATCH v2 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T15:48:26Z","receivedAt":"2022-06-27T15:48:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 27, 2022 at 10:18:51AM -0400, Derrick Stolee wrote:\n> On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> > +\t\t\ttriplets. The format and meaning of the table is described\n> > +\t\t\tbelow.\n> > ++\n> > +NOTE: This xor_offset is different from the bitmap's xor_offset.\n> > +Bitmap's xor_offset is relative i.e. it tells how many bitmaps we have\n> > +to go back from the current bitmap. Lookup table's xor_offset tells the\n> > +position of the triplet in the list whose bitmap the current commit's\n> > +bitmap have to xor with.\n>\n> I found this difficult to parse. Here is an attempt at a rewording. Please\n> let me know if I misunderstood something when reading your version:\n>\n> NOTE: The xor_offset stored in the BITMAP_OPT_LOOKUP_TABLE is different\n> from the xor_offset used in the bitmap data table. The xor_offset in this\n> table indicates the row number within this table of the commit whose\n> bitmap is used for the XOR computation with the current commit's stored\n> bitmap to create the proper logical reachability bitmap.\n>\n> This does make me think that \"xor_offset\" should really be \"xor_row\" or\n> something like that.\n\nTo be fair, I found Stolee's version equally difficult to parse. I\nwonder if something like the following would be clearer:\n\n NOTE: Unlike the xor_offset used to compress an individual bitmap,\n this value stores an *absolute* index into the lookup table, not a\n location relative to the current entry.\n\n> > +For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n> > +contains a list of `nr_entries` triplets.\n> > +The content of i'th triplet is -\n> > +\n> > +\t* {empty}\n> > +\tcommit pos (4 byte integer, network byte order): ::\n> > +\tIt stores the object position of the commit (in the midx or pack index)\n> > +\tto which the i'th bitmap in the bitmap entries belongs.\n>\n> Ok, we are saving some space here, but relying on looking into the pack-index\n> or multi-pack-index to get the actual commit OID.\n>\n> Since this is sorted by the order that stores the bitmaps, binary search will\n> no longer work on this list (unless we enforce that on the rest of the bitmap\n> file). I am going to expect that you parse this table into a hashmap in order\n> to allow fast commit lookups. I'll keep an eye out for that implementation.\n\nThe main purpose of this series is to avoid having to construct such a\ntable ahead of time. This is more or less akin to what the existing\nimplementation already does in load_bitmap_entries_v1(), though that\nfunction has to read (but not decompress!) all bitmaps.\n\nBut I disagree that this isn't binary searchable. The object positions\nare in MIDX or pack .idx order, so they are sorted lexicographically.\nThe comparator implementation could either take as its key an object_id,\nand then convert each of the \"commit pos\" fields themselves to\nobject_ids and call oidcmp().\n\nOr we could go the other way (as it looks like Abhradeep did in a later\npatch) and convert the key's object_id into the index or MIDX-relative\nposition, and search for that.\n\n> > +\t* {empty}\n> > +\toffset (8 byte integer, network byte order): ::\n> > +\tThe offset from which that commit's bitmap can be read.\n> > +\n> > +\t* {empty}\n> > +\txor offset (4 byte integer, network byte order): ::\n> > +\tIt holds the position of the triplet with whose bitmap the\n> > +\tcurrent bitmap need to xor. If the current triplet's bitmap\n> > +\tdo not have any xor bitmap, it defaults to 0xffffffff.\n>\n> This last sentence seems backward. Perhaps:\n>\n> If the value is 0xffffffff, then the current bitmap has no xor bitmap.\n\nPerhaps even more concisely:\n\n The position of a triplet whose bitmap is used to compress this one,\n or 0xffffffff if no such bitmap exists.\n\nThanks,\nTaylor\n"},{"id":"457913","messageId":"YrnVRLvbG4TK8pXY@nand.local","threadId":"58038","inReplyTo":"d118f1d45e6202925d4efd5435acdd08545bf132.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T16:05:24Z","receivedAt":"2022-06-27T16:05:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 26, 2022 at 01:10:13PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> The bitmap lookup table extension was documentated by an earlier\n> change, but Git does not yet knowhow to write that extension.\n>\n> Teach git to write bitmap lookup table extension. The table contains\n> the list of `N` ` triplets. These\n> triplets are sorted according to their commit pos (ascending order).\n> The meaning of each data in the i'th triplet is given below:\n>\n> - Commit pos is the position of the commit in the pack-index\n> (or midx) to which the i'th bitmap belongs. It is a 4 byte\n> network byte order integer.\n>\n> - offset is the position of the i'th bitmap.\n>\n> - xor offset denotes the position of the triplet with whose\n> bitmap the current triplet's bitmap need to xor with.\n>\n> Co-authored-by: Taylor Blau \n> Mentored-by: Taylor Blau \n> Co-mentored-by: Kaartic Sivaraam \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++++++++++++++--\n> pack-bitmap.h | 5 ++--\n> 2 files changed, 73 insertions(+), 4 deletions(-)\n>\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index c43375bd344..899a4a941e1 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -650,7 +650,9 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n>\n> static void write_selected_commits_v1(struct hashfile *f,\n> \t\t\t\t struct pack_idx_entry **index,\n> -\t\t\t\t uint32_t index_nr)\n> +\t\t\t\t uint32_t index_nr,\n> +\t\t\t\t uint64_t *offsets,\n\nWe should probably leave this as a pointer to an off_t, since that is a\nmore appropriate type for keeping track of an offset within a file (and\nindeed it is the return type of hashfile_total()).\n\nBut since it's platform-dependent, we should make sure to cast it to a\nuint64_t before writing it as part of the lookup table.\n\n> +\t\t\t\t uint32_t *commit_positions)\n> {\n> \tint i;\n>\n> @@ -663,6 +665,11 @@ static void write_selected_commits_v1(struct hashfile *f,\n> \t\tif (commit_pos < 0)\n> \t\t\tBUG(\"trying to write commit not in index\");\n>\n> +\t\tif (offsets)\n> +\t\t\toffsets[i] = hashfile_total(f);\n\nThis makes sense to store here, since we can't easily recover this\ninformation later on.\n\n> +\t\tif (commit_positions)\n> +\t\t\tcommit_positions[i] = commit_pos;\n\nThis one I'm not as sure about. It would be nice to not have\nwrite_selected_commits_v1() be responsible for writing this down, too.\nAnd I think it's easy enough to recover later on, since we're just doing\na search over \"index\" (see above the \"oid_pos\" call).\n\nI think that oid_pos() call could be hidden behind a function that takes\nan object_id pointer, an index (double pointer) of pack_idx_entry\nstructs, and a length.\n\nIts implementation would be something like:\n\n static int commit_bitmap_writer_pos(struct object_id *oid,\n struct pack_idx_entry **index,\n uint32_t index_nr)\n {\n return oid_pos(oid, index, index_nr, oid_access);\n }\n\nand then we could replace any calls like commit_positions[i] with one\nthat first takes `i` to the appropriate object_id in selected commit\norder.\n\nThat would be strictly less efficient, but not in a way that I think\nmatters, and it would definitely be cleaner to not rely on a side-effect\nof write_selected_commits_v1().\n\nSomething in the middle there would be to have write_lookup_table()\nassemble that list of commit_positions itself, something like:\n\n uint32_t *commit_positions;\n\n ALLOC_ARRAY(commit_positions, writer.selected_nr);\n\n for (i = 0; i < writer.selected_nr; i++) {\n int pos = oid_pos(&writer.selected[i].commit->object.oid,\n index, index_nr);\n if (pos < 0)\n BUG(\"trying to write commit not in index\");\n commit_positions[i] = pos;\n }\n\n ...\n\n free(commit_positions);\n\nThat at least removes a side-effect from the implementation of\nwrite_selected_commits_v1() and brings the creation of the\ncommit_positions array closer to where it's being used, while still\nmaintaining the constant-time lookups. So that may be a good\nalternative, but I'm curious of your thoughts.\n\n> +static int table_cmp(const void *_va, const void *_vb, void *commit_positions)\n\nOK, so this is sorting the table in order of the commit positions. I\nwould rename the commit_positions parameter to something like \"void\n*_data\", and then have commit_positions be the result of the cast, like\n\"uint32_t *commit_positions = _data\";\n\n> +{\n> +\tint8_t result = 0;\n\nint8_t isn't an often used type in Git's codebase, but we can get rid of\nthis variable altogether and just return immediately from each case,\ne.g.:\n\n if (a < b)\n return -1;\n else if (a > b)\n return 1;\n return 0;\n\nor similar.\n\n> +\tuint32_t *positions = (uint32_t *) commit_positions;\n\nExplicit cast isn't need here since you're going up from void*.\n\n> +static void write_lookup_table(struct hashfile *f,\n> +\t\t\t uint64_t *offsets,\n> +\t\t\t uint32_t *commit_positions)\n> +{\n> +\tuint32_t i;\n> +\tuint32_t *table, *table_inv;\n> +\n> +\tALLOC_ARRAY(table, writer.selected_nr);\n> +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n> +\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable[i] = i;\n> +\n> +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n\nI think the construction of table and table_inv could definitely benefit\nfrom a comment here indicating what they're used for and what they\ncontain (e.g., \"table maps abc to xyz\").\n\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable_inv[table[i]] = i;\n> +\n> +\tfor (i = 0; i < writer.selected_nr; i++) {\n> +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> +\t\tuint32_t xor_offset = selected->xor_offset;\n> +\n> +\t\thashwrite_be32(f, commit_positions[table[i]]);\n> +\t\thashwrite_be64(f, offsets[table[i]]);\n> +\t\thashwrite_be32(f, xor_offset ?\n> +\t\t\t\ttable_inv[table[i] - xor_offset]: 0xffffffff);\n\nNit: missing space before ':'.\n\nThanks,\nTaylor\n"},{"id":"457915","messageId":"YrnW81NV7yY68Z2v@nand.local","threadId":"58038","inReplyTo":"560d1802-8565-04eb-9faf-0f821861d321@github.com","subject":"Re: [PATCH v2 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T16:12:35Z","receivedAt":"2022-06-27T16:14:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 27, 2022 at 10:35:25AM -0400, Derrick Stolee wrote:\n> On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n>\n> > +\tuint32_t a = positions[*(uint32_t *)_va];\n> > +\tuint32_t b = positions[*(uint32_t *)_vb];\n> > +\n> > +\tif (a > b)\n> > +\t\tresult = 1;\n> > +\telse if (a < b)\n> > +\t\tresult = -1;\n> > +\telse\n> > +\t\tresult = 0;\n> > +\n> > +\treturn result;\n> > +}\n>\n> Ok, here you are sorting by commit OID (indirectly by the order in the\n> [multi-]pack-index). I suppose that I misunderstood in the previous\n> patch, so that could use some more specific language, maybe.\n\nYeah, I agree that some more specific language could be used, with the\nmain idea being there that we make it clearer that the list of tuples is\nstill sorted (and can be binary searched).\n\n> > +\tfor (i = 0; i < writer.selected_nr; i++)\n> > +\t\ttable[i] = i;\n> > +\n> > +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n>\n> At the end of this sort, table[j] = i means that the ith bitmap corresponds\n> to the jth bitmapped commit in lex order of OIDs.\n>\n> > +\tfor (i = 0; i < writer.selected_nr; i++)\n> > +\t\ttable_inv[table[i]] = i;\n>\n> And table_inv helps us discover that relationship (ith bitmap to jth commit\n> by j = table_inv[i]).\n\nThese are both great descriptions and should give an idea of what sort\nof information is worth putting into a comment.\n>\n> > +\tfor (i = 0; i < writer.selected_nr; i++) {\n> > +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> > +\t\tuint32_t xor_offset = selected->xor_offset;\n>\n> Here, xor_offset is \"number of bitmaps in relationship to the current bitmap\"\n\nIt's an offset to an earlier commit which must be used to XOR-decompress the\ncurrent one (if any).\n\n> > +\t\thashwrite_be32(f, commit_positions[table[i]]);\n> > +\t\thashwrite_be64(f, offsets[table[i]]);\n> > +\t\thashwrite_be32(f, xor_offset ?\n> > +\t\t\t\ttable_inv[table[i] - xor_offset]: 0xffffffff);\n>\n> Which means that if \"k = table[i] - xor_offset\" that the xor base is the kth\n> bitmap. table_inv[k] gets us the position in this table of that bitmap's\n> commit.\n\nYes, exactly. Abhradeep: this is also worth commenting ;-).\n\n> (It's also strange to me that the offset is being _subtracted_, but I guess\n> the bitmap format requires the xor base to appear first so the offset does\n> not need to be a negative number ever.)\n\nYou're right, this follows from the fact that the XOR bases must come\nbefore the commits who must use them to decompress themselves. From\nDocumentation/technical/bitmap-format.txt:\n\n This number is always positive, and hence entries are always xor'ed\n with **previous** bitmaps, not bitmaps that will come afterwards in\n the index.\n\n> This last line is a bit complex.\n>\n> \tuint32_t xor_offset = selected->xor_offset;\n> \tuint32_t xor_row = 0xffffffff;\n>\n> \tif (xor_offset) {\n> \t\tuint32_t xor_order = table[i] - xor_offset;\n> \t\txor_row = table_inf[xor_order];\n> \t}\n>\n> ...then we can \"hashwrite_be32(f, xor_row);\" when necessary. I'm not sure\n> that we need the \"uint32_t xor_order\" inside the \"if (xor_offset)\" block,\n> but splitting it helps add clarity to the multi-step computation.\n\nI had the same thought, though I would also say that xor_row should be\ndeclared, not initialized, and the \"else\" block of \"if (xor_offset)\"\nshould set it to 0xffffffff to make the relationship between xor_offset\nand the value written a little clearer.\n\nThanks,\nTaylor\n"},{"id":"457916","messageId":"20220627165100.16178-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"b2c947c6-b33b-066e-a578-65f769ef4f75@github.com","subject":"Re: [PATCH v2 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-27T16:51:00Z","receivedAt":"2022-06-27T16:51:56Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stolee wrote:\n\n> I found this difficult to parse. Here is an attempt at a rewording. Please\n> let me know if I misunderstood something when reading your version:\n>\n> NOTE: The xor_offset stored in the BITMAP_OPT_LOOKUP_TABLE is different\n> from the xor_offset used in the bitmap data table. The xor_offset in this\n> table indicates the row number within this table of the commit whose\n> bitmap is used for the XOR computation with the current commit's stored\n> bitmap to create the proper logical reachability bitmap.\n>\n> This does make me think that \"xor_offset\" should really be \"xor_row\" or\n> something like that.\n\nThanks. `xor_row` seems nice to me.\n\n> > +\t* {empty}\n> > +\tcommit pos (4 byte integer, network byte order): ::\n> > +\tIt stores the object position of the commit (in the midx or pack index)\n> > +\tto which the i'th bitmap in the bitmap entries belongs.\n>\n> Ok, we are saving some space here, but relying on looking into the pack-index\n> or multi-pack-index to get the actual commit OID.\n\nSeems like I didn't update this particular part. At the time of writing this\npatch, I was clear that I would store these triplets in the bitmap's order.\nBut when I started to implement the \"read\" part, I realised that these triplets\nneed to be ordered in ascending order. So I did update the \"write extension\"\npatch but somehow missed this particular part.\n\nJust to be clear, bitmaps are sorted by their commit's date (as far as I know).\nBitmaps for recent commits comes before bitmaps for older commits. So these\ntwo orders are not same. Thus hashmap would not work here.\n\nWill update this portion.\n\nThanks :)\n"},{"id":"457917","messageId":"20220627171008.16213-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"560d1802-8565-04eb-9faf-0f821861d321@github.com","subject":"Re: [PATCH v2 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-27T17:10:08Z","receivedAt":"2022-06-27T17:10:39Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stolee wrote:\n\n> Which means that if \"k = table[i] - xor_offset\" that the xor base is the kth\n> bitmap. table_inv[k] gets us the position in this table of that bitmap's\n> commit.\n>\n> (It's also strange to me that the offset is being _subtracted_, but I guess\n> the bitmap format requires the xor base to appear first so the offset does\n> not need to be a negative number ever.)\n>\n> This last line is a bit complex.\n>\n> \tuint32_t xor_offset = selected->xor_offset;\n> \tuint32_t xor_row = 0xffffffff;\n>\n>\tif (xor_offset) {\n>\t\tuint32_t xor_order = table[i] - xor_offset;\n>\t\txor_row = table_inf[xor_order];\n>\t}\n>\n> ...then we can \"hashwrite_be32(f, xor_row);\" when necessary. I'm not sure\n> that we need the \"uint32_t xor_order\" inside the \"if (xor_offset)\" block,\n> but splitting it helps add clarity to the multi-step computation.\n\nGot it. Will add comments too.\n\nThanks :)\n"},{"id":"457919","messageId":"20220627174230.16253-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"05edd01a-8b6f-b25d-0cd1-b1a46ca7c219@github.com","subject":"[PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-27T17:42:30Z","receivedAt":"2022-06-27T17:42:46Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stolee wrote:\n\n> I wonder if it makes sense to have it default to 'false' for now, but to\n> change that default after the feature has been shipped and running in\n> production for a while.\n\nI do not have any opinion. If most reviewers agree on it, I will surely\nSet it to false.\n\n> I think you should either use \"Git\" when talking about the software\n> generally, OR use \"`git repack --write-bitmap-index` will include...\"\n\nOhh, yeah! Thanks for pointing out.\n\n> s/which/that/\n>\n> (I'm pretty sure that \"that\" is better. We're trying to restrict the set\n> of repositories we are talking about, not implying that all repositories\n> have this property.)\n\nOk.\n\n> These lines seem misplaced. Maybe they were meant for the previous\n> patch?\n\nI mainly used it for testing purpose. That's why I included it in\nThis patch. But I got your point and will move it to the previous\npatch.\n\nThanks :)\n"},{"id":"457920","messageId":"YrntSpG5asIPNdZz@nand.local","threadId":"58038","inReplyTo":"7786dc879f006c8316c33dd70e98888ceb50a014.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T17:47:54Z","receivedAt":"2022-06-27T17:48:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 26, 2022 at 01:10:14PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Teach git to provide a way for users to enable/disable bitmap lookup\n> table extension by providing a config option named 'writeBitmapLookupTable'.\n> Default is true.\n>\n> Also add test to verify writting of lookup table.\n>\n> Co-Authored-by: Taylor Blau \n> Signed-off-by: Abhradeep Chakraborty \n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n\nI think that this was covered earlier in the review of this round, but\nin general your Signed-off-by (often abbreviated as \"S-o-b\") should come\nlast. The order should be chronological, so I'd probably suggest\nsomething like:\n\n Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n Co-Authored-by: Taylor Blau \n Signed-off-by: Abhradeep Chakraborty \n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 5edbb7fe86e..3757616f09c 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n> \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n> \t}\n>\n> +\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n> +\t\tif (git_config_bool(var, value))\n> +\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n> +\t\telse\n> +\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n> +\t}\n> +\n> \t/*\n> \t * We should never make a fall-back call to 'git_default_config', since\n> \t * this was already called in 'cmd_multi_pack_index()'.\n> @@ -123,6 +130,7 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n> \t};\n>\n> \topts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE;\n> +\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n\nI wonder if this should respect pack.writeBitmapLookupTable, too.\nProbably both of them should take into account their separate\nconfiguration values, but cleaning up the hashcache one can be done\nseparately outside of this series.\n\nEverything else looks good.\n\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 899a4a941e1..79be0cf80e6 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -713,6 +713,7 @@ static void write_lookup_table(struct hashfile *f,\n> \tfor (i = 0; i < writer.selected_nr; i++)\n> \t\ttable_inv[table[i]] = i;\n>\n> +\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n> \tfor (i = 0; i < writer.selected_nr; i++) {\n> \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> \t\tuint32_t xor_offset = selected->xor_offset;\n> @@ -725,6 +726,7 @@ static void write_lookup_table(struct hashfile *f,\n>\n> \tfree(table);\n> \tfree(table_inv);\n> +\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n\nThis region may make more sense to include in the previous commit,\nthough I don't have a strong feeling about it.\n\nThanks,\nTaylor\n"},{"id":"457921","messageId":"Yrntsj06L1G0joIz@nand.local","threadId":"58038","inReplyTo":"20220627174230.16253-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T17:49:38Z","receivedAt":"2022-06-27T17:49:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 27, 2022 at 11:12:30PM +0530, Abhradeep Chakraborty wrote:\n> Derrick Stolee wrote:\n>\n> > I wonder if it makes sense to have it default to 'false' for now, but to\n> > change that default after the feature has been shipped and running in\n> > production for a while.\n>\n> I do not have any opinion. If most reviewers agree on it, I will surely\n> Set it to false.\n\nI think it's definitely a safe approach. I don't have a huge concern\nabout enabling it earlier, but I don't think we're in a huge rush to add\na new feature here, either.\n\nSo I'd be fine to ship this with the default being disabled (IOW, *not*\nwriting the lookup table). That should give us a window where we can\nshake out whatever bugs there are, as is often the case when working\nwith the bitmap code ;).\n\nThanks,\nTaylor\n"},{"id":"457922","messageId":"20220627180618.16291-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"cebad3da-5779-5908-15d5-63d4c590c20b@github.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-27T18:06:18Z","receivedAt":"2022-06-27T18:06:38Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Derrick Stolee wrote:\n\n> I didn't check the previous patches, but your sign-off should be the\n> last line of the message. (You are singing off on all previous content,\n> and any later content is not covered by your sign-off.)\n\nOhhh, got it. I didn't know about it before.\n\n> nit: This alignment should use four spaces at the end so the second phrase\n> matches the start of the previous phrase. Like this:\n>\n>\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n>\t\t git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n>\n> Perhaps it looked right in your editor because it renders tabs as 4 spaces\n> instead of 8 spaces.\n\nI don't know why but my editor sometimes do some weird things for alignments.\nI generally use VS Code. But for alignment related problems, sometimes I have\nto use vi editor.\n\n> Here, we _do_ want to keep the st_mult(). Is the st_add() still necessary? It\n> seems this is a leftover from the previous version that had the 4-byte flag\n> data.\n>\n> We set table_size to zero above. We could drop that initialization and instead\n> have this after the \"size_t triplet_sz\" definition:\n>\n>\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n>\t\t\t\t\t\t triplet_sz));\n\nYes, you're right. Will update.\n\n> I expected something different: binary search on the triplets where the comparison is\n> made by looking up the OID from the [multi-]pack-index and comparing that OID to the\n> commit OID we are looking for.\n>\n> I'm not convinced that the binary search I had in mind is meaningfully faster than\n> what you've implemented here, so I'm happy to leave it as you have it. We can investigate\n> if that full search on the pack-index matters at all (it probably doesn't).\n\nGood idea! Thanks!\n\n> While there is potential that this is wasteful, it's probably not that huge,\n> so we can start with the \"maximum XOR depth\" and then reconsider a smaller\n> allocation in the future.\n\nOk.\n\n> We should consider ensuring that also \"size < bitmap_git->entry_count\".\n> Better yet, create an xor_positions_alloc variable that is initialized\n> to the entry_count value.\n>\n> \"size\" should probably be xor_positions_nr.\n> \n> > +\t\t\txor_positions[size++] = xor_pos;\n> > +\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n> > +\t\t\txor_pos = triplet_get_xor_pos(triplet);\n> > +\t\t}\n> \n> (at this point, \"if (xor_positions_nr >= xor_positions_alloc)\", then error\n> out since the file must be malformed with an XOR loop.)\n\nGot it.\n\n> Since we are storing the bitmap here as we \"pop\" the stack, should we be\n> looking for a stored bitmap while pushing to the stack in the previous loop?\n> That would save time when using multiple bitmaps with common XOR bases.\n\nYeah, I also am thinking about it. Will make a try.\n\nThanks :)\n"},{"id":"457930","messageId":"20220627182930.16347-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrnVRLvbG4TK8pXY@nand.local","subject":"Re: [PATCH v2 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-27T18:29:30Z","receivedAt":"2022-06-27T18:34:51Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> We should probably leave this as a pointer to an off_t, since that is a\n> more appropriate type for keeping track of an offset within a file (and\n> indeed it is the return type of hashfile_total()).\n> \n> But since it's platform-dependent, we should make sure to cast it to a\n> uint64_t before writing it as part of the lookup table.\n\nHmm, will make the necessary changes.\n\n> That at least removes a side-effect from the implementation of\n> write_selected_commits_v1() and brings the creation of the\n> commit_positions array closer to where it's being used, while still\n> maintaining the constant-time lookups. So that may be a good\n> alternative, but I'm curious of your thoughts.\n\nSounds good to me :)\n\n> I think the construction of table and table_inv could definitely benefit\n> from a comment here indicating what they're used for and what they\n> contain (e.g., \"table maps abc to xyz\").\n\nYeah, true. Will add comments.\n\nThanks for the other suggestions also :)\n"},{"id":"457948","messageId":"22428078-5e08-5788-1092-02b0381b6c6d@github.com","threadId":"58038","inReplyTo":"20220627180618.16291-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-27T18:32:00Z","receivedAt":"2022-06-27T18:38:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/27/2022 2:06 PM, Abhradeep Chakraborty wrote:\n> Derrick Stolee wrote:\n>> nit: This alignment should use four spaces at the end so the second phrase\n>> matches the start of the previous phrase. Like this:\n>>\n>> \t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n>> \t\t git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n>>\n>> Perhaps it looked right in your editor because it renders tabs as 4 spaces\n>> instead of 8 spaces.\n> \n> I don't know why but my editor sometimes do some weird things for alignments.\n> I generally use VS Code. But for alignment related problems, sometimes I have\n> to use vi editor.\n\nI also use VS Code, and I noticed a few spacing issues recently, especially\nin .txt files.\n\nI submitted a patch [1] to improve the contrib/vscode/init.sh script, which\nadds some helpful config settings to your Git workspace. Please take a look\nand see how it works for you.\n\nThanks,\n-Stolee\n\n[1] https://lore.kernel.org/git/pull.1271.git.1656354587496.gitgitgadget@gmail.com\n"},{"id":"457955","messageId":"20220627183924.16369-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrntSpG5asIPNdZz@nand.local","subject":"Re: [PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-27T18:39:23Z","receivedAt":"2022-06-27T18:39:57Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> Probably both of them should take into account their separate\n> configuration values, but cleaning up the hashcache one can be done\n> separately outside of this series.\n\nActually, it does respect the `pack.writebitmaplookuptable` config.\nAs pack.writebitmaplookuptable is by default true (for this patch\nSeries), this line enables it by default. If `pack.writebitmaplookuptable`\nSet to false, the proposed change in the `git_multi_pack_index_write_config`\nfunction disables this flag.\n\n> This region may make more sense to include in the previous commit,\n> though I don't have a strong feeling about it.\n\nOk. Will move it to the previous patch.\n\nThanks :)\n"},{"id":"457966","messageId":"YrojV5aYCzxXlV3c@nand.local","threadId":"58038","inReplyTo":"4fbfcff8a208798146cd561b0185e094a116cf0e.1656249017.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T21:38:31Z","receivedAt":"2022-06-27T21:38:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 26, 2022 at 01:10:15PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Earlier change teaches Git to write bitmap lookup table. But Git\n> does not know how to parse them.\n>\n> Teach Git to parse the existing bitmap lookup table. The older\n> versions of git are not affected by it. Those versions ignore the\n\ns/git/Git\n\n> lookup table.\n>\n> Signed-off-by: Abhradeep Chakraborty \n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n> ---\n> pack-bitmap.c | 193 ++++++++++++++++++++++++++++++++--\n> t/t5310-pack-bitmaps.sh | 7 ++\n> t/t5326-multi-pack-bitmaps.sh | 1 +\n> 3 files changed, 191 insertions(+), 10 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 36134222d7a..9e09c5824fc 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -82,6 +82,12 @@ struct bitmap_index {\n> \t/* The checksum of the packfile or MIDX; points into map. */\n> \tconst unsigned char *checksum;\n>\n> +\t/*\n> +\t * If not NULL, this point into the commit table extension\n> +\t * (within map).\n\nIt may be worth replacing \"within map\" to \"within the memory mapped\nregion `map`\" to make clear that this points somewhere within the mmap.\n\n> +\t */\n> +\tunsigned char *table_lookup;\n> +\n\n\n> @@ -185,6 +191,22 @@ static int load_bitmap_header(struct bitmap_index *index)\n> \t\t\tindex->hashes = (void *)(index_end - cache_size);\n> \t\t\tindex_end -= cache_size;\n> \t\t}\n> +\n> +\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n> +\t\t\tgit_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n\nI should have commented on this in an earlier round, but I wonder what\nthe behavior should be when we have BITMAP_OPT_LOOKUP_TABLE in our\nflags, but GIT_TEST_READ_COMMIT_TABLE is disabled.\n\nRight now, it doesn't matter, since there aren't any flags in bits above\nBITMAP_OPT_LOOKUP_TABLE. But in the future, if there was some\nBITMAP_OPT_FOO that was newer than BITMAP_OPT_LOOKUP_TABLE, we would\nwant to be able to read it without needing to read the lookup table.\n\nAt least, I think that should be true, though I would be interested to\nhear if anybody has a differing opinion there.\n\n> +\t\t\tsize_t table_size = 0;\n> +\t\t\tsize_t triplet_sz = st_add3(sizeof(uint32_t), /* commit position */\n> +\t\t\t\t\t\t\tsizeof(uint64_t), /* offset */\n> +\t\t\t\t\t\t\tsizeof(uint32_t)); /* xor offset */\n\nI don't think we need a st_add3() call here, since the size of these\nthree types is known to be small and thus won't overflow the available\nrange of size_t.\n\n> +\t\t\ttable_size = st_add(table_size,\n> +\t\t\t\t\tst_mult(ntohl(header->entry_count),\n> +\t\t\t\t\t\ttriplet_sz));\n\nAnd table_size here is going to start off at zero, so the outer st_add()\ncall isn't necessary, either. This should instead be:\n\n size_t table_size = st_mult(ntohl(header->entry_count),\n sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t));\n\nIt might be nice to have triplet_sz #define'd somewhere else, since\nthere are a handful of declarations in this patch that are all\nidentical. Probably something like:\n\n #define BITMAP_LOOKUP_TABLE_RECORD_WIDTH (sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uin32_t))\n\nor even:\n\n /*\n * The width in bytes of a single record in the lookup table\n * extension:\n *\n * (commit_pos, offset, xor_pos)\n *\n * whose fields are 32-, 64-, and 32-bits wide, respectively.\n */\n #define BITMAP_LOOKUP_TABLE_RECORD_WIDTH (16)\n\n> +\t\t\tif (table_size > index_end - index->map - header_size)\n> +\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit lookup table)\");\n\nif we decide to still recognize the lookup table extension without\n*reading* from it when GIT_TEST_READ_COMMIT_TABLE is unset, I think we\nshould do something like:\n\n if (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n index->table_lookup = (void *)(index_end - table_size);\n index_end -= table_size;\n\n...where the subtraction on index_end happens unconditionally.\n\n> +static inline const void *bitmap_get_triplet(struct bitmap_index *bitmap_git, uint32_t xor_pos)\n> +{\n> +\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n\nSame note about the #define constant here.\n\n> +\tconst void *p = bitmap_git->table_lookup + st_mult(xor_pos, triplet_sz);\n\nAnd this can be returned directly. Just:\n\n return bitmap_git->table_lookup + st_mult(xor_pos, BITMAP_LOOKUP_TABLE_RECORD_WIDTH);\n\nalthough I wonder: why \"xor_pos\" and not just \"pos\" here?\n\n> +static uint64_t triplet_get_offset(const void *triplet)\n> +{\n> +\tconst void *p = (unsigned char*) triplet + sizeof(uint32_t);\n> +\treturn get_be64(p);\n> +}\n> +\n> +static uint32_t triplet_get_xor_pos(const void *triplet)\n> +{\n> +\tconst void *p = (unsigned char*) triplet + st_add(sizeof(uint32_t), sizeof(uint64_t));\n> +\treturn get_be32(p);\n> +}\n\nI wonder if we could get rid of these functions altogether and return a\nsmall structure like:\n\n struct bitmap_lookup_table_record {\n uint32_t commit_pos;\n uint64_t offset;\n uint32_t xor_pos;\n };\n\nor similar.\n\n> +static int triplet_cmp(const void *va, const void *vb)\n> +{\n> +\tint result = 0;\n> +\tuint32_t *a = (uint32_t *) va;\n> +\tuint32_t b = get_be32(vb);\n\nHmm. This is a little tricky to read. Here we're expecting \"va\" to hold\ncommit_pos from below, and \"vb\" to be a pointer at a lookup record.\nEverything here is right, though I wonder if a comment or two might\nclarify why one is \"*(uint32_t *)va\" and the other is \"get_be32(vb)\".\n\n> +\tif (*a > b)\n> +\t\tresult = 1;\n> +\telse if (*a < b)\n> +\t\tresult = -1;\n> +\telse\n> +\t\tresult = 0;\n\nLet's just return the result of the comparison directly here. And while\nI'm looking at it, I think we can avoid dereferencing \"a\" on each use,\nand instead just dereference va on assignment after casting, e.g.:\n\n uint32_t a = *(uint32_t*)va;\n\n> +static uint32_t bsearch_pos(struct bitmap_index *bitmap_git, struct object_id *oid,\n> +\t\t\t\t\t\tuint32_t *result)\n> +{\n> +\tint found;\n> +\n> +\tif (bitmap_git->midx)\n\nNit: let's use the bitmap_is_midx() helper here instead of looking at\nbitamp_git->midx directly.\n\n> +\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n> +\telse\n> +\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n> +\n> +\treturn found;\n> +}\n\nMakes sense.\n\n> +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t struct commit *commit)\n> +{\n> +\tuint32_t commit_pos, xor_pos;\n> +\tuint64_t offset;\n> +\tint flags;\n> +\tconst void *triplet = NULL;\n> +\tstruct object_id *oid = &commit->object.oid;\n> +\tstruct ewah_bitmap *bitmap;\n> +\tstruct stored_bitmap *xor_bitmap = NULL;\n> +\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n> +\n> +\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n> +\n> +\tif (!found)\n> +\t\treturn NULL;\n> +\n> +\ttriplet = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n> +\t\t\t\t\t\ttriplet_sz, triplet_cmp);\n> +\tif (!triplet)\n> +\t\treturn NULL;\n\nOK. If you don't mind, I'm going to \"think aloud\" while I read through\nthis function to make sure that we're on the same page.\n\nFirst thing is to convert the commit OID we're looking for into its\nposition within the corresponding pack index or MIDX file so that we can\nuse it as a search key to locate in the lookup table. If we didn't find\nanything, or the commit doesn't exist in our pack / MIDX, nothing to do.\n\n> +\n> +\toffset = triplet_get_offset(triplet);\n> +\txor_pos = triplet_get_xor_pos(triplet);\n\nOtherwise, record its offset and XOR \"offset\".\n\n> +\n> +\tif (xor_pos != 0xffffffff) {\n> +\t\tint xor_flags;\n> +\t\tuint64_t offset_xor;\n> +\t\tuint32_t *xor_positions;\n> +\t\tstruct object_id xor_oid;\n> +\t\tsize_t size = 0;\n> +\n> +\t\tALLOC_ARRAY(xor_positions, bitmap_git->entry_count);\n\nIf we are XOR'd with another bitmap, make a stack of those bitmaps so\nthat we can decompress ourself.\n\nI'm a little surprised that we're allocating an array as large as\nbitmap_git->entry_count. It's not wrong, but it does waste some bytes\nsince we likely don't often have these long chains of XOR'd bitmaps.\n\nWe should instead allocate a smaller array and grow it over time (search\nfor examples of ALLOC_GROW() to see the canonical way to do this in\nGit's codebase).\n\n> +\t\twhile (xor_pos != 0xffffffff) {\n> +\t\t\txor_positions[size++] = xor_pos;\n> +\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n> +\t\t\txor_pos = triplet_get_xor_pos(triplet);\n> +\t\t}\n> +\n> +\t\twhile (size){\n\nNit: missing space after \")\" and before \"{\".\n\n> +\t\t\txor_pos = xor_positions[size - 1];\n> +\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n\nWe already have to get the triplets in the loop above, and then we dig\nthem back out here. Would it be easier to keep track of a list of\npointers into the mmaped region instead of looking up these triplets\neach time?\n\n> +\t\t\tcommit_pos = get_be32(triplet);\n> +\t\t\toffset_xor = triplet_get_offset(triplet);\n> +\n> +\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n\nShould it be an error if we can't look up the object's ID here? I'd\nthink so.\n\n> +\t\t\t\tfree(xor_positions);\n> +\t\t\t\treturn NULL;\n> +\t\t\t}\n> +\n> +\t\t\tbitmap_git->map_pos = offset_xor + sizeof(uint32_t) + sizeof(uint8_t);\n> +\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\t\t\tbitmap = read_bitmap_1(bitmap_git);\n> +\n> +\t\t\tif (!bitmap){\n\nNit: missing space between \")\" and \"{\".\n\n> +\t\t\t\tfree(xor_positions);\n> +\t\t\t\treturn NULL;\n> +\t\t\t}\n> +\n> +\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_oid, xor_bitmap, xor_flags);\n> +\t\t\tsize--;\n\nMakes sense. Nicely done!\n\n> +\t\t}\n> +\n> +\t\tfree(xor_positions);\n> +\t}\n> +\n> +\tbitmap_git->map_pos = offset + sizeof(uint32_t) + sizeof(uint8_t);\n> +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\tbitmap = read_bitmap_1(bitmap_git);\n\nGreat, and now we can finally read the original bitmap that we wanted\nto...\n\n> +\tif (!bitmap)\n> +\t\treturn NULL;\n> +\n> +\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n\n...and XOR it with the thing we built up in the loop. Very nicely done.\nDo we have a good way to make sure that we're testing this code in CI?\nIt *seems* correct to me, but of course, we should have a computer check\nthat this produces OK results, not a human ;).\n\n> +}\n> +\n> struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n> \t\t\t\t struct commit *commit)\n> {\n> \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n> \t\t\t\t\t commit->object.oid);\n> -\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n> -\t\treturn NULL;\n> +\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n> +\t\tstruct stored_bitmap *bitmap = NULL;\n> +\t\tif (!bitmap_git->table_lookup)\n> +\t\t\treturn NULL;\n> +\n> +\t\t/* NEEDSWORK: cache misses aren't recorded */\n\nFor what it's worth, I think that it's completely fine to leave this as\na NEEDSWORK for the purposes of this series. I think we plausibly could\nimprove this in certain scenarios by finding some threshold on cache\nmisses when we should just fault in all bitmaps, but that can easily be\ndone on top.\n\n> +\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n> +\t\tif(!bitmap)\n\nNit: missing space between \"if\" and \"(\".\n\n> +\t\t\treturn NULL;\n> +\t\treturn lookup_stored_bitmap(bitmap);\n> +\t}\n> \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n> }\n>\n> @@ -1699,9 +1861,13 @@ void test_bitmap_walk(struct rev_info *revs)\n> \tif (revs->pending.nr != 1)\n> \t\tdie(\"you must specify exactly one commit to test\");\n>\n> -\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n> +\tfprintf(stderr, \"Bitmap v%d test (%d entries)\\n\",\n> \t\tbitmap_git->version, bitmap_git->entry_count);\n>\n> +\tif (!bitmap_git->table_lookup)\n> +\t\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n> +\t\t\tbitmap_git->version, bitmap_git->entry_count);\n> +\n\nI think we should probably print just one or the other here, perhaps\nlike:\n\n fprintf(stderr, \"Bitmap v%d test (%d entries%s)\",\n bitmap_git->version,\n bitmap_git->entry_count,\n bitmap_git->table_lookup ? \"\" : \" loaded\");\n\n> \troot = revs->pending.objects[0].item;\n> \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n>\n> @@ -1753,10 +1919,16 @@ void test_bitmap_walk(struct rev_info *revs)\n>\n> int test_bitmap_commits(struct repository *r)\n> {\n> -\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n> +\tstruct bitmap_index *bitmap_git = NULL;\n> \tstruct object_id oid;\n> \tMAYBE_UNUSED void *value;\n>\n> +\t/* As this function is only used to print bitmap selected\n> +\t * commits, we don't have to read the commit table.\n> +\t */\n> +\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"0\", 1);\n> +\n> +\tbitmap_git = prepare_bitmap_git(r);\n> \tif (!bitmap_git)\n> \t\tdie(\"failed to load bitmap indexes\");\n>\n> @@ -1764,6 +1936,7 @@ int test_bitmap_commits(struct repository *r)\n> \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n> \t});\n>\n> +\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"1\", 1);\n> \tfree_bitmap_index(bitmap_git);\n\nHmm. I'm not sure I follow the purpose of tweaking\nGIT_TEST_READ_COMMIT_TABLE like this with setenv(). Are we trying to\navoid reading the lookup table? If so, why? I'd rather avoid\nmanipulating the environment directly like this, and instead have a\nfunction we could call to fault in all of the bitmaps (when a lookup\ntable exists, otherwise do nothing).\n\nThanks,\nTaylor\n"},{"id":"457967","messageId":"Yrol2tY4emxmYh9n@nand.local","threadId":"58038","inReplyTo":"cebad3da-5779-5908-15d5-63d4c590c20b@github.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T21:49:14Z","receivedAt":"2022-06-27T21:49:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 27, 2022 at 11:12:09AM -0400, Derrick Stolee wrote:\n> On 6/26/2022 9:10 AM, Abhradeep Chakraborty via GitGitGadget wrote:\n> > +\t\t\ttable_size = st_add(table_size,\n> > +\t\t\t\t\tst_mult(ntohl(header->entry_count),\n> > +\t\t\t\t\t\ttriplet_sz));\n>\n> Here, we _do_ want to keep the st_mult(). Is the st_add() still necessary? It\n> seems this is a leftover from the previous version that had the 4-byte flag\n> data.\n>\n> We set table_size to zero above. We could drop that initialization and instead\n> have this after the \"size_t triplet_sz\" definition:\n>\n> \t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n> \t\t\t\t\t\t triplet_sz));\n\nWell put, thank you.\n\n> > +\t\t\tif (table_size > index_end - index->map - header_size)\n> > +\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit lookup table)\");\n>\n> Please add \"_(...)\" around the error message so it can be translated.\n\nI missed this in my own review, but yes: this is a good practice.\n\n> > +\tif (bitmap_git->midx)\n> > +\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n> > +\telse\n> > +\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n> > +\n> > +\treturn found;\n>\n> Here, we are doing a binary search on the entire list of packed objects, which could\n> use quite a few more hops than a binary search on the bitmapped commits.\n\nI think this is the best we can do if we make the key to our bsearch\nthrough the lookup table be an index into the pack index / MIDX. But...\n\n> > +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> > +\t\t\t\t\t struct commit *commit)\n> ...\n> > +\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n> > +\n> > +\tif (!found)\n> > +\t\treturn NULL;\n> > +\n> > +\ttriplet = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n> > +\t\t\t\t\t\ttriplet_sz, triplet_cmp);\n>\n> But I see, you are searching the pack-index for the position in the index, and _then_\n> searching the bitmap lookup table based on that position value.\n>\n> I expected something different: binary search on the triplets where the comparison is\n> made by looking up the OID from the [multi-]pack-index and comparing that OID to the\n> commit OID we are looking for.\n>\n> I'm not convinced that the binary search I had in mind is meaningfully faster than\n> what you've implemented here, so I'm happy to leave it as you have it. We can investigate\n> if that full search on the pack-index matters at all (it probably doesn't).\n\n...exactly my thoughts, too. It's possible that it would be faster to\nkey this search on the object_id \"oid\" above, and then convert each of\nthe entries in the lookup table from a uint32_t into an object_id by\ncalling nth_bitmap_object_oid() repeatedly.\n\nI *think* that what Abhradeep wrote here is going to be faster more\noften than not since it makes more efficient use of the page cache\nrather than switching between reads across different memory mapped\nregions at each point in the binary search.\n\nBut of course that depends on a number of factors. Abhradeep: if you're\nup for it, I think it would be worth trying it both ways and seeing if\none produces a meaningful speed-up or slow-down over the other. Like I\nsaid: my guess is that what you have now will be faster, but I don't\nhave a clear sense that that is true without trying it both ways ;-).\n\n> > +\tif (!triplet)\n> > +\t\treturn NULL;\n> > +\n> > +\toffset = triplet_get_offset(triplet);\n> > +\txor_pos = triplet_get_xor_pos(triplet);\n> > +\n> > +\tif (xor_pos != 0xffffffff) {\n> > +\t\tint xor_flags;\n> > +\t\tuint64_t offset_xor;\n> > +\t\tuint32_t *xor_positions;\n> > +\t\tstruct object_id xor_oid;\n> > +\t\tsize_t size = 0;\n> > +\n> > +\t\tALLOC_ARRAY(xor_positions, bitmap_git->entry_count);\n>\n> While there is potential that this is wasteful, it's probably not that huge,\n> so we can start with the \"maximum XOR depth\" and then reconsider a smaller\n> allocation in the future.\n\nThere is no maximum XOR depth, to my knowledge. We do have a maximum XOR\n*offset*, which says we cannot XOR-compress a bitmap with an entry more\nthan 160 entries away from the current one. But in theory every commit\ncould be XOR compressed with the one immediately proceeding it, so the\nmaximum depth could be as long as the entry_count itself.\n\nI think starting off with a small array and then letting it grow\naccording to alloc_nr() would be fine here, since it will grow more and\nmore each time, so the amount of times we have to reallocate the buffer\nwill tail off over time.\n\nIf we were really concerned about it, we could treat the buffer as a\nstatic pointer and reuse it over time (making sure to clear out the\nportions of it we're going to reuse, or otherwise ensuring that we don't\nread old data). But I doubt it matters much either way in practice: the\nindividual records are small (at just 4 bytes each) and entry_count is\noften less than 1,000, so I think this probably has a vanishingly small\nimpact.\n\nThanks,\nTaylor\n"},{"id":"457968","messageId":"YromNwFsUNGtW9pJ@nand.local","threadId":"58038","inReplyTo":"fe556b58814405baf5f19f4dd3e89883d08edb8e.1656249018.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/6] p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T21:50:47Z","receivedAt":"2022-06-27T21:51:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 26, 2022 at 01:10:17PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Enable pack.writeReverseIndex to true to see the effect of writing\n> the reverse index in the existing bitmap tests (with and without\n> lookup table).\n\nI think we should swap the order of these final two patches, since we're\nprimarily interested in the difference between using a reverse index\nwith and without the lookup table.\n\nThanks,\nTaylor\n"},{"id":"457969","messageId":"Yrom04Go0tCAZWT8@nand.local","threadId":"58038","inReplyTo":"96c0041688f6139c17611203f98274988ced25ab.1656249018.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-27T21:53:23Z","receivedAt":"2022-06-27T21:53:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 26, 2022 at 01:10:16PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Add performance tests to verify the performance of lookup table.\n>\n> Lookup table makes Git run faster in most of the cases. Below is the\n> result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\n> gives similar result. The repository used in the test is linux kernel.\n>\n> Test this tree\n> --------------------------------------------------------------------------\n> 5310.4: repack to disk (lookup=false) 295.94(250.45+15.24)\n> 5310.5: simulated clone 12.52(5.07+1.40)\n> 5310.6: simulated fetch 1.89(2.94+0.24)\n> 5310.7: pack to file (bitmap) 41.39(20.33+7.20)\n> 5310.8: rev-list (commits) 0.98(0.59+0.12)\n> 5310.9: rev-list (objects) 3.40(3.27+0.10)\n> 5310.10: rev-list with tag negated via --not\t\t0.07(0.02+0.04)\n> --all (objects)\n> 5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n> 5310.12: rev-list count with blob:none 0.26(0.18+0.07)\n> 5310.13: rev-list count with blob:limit=1k 6.45(5.94+0.37)\n> 5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n> 5310.15: simulated partial clone 4.99(3.19+0.45)\n> 5310.19: repack to disk (lookup=true) 269.67(174.70+21.33)\n> 5310.20: simulated clone 11.03(5.07+1.11)\n> 5310.21: simulated fetch 0.79(0.79+0.17)\n> 5310.22: pack to file (bitmap) 43.03(20.28+7.43)\n> 5310.23: rev-list (commits) 0.86(0.54+0.09)\n> 5310.24: rev-list (objects) 3.35(3.26+0.07)\n> 5310.25: rev-list with tag negated via --not\t\t0.05(0.00+0.03)\n> \t --all (objects)\n> 5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n> 5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n> 5310.28: rev-list count with blob:limit=1k 6.45(5.87+0.31)\n> 5310.29: rev-list count with tree:0 0.22(0.16+0.05)\n> 5310.30: simulated partial clone 5.17(3.12+0.48)\n>\n> Test 4-15 are tested without using lookup table. Same tests are\n> repeated in 16-30 (using lookup table).\n>\n> Signed-off-by: Abhradeep Chakraborty \n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n> ---\n> t/perf/p5310-pack-bitmaps.sh | 77 ++++++++++++++-----------\n> t/perf/p5326-multi-pack-bitmaps.sh | 93 ++++++++++++++++--------------\n> 2 files changed, 94 insertions(+), 76 deletions(-)\n>\n> diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\n> index 7ad4f237bc3..6ff42bdd391 100755\n> --- a/t/perf/p5310-pack-bitmaps.sh\n> +++ b/t/perf/p5310-pack-bitmaps.sh\n> @@ -16,39 +16,48 @@ test_expect_success 'setup bitmap config' '\n> \tgit config pack.writebitmaps true\n> '\n>\n> -# we need to create the tag up front such that it is covered by the repack and\n> -# thus by generated bitmaps.\n> -test_expect_success 'create tags' '\n> -\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n> -'\n> -\n> -test_perf 'repack to disk' '\n> -\tgit repack -ad\n> -'\n> -\n> -test_full_bitmap\n> -\n> -test_expect_success 'create partial bitmap state' '\n> -\t# pick a commit to represent the repo tip in the past\n> -\tcutoff=$(git rev-list HEAD~100 -1) &&\n> -\torig_tip=$(git rev-parse HEAD) &&\n> -\n> -\t# now kill off all of the refs and pretend we had\n> -\t# just the one tip\n> -\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n> -\tgit update-ref HEAD $cutoff &&\n> -\n> -\t# and then repack, which will leave us with a nice\n> -\t# big bitmap pack of the \"old\" history, and all of\n> -\t# the new history will be loose, as if it had been pushed\n> -\t# up incrementally and exploded via unpack-objects\n> -\tgit repack -Ad &&\n> -\n> -\t# and now restore our original tip, as if the pushes\n> -\t# had happened\n> -\tgit update-ref HEAD $orig_tip\n> -'\n> -\n> -test_partial_bitmap\n> +test_bitmap () {\n> + local enabled=\"$1\"\n> +\n> +\t# we need to create the tag up front such that it is covered by the repack and\n> +\t# thus by generated bitmaps.\n> +\ttest_expect_success 'create tags' '\n> +\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n> +\t'\n\nI think this \"create tags\" step can happen outside of the test_bitmap()\nfunction, since it should only need to be done once, right?\n\n> +\ttest_expect_success \"use lookup table: $enabled\" '\n> +\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n> +\t'\n> +\n> +\ttest_perf \"repack to disk (lookup=$enabled)\" '\n> +\t\tgit repack -ad\n> +\t'\n\nAnd I think these two tests could be combined, since this could just\nbecome:\n\n git -c pack.writeBitmapLookupTable \"$enabled\" repack -ad\n\nright?\n\n> +\ttest_full_bitmap\n> +\n> + test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n\nThere is some funky spacing going on here, at least in my email client.\nCould you double check that tabs are used consistently here?\n\nThanks,\nTaylor\n"},{"id":"457990","messageId":"20220628075843.19170-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"Yrom04Go0tCAZWT8@nand.local","subject":"Re: [PATCH v2 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-28T07:58:43Z","receivedAt":"2022-06-28T07:59:10Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> I think this \"create tags\" step can happen outside of the test_bitmap()\n> function, since it should only need to be done once, right?\n\nYeah, I also think the same. That's why I tried to not include in the\nFunction but for some reason, one test is failing -\n\n perf 24 - rev-list with tag negated via --not --all (objects):\n running: \n \t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n\t\n fatal: ambiguous argument 'perf-tag': unknown revision or path not in the working tree.\n Use '--' to separate paths from revisions, like this:\n 'git [...] -- [...]'\n not ok 24 - rev-list with tag negated via --not --all (objects)\n\nOne thing to note here is that the first `test_bitmap` call always\nPasses. But the second `test_bitmap` call fails due to above error.\nIt throws error irrespective of any parameters for second `test_bitmap`.\n\nIf I put it inside the function it doesn't throw any error! \n\nFor this reason, I put it into the function. Do you have any idea\nwhy this happend?\n\n> And I think these two tests could be combined, since this could just\n> become:\n>\n> git -c pack.writeBitmapLookupTable \"$enabled\" repack -ad\n>\n> right?\n\nYeah, sure.\n\n> There is some funky spacing going on here, at least in my email client.\n> Could you double check that tabs are used consistently here?\n\nThis is due to my editor's spacing issues. All seems fine when I look at\nit in my editor. But actually it is not. Fixing it.\n\nThanks :)\n"},{"id":"457991","messageId":"20220628080130.19199-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YromNwFsUNGtW9pJ@nand.local","subject":"Re: [PATCH v2 6/6] p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-28T08:01:30Z","receivedAt":"2022-06-28T08:01:46Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> I think we should swap the order of these final two patches, since we're\n> primarily interested in the difference between using a reverse index\n> with and without the lookup table.\n\nOk. Thanks :)\n"},{"id":"458000","messageId":"20220628085950.19288-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"Yrol2tY4emxmYh9n@nand.local","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-28T08:59:50Z","receivedAt":"2022-06-28T09:00:11Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Taylor Blau wrote:\n\n> ...exactly my thoughts, too. It's possible that it would be faster to\n> key this search on the object_id \"oid\" above, and then convert each of\n> the entries in the lookup table from a uint32_t into an object_id by\n> calling nth_bitmap_object_oid() repeatedly.\n>\n> I *think* that what Abhradeep wrote here is going to be faster more\n> often than not since it makes more efficient use of the page cache\n> rather than switching between reads across different memory mapped\n> regions at each point in the binary search.\n>\n> But of course that depends on a number of factors. Abhradeep: if you're\n> up for it, I think it would be worth trying it both ways and seeing if\n> one produces a meaningful speed-up or slow-down over the other. Like I\n> said: my guess is that what you have now will be faster, but I don't\n> have a clear sense that that is true without trying it both ways ;-).\n\nOk. Let me try both the ways. In my opinion, I think my version has\nless searching and less computation. So, I want to stick with this\nversion. But I also like to try the other one once so that we can\nget the best out of these two.\n\n> I think starting off with a small array and then letting it grow\n> according to alloc_nr() would be fine here, since it will grow more and\n> more each time, so the amount of times we have to reallocate the buffer\n> will tail off over time.\n\nWhat should be the size of that array?\n\n> If we were really concerned about it, we could treat the buffer as a\n> static pointer and reuse it over time (making sure to clear out the\n> portions of it we're going to reuse, or otherwise ensuring that we don't\n> read old data). But I doubt it matters much either way in practice: the\n> individual records are small (at just 4 bytes each) and entry_count is\n> often less than 1,000, so I think this probably has a vanishingly small\n> impact.\n\nBefore submitting it to the mailing list, I did use the ALLOC_GROW macro\nfunction. But my version was worse than yours. For every iteration I was\nreallocating the array to support `size+1` positions. But later I drop\nthe code as this might be very much expensive.\n\nThen I wrote this code. As `table` array and `table_inv` array allocate\nthis size of arrays (though all the indices are used), I thought it\nwould not be a problem if I use an array of this size for a small amount\nof time.\n\nHonestly, I don't like to realloc arrays. Because as far as I can remember,\nrealloc allocates a new array internally and copies the items from the old\narray to the new array. This irritates me.\n\nBut at the same time, it is also true that in most cases we might not need\nthis amount of space.\n\nThanks :)\n"},{"id":"458061","messageId":"20220628192555.23565-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"YrojV5aYCzxXlV3c@nand.local","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-28T19:25:55Z","receivedAt":"2022-06-28T19:29:30Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nOhh, sorry! Looks like I missed this comment!\n\nTaylor Blau wrote:\n\n> It may be worth replacing \"within map\" to \"within the memory mapped\n> region `map`\" to make clear that this points somewhere within the mmap.\n\nOk.\n\n> I should have commented on this in an earlier round, but I wonder what\n> the behavior should be when we have BITMAP_OPT_LOOKUP_TABLE in our\n> flags, but GIT_TEST_READ_COMMIT_TABLE is disabled.\n>\n> Right now, it doesn't matter, since there aren't any flags in bits above\n> BITMAP_OPT_LOOKUP_TABLE. But in the future, if there was some\n> BITMAP_OPT_FOO that was newer than BITMAP_OPT_LOOKUP_TABLE, we would\n> want to be able to read it without needing to read the lookup table.\n>\n> At least, I think that should be true, though I would be interested to\n> hear if anybody has a differing opinion there.\n\nOh right! I didn't think about it. In that case, we should still subtract\nThe table size from the last index_size. In that way, These sections will\nNot be overlapped.\n\n> And table_size here is going to start off at zero, so the outer st_add()\n> call isn't necessary, either. This should instead be:\n>\n> size_t table_size = st_mult(ntohl(header->entry_count),\n> sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t));\n>\n> It might be nice to have triplet_sz #define'd somewhere else, since\n> there are a handful of declarations in this patch that are all\n> identical. Probably something like:\n>\n> #define BITMAP_LOOKUP_TABLE_RECORD_WIDTH (sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uin32_t))\n>\n> or even:\n>\n> /*\n> * The width in bytes of a single record in the lookup table\n> * extension:\n> *\n> * (commit_pos, offset, xor_pos)\n> *\n> * whose fields are 32-, 64-, and 32-bits wide, respectively.\n> */\n> #define BITMAP_LOOKUP_TABLE_RECORD_WIDTH (16)\n\nSeems perfect to me.\n\n> if we decide to still recognize the lookup table extension without\n> *reading* from it when GIT_TEST_READ_COMMIT_TABLE is unset, I think we\n> should do something like:\n>\n> if (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n> index->table_lookup = (void *)(index_end - table_size);\n> index_end -= table_size;\n>\n> ...where the subtraction on index_end happens unconditionally.\n\nRight. Thanks!\n\n> I wonder if we could get rid of these functions altogether and return a\n> small structure like:\n>\n> struct bitmap_lookup_table_record {\n> uint32_t commit_pos;\n> uint64_t offset;\n> uint32_t xor_pos;\n> };\n>\n> or similar.\n\nOk.\n\n> Hmm. This is a little tricky to read. Here we're expecting \"va\" to hold\n> commit_pos from below, and \"vb\" to be a pointer at a lookup record.\n> Everything here is right, though I wonder if a comment or two might\n> clarify why one is \"*(uint32_t *)va\" and the other is \"get_be32(vb)\".\n\nSure. Will add comments.\n\n> Nit: let's use the bitmap_is_midx() helper here instead of looking at\n> bitamp_git->midx directly.\n\nOk.\n\n> First thing is to convert the commit OID we're looking for into its\n> position within the corresponding pack index or MIDX file so that we can\n> use it as a search key to locate in the lookup table. If we didn't find\n> anything, or the commit doesn't exist in our pack / MIDX, nothing to do.\n>\n> > +\n> > +\toffset = triplet_get_offset(triplet);\n> > +\txor_pos = triplet_get_xor_pos(triplet);\n>\n> Otherwise, record its offset and XOR \"offset\".\n\nExactly!\n\n> We already have to get the triplets in the loop above, and then we dig\n> them back out here. Would it be easier to keep track of a list of\n> pointers into the mmaped region instead of looking up these triplets\n> each time?\n\nSure. It might be a good idea. Thanks.\n\n> > +\t\t\tcommit_pos = get_be32(triplet);\n> > +\t\t\toffset_xor = triplet_get_offset(triplet);\n> > +\n> > +\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n>\n> Should it be an error if we can't look up the object's ID here? I'd\n> think so.\n\nI also am not sure about it. Morally, I think it is better to throw\nAn error here.\n\n> Do we have a good way to make sure that we're testing this code in CI?\n> It *seems* correct to me, but of course, we should have a computer check\n> that this produces OK results, not a human ;).\n\nMy current test file changes should test this code. As for now, the lookup\nTable is enabled by default, all the existing tests that include write and\nread bitmaps uses this lookup table. So, all the test case scenarios should\nPass. So, I think it is being tested in CI. Do you have a good idea to test\nIt better?\n\n> Hmm. I'm not sure I follow the purpose of tweaking\n> GIT_TEST_READ_COMMIT_TABLE like this with setenv(). Are we trying to\n> avoid reading the lookup table? If so, why? I'd rather avoid\n> manipulating the environment directly like this, and instead have a\n> function we could call to fault in all of the bitmaps (when a lookup\n> table exists, otherwise do nothing).\n\nThe problem was that the `test-tool bitmap list-commit` command was\nNot printing any commits (the error that I notified you before). It\nis because of this function. As lookup table is enabled by default,\n`prepare_bitmap_git` function doesn't load each bitmap entries and\nthus the below code in this function doesn't provide the bitmapped\ncommit list (because Hashtable didn't generated).\n\n kh_foreach(bitmap_git->bitmaps, oid, value, {\n\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n\t});\n\nSo, the simplest fix I found was this. Should I make a function then\n(Which you suggested here)?\n\nThanks :)\n"},{"id":"458121","messageId":"YryyCGSvR2Om3UpH@nand.local","threadId":"58038","inReplyTo":"20220627183924.16369-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v2 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-29T20:11:52Z","receivedAt":"2022-06-29T20:11:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 28, 2022 at 12:09:23AM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n>\n> > Probably both of them should take into account their separate\n> > configuration values, but cleaning up the hashcache one can be done\n> > separately outside of this series.\n>\n> Actually, it does respect the `pack.writebitmaplookuptable` config.\n> As pack.writebitmaplookuptable is by default true (for this patch\n> Series), this line enables it by default. If `pack.writebitmaplookuptable`\n> Set to false, the proposed change in the `git_multi_pack_index_write_config`\n> function disables this flag.\n\nAha, you're absolutely right. I missed the earlier hunk. Thanks for\npointing it out, this part looks fine to me.\n\nThanks,\nTaylor\n"},{"id":"458122","messageId":"Yry0bKgayLB3GdsW@nand.local","threadId":"58038","inReplyTo":"20220628085950.19288-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-29T20:22:04Z","receivedAt":"2022-06-29T20:22:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 28, 2022 at 02:29:50PM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n>\n> > ...exactly my thoughts, too. It's possible that it would be faster to\n> > key this search on the object_id \"oid\" above, and then convert each of\n> > the entries in the lookup table from a uint32_t into an object_id by\n> > calling nth_bitmap_object_oid() repeatedly.\n> >\n> > I *think* that what Abhradeep wrote here is going to be faster more\n> > often than not since it makes more efficient use of the page cache\n> > rather than switching between reads across different memory mapped\n> > regions at each point in the binary search.\n> >\n> > But of course that depends on a number of factors. Abhradeep: if you're\n> > up for it, I think it would be worth trying it both ways and seeing if\n> > one produces a meaningful speed-up or slow-down over the other. Like I\n> > said: my guess is that what you have now will be faster, but I don't\n> > have a clear sense that that is true without trying it both ways ;-).\n>\n> Ok. Let me try both the ways. In my opinion, I think my version has\n> less searching and less computation. So, I want to stick with this\n> version. But I also like to try the other one once so that we can\n> get the best out of these two.\n\nYeah, I agree with your general sense that the version as written is\ngoing to be faster. We're comparing a smaller datatype (IOW, a 4-byte\ninteger that can be checked for equality in a single instruction,\ninstead of comparing two 20-byte OIDs), and likely flushing the cache\nfar less often.\n\nBut having two concrete implementations to compare will help us know for\na fact that our intuition is correct.\n\nI'll be curious to see what you find here!\n\n> > I think starting off with a small array and then letting it grow\n> > according to alloc_nr() would be fine here, since it will grow more and\n> > more each time, so the amount of times we have to reallocate the buffer\n> > will tail off over time.\n>\n> What should be the size of that array?\n\nI think some small, power of 2 would be a reasonable choice here.\n\n> > If we were really concerned about it, we could treat the buffer as a\n> > static pointer and reuse it over time (making sure to clear out the\n> > portions of it we're going to reuse, or otherwise ensuring that we don't\n> > read old data). But I doubt it matters much either way in practice: the\n> > individual records are small (at just 4 bytes each) and entry_count is\n> > often less than 1,000, so I think this probably has a vanishingly small\n> > impact.\n>\n> Before submitting it to the mailing list, I did use the ALLOC_GROW macro\n> function. But my version was worse than yours. For every iteration I was\n> reallocating the array to support `size+1` positions. But later I drop\n> the code as this might be very much expensive.\n\nThat shouldn't be the case. When you have a chance, take a look at the\nalloc_nr macro, which shows how much memory we allocate at each\nstep:\n\n #define alloc_nr(x) (((x)+16)*3/2)\n\nSuppose we allocated 16 slots initially, so nr (the number of entries\nstored in the list) is 0 and alloc (the number of entries allocated) is\n16. Then when we try to add the 17th item, we'll pass 16 to alloc_nr\nwhich will allocate 48 slots. Then 96, then 168, and so on.\n\nWe only have to reallocate and copy the array when nr > alloc, which\nshould be fairly infrequently, and happens less and less often the\nlarger the array grows.\n\nThanks,\nTaylor\n"},{"id":"458123","messageId":"Yry4ElVFQEsVbqse@nand.local","threadId":"58038","inReplyTo":"20220628192555.23565-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-29T20:37:38Z","receivedAt":"2022-06-29T20:37:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 29, 2022 at 12:55:55AM +0530, Abhradeep Chakraborty wrote:\n> > > +\t\t\tcommit_pos = get_be32(triplet);\n> > > +\t\t\toffset_xor = triplet_get_offset(triplet);\n> > > +\n> > > +\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n> >\n> > Should it be an error if we can't look up the object's ID here? I'd\n> > think so.\n>\n> I also am not sure about it. Morally, I think it is better to throw\n> An error here.\n\nYeah.\n\n> > Do we have a good way to make sure that we're testing this code in CI?\n> > It *seems* correct to me, but of course, we should have a computer check\n> > that this produces OK results, not a human ;).\n>\n> My current test file changes should test this code. As for now, the lookup\n> Table is enabled by default, all the existing tests that include write and\n> read bitmaps uses this lookup table. So, all the test case scenarios should\n> Pass. So, I think it is being tested in CI. Do you have a good idea to test\n> It better?\n\nI think having some indication (maybe via a trace2 region?) that we're\nactually executing this code would be good. Although it's going to be\n*really* noisy, so probably not a good idea to do that in general.\n\nStolee runs some coverage tests that show lines that we aren't\nexercising via tests. So making sure that this doesn't show up in that\nreport when you run it locally would be good.\n\nSee some information from him about how to run those tests locally here:\n\n https://lore.kernel.org/git/00a57a1d-0566-8f54-26b2-0f3558bde88d@github.com/\n\n(TL;DR: run `make coverage-test` and make sure that these lines don't\nshow up ;-)).\n\n> > Hmm. I'm not sure I follow the purpose of tweaking\n> > GIT_TEST_READ_COMMIT_TABLE like this with setenv(). Are we trying to\n> > avoid reading the lookup table? If so, why? I'd rather avoid\n> > manipulating the environment directly like this, and instead have a\n> > function we could call to fault in all of the bitmaps (when a lookup\n> > table exists, otherwise do nothing).\n>\n> The problem was that the `test-tool bitmap list-commit` command was\n> Not printing any commits (the error that I notified you before). It\n> is because of this function. As lookup table is enabled by default,\n> `prepare_bitmap_git` function doesn't load each bitmap entries and\n> thus the below code in this function doesn't provide the bitmapped\n> commit list (because Hashtable didn't generated).\n>\n> kh_foreach(bitmap_git->bitmaps, oid, value, {\n> \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n> \t});\n>\n> So, the simplest fix I found was this. Should I make a function then\n> (Which you suggested here)?\n\nI see. I remember that issue, but I think we should go about fixing it\nin a different way. Instead of tricking the code into loading all\nbitmaps by pretending the lookup table doesn't exist, we should have a\nfunction that forces loading in all bitmaps from the lookup table, if\none exists. If the lookup table doesn't exist, or we have already loaded\nits entries, then that function can be noop.\n\nIf we had something like that, we could call that function from within\n`test_bitmap_commits()` before reading the keys and values out of\n`bitmap_git->bitmaps`.\n\nAn alternative approach would be to read the table directly when it\nexists, perhaps something like this:\n\n--- 8< ---\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9e09c5824f..3bda059b9f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1921,22 +1921,28 @@ int test_bitmap_commits(struct repository *r)\n {\n \tstruct bitmap_index *bitmap_git = NULL;\n \tstruct object_id oid;\n-\tMAYBE_UNUSED void *value;\n-\n-\t/* As this function is only used to print bitmap selected\n-\t * commits, we don't have to read the commit table.\n-\t */\n-\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"0\", 1);\n\n \tbitmap_git = prepare_bitmap_git(r);\n \tif (!bitmap_git)\n \t\tdie(\"failed to load bitmap indexes\");\n\n-\tkh_foreach(bitmap_git->bitmaps, oid, value, {\n-\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n-\t});\n+\tif (bitmap_git->table_lookup) {\n+\t\tuint32_t i, commit_pos;\n+\t\tfor (i = 0; i < bitmap_git->entry_count; i++) {\n+\t\t\tcommit_pos = get_be32(bitmap_get_triplet(bitmap_git, i));\n+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &oid,\n+\t\t\t\t\t\t commit_pos) < 0)\n+\t\t\t\treturn error(\"could not find commit at \"\n+\t\t\t\t\t \"position %\"PRIu32, commit_pos);\n+\t\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\t}\n+\t} else {\n+\t\tMAYBE_UNUSED void *value;\n+\t\tkh_foreach(bitmap_git->bitmaps, oid, value, {\n+\t\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\t});\n+\t}\n\n-\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"1\", 1);\n \tfree_bitmap_index(bitmap_git);\n\n \treturn 0;\n\n--- >8 ---\n\nThanks,\nTaylor\n"},{"id":"458125","messageId":"Yry40c0lqOoU8p4G@nand.local","threadId":"58038","inReplyTo":"20220628075843.19170-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v2 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-29T20:40:49Z","receivedAt":"2022-06-29T20:40:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 28, 2022 at 01:28:43PM +0530, Abhradeep Chakraborty wrote:\n> Taylor Blau wrote:\n>\n> > I think this \"create tags\" step can happen outside of the test_bitmap()\n> > function, since it should only need to be done once, right?\n>\n> Yeah, I also think the same. That's why I tried to not include in the\n> Function but for some reason, one test is failing -\n>\n> perf 24 - rev-list with tag negated via --not --all (objects):\n> running:\n> \t\tgit rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null\n>\n> fatal: ambiguous argument 'perf-tag': unknown revision or path not in the working tree.\n> Use '--' to separate paths from revisions, like this:\n> 'git [...] -- [...]'\n> not ok 24 - rev-list with tag negated via --not --all (objects)\n>\n> One thing to note here is that the first `test_bitmap` call always\n> Passes. But the second `test_bitmap` call fails due to above error.\n> It throws error irrespective of any parameters for second `test_bitmap`.\n>\n> If I put it inside the function it doesn't throw any error!\n>\n> For this reason, I put it into the function. Do you have any idea\n> why this happend?\n\nI think that it's because we delete all of the refs in the test that\ncreates a partial bitmap state. So anything that relies on perf-tag\nexisting after that test runs will definitely not work :).\n\nMy original suggestion was misguided there, unless we wanted to make the\naforementioned test (the one that creates the partial bitmap state)\nrestore the ref state after it finishes running, but I don't think\nthat's worthwhile.\n\nThanks,\nTaylor\n"},{"id":"458132","messageId":"Yry4/W6ENxnljI60@nand.local","threadId":"58038","inReplyTo":"Yry4ElVFQEsVbqse@nand.local","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-06-29T20:41:33Z","receivedAt":"2022-06-29T20:41:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 29, 2022 at 04:37:38PM -0400, Taylor Blau wrote:\n> +\t\t\t\treturn error(\"could not find commit at \"\n> +\t\t\t\t\t \"position %\"PRIu32, commit_pos);\n\nOops. Pretend that I marked this string for translation ;-).\n\nThanks,\nTaylor\n"},{"id":"458165","messageId":"20220630065833.6333-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"Yry0bKgayLB3GdsW@nand.local","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-30T06:58:33Z","receivedAt":"2022-06-30T06:59:13Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nTaylor Blau wrote:\n\n> That shouldn't be the case. When you have a chance, take a look at the\n> alloc_nr macro, which shows how much memory we allocate at each\n> step:\n>\n> #define alloc_nr(x) (((x)+16)*3/2)\n>\n> Suppose we allocated 16 slots initially, so nr (the number of entries\n> stored in the list) is 0 and alloc (the number of entries allocated) is\n> 16. Then when we try to add the 17th item, we'll pass 16 to alloc_nr\n> which will allocate 48 slots. Then 96, then 168, and so on.\n>\n> We only have to reallocate and copy the array when nr > alloc, which\n> should be fairly infrequently, and happens less and less often the\n> larger the array grows.\n\nOhh, I misunderstood the ALLOC_GROW function. Thanks!\n"},{"id":"458167","messageId":"20220630083517.6411-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"Yry4ElVFQEsVbqse@nand.local","subject":"Re: [PATCH v2 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-06-30T08:35:17Z","receivedAt":"2022-06-30T08:35:47Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nTaylor Blau wrote:\n\n> I think having some indication (maybe via a trace2 region?) that we're\n> actually executing this code would be good. Although it's going to be\n> *really* noisy, so probably not a good idea to do that in general.\n>\n> Stolee runs some coverage tests that show lines that we aren't\n> exercising via tests. So making sure that this doesn't show up in that\n> report when you run it locally would be good.\n>\n> See some information from him about how to run those tests locally here:\n>\n> https://lore.kernel.org/git/00a57a1d-0566-8f54-26b2-0f3558bde88d@github.com/\n>\n> (TL;DR: run `make coverage-test` and make sure that these lines don't\n> show up ;-)).\n\nGot it. Thanks.\n\n> If we had something like that, we could call that function from within\n> `test_bitmap_commits()` before reading the keys and values out of\n> `bitmap_git->bitmaps`.\n>\n> An alternative approach would be to read the table directly when it\n> exists, perhaps something like this:\n\nI think we have a simpler fix than what you suggested here. What if\nWe do it like this way -\n\n if (bitmap_git->table_lookup) {\n\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n\t die(_(\"failed to load bitmap indexes\"));\n }\n\nIs this okay for you?\n"},{"id":"458436","messageId":"5e9b985e39b0b9edee7af55dd8b0698a20062cf7.1656924376.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:12Z","receivedAt":"2022-07-04T08:46:30Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe bitmap lookup table extension was documented by an earlier\nchange, but Git does not yet know how to write that extension.\n\nTeach Git to write bitmap lookup table extension. The table contains\nthe list of `N` ` triplets. These\ntriplets are sorted according to their commit pos (ascending order).\nThe meaning of each data in the i'th triplet is given below:\n\n - commit_pos stores commit position (in the pack-index or midx).\n It is a 4 byte network byte order unsigned integer.\n\n - offset is the position (in the bitmap file) from which that\n commit's bitmap can be read.\n\n - xor_row is the position of the triplet in the lookup table\n whose bitmap is used to compress this bitmap, or `0xffffffff`\n if no such bitmap exists.\n\nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nCo-authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 112 ++++++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 5 +-\n 2 files changed, 112 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c43375bd344..4a0edd746bc 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -648,9 +648,17 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \treturn &index[pos]->oid;\n }\n \n+static int commit_bitmap_writer_pos(struct object_id *oid,\n+\t\t\t\t struct pack_idx_entry **index,\n+\t\t\t\t uint32_t index_nr)\n+{\n+\treturn oid_pos(oid, index, index_nr, oid_access);\n+}\n+\n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t off_t *offsets)\n {\n \tint i;\n \n@@ -658,11 +666,14 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n \t\tint commit_pos =\n-\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\t\t\tcommit_bitmap_writer_pos(&stored->commit->object.oid, index, index_nr);\n \n \t\tif (commit_pos < 0)\n \t\t\tBUG(\"trying to write commit not in index\");\n \n+\t\tif (offsets)\n+\t\t\toffsets[i] = hashfile_total(f);\n+\n \t\thashwrite_be32(f, commit_pos);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n@@ -671,6 +682,92 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static int table_cmp(const void *_va, const void *_vb, void *_data)\n+{\n+\tuint32_t *commit_positions = _data;\n+\tuint32_t a = commit_positions[*(uint32_t *)_va];\n+\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n+\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static void write_lookup_table(struct hashfile *f,\n+\t\t\t struct pack_idx_entry **index,\n+\t\t\t uint32_t index_nr,\n+\t\t\t off_t *offsets)\n+{\n+\tuint32_t i;\n+\tuint32_t *table, *table_inv, *commit_positions;\n+\n+\tALLOC_ARRAY(table, writer.selected_nr);\n+\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n+\n+\t/* store the index positions of the commits */\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tint pos = commit_bitmap_writer_pos(&writer.selected[i].commit->object.oid,\n+\t\t\t\t\t\t index, index_nr);\n+\t\tif (pos < 0)\n+\t\t\tBUG(_(\"trying to write commit not in index\"));\n+\n+\t\tcommit_positions[i] = pos;\n+\t}\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable[i] = i;\n+\n+\t/*\n+\t * At the end of this sort table[j] = i means that the i'th\n+\t * bitmap corresponds to j'th bitmapped commit in lex order of\n+\t * OIDs.\n+\t */\n+\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\n+\t/* table_inv helps us discover that relationship (i'th bitmap\n+\t * to j'th commit by j = table_inv[i])\n+\t */\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable_inv[table[i]] = i;\n+\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\t\tuint32_t xor_offset = selected->xor_offset;\n+\t\tuint32_t xor_row;\n+\n+\t\tif (xor_offset) {\n+\t\t\t/*\n+\t\t\t * xor_index stores the index (in the bitmap entries)\n+\t\t\t * of the corresponding xor bitmap. But we need to convert\n+\t\t\t * this index into lookup table's index. So, table_inv[xor_index]\n+\t\t\t * gives us the index position w.r.t. the lookup table.\n+\t\t\t *\n+\t\t\t * If \"k = table[i] - xor_offset\" then the xor base is the k'th\n+\t\t\t * bitmap. `table_inv[k]` gives us the position of that bitmap\n+\t\t\t * in the lookup table.\n+\t\t\t */\n+\t\t\tuint32_t xor_index = table[i] - xor_offset;\n+\t\t\txor_row = table_inv[xor_index];\n+\t\t} else {\n+\t\t\txor_row = 0xffffffff;\n+\t\t}\n+\n+\t\thashwrite_be32(f, commit_positions[table[i]]);\n+\t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n+\t\thashwrite_be32(f, xor_row);\n+\t}\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\n+\tfree(table);\n+\tfree(table_inv);\n+\tfree(commit_positions);\n+}\n+\n static void write_hash_cache(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n@@ -695,6 +792,7 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n+\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \n@@ -715,7 +813,14 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n-\twrite_selected_commits_v1(f, index, index_nr);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\tCALLOC_ARRAY(offsets, index_nr);\n+\n+\twrite_selected_commits_v1(f, index, index_nr, offsets);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\twrite_lookup_table(f, index, index_nr, offsets);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n@@ -730,4 +835,5 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n+\tfree(offsets);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3d3ddd77345..67a9d0fc303 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -24,8 +24,9 @@ struct bitmap_disk_header {\n #define NEEDS_BITMAP (1u<<22)\n \n enum pack_bitmap_opts {\n-\tBITMAP_OPT_FULL_DAG = 1,\n-\tBITMAP_OPT_HASH_CACHE = 4,\n+\tBITMAP_OPT_FULL_DAG = 0x1,\n+\tBITMAP_OPT_HASH_CACHE = 0x4,\n+\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n };\n \n enum pack_bitmap_flags {\n-- \ngitgitgadget\n\n"},{"id":"458437","messageId":"f72bf11e6efb4690ae808c0b56c3991c2b1ef266.1656924376.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:11Z","receivedAt":"2022-07-04T08:46:31Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nWhen reading bitmap file, Git loads each and every bitmap one by one\neven if all the bitmaps are not required. A \"bitmap lookup table\"\nextension to the bitmap format can reduce the overhead of loading\nbitmaps which stores a list of bitmapped commit id pos (in the midx\nor pack, along with their offset and xor offset. This way git can\nload only the necessary bitmaps without loading the previous bitmaps.\n\nOlder versions of Git ignore the lookup table extension and don't\nthrow any kind of warning or error while parsing the bitmap file.\n\nAdd some information for the new \"bitmap lookup table\" extension in the\nbitmap-format documentation.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/technical/bitmap-format.txt | 39 +++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 04b3ec21785..c30dc177643 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -67,6 +67,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n \t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n+\t\t\t** {empty}\n+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n+\t\t\tIf present, the end of the bitmap file contains a table\n+\t\t\tcontaining a list of `N` \n+\t\t\ttriplets. The format and meaning of the table is described\n+\t\t\tbelow.\n++\n+NOTE: Unlike the xor_offset used to compress an individual bitmap,\n+`xor_row` stores an *absolute* index into the lookup table, not a location\n+relative to the current entry.\n+\n \t\t4-byte entry count (network byte order)\n \n \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n@@ -205,3 +216,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n hashes made under two different schemes would be pointless).\n+\n+Commit lookup table\n+-------------------\n+\n+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n+bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`\n+file contains a lookup table specifying the information needed to get\n+the desired bitmap from the entries without parsing previous unnecessary\n+bitmaps.\n+\n+For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n+contains a list of `nr_entries` triplets\n+(sorted in the ascending order of `commit_pos`). The content of i'th\n+triplet is -\n+\n+\t* {empty}\n+\tcommit_pos (4 byte integer, network byte order): ::\n+\tIt stores the object position of a commit (in the midx or pack\n+\tindex).\n+\n+\t* {empty}\n+\toffset (8 byte integer, network byte order): ::\n+\tThe offset from which that commit's bitmap can be read.\n+\n+\t* {empty}\n+\txor_row (4 byte integer, network byte order): ::\n+\tThe position of the triplet whose bitmap is used to compress\n+\tthis one, or `0xffffffff` if no such bitmap exists.\n-- \ngitgitgadget\n\n"},{"id":"458438","messageId":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v2.git.1656249017.gitgitgadget@gmail.com","subject":"[PATCH v3 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:10Z","receivedAt":"2022-07-04T08:46:35Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"When parsing the .bitmap file, git loads all the bitmaps one by one even if\nsome of the bitmaps are not necessary. We can remove this overhead by\nloading only the necessary bitmaps. A look up table extension can solve this\nissue.\n\nChanges since v1:\n\nThis is the second version which addressed all (I think) the reviews. Please\nnotify me if some reviews are not addressed :)\n\n * The table size is decreased and the format has also changed. It now\n contains nr_entries triplets of size 4+8+4 bytes. Each triplet contains\n the following things - (1) 4 byte commit position (in the pack-index or\n midx) (2) 8 byte offset and (3) 4 byte xor triplet (i.e. with whose\n bitmap the current triplet's bitmap has to xor) position.\n * Performance tests are splitted into two commits. First contains the\n actual performance tests and second enables the pack.writeReverseIndex\n (as suggested by Taylor).\n * st_*() functions are used.\n * commit order is changed according to Derrick's suggestion.\n * Iterative approach is used instead of recursive approach to parse xor\n bitmaps. (As suggested by Derrick).\n * Some minor bug fixes of previous version.\n\nInitial version:\n\nThe proposed table has:\n\n * a list of nr_entries object ids. These objects are commits that has\n bitmaps. Ids are stored in lexicographic order (for better searching).\n * a list of pairs (4-byte integers, network-byte\n order). The i'th pair denotes the offset and xor-offset(respectively) of\n the bitmap of i'th commit in the previous list. These two informations\n are necessary because only in this way bitmaps can be found without\n parsing all the bitmap.\n * a 4-byte integer for table specific flags (none exists currently).\n\nWhenever git want to parse the bitmap for a specific commit, it will first\nrefer to the table and will look for the offset and xor-offset for that\ncommit. Git will then try to parse the bitmap located at the offset\nposition. The xor-offset can be used to find the xor-bitmap for the\nbitmap(if any).\n\nAbhradeep Chakraborty (6):\n Documentation/technical: describe bitmap lookup table extension\n pack-bitmap-write.c: write lookup table extension\n pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n pack-bitmap: prepare to read lookup table extension\n bitmap-lookup-table: add performance tests for lookup table\n p5310-pack-bitmaps.sh: remove pack.writeReverseIndex\n\n Documentation/config/pack.txt | 7 +\n Documentation/technical/bitmap-format.txt | 39 ++\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n pack-bitmap-write.c | 112 ++-\n pack-bitmap.c | 266 +++++++-\n pack-bitmap.h | 14 +-\n t/perf/p5310-pack-bitmaps.sh | 77 ++-\n t/perf/p5326-multi-pack-bitmaps.sh | 93 +--\n t/t5310-pack-bitmaps.sh | 786 ++++++++++++----------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++-----\n t/t5327-multi-pack-bitmaps-rev.sh | 9 +\n 15 files changed, 1238 insertions(+), 658 deletions(-)\n\n\nbase-commit: 39c15e485575089eb77c769f6da02f98a55905e0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1266%2FAbhra303%2Fbitmap-commit-table-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1266/Abhra303/bitmap-commit-table-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1266\n\nRange-diff vs v2:\n\n 1: 4d11be66cfa ! 1: f72bf11e6ef Documentation/technical: describe bitmap lookup table extension\n @@ Metadata\n ## Commit message ##\n Documentation/technical: describe bitmap lookup table extension\n \n - When reading bitmap file, git loads each and every bitmap one by one\n + When reading bitmap file, Git loads each and every bitmap one by one\n even if all the bitmaps are not required. A \"bitmap lookup table\"\n extension to the bitmap format can reduce the overhead of loading\n bitmaps which stores a list of bitmapped commit id pos (in the midx\n or pack, along with their offset and xor offset. This way git can\n - load only the neccesary bitmaps without loading the previous bitmaps.\n + load only the necessary bitmaps without loading the previous bitmaps.\n \n - The older version of Git ignores the lookup table extension and doesn't\n + Older versions of Git ignore the lookup table extension and don't\n throw any kind of warning or error while parsing the bitmap file.\n \n Add some information for the new \"bitmap lookup table\" extension in the\n bitmap-format documentation.\n \n - Co-Authored-by: Taylor Blau \n Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n + Co-Authored-by: Taylor Blau \n Signed-off-by: Abhradeep Chakraborty \n \n ## Documentation/technical/bitmap-format.txt ##\n @@ Documentation/technical/bitmap-format.txt: MIDXs, both the bit-cache and rev-cac\n +\t\t\t** {empty}\n +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n +\t\t\tIf present, the end of the bitmap file contains a table\n -+\t\t\tcontaining a list of `N` \n ++\t\t\tcontaining a list of `N` \n +\t\t\ttriplets. The format and meaning of the table is described\n +\t\t\tbelow.\n ++\n -+NOTE: This xor_offset is different from the bitmap's xor_offset.\n -+Bitmap's xor_offset is relative i.e. it tells how many bitmaps we have\n -+to go back from the current bitmap. Lookup table's xor_offset tells the\n -+position of the triplet in the list whose bitmap the current commit's\n -+bitmap have to xor with.\n ++NOTE: Unlike the xor_offset used to compress an individual bitmap,\n ++`xor_row` stores an *absolute* index into the lookup table, not a location\n ++relative to the current entry.\n +\n \t\t4-byte entry count (network byte order)\n \n @@ Documentation/technical/bitmap-format.txt: Note that this hashing scheme is tied\n +-------------------\n +\n +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n -+(preceding the name-hash cache and trailing hash) of the `.bitmap` file\n -+contains a lookup table specifying the information needed to get the\n -+desired bitmap from the entries without parsing previous unnecessary\n ++bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`\n ++file contains a lookup table specifying the information needed to get\n ++the desired bitmap from the entries without parsing previous unnecessary\n +bitmaps.\n +\n +For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n -+contains a list of `nr_entries` triplets.\n -+The content of i'th triplet is -\n ++contains a list of `nr_entries` triplets\n ++(sorted in the ascending order of `commit_pos`). The content of i'th\n ++triplet is -\n +\n +\t* {empty}\n -+\tcommit pos (4 byte integer, network byte order): ::\n -+\tIt stores the object position of the commit (in the midx or pack index)\n -+\tto which the i'th bitmap in the bitmap entries belongs.\n ++\tcommit_pos (4 byte integer, network byte order): ::\n ++\tIt stores the object position of a commit (in the midx or pack\n ++\tindex).\n +\n +\t* {empty}\n +\toffset (8 byte integer, network byte order): ::\n +\tThe offset from which that commit's bitmap can be read.\n +\n +\t* {empty}\n -+\txor offset (4 byte integer, network byte order): ::\n -+\tIt holds the position of the triplet with whose bitmap the\n -+\tcurrent bitmap need to xor. If the current triplet's bitmap\n -+\tdo not have any xor bitmap, it defaults to 0xffffffff.\n ++\txor_row (4 byte integer, network byte order): ::\n ++\tThe position of the triplet whose bitmap is used to compress\n ++\tthis one, or `0xffffffff` if no such bitmap exists.\n 2: d118f1d45e6 ! 2: 5e9b985e39b pack-bitmap-write.c: write lookup table extension\n @@ Metadata\n ## Commit message ##\n pack-bitmap-write.c: write lookup table extension\n \n - The bitmap lookup table extension was documentated by an earlier\n - change, but Git does not yet knowhow to write that extension.\n + The bitmap lookup table extension was documented by an earlier\n + change, but Git does not yet know how to write that extension.\n \n - Teach git to write bitmap lookup table extension. The table contains\n - the list of `N` ` triplets. These\n + Teach Git to write bitmap lookup table extension. The table contains\n + the list of `N` ` triplets. These\n triplets are sorted according to their commit pos (ascending order).\n The meaning of each data in the i'th triplet is given below:\n \n - - Commit pos is the position of the commit in the pack-index\n - (or midx) to which the i'th bitmap belongs. It is a 4 byte\n - network byte order integer.\n + - commit_pos stores commit position (in the pack-index or midx).\n + It is a 4 byte network byte order unsigned integer.\n \n - - offset is the position of the i'th bitmap.\n + - offset is the position (in the bitmap file) from which that\n + commit's bitmap can be read.\n \n - - xor offset denotes the position of the triplet with whose\n - bitmap the current triplet's bitmap need to xor with.\n + - xor_row is the position of the triplet in the lookup table\n + whose bitmap is used to compress this bitmap, or `0xffffffff`\n + if no such bitmap exists.\n \n - Co-authored-by: Taylor Blau \n Mentored-by: Taylor Blau \n Co-mentored-by: Kaartic Sivaraam \n + Co-authored-by: Taylor Blau \n Signed-off-by: Abhradeep Chakraborty \n \n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: static const struct object_id *oid_access(size_t pos, const void *table)\n + \treturn &index[pos]->oid;\n + }\n \n ++static int commit_bitmap_writer_pos(struct object_id *oid,\n ++\t\t\t\t struct pack_idx_entry **index,\n ++\t\t\t\t uint32_t index_nr)\n ++{\n ++\treturn oid_pos(oid, index, index_nr, oid_access);\n ++}\n ++\n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n -\t\t\t\t uint32_t index_nr)\n +\t\t\t\t uint32_t index_nr,\n -+\t\t\t\t uint64_t *offsets,\n -+\t\t\t\t uint32_t *commit_positions)\n ++\t\t\t\t off_t *offsets)\n {\n \tint i;\n \n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n + \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n + \n + \t\tint commit_pos =\n +-\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n ++\t\t\tcommit_bitmap_writer_pos(&stored->commit->object.oid, index, index_nr);\n + \n \t\tif (commit_pos < 0)\n \t\t\tBUG(\"trying to write commit not in index\");\n \n +\t\tif (offsets)\n +\t\t\toffsets[i] = hashfile_total(f);\n -+\t\tif (commit_positions)\n -+\t\t\tcommit_positions[i] = commit_pos;\n +\n \t\thashwrite_be32(f, commit_pos);\n \t\thashwrite_u8(f, stored->xor_offset);\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n -+static int table_cmp(const void *_va, const void *_vb, void *commit_positions)\n ++static int table_cmp(const void *_va, const void *_vb, void *_data)\n +{\n -+\tint8_t result = 0;\n -+\tuint32_t *positions = (uint32_t *) commit_positions;\n -+\tuint32_t a = positions[*(uint32_t *)_va];\n -+\tuint32_t b = positions[*(uint32_t *)_vb];\n ++\tuint32_t *commit_positions = _data;\n ++\tuint32_t a = commit_positions[*(uint32_t *)_va];\n ++\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n +\n +\tif (a > b)\n -+\t\tresult = 1;\n ++\t\treturn 1;\n +\telse if (a < b)\n -+\t\tresult = -1;\n -+\telse\n -+\t\tresult = 0;\n ++\t\treturn -1;\n +\n -+\treturn result;\n ++\treturn 0;\n +}\n +\n +static void write_lookup_table(struct hashfile *f,\n -+\t\t\t uint64_t *offsets,\n -+\t\t\t uint32_t *commit_positions)\n ++\t\t\t struct pack_idx_entry **index,\n ++\t\t\t uint32_t index_nr,\n ++\t\t\t off_t *offsets)\n +{\n +\tuint32_t i;\n -+\tuint32_t *table, *table_inv;\n ++\tuint32_t *table, *table_inv, *commit_positions;\n +\n +\tALLOC_ARRAY(table, writer.selected_nr);\n +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n ++\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n ++\n ++\t/* store the index positions of the commits */\n ++\tfor (i = 0; i < writer.selected_nr; i++) {\n ++\t\tint pos = commit_bitmap_writer_pos(&writer.selected[i].commit->object.oid,\n ++\t\t\t\t\t\t index, index_nr);\n ++\t\tif (pos < 0)\n ++\t\t\tBUG(_(\"trying to write commit not in index\"));\n ++\n ++\t\tcommit_positions[i] = pos;\n ++\t}\n +\n +\tfor (i = 0; i < writer.selected_nr; i++)\n +\t\ttable[i] = i;\n +\n ++\t/*\n ++\t * At the end of this sort table[j] = i means that the i'th\n ++\t * bitmap corresponds to j'th bitmapped commit in lex order of\n ++\t * OIDs.\n ++\t */\n +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n +\n ++\t/* table_inv helps us discover that relationship (i'th bitmap\n ++\t * to j'th commit by j = table_inv[i])\n ++\t */\n +\tfor (i = 0; i < writer.selected_nr; i++)\n +\t\ttable_inv[table[i]] = i;\n +\n ++\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n +\tfor (i = 0; i < writer.selected_nr; i++) {\n +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n +\t\tuint32_t xor_offset = selected->xor_offset;\n ++\t\tuint32_t xor_row;\n ++\n ++\t\tif (xor_offset) {\n ++\t\t\t/*\n ++\t\t\t * xor_index stores the index (in the bitmap entries)\n ++\t\t\t * of the corresponding xor bitmap. But we need to convert\n ++\t\t\t * this index into lookup table's index. So, table_inv[xor_index]\n ++\t\t\t * gives us the index position w.r.t. the lookup table.\n ++\t\t\t *\n ++\t\t\t * If \"k = table[i] - xor_offset\" then the xor base is the k'th\n ++\t\t\t * bitmap. `table_inv[k]` gives us the position of that bitmap\n ++\t\t\t * in the lookup table.\n ++\t\t\t */\n ++\t\t\tuint32_t xor_index = table[i] - xor_offset;\n ++\t\t\txor_row = table_inv[xor_index];\n ++\t\t} else {\n ++\t\t\txor_row = 0xffffffff;\n ++\t\t}\n +\n +\t\thashwrite_be32(f, commit_positions[table[i]]);\n -+\t\thashwrite_be64(f, offsets[table[i]]);\n -+\t\thashwrite_be32(f, xor_offset ?\n -+\t\t\t\ttable_inv[table[i] - xor_offset]: 0xffffffff);\n ++\t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n ++\t\thashwrite_be32(f, xor_row);\n +\t}\n ++\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n +\n +\tfree(table);\n +\tfree(table_inv);\n ++\tfree(commit_positions);\n +}\n +\n static void write_hash_cache(struct hashfile *f,\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n -+\tuint64_t *offsets = NULL;\n -+\tuint32_t *commit_positions = NULL;\n ++\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n -\twrite_selected_commits_v1(f, index, index_nr);\n - \n -+\tif (options & BITMAP_OPT_LOOKUP_TABLE) {\n ++\n ++\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n +\t\tCALLOC_ARRAY(offsets, index_nr);\n -+\t\tCALLOC_ARRAY(commit_positions, index_nr);\n -+\t}\n +\n -+\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n ++\twrite_selected_commits_v1(f, index, index_nr, offsets);\n +\n +\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n -+\t\twrite_lookup_table(f, offsets, commit_positions);\n ++\t\twrite_lookup_table(f, index, index_nr, offsets);\n + \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n - \n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n +\tfree(offsets);\n -+\tfree(commit_positions);\n }\n \n ## pack-bitmap.h ##\n 3: 7786dc879f0 ! 3: 3dc40cc7f73 pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n @@ Metadata\n ## Commit message ##\n pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n \n - Teach git to provide a way for users to enable/disable bitmap lookup\n + Teach Git to provide a way for users to enable/disable bitmap lookup\n table extension by providing a config option named 'writeBitmapLookupTable'.\n - Default is true.\n + Default is false.\n \n Also add test to verify writting of lookup table.\n \n - Co-Authored-by: Taylor Blau \n - Signed-off-by: Abhradeep Chakraborty \n Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n + Co-Authored-by: Taylor Blau \n + Signed-off-by: Abhradeep Chakraborty \n \n ## Documentation/config/pack.txt ##\n @@ Documentation/config/pack.txt: When writing a multi-pack reachability bitmap, no new namehashes are\n @@ Documentation/config/pack.txt: When writing a multi-pack reachability bitmap, no\n permuted into their appropriate location when writing a new bitmap.\n \n +pack.writeBitmapLookupTable::\n -+\tWhen true, git will include a \"lookup table\" section in the\n ++\tWhen true, Git will include a \"lookup table\" section in the\n +\tbitmap index (if one is written). This table is used to defer\n +\tloading individual bitmaps as late as possible. This can be\n -+\tbeneficial in repositories which have relatively large bitmap\n -+\tindexes. Defaults to true.\n ++\tbeneficial in repositories that have relatively large bitmap\n ++\tindexes. Defaults to false.\n +\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n @@ builtin/multi-pack-index.c: static int git_multi_pack_index_write_config(const c\n \t/*\n \t * We should never make a fall-back call to 'git_default_config', since\n \t * this was already called in 'cmd_multi_pack_index()'.\n -@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv)\n - \t};\n - \n - \topts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE;\n -+\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n - \n - \tgit_config(git_multi_pack_index_write_config, NULL);\n - \n \n ## builtin/pack-objects.c ##\n -@@ builtin/pack-objects.c: static enum {\n - \tWRITE_BITMAP_QUIET,\n - \tWRITE_BITMAP_TRUE,\n - } write_bitmap_index;\n --static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE;\n -+static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE | BITMAP_OPT_LOOKUP_TABLE;\n - \n - static int exclude_promisor_objects;\n - \n @@ builtin/pack-objects.c: static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n @@ midx.h: struct multi_pack_index {\n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\n \n - ## pack-bitmap-write.c ##\n -@@ pack-bitmap-write.c: static void write_lookup_table(struct hashfile *f,\n - \tfor (i = 0; i < writer.selected_nr; i++)\n - \t\ttable_inv[table[i]] = i;\n - \n -+\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n - \tfor (i = 0; i < writer.selected_nr; i++) {\n - \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n - \t\tuint32_t xor_offset = selected->xor_offset;\n -@@ pack-bitmap-write.c: static void write_lookup_table(struct hashfile *f,\n - \n - \tfree(table);\n - \tfree(table_inv);\n -+\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n + ## t/t5310-pack-bitmaps.sh ##\n +@@ t/t5310-pack-bitmaps.sh: has_any () {\n + \tgrep -Ff \"$1\" \"$2\"\n }\n \n - static void write_hash_cache(struct hashfile *f,\n -\n - ## t/t5310-pack-bitmaps.sh ##\n -@@ t/t5310-pack-bitmaps.sh: test_expect_success 'full repack creates bitmaps' '\n - \tls .git/objects/pack/ | grep bitmap >output &&\n - \ttest_line_count = 1 output &&\n - \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n +-setup_bitmap_history\n +-\n +-test_expect_success 'setup writing bitmaps during repack' '\n +-\tgit config repack.writeBitmaps true\n +-'\n +-\n +-test_expect_success 'full repack creates bitmaps' '\n +-\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n ++test_bitmap_cases () {\n ++\twriteLookupTable=false\n ++\tfor i in \"$@\"\n ++\tdo\n ++\t\tcase \"$i\" in\n ++\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n ++\t\tesac\n ++\tdone\n ++\n ++\ttest_expect_success 'setup test repository' '\n ++\t\trm -fr * .git &&\n ++\t\tgit init &&\n ++\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n ++\t'\n ++\tsetup_bitmap_history\n ++\n ++\ttest_expect_success 'setup writing bitmaps during repack' '\n ++\t\tgit config repack.writeBitmaps true\n ++\t'\n ++\n ++\ttest_expect_success 'full repack creates bitmaps' '\n ++\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n ++\t\t\tgit repack -ad &&\n ++\t\tls .git/objects/pack/ | grep bitmap >output &&\n ++\t\ttest_line_count = 1 output &&\n ++\t\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n ++\t\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n ++\t'\n ++\n ++\tbasic_bitmap_tests\n ++\n ++\ttest_expect_success 'pack-objects respects --local (non-local loose)' '\n ++\t\tgit init --bare alt.git &&\n ++\t\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n ++\t\techo content1 >file1 &&\n ++\t\t# non-local loose object which is not present in bitmapped pack\n ++\t\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n ++\t\t# non-local loose object which is also present in bitmapped pack\n ++\t\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n ++\t\tgit add file1 &&\n ++\t\ttest_tick &&\n ++\t\tgit commit -m commit_file1 &&\n ++\t\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n ++\t\tgit index-pack 1.pack &&\n ++\t\tlist_packed_objects 1.idx >1.objects &&\n ++\t\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n ++\t\t! has_any nonlocal-loose 1.objects\n ++\t'\n ++\n ++\ttest_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n ++\t\techo content2 >file2 &&\n ++\t\tblob2=$(git hash-object -w file2) &&\n ++\t\tgit add file2 &&\n ++\t\ttest_tick &&\n ++\t\tgit commit -m commit_file2 &&\n ++\t\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n ++\t\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n ++\t\trm $(objpath $blob2) &&\n ++\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n ++\t\tgit index-pack 2a.pack &&\n ++\t\tlist_packed_objects 2a.idx >2a.objects &&\n ++\t\t! has_any keepobjects 2a.objects\n ++\t'\n ++\n ++\ttest_expect_success 'pack-objects respects --local (non-local pack)' '\n ++\t\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n ++\t\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n ++\t\tgit index-pack 2b.pack &&\n ++\t\tlist_packed_objects 2b.idx >2b.objects &&\n ++\t\t! has_any keepobjects 2b.objects\n ++\t'\n ++\n ++\ttest_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n ++\t\tls .git/objects/pack/ | grep bitmap >output &&\n ++\t\ttest_line_count = 1 output &&\n ++\t\tpackbitmap=$(basename $(cat output) .bitmap) &&\n ++\t\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n ++\t\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n ++\t\t>.git/objects/pack/$packbitmap.keep &&\n ++\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n ++\t\tgit index-pack 3a.pack &&\n ++\t\tlist_packed_objects 3a.idx >3a.objects &&\n ++\t\t! has_any packbitmap.objects 3a.objects\n ++\t'\n ++\n ++\ttest_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n ++\t\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n ++\t\trm -f .git/objects/pack/multi-pack-index &&\n ++\t\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n ++\t\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n ++\t\tgit index-pack 3b.pack &&\n ++\t\tlist_packed_objects 3b.idx >3b.objects &&\n ++\t\t! has_any packbitmap.objects 3b.objects\n ++\t'\n ++\n ++\ttest_expect_success 'pack-objects to file can use bitmap' '\n ++\t\t# make sure we still have 1 bitmap index from previous tests\n ++\t\tls .git/objects/pack/ | grep bitmap >output &&\n ++\t\ttest_line_count = 1 output &&\n ++\t\t# verify equivalent packs are generated with/without using bitmap index\n ++\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n ++\t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n ++\t\ttest_cmp packa.objects packb.objects\n ++\t'\n ++\n ++\ttest_expect_success 'full repack, reusing previous bitmaps' '\n + \t\tgit repack -ad &&\n +-\tls .git/objects/pack/ | grep bitmap >output &&\n +-\ttest_line_count = 1 output &&\n +-\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n -\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n -+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace &&\n -+\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n +-'\n ++\t\tls .git/objects/pack/ | grep bitmap >output &&\n ++\t\ttest_line_count = 1 output\n ++\t'\n ++\n ++\ttest_expect_success 'fetch (full bitmap)' '\n ++\t\tgit --git-dir=clone.git fetch origin second:second &&\n ++\t\tgit rev-parse HEAD >expect &&\n ++\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n ++\t\ttest_cmp expect actual\n ++\t'\n ++\n ++\ttest_expect_success 'create objects for missing-HAVE tests' '\n ++\t\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n ++\t\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n ++\t\tparent=$(echo parent | git commit-tree $tree) &&\n ++\t\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n ++\t\tcat >revs <<-EOF\n ++\t\tHEAD\n ++\t\t^HEAD^\n ++\t\t^$commit\n ++\t\tEOF\n ++\t'\n ++\n ++\ttest_expect_success 'pack-objects respects --incremental' '\n ++\t\tcat >revs2 <<-EOF &&\n ++\t\tHEAD\n ++\t\t$commit\n ++\t\tEOF\n ++\t\tgit pack-objects --incremental --stdout --revs 4.pack &&\n ++\t\tgit index-pack 4.pack &&\n ++\t\tlist_packed_objects 4.idx >4.objects &&\n ++\t\ttest_line_count = 4 4.objects &&\n ++\t\tgit rev-list --objects $commit >revlist &&\n ++\t\tcut -d\" \" -f1 revlist |sort >objects &&\n ++\t\ttest_cmp 4.objects objects\n ++\t'\n ++\n ++\ttest_expect_success 'pack with missing blob' '\n ++\t\trm $(objpath $blob) &&\n ++\t\tgit pack-objects --stdout --revs /dev/null\n ++\t'\n ++\n ++\ttest_expect_success 'pack with missing tree' '\n ++\t\trm $(objpath $tree) &&\n ++\t\tgit pack-objects --stdout --revs /dev/null\n ++\t'\n ++\n ++\ttest_expect_success 'pack with missing parent' '\n ++\t\trm $(objpath $parent) &&\n ++\t\tgit pack-objects --stdout --revs /dev/null\n ++\t'\n ++\n ++\ttest_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n ++\t\tgit clone --bare . compat-jgit.git &&\n ++\t\t(\n ++\t\t\tcd compat-jgit.git &&\n ++\t\t\trm -f objects/pack/*.bitmap &&\n ++\t\t\tjgit gc &&\n ++\t\t\tgit rev-list --test-bitmap HEAD\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n ++\t\tgit clone --bare . compat-us.git &&\n ++\t\t(\n ++\t\t\tcd compat-us.git &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n ++\t\t\tgit repack -adb &&\n ++\t\t\t# jgit gc will barf if it does not like our bitmaps\n ++\t\t\tjgit gc\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success 'splitting packs does not generate bogus bitmaps' '\n ++\t\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n ++\t\tgit add rand &&\n ++\t\tgit commit -m \"commit with big file\" &&\n ++\t\tgit -c pack.packSizeLimit=500k repack -adb &&\n ++\t\tgit init --bare no-bitmaps.git &&\n ++\t\tgit -C no-bitmaps.git fetch .. HEAD\n ++\t'\n ++\n ++\ttest_expect_success 'set up reusable pack' '\n ++\t\trm -f .git/objects/pack/*.keep &&\n ++\t\tgit repack -adb &&\n ++\t\treusable_pack () {\n ++\t\t\tgit for-each-ref --format=\"%(objectname)\" |\n ++\t\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n ++\t\t}\n ++\t'\n ++\n ++\ttest_expect_success 'pack reuse respects --honor-pack-keep' '\n ++\t\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n ++\t\tfor i in .git/objects/pack/*.pack\n ++\t\tdo\n ++\t\t\t>${i%.pack}.keep || return 1\n ++\t\tdone &&\n ++\t\treusable_pack --honor-pack-keep >empty.pack &&\n ++\t\tgit index-pack empty.pack &&\n ++\t\tgit show-index actual &&\n ++\t\ttest_must_be_empty actual\n ++\t'\n ++\n ++\ttest_expect_success 'pack reuse respects --local' '\n ++\t\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n ++\t\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n ++\t\treusable_pack --local >empty.pack &&\n ++\t\tgit index-pack empty.pack &&\n ++\t\tgit show-index actual &&\n ++\t\ttest_must_be_empty actual\n ++\t'\n ++\n ++\ttest_expect_success 'pack reuse respects --incremental' '\n ++\t\treusable_pack --incremental >empty.pack &&\n ++\t\tgit index-pack empty.pack &&\n ++\t\tgit show-index actual &&\n ++\t\ttest_must_be_empty actual\n ++\t'\n ++\n ++\ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n ++\t\ttest_config pack.writebitmaphashcache false &&\n ++\t\tgit repack -ad &&\n ++\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n ++\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n ++\t\ttest_when_finished \"rm -f $bitmap\" &&\n ++\t\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n ++\t\tmv -f $bitmap.tmp $bitmap &&\n ++\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n ++\t\ttest_cmp expect actual &&\n ++\t\ttest_i18ngrep corrupt.ewah.bitmap stderr\n ++\t'\n ++\n ++\ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n ++\t\tgit repack -ad &&\n ++\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n ++\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n ++\t\ttest_when_finished \"rm -f $bitmap\" &&\n ++\t\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n ++\t\tmv -f $bitmap.tmp $bitmap &&\n ++\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n ++\t\ttest_cmp expect actual &&\n ++\t\ttest_i18ngrep corrupted.bitmap.index stderr\n ++\t'\n ++\n ++\t# Create a state of history with these properties:\n ++\t#\n ++\t# - refs that allow a client to fetch some new history, while sharing some old\n ++\t# history with the server; we use branches delta-reuse-old and\n ++\t# delta-reuse-new here\n ++\t#\n ++\t# - the new history contains an object that is stored on the server as a delta\n ++\t# against a base that is in the old history\n ++\t#\n ++\t# - the base object is not immediately reachable from the tip of the old\n ++\t# history; finding it would involve digging down through history we know the\n ++\t# other side has\n ++\t#\n ++\t# This should result in a state where fetching from old->new would not\n ++\t# traditionally reuse the on-disk delta (because we'd have to dig to realize\n ++\t# that the client has it), but we will do so if bitmaps can tell us cheaply\n ++\t# that the other side has it.\n ++\ttest_expect_success 'set up thin delta-reuse parent' '\n ++\t\t# This first commit contains the buried base object.\n ++\t\ttest-tool genrandom delta 16384 >file &&\n ++\t\tgit add file &&\n ++\t\tgit commit -m \"delta base\" &&\n ++\t\tbase=$(git rev-parse --verify HEAD:file) &&\n ++\n ++\t\t# These intermediate commits bury the base back in history.\n ++\t\t# This becomes the \"old\" state.\n ++\t\tfor i in 1 2 3 4 5\n ++\t\tdo\n ++\t\t\techo $i >file &&\n ++\t\t\tgit commit -am \"intermediate $i\" || return 1\n ++\t\tdone &&\n ++\t\tgit branch delta-reuse-old &&\n ++\n ++\t\t# And now our new history has a delta against the buried base. Note\n ++\t\t# that this must be smaller than the original file, since pack-objects\n ++\t\t# prefers to create deltas from smaller objects to larger.\n ++\t\ttest-tool genrandom delta 16300 >file &&\n ++\t\tgit commit -am \"delta result\" &&\n ++\t\tdelta=$(git rev-parse --verify HEAD:file) &&\n ++\t\tgit branch delta-reuse-new &&\n ++\n ++\t\t# Repack with bitmaps and double check that we have the expected delta\n ++\t\t# relationship.\n ++\t\tgit repack -adb &&\n ++\t\thave_delta $delta $base\n ++\t'\n ++\n ++\t# Now we can sanity-check the non-bitmap behavior (that the server is not able\n ++\t# to reuse the delta). This isn't strictly something we care about, so this\n ++\t# test could be scrapped in the future. But it makes sure that the next test is\n ++\t# actually triggering the feature we want.\n ++\t#\n ++\t# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n ++\t# we actually perform the fetch, retain the resulting pack, and inspect the\n ++\t# result.\n ++\ttest_expect_success 'fetch without bitmaps ignores delta against old base' '\n ++\t\ttest_config pack.usebitmaps false &&\n ++\t\ttest_when_finished \"rm -rf client.git\" &&\n ++\t\tgit init --bare client.git &&\n ++\t\t(\n ++\t\t\tcd client.git &&\n ++\t\t\tgit config transfer.unpackLimit 1 &&\n ++\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n ++\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n ++\t\t\thave_delta $delta $ZERO_OID\n ++\t\t)\n ++\t'\n ++\n ++\t# And do the same for the bitmap case, where we do expect to find the delta.\n ++\ttest_expect_success 'fetch with bitmaps can reuse old base' '\n ++\t\ttest_config pack.usebitmaps true &&\n ++\t\ttest_when_finished \"rm -rf client.git\" &&\n ++\t\tgit init --bare client.git &&\n ++\t\t(\n ++\t\t\tcd client.git &&\n ++\t\t\tgit config transfer.unpackLimit 1 &&\n ++\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n ++\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n ++\t\t\thave_delta $delta $base\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success 'pack.preferBitmapTips' '\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n ++\n ++\t\t\t# create enough commits that not all are receive bitmap\n ++\t\t\t# coverage even if they are all at the tip of some reference.\n ++\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n ++\n ++\t\t\tgit rev-list HEAD >commits.raw &&\n ++\t\t\tsort commits &&\n ++\n ++\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n ++\t\t\tgit update-ref --stdin bitmaps &&\n ++\n ++\t\t\t# remember which commits did not receive bitmaps\n ++\t\t\tcomm -13 bitmaps commits >before &&\n ++\t\t\ttest_file_not_empty before &&\n ++\n ++\t\t\t# mark the commits which did not receive bitmaps as preferred,\n ++\t\t\t# and generate the bitmap again\n ++\t\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n ++\t\t\tcomm -13 bitmaps commits >after &&\n ++\n ++\t\t\t! test_cmp before after\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success 'complains about multiple pack bitmaps' '\n ++\t\trm -fr repo &&\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n ++\n ++\t\t\ttest_commit base &&\n ++\n ++\t\t\tgit repack -adb &&\n ++\t\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n ++\t\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n ++\n ++\t\t\ttest_commit other &&\n ++\t\t\tgit repack -ab &&\n ++\n ++\t\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n ++\n ++\t\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n ++\t\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n ++\t\t\ttest_line_count = 2 packs &&\n ++\t\t\ttest_line_count = 2 bitmaps &&\n ++\n ++\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n ++\t\t\tgrep \"ignoring extra bitmap file\" err\n ++\t\t)\n ++\t'\n ++}\n + \n +-basic_bitmap_tests\n ++test_bitmap_cases\n + \n + test_expect_success 'incremental repack fails when bitmaps are requested' '\n + \ttest_commit more-1 &&\n +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'incremental repack can disable bitmaps' '\n + \tgit repack -d --no-write-bitmap-index\n + '\n + \n +-test_expect_success 'pack-objects respects --local (non-local loose)' '\n +-\tgit init --bare alt.git &&\n +-\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n +-\techo content1 >file1 &&\n +-\t# non-local loose object which is not present in bitmapped pack\n +-\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n +-\t# non-local loose object which is also present in bitmapped pack\n +-\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n +-\tgit add file1 &&\n +-\ttest_tick &&\n +-\tgit commit -m commit_file1 &&\n +-\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n +-\tgit index-pack 1.pack &&\n +-\tlist_packed_objects 1.idx >1.objects &&\n +-\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n +-\t! has_any nonlocal-loose 1.objects\n +-'\n +-\n +-test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n +-\techo content2 >file2 &&\n +-\tblob2=$(git hash-object -w file2) &&\n +-\tgit add file2 &&\n +-\ttest_tick &&\n +-\tgit commit -m commit_file2 &&\n +-\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n +-\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n +-\trm $(objpath $blob2) &&\n +-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n +-\tgit index-pack 2a.pack &&\n +-\tlist_packed_objects 2a.idx >2a.objects &&\n +-\t! has_any keepobjects 2a.objects\n +-'\n +-\n +-test_expect_success 'pack-objects respects --local (non-local pack)' '\n +-\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n +-\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n +-\tgit index-pack 2b.pack &&\n +-\tlist_packed_objects 2b.idx >2b.objects &&\n +-\t! has_any keepobjects 2b.objects\n +-'\n +-\n +-test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n +-\tls .git/objects/pack/ | grep bitmap >output &&\n +-\ttest_line_count = 1 output &&\n +-\tpackbitmap=$(basename $(cat output) .bitmap) &&\n +-\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n +-\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n +-\t>.git/objects/pack/$packbitmap.keep &&\n +-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n +-\tgit index-pack 3a.pack &&\n +-\tlist_packed_objects 3a.idx >3a.objects &&\n +-\t! has_any packbitmap.objects 3a.objects\n +-'\n +-\n +-test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n +-\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n +-\trm -f .git/objects/pack/multi-pack-index &&\n +-\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n +-\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n +-\tgit index-pack 3b.pack &&\n +-\tlist_packed_objects 3b.idx >3b.objects &&\n +-\t! has_any packbitmap.objects 3b.objects\n +-'\n +-\n +-test_expect_success 'pack-objects to file can use bitmap' '\n +-\t# make sure we still have 1 bitmap index from previous tests\n +-\tls .git/objects/pack/ | grep bitmap >output &&\n +-\ttest_line_count = 1 output &&\n +-\t# verify equivalent packs are generated with/without using bitmap index\n +-\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n +-\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n +-\ttest_cmp packa.objects packb.objects\n +-'\n +-\n +-test_expect_success 'full repack, reusing previous bitmaps' '\n +-\tgit repack -ad &&\n +-\tls .git/objects/pack/ | grep bitmap >output &&\n +-\ttest_line_count = 1 output\n +-'\n +-\n +-test_expect_success 'fetch (full bitmap)' '\n +-\tgit --git-dir=clone.git fetch origin second:second &&\n +-\tgit rev-parse HEAD >expect &&\n +-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n +-\ttest_cmp expect actual\n +-'\n +-\n +-test_expect_success 'create objects for missing-HAVE tests' '\n +-\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n +-\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n +-\tparent=$(echo parent | git commit-tree $tree) &&\n +-\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n +-\tcat >revs <<-EOF\n +-\tHEAD\n +-\t^HEAD^\n +-\t^$commit\n +-\tEOF\n +-'\n +-\n +-test_expect_success 'pack-objects respects --incremental' '\n +-\tcat >revs2 <<-EOF &&\n +-\tHEAD\n +-\t$commit\n +-\tEOF\n +-\tgit pack-objects --incremental --stdout --revs 4.pack &&\n +-\tgit index-pack 4.pack &&\n +-\tlist_packed_objects 4.idx >4.objects &&\n +-\ttest_line_count = 4 4.objects &&\n +-\tgit rev-list --objects $commit >revlist &&\n +-\tcut -d\" \" -f1 revlist |sort >objects &&\n +-\ttest_cmp 4.objects objects\n +-'\n +-\n +-test_expect_success 'pack with missing blob' '\n +-\trm $(objpath $blob) &&\n +-\tgit pack-objects --stdout --revs /dev/null\n +-'\n ++test_bitmap_cases \"pack.writeBitmapLookupTable\"\n + \n +-test_expect_success 'pack with missing tree' '\n +-\trm $(objpath $tree) &&\n +-\tgit pack-objects --stdout --revs /dev/null\n +-'\n +-\n +-test_expect_success 'pack with missing parent' '\n +-\trm $(objpath $parent) &&\n +-\tgit pack-objects --stdout --revs /dev/null\n +-'\n +-\n +-test_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n +-\tgit clone --bare . compat-jgit.git &&\n +-\t(\n +-\t\tcd compat-jgit.git &&\n +-\t\trm -f objects/pack/*.bitmap &&\n +-\t\tjgit gc &&\n +-\t\tgit rev-list --test-bitmap HEAD\n +-\t)\n +-'\n +-\n +-test_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n +-\tgit clone --bare . compat-us.git &&\n +-\t(\n +-\t\tcd compat-us.git &&\n +-\t\tgit repack -adb &&\n +-\t\t# jgit gc will barf if it does not like our bitmaps\n +-\t\tjgit gc\n +-\t)\n +-'\n +-\n +-test_expect_success 'splitting packs does not generate bogus bitmaps' '\n +-\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n +-\tgit add rand &&\n +-\tgit commit -m \"commit with big file\" &&\n +-\tgit -c pack.packSizeLimit=500k repack -adb &&\n +-\tgit init --bare no-bitmaps.git &&\n +-\tgit -C no-bitmaps.git fetch .. HEAD\n +-'\n +-\n +-test_expect_success 'set up reusable pack' '\n +-\trm -f .git/objects/pack/*.keep &&\n +-\tgit repack -adb &&\n +-\treusable_pack () {\n +-\t\tgit for-each-ref --format=\"%(objectname)\" |\n +-\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n +-\t}\n +-'\n +-\n +-test_expect_success 'pack reuse respects --honor-pack-keep' '\n +-\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n +-\tfor i in .git/objects/pack/*.pack\n +-\tdo\n +-\t\t>${i%.pack}.keep || return 1\n +-\tdone &&\n +-\treusable_pack --honor-pack-keep >empty.pack &&\n +-\tgit index-pack empty.pack &&\n +-\tgit show-index actual &&\n +-\ttest_must_be_empty actual\n +-'\n +-\n +-test_expect_success 'pack reuse respects --local' '\n +-\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n +-\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n +-\treusable_pack --local >empty.pack &&\n +-\tgit index-pack empty.pack &&\n +-\tgit show-index actual &&\n +-\ttest_must_be_empty actual\n +-'\n +-\n +-test_expect_success 'pack reuse respects --incremental' '\n +-\treusable_pack --incremental >empty.pack &&\n +-\tgit index-pack empty.pack &&\n +-\tgit show-index actual &&\n +-\ttest_must_be_empty actual\n +-'\n +-\n +-test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n +-\ttest_config pack.writebitmaphashcache false &&\n +-\tgit repack -ad &&\n +-\tgit rev-list --use-bitmap-index --count --all >expect &&\n +-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n +-\ttest_when_finished \"rm -f $bitmap\" &&\n +-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n +-\tmv -f $bitmap.tmp $bitmap &&\n +-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n +-\ttest_cmp expect actual &&\n +-\ttest_i18ngrep corrupt.ewah.bitmap stderr\n +-'\n +-\n +-test_expect_success 'truncated bitmap fails gracefully (cache)' '\n +-\tgit repack -ad &&\n +-\tgit rev-list --use-bitmap-index --count --all >expect &&\n +-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n +-\ttest_when_finished \"rm -f $bitmap\" &&\n +-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n +-\tmv -f $bitmap.tmp $bitmap &&\n +-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n +-\ttest_cmp expect actual &&\n +-\ttest_i18ngrep corrupted.bitmap.index stderr\n +-'\n +-\n +-# Create a state of history with these properties:\n +-#\n +-# - refs that allow a client to fetch some new history, while sharing some old\n +-# history with the server; we use branches delta-reuse-old and\n +-# delta-reuse-new here\n +-#\n +-# - the new history contains an object that is stored on the server as a delta\n +-# against a base that is in the old history\n +-#\n +-# - the base object is not immediately reachable from the tip of the old\n +-# history; finding it would involve digging down through history we know the\n +-# other side has\n +-#\n +-# This should result in a state where fetching from old->new would not\n +-# traditionally reuse the on-disk delta (because we'd have to dig to realize\n +-# that the client has it), but we will do so if bitmaps can tell us cheaply\n +-# that the other side has it.\n +-test_expect_success 'set up thin delta-reuse parent' '\n +-\t# This first commit contains the buried base object.\n +-\ttest-tool genrandom delta 16384 >file &&\n +-\tgit add file &&\n +-\tgit commit -m \"delta base\" &&\n +-\tbase=$(git rev-parse --verify HEAD:file) &&\n +-\n +-\t# These intermediate commits bury the base back in history.\n +-\t# This becomes the \"old\" state.\n +-\tfor i in 1 2 3 4 5\n +-\tdo\n +-\t\techo $i >file &&\n +-\t\tgit commit -am \"intermediate $i\" || return 1\n +-\tdone &&\n +-\tgit branch delta-reuse-old &&\n +-\n +-\t# And now our new history has a delta against the buried base. Note\n +-\t# that this must be smaller than the original file, since pack-objects\n +-\t# prefers to create deltas from smaller objects to larger.\n +-\ttest-tool genrandom delta 16300 >file &&\n +-\tgit commit -am \"delta result\" &&\n +-\tdelta=$(git rev-parse --verify HEAD:file) &&\n +-\tgit branch delta-reuse-new &&\n +-\n +-\t# Repack with bitmaps and double check that we have the expected delta\n +-\t# relationship.\n +-\tgit repack -adb &&\n +-\thave_delta $delta $base\n +-'\n +-\n +-# Now we can sanity-check the non-bitmap behavior (that the server is not able\n +-# to reuse the delta). This isn't strictly something we care about, so this\n +-# test could be scrapped in the future. But it makes sure that the next test is\n +-# actually triggering the feature we want.\n +-#\n +-# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n +-# we actually perform the fetch, retain the resulting pack, and inspect the\n +-# result.\n +-test_expect_success 'fetch without bitmaps ignores delta against old base' '\n +-\ttest_config pack.usebitmaps false &&\n +-\ttest_when_finished \"rm -rf client.git\" &&\n +-\tgit init --bare client.git &&\n +-\t(\n +-\t\tcd client.git &&\n +-\t\tgit config transfer.unpackLimit 1 &&\n +-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n +-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n +-\t\thave_delta $delta $ZERO_OID\n +-\t)\n +-'\n +-\n +-# And do the same for the bitmap case, where we do expect to find the delta.\n +-test_expect_success 'fetch with bitmaps can reuse old base' '\n +-\ttest_config pack.usebitmaps true &&\n +-\ttest_when_finished \"rm -rf client.git\" &&\n +-\tgit init --bare client.git &&\n +-\t(\n +-\t\tcd client.git &&\n +-\t\tgit config transfer.unpackLimit 1 &&\n +-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n +-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n +-\t\thave_delta $delta $base\n +-\t)\n +-'\n +-\n +-test_expect_success 'pack.preferBitmapTips' '\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n +-\n +-\t\t# create enough commits that not all are receive bitmap\n +-\t\t# coverage even if they are all at the tip of some reference.\n +-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n +-\n +-\t\tgit rev-list HEAD >commits.raw &&\n +-\t\tsort commits &&\n +-\n +-\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n +-\t\tgit update-ref --stdin bitmaps &&\n +-\n +-\t\t# remember which commits did not receive bitmaps\n +-\t\tcomm -13 bitmaps commits >before &&\n +-\t\ttest_file_not_empty before &&\n +-\n +-\t\t# mark the commits which did not receive bitmaps as preferred,\n +-\t\t# and generate the bitmap again\n +-\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n +-\t\tcomm -13 bitmaps commits >after &&\n +-\n +-\t\t! test_cmp before after\n +-\t)\n +-'\n +-\n +-test_expect_success 'complains about multiple pack bitmaps' '\n +-\trm -fr repo &&\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n +-\n +-\t\ttest_commit base &&\n +-\n +-\t\tgit repack -adb &&\n +-\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n +-\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n +-\n +-\t\ttest_commit other &&\n +-\t\tgit repack -ab &&\n +-\n +-\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n +-\n +-\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n +-\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n +-\t\ttest_line_count = 2 packs &&\n +-\t\ttest_line_count = 2 bitmaps &&\n +-\n +-\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n +-\t\tgrep \"ignoring extra bitmap file\" err\n +-\t)\n ++test_expect_success 'verify writing bitmap lookup table when enabled' '\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace2\" \\\n ++\t\tgit repack -ad &&\n ++\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n - basic_bitmap_tests\n + test_done\n +\n + ## t/t5311-pack-bitmaps-shallow.sh ##\n +@@ t/t5311-pack-bitmaps-shallow.sh: test_description='check bitmap operation with shallow repositories'\n + # the tree for A. But in a shallow one, we've grafted away\n + # A, and fetching A to B requires that the other side send\n + # us the tree for file=1.\n +-test_expect_success 'setup shallow repo' '\n +-\techo 1 >file &&\n +-\tgit add file &&\n +-\tgit commit -m orig &&\n +-\techo 2 >file &&\n +-\tgit commit -a -m update &&\n +-\tgit clone --no-local --bare --depth=1 . shallow.git &&\n +-\techo 1 >file &&\n +-\tgit commit -a -m repeat\n +-'\n +-\n +-test_expect_success 'turn on bitmaps in the parent' '\n +-\tgit repack -adb\n +-'\n +-\n +-test_expect_success 'shallow fetch from bitmapped repo' '\n +-\t(cd shallow.git && git fetch)\n +-'\n ++test_shallow_bitmaps () {\n ++\twriteLookupTable=false\n ++\n ++\tfor i in \"$@\"\n ++\tdo\n ++\t\tcase $i in\n ++\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n ++\t\tesac\n ++\tdone\n ++\n ++\ttest_expect_success 'setup shallow repo' '\n ++\t\trm -rf * .git &&\n ++\t\tgit init &&\n ++\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n ++\t\techo 1 >file &&\n ++\t\tgit add file &&\n ++\t\tgit commit -m orig &&\n ++\t\techo 2 >file &&\n ++\t\tgit commit -a -m update &&\n ++\t\tgit clone --no-local --bare --depth=1 . shallow.git &&\n ++\t\techo 1 >file &&\n ++\t\tgit commit -a -m repeat\n ++\t'\n ++\n ++\ttest_expect_success 'turn on bitmaps in the parent' '\n ++\t\tgit repack -adb\n ++\t'\n ++\n ++\ttest_expect_success 'shallow fetch from bitmapped repo' '\n ++\t\t(cd shallow.git && git fetch)\n ++\t'\n ++}\n ++\n ++test_shallow_bitmaps\n ++\n + \n + test_done\n \n ## t/t5326-multi-pack-bitmaps.sh ##\n -@@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'graceful fallback when missing reverse index' '\n - \t)\n - '\n +@@ t/t5326-multi-pack-bitmaps.sh: GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n + sane_unset GIT_TEST_MIDX_WRITE_REV\n + sane_unset GIT_TEST_MIDX_READ_RIDX\n \n +-midx_bitmap_core\n +-\n + bitmap_reuse_tests() {\n + \tfrom=$1\n + \tto=$2\n ++\twriteLookupTable=false\n ++\n ++\tfor i in $3-${$#}\n ++\tdo\n ++\t\tcase $i in\n ++\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n ++\t\tesac\n ++\tdone\n + \n + \ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n + \t\trm -fr repo &&\n + \t\tgit init repo &&\n + \t\t(\n + \t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \t\t\ttest_commit_bulk 16 &&\n + \t\t\tgit tag old-tip &&\n + \n +@@ t/t5326-multi-pack-bitmaps.sh: bitmap_reuse_tests() {\n + \ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n + \t\t(\n + \t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \t\t\ttest_commit_bulk --id=further 16 &&\n + \t\t\tgit tag new-tip &&\n + \n +@@ t/t5326-multi-pack-bitmaps.sh: bitmap_reuse_tests() {\n + \ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n + \t\t(\n + \t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \t\t\tgit for-each-ref &&\n + \t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n + \t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n +@@ t/t5326-multi-pack-bitmaps.sh: bitmap_reuse_tests() {\n + \t'\n + }\n + \n +-bitmap_reuse_tests 'pack' 'MIDX'\n +-bitmap_reuse_tests 'MIDX' 'pack'\n +-bitmap_reuse_tests 'MIDX' 'MIDX'\n ++test_midx_bitmap_cases () {\n ++\twriteLookupTable=false\n ++\twriteBitmapLookupTable=\n ++\n ++\tfor i in \"$@\"\n ++\tdo\n ++\t\tcase $i in\n ++\t\t\"pack.writeBitmapLookupTable\")\n ++\t\t\twriteLookupTable=true\n ++\t\t\twriteBitmapLookupTable=\"$i\"\n ++\t\t\t;;\n ++\t\tesac\n ++\tdone\n ++\n ++\ttest_expect_success 'setup test_repository' '\n ++\t\trm -rf * .git &&\n ++\t\tgit init &&\n ++\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n ++\t'\n + \n +-test_expect_success 'missing object closure fails gracefully' '\n +-\trm -fr repo &&\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n ++\tmidx_bitmap_core\n + \n +-\t\ttest_commit loose &&\n +-\t\ttest_commit packed &&\n ++\tbitmap_reuse_tests 'pack' 'MIDX' \"$writeBitmapLookupTable\"\n ++\tbitmap_reuse_tests 'MIDX' 'pack' \"$writeBitmapLookupTable\"\n ++\tbitmap_reuse_tests 'MIDX' 'MIDX' \"$writeBitmapLookupTable\"\n + \n +-\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n +-\t\t# commit.\n +-\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n +-\t\t$(git rev-parse packed)\n +-\t\tEOF\n ++\ttest_expect_success 'missing object closure fails gracefully' '\n ++\t\trm -fr repo &&\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n +-\t\tgrep \"doesn.t have full closure\" err &&\n +-\t\ttest_path_is_missing $midx\n +-\t)\n +-'\n ++\t\t\ttest_commit loose &&\n ++\t\t\ttest_commit packed &&\n + \n +-midx_bitmap_partial_tests\n ++\t\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n ++\t\t\t# commit.\n ++\t\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n ++\t\t\t$(git rev-parse packed)\n ++\t\t\tEOF\n + \n +-test_expect_success 'removing a MIDX clears stale bitmaps' '\n +-\trm -fr repo &&\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n +-\t\ttest_commit base &&\n +-\t\tgit repack &&\n +-\t\tgit multi-pack-index write --bitmap &&\n ++\t\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n ++\t\t\tgrep \"doesn.t have full closure\" err &&\n ++\t\t\ttest_path_is_missing $midx\n ++\t\t)\n ++\t'\n + \n +-\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n +-\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n +-\t\trm $midx &&\n ++\tmidx_bitmap_partial_tests\n + \n +-\t\t# Then write a new MIDX.\n +-\t\ttest_commit new &&\n +-\t\tgit repack &&\n +-\t\tgit multi-pack-index write --bitmap &&\n ++\ttest_expect_success 'removing a MIDX clears stale bitmaps' '\n ++\t\trm -fr repo &&\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n ++\t\t\ttest_commit base &&\n ++\t\t\tgit repack &&\n ++\t\t\tgit multi-pack-index write --bitmap &&\n ++\n ++\t\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n ++\t\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n ++\t\t\trm $midx &&\n ++\n ++\t\t\t# Then write a new MIDX.\n ++\t\t\ttest_commit new &&\n ++\t\t\tgit repack &&\n ++\t\t\tgit multi-pack-index write --bitmap &&\n ++\n ++\t\t\ttest_path_is_file $midx &&\n ++\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\t\t\ttest_path_is_missing $stale_bitmap\n ++\t\t)\n ++\t'\n + \n +-\t\ttest_path_is_file $midx &&\n +-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n +-\t\ttest_path_is_missing $stale_bitmap\n +-\t)\n +-'\n ++\ttest_expect_success 'pack.preferBitmapTips' '\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-test_expect_success 'pack.preferBitmapTips' '\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n ++\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n + \n +-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n ++\t\t\tgit log --format=\"%H\" >commits.raw &&\n ++\t\t\tsort commits &&\n + \n +-\t\tgit log --format=\"%H\" >commits.raw &&\n +-\t\tsort commits &&\n ++\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n ++\t\t\tgit update-ref --stdin refs &&\n +-\t\tgit update-ref --stdin bitmaps &&\n ++\t\t\tcomm -13 bitmaps commits >before &&\n ++\t\t\ttest_line_count = 1 before &&\n + \n +-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +-\t\tcomm -13 bitmaps commits >before &&\n +-\t\ttest_line_count = 1 before &&\n ++\t\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n ++\t\t\t\tbitmaps &&\n ++\t\t\tcomm -13 bitmaps commits >after &&\n + \n +-\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n +-\t\t\tmulti-pack-index write --bitmap &&\n +-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +-\t\tcomm -13 bitmaps commits >after &&\n ++\t\t\t! test_cmp before after\n ++\t\t)\n ++\t'\n + \n +-\t\t! test_cmp before after\n +-\t)\n +-'\n ++\ttest_expect_success 'writing a bitmap with --refs-snapshot' '\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-test_expect_success 'writing a bitmap with --refs-snapshot' '\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n ++\t\t\ttest_commit one &&\n ++\t\t\ttest_commit two &&\n + \n +-\t\ttest_commit one &&\n +-\t\ttest_commit two &&\n ++\t\t\tgit rev-parse one >snapshot &&\n + \n +-\t\tgit rev-parse one >snapshot &&\n ++\t\t\tgit repack -ad &&\n + \n +-\t\tgit repack -ad &&\n ++\t\t\t# First, write a MIDX which see both refs/tags/one and\n ++\t\t\t# refs/tags/two (causing both of those commits to receive\n ++\t\t\t# bitmaps).\n ++\t\t\tgit multi-pack-index write --bitmap &&\n + \n +-\t\t# First, write a MIDX which see both refs/tags/one and\n +-\t\t# refs/tags/two (causing both of those commits to receive\n +-\t\t# bitmaps).\n +-\t\tgit multi-pack-index write --bitmap &&\n ++\t\t\ttest_path_is_file $midx &&\n ++\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n + \n +-\t\ttest_path_is_file $midx &&\n +-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n ++\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n ++\t\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n + \n +-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n +-\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n ++\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n ++\t\t\trm -fr $midx &&\n + \n +-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n +-\t\trm -fr $midx &&\n ++\t\t\t# Then again, but with a refs snapshot which only sees\n ++\t\t\t# refs/tags/one.\n ++\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n + \n +-\t\t# Then again, but with a refs snapshot which only sees\n +-\t\t# refs/tags/one.\n +-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n ++\t\t\ttest_path_is_file $midx &&\n ++\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n + \n +-\t\ttest_path_is_file $midx &&\n +-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n ++\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n ++\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n ++\t\t\t! grep \"$(git rev-parse two)\" bitmaps\n ++\t\t)\n ++\t'\n + \n +-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n +-\t\t! grep \"$(git rev-parse two)\" bitmaps\n +-\t)\n +-'\n ++\ttest_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n ++\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n + \n +-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n ++\t\t\tgit log --format=\"%H\" >commits.raw &&\n ++\t\t\tsort commits &&\n + \n +-\t\tgit log --format=\"%H\" >commits.raw &&\n +-\t\tsort commits &&\n ++\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n ++\t\t\tgit update-ref --stdin refs &&\n +-\t\tgit update-ref --stdin bitmaps &&\n ++\t\t\tcomm -13 bitmaps commits >before &&\n ++\t\t\ttest_line_count = 1 before &&\n + \n +-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +-\t\tcomm -13 bitmaps commits >before &&\n +-\t\ttest_line_count = 1 before &&\n ++\t\t\t(\n ++\t\t\t\tgrep -vf before commits.raw &&\n ++\t\t\t\t# mark missing commits as preferred\n ++\t\t\t\tsed \"s/^/+/\" before\n ++\t\t\t) >snapshot &&\n + \n ++\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n ++\t\t\trm -fr $midx &&\n ++\n ++\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n ++\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n ++\t\t\tcomm -13 bitmaps commits >after &&\n ++\n ++\t\t\t! test_cmp before after\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success 'hash-cache values are propagated from pack bitmaps' '\n ++\t\trm -fr repo &&\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n + \t\t(\n +-\t\t\tgrep -vf before commits.raw &&\n +-\t\t\t# mark missing commits as preferred\n +-\t\t\tsed \"s/^/+/\" before\n +-\t\t) >snapshot &&\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n +-\t\trm -fr $midx &&\n ++\t\t\ttest_commit base &&\n ++\t\t\ttest_commit base2 &&\n ++\t\t\tgit repack -adb &&\n + \n +-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n +-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n +-\t\tcomm -13 bitmaps commits >after &&\n ++\t\t\ttest-tool bitmap dump-hashes >pack.raw &&\n ++\t\t\ttest_file_not_empty pack.raw &&\n ++\t\t\tsort pack.raw >pack.hashes &&\n + \n +-\t\t! test_cmp before after\n +-\t)\n +-'\n ++\t\t\ttest_commit new &&\n ++\t\t\tgit repack &&\n ++\t\t\tgit multi-pack-index write --bitmap &&\n + \n +-test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n +-\trm -fr repo &&\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n ++\t\t\ttest-tool bitmap dump-hashes >midx.raw &&\n ++\t\t\tsort midx.raw >midx.hashes &&\n + \n +-\t\ttest_commit base &&\n +-\t\ttest_commit base2 &&\n +-\t\tgit repack -adb &&\n ++\t\t\t# ensure that every namehash in the pack bitmap can be found in\n ++\t\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n ++\t\t\t# unique to the pack bitmap).\n ++\t\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n ++\t\t\ttest_must_be_empty dropped.hashes\n ++\t\t)\n ++\t'\n + \n +-\t\ttest-tool bitmap dump-hashes >pack.raw &&\n +-\t\ttest_file_not_empty pack.raw &&\n +-\t\tsort pack.raw >pack.hashes &&\n ++\ttest_expect_success 'no .bitmap is written without any objects' '\n ++\t\trm -fr repo &&\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-\t\ttest_commit new &&\n +-\t\tgit repack &&\n +-\t\tgit multi-pack-index write --bitmap &&\n ++\t\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n ++\t\t\tpack-$empty.idx\n ++\t\t\tEOF\n + \n +-\t\ttest-tool bitmap dump-hashes >midx.raw &&\n +-\t\tsort midx.raw >midx.hashes &&\n ++\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n ++\t\t\t\terr &&\n + \n +-\t\t# ensure that every namehash in the pack bitmap can be found in\n +-\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n +-\t\t# unique to the pack bitmap).\n +-\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n +-\t\ttest_must_be_empty dropped.hashes\n +-\t)\n +-'\n ++\t\t\tgrep \"bitmap without any objects\" err &&\n + \n +-test_expect_success 'no .bitmap is written without any objects' '\n +-\trm -fr repo &&\n +-\tgit init repo &&\n +-\ttest_when_finished \"rm -fr repo\" &&\n +-\t(\n +-\t\tcd repo &&\n ++\t\t\ttest_path_is_file $midx &&\n ++\t\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success 'graceful fallback when missing reverse index' '\n ++\t\trm -fr repo &&\n ++\t\tgit init repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \n +-\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n +-\t\tpack-$empty.idx\n +-\t\tEOF\n ++\t\t\ttest_commit base &&\n + \n +-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n +-\t\t\terr &&\n ++\t\t\t# write a pack and MIDX bitmap containing base\n ++\t\t\tgit repack -adb &&\n ++\t\t\tgit multi-pack-index write --bitmap &&\n + \n +-\t\tgrep \"bitmap without any objects\" err &&\n ++\t\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n ++\t\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n ++\t\t\t! grep \"ignoring extra bitmap file\" err\n ++\t\t)\n ++\t'\n ++}\n + \n +-\t\ttest_path_is_file $midx &&\n +-\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n +-\t)\n +-'\n ++test_midx_bitmap_cases\n ++\n ++test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n + \n +-test_expect_success 'graceful fallback when missing reverse index' '\n +test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n -+\trm -fr repo &&\n -+\tgit init repo &&\n -+\ttest_when_finished \"rm -fr repo\" &&\n -+\t(\n -+\t\tcd repo &&\n -+\t\ttest_commit base &&\n + \trm -fr repo &&\n + \tgit init repo &&\n + \ttest_when_finished \"rm -fr repo\" &&\n + \t(\n + \t\tcd repo &&\n +-\n + \t\ttest_commit base &&\n +-\n +-\t\t# write a pack and MIDX bitmap containing base\n +-\t\tgit repack -adb &&\n +-\t\tgit multi-pack-index write --bitmap &&\n +-\n +-\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n +-\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n +-\t\t! grep \"ignoring extra bitmap file\" err\n ++\t\tgit config pack.writeBitmapLookupTable true &&\n +\t\tgit repack -ad &&\n +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n +\t\t\tgit multi-pack-index write --bitmap &&\n +\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n -+\t)\n + \t)\n + '\n + \n +\n + ## t/t5327-multi-pack-bitmaps-rev.sh ##\n +@@ t/t5327-multi-pack-bitmaps-rev.sh: export GIT_TEST_MIDX_READ_RIDX\n + midx_bitmap_core rev\n + midx_bitmap_partial_tests rev\n + \n ++test_expect_success 'reinitialize the repository with lookup table enabled' '\n ++ rm -fr * .git &&\n ++ git init &&\n ++ git config pack.writeBitmapLookupTable true\n +'\n ++\n ++midx_bitmap_core rev\n ++midx_bitmap_partial_tests rev\n ++\n test_done\n 4: 4fbfcff8a20 ! 4: e64362621d2 pack-bitmap: prepare to read lookup table extension\n @@ Commit message\n does not know how to parse them.\n \n Teach Git to parse the existing bitmap lookup table. The older\n - versions of git are not affected by it. Those versions ignore the\n + versions of Git are not affected by it. Those versions ignore the\n lookup table.\n \n - Signed-off-by: Abhradeep Chakraborty \n Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n + Signed-off-by: Abhradeep Chakraborty \n \n ## pack-bitmap.c ##\n @@ pack-bitmap.c: struct bitmap_index {\n @@ pack-bitmap.c: struct bitmap_index {\n \n +\t/*\n +\t * If not NULL, this point into the commit table extension\n -+\t * (within map).\n ++\t * (within the memory mapped region `map`).\n +\t */\n +\tunsigned char *table_lookup;\n +\n @@ pack-bitmap.c: static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex_end -= cache_size;\n \t\t}\n +\n -+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE &&\n -+\t\t\tgit_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1)) {\n -+\t\t\tsize_t table_size = 0;\n -+\t\t\tsize_t triplet_sz = st_add3(sizeof(uint32_t), /* commit position */\n -+\t\t\t\t\t\t\tsizeof(uint64_t), /* offset */\n -+\t\t\t\t\t\t\tsizeof(uint32_t)); /* xor offset */\n -+\n -+\t\t\ttable_size = st_add(table_size,\n -+\t\t\t\t\tst_mult(ntohl(header->entry_count),\n -+\t\t\t\t\t\ttriplet_sz));\n ++\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE) {\n ++\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n ++\t\t\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n +\t\t\tif (table_size > index_end - index->map - header_size)\n -+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit lookup table)\");\n -+\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n ++\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit lookup table)\"));\n ++\t\t\tif (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n ++\t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n +\t\t\tindex_end -= table_size;\n +\t\t}\n \t}\n @@ pack-bitmap.c: static struct stored_bitmap *store_bitmap(struct bitmap_index *in\n -\t/* a 0 return code means the insertion succeeded with no changes,\n -\t * because the SHA1 already existed on the map. this is bad, there\n -\t * shouldn't be duplicated commits in the index */\n -+\t/* A 0 return code means the insertion succeeded with no changes,\n -+\t * because the SHA1 already existed on the map. If lookup table\n -+\t * is NULL, this is bad, there shouldn't be duplicated commits\n -+\t * in the index.\n -+\t *\n -+\t * If table_lookup exists, that means the desired bitmap is already\n -+\t * loaded. Either this bitmap has been stored directly or another\n -+\t * bitmap has a direct or indirect xor relation with it. */\n ++\t/*\n ++\t * A 0 return code means the insertion succeeded with no changes,\n ++\t * because the SHA1 already existed on the map. This is bad, there\n ++\t * shouldn't be duplicated commits in the index.\n ++\t */\n \tif (ret == 0) {\n -\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n --\t\treturn NULL;\n -+\t\tif (!index->table_lookup) {\n -+\t\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n -+\t\t\treturn NULL;\n -+\t\t}\n -+\t\treturn kh_value(index->bitmaps, hash_pos);\n ++\t\terror(_(\"duplicate entry in bitmap index: %s\"), oid_to_hex(oid));\n + \t\treturn NULL;\n \t}\n \n - \tkh_value(index->bitmaps, hash_pos) = stored;\n @@ pack-bitmap.c: static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n @@ pack-bitmap.c: struct include_data {\n \tstruct bitmap *seen;\n };\n \n -+static inline const void *bitmap_get_triplet(struct bitmap_index *bitmap_git, uint32_t xor_pos)\n -+{\n -+\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n -+\tconst void *p = bitmap_git->table_lookup + st_mult(xor_pos, triplet_sz);\n -+\treturn p;\n -+}\n -+\n -+static uint64_t triplet_get_offset(const void *triplet)\n -+{\n -+\tconst void *p = (unsigned char*) triplet + sizeof(uint32_t);\n -+\treturn get_be64(p);\n -+}\n ++struct bitmap_lookup_table_triplet {\n ++\tuint32_t commit_pos;\n ++\tuint64_t offset;\n ++\tuint32_t xor_row;\n ++};\n +\n -+static uint32_t triplet_get_xor_pos(const void *triplet)\n ++struct bitmap_lookup_table_xor_item {\n ++\tstruct object_id oid;\n ++\tuint64_t offset;\n ++};\n ++\n ++/*\n ++ * This function gets the raw triplet from `row`'th row in the\n ++ * lookup table and fills that data to the `triplet`.\n ++ */\n ++static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n ++\t\t\t\t uint32_t pos,\n ++\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n +{\n -+\tconst void *p = (unsigned char*) triplet + st_add(sizeof(uint32_t), sizeof(uint64_t));\n -+\treturn get_be32(p);\n ++\tunsigned char *p = NULL;\n ++\tif (pos >= bitmap_git->entry_count)\n ++\t\treturn error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n ++\n ++\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n ++\n ++\ttriplet->commit_pos = get_be32(p);\n ++\tp += sizeof(uint32_t);\n ++\ttriplet->offset = get_be64(p);\n ++\tp += sizeof(uint64_t);\n ++\ttriplet->xor_row = get_be32(p);\n ++\treturn 0;\n +}\n +\n ++/*\n ++ * Searches for a matching triplet. `va` is a pointer\n ++ * to the wanted commit position value. `vb` points to\n ++ * a triplet in lookup table. The first 4 bytes of each\n ++ * triplet (pointed by `vb`) are compared with `*va`.\n ++ */\n +static int triplet_cmp(const void *va, const void *vb)\n +{\n -+\tint result = 0;\n -+\tuint32_t *a = (uint32_t *) va;\n ++\n ++\tuint32_t a = *(uint32_t *)va;\n +\tuint32_t b = get_be32(vb);\n -+\tif (*a > b)\n -+\t\tresult = 1;\n -+\telse if (*a < b)\n -+\t\tresult = -1;\n -+\telse\n -+\t\tresult = 0;\n ++\tif (a > b)\n ++\t\treturn 1;\n ++\telse if (a < b)\n ++\t\treturn -1;\n +\n -+\treturn result;\n ++\treturn 0;\n +}\n +\n -+static uint32_t bsearch_pos(struct bitmap_index *bitmap_git, struct object_id *oid,\n -+\t\t\t\t\t\tuint32_t *result)\n ++static uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n ++\t\t\t struct object_id *oid,\n ++\t\t\t uint32_t *result)\n +{\n +\tint found;\n +\n -+\tif (bitmap_git->midx)\n ++\tif (bitmap_is_midx(bitmap_git))\n +\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n +\telse\n +\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n @@ pack-bitmap.c: struct include_data {\n +\treturn found;\n +}\n +\n ++/*\n ++ * `bsearch_triplet` function searches for the raw triplet having\n ++ * commit position same as `commit_pos` and fills `triplet`\n ++ * object from the raw triplet. Returns 1 on success and 0\n ++ * on failure.\n ++ */\n ++static int bsearch_triplet(uint32_t *commit_pos,\n ++\t\t\t struct bitmap_index *bitmap_git,\n ++\t\t\t struct bitmap_lookup_table_triplet *triplet)\n ++{\n ++\tunsigned char *p = bsearch(commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n ++\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n ++\n ++\tif (!p)\n ++\t\treturn 0;\n ++\ttriplet->commit_pos = get_be32(p);\n ++\tp += sizeof(uint32_t);\n ++\ttriplet->offset = get_be64(p);\n ++\tp += sizeof(uint64_t);\n ++\ttriplet->xor_row = get_be32(p);\n ++\treturn 1;\n ++}\n ++\n +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n +\t\t\t\t\t struct commit *commit)\n +{\n -+\tuint32_t commit_pos, xor_pos;\n ++\tuint32_t commit_pos, xor_row;\n +\tuint64_t offset;\n +\tint flags;\n -+\tconst void *triplet = NULL;\n ++\tstruct bitmap_lookup_table_triplet triplet;\n +\tstruct object_id *oid = &commit->object.oid;\n +\tstruct ewah_bitmap *bitmap;\n +\tstruct stored_bitmap *xor_bitmap = NULL;\n -+\tsize_t triplet_sz = st_add3(sizeof(uint32_t), sizeof(uint64_t), sizeof(uint32_t));\n +\n +\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n +\n +\tif (!found)\n +\t\treturn NULL;\n +\n -+\ttriplet = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n -+\t\t\t\t\t\ttriplet_sz, triplet_cmp);\n -+\tif (!triplet)\n ++\tif (!bsearch_triplet(&commit_pos, bitmap_git, &triplet))\n +\t\treturn NULL;\n +\n -+\toffset = triplet_get_offset(triplet);\n -+\txor_pos = triplet_get_xor_pos(triplet);\n ++\toffset = triplet.offset;\n ++\txor_row = triplet.xor_row;\n +\n -+\tif (xor_pos != 0xffffffff) {\n ++\tif (xor_row != 0xffffffff) {\n +\t\tint xor_flags;\n ++\t\tkhiter_t hash_pos;\n +\t\tuint64_t offset_xor;\n -+\t\tuint32_t *xor_positions;\n -+\t\tstruct object_id xor_oid;\n -+\t\tsize_t size = 0;\n -+\n -+\t\tALLOC_ARRAY(xor_positions, bitmap_git->entry_count);\n -+\t\twhile (xor_pos != 0xffffffff) {\n -+\t\t\txor_positions[size++] = xor_pos;\n -+\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n -+\t\t\txor_pos = triplet_get_xor_pos(triplet);\n ++\t\tstruct bitmap_lookup_table_xor_item *xor_items;\n ++\t\tstruct bitmap_lookup_table_xor_item xor_item;\n ++\t\tsize_t xor_items_nr = 0, xor_items_alloc = 64;\n ++\n ++\t\tALLOC_ARRAY(xor_items, xor_items_alloc);\n ++\t\twhile (xor_row != 0xffffffff) {\n ++\t\t\tstruct object_id xor_oid;\n ++\n ++\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n ++\t\t\t\tfree(xor_items);\n ++\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n ++\t\t\t\treturn NULL;\n ++\t\t\t}\n ++\n ++\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n ++\t\t\t\treturn NULL;\n ++\n ++\t\t\toffset_xor = triplet.offset;\n ++\n ++\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, triplet.commit_pos) < 0) {\n ++\t\t\t\tfree(xor_items);\n ++\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n ++\t\t\t\t\ttriplet.commit_pos);\n ++\t\t\t\treturn NULL;\n ++\t\t\t}\n ++\n ++\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_oid);\n ++\n ++\t\t\t/*\n ++\t\t\t * If desired bitmap is already stored, we don't need\n ++\t\t\t * to iterate further. Because we know that bitmaps\n ++\t\t\t * that are needed to be parsed to parse this bitmap\n ++\t\t\t * has already been stored. So, assign this stored bitmap\n ++\t\t\t * to the xor_bitmap.\n ++\t\t\t */\n ++\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n ++\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n ++\t\t\t\tbreak;\n ++\n ++\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n ++\t\t\txor_items[xor_items_nr++] = (struct bitmap_lookup_table_xor_item) {.oid = xor_oid,\n ++\t\t\t\t\t\t\t\t\t\t\t .offset = offset_xor};\n ++\t\t\txor_row = triplet.xor_row;\n +\t\t}\n +\n -+\t\twhile (size){\n -+\t\t\txor_pos = xor_positions[size - 1];\n -+\t\t\ttriplet = bitmap_get_triplet(bitmap_git, xor_pos);\n -+\t\t\tcommit_pos = get_be32(triplet);\n -+\t\t\toffset_xor = triplet_get_offset(triplet);\n ++\t\twhile (xor_items_nr) {\n ++\t\t\txor_item = xor_items[xor_items_nr - 1];\n ++\t\t\toffset_xor = xor_item.offset;\n +\n -+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, commit_pos) < 0) {\n -+\t\t\t\tfree(xor_positions);\n ++\t\t\tbitmap_git->map_pos = offset_xor;\n ++\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n ++\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n ++\t\t\t\t\toid_to_hex(&xor_item.oid));\n ++\t\t\t\tfree(xor_items);\n +\t\t\t\treturn NULL;\n +\t\t\t}\n +\n -+\t\t\tbitmap_git->map_pos = offset_xor + sizeof(uint32_t) + sizeof(uint8_t);\n ++\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n +\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n +\t\t\tbitmap = read_bitmap_1(bitmap_git);\n +\n -+\t\t\tif (!bitmap){\n -+\t\t\t\tfree(xor_positions);\n ++\t\t\tif (!bitmap) {\n ++\t\t\t\tfree(xor_items);\n +\t\t\t\treturn NULL;\n +\t\t\t}\n +\n -+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_oid, xor_bitmap, xor_flags);\n -+\t\t\tsize--;\n ++\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item.oid, xor_bitmap, xor_flags);\n ++\t\t\txor_items_nr--;\n +\t\t}\n +\n -+\t\tfree(xor_positions);\n ++\t\tfree(xor_items);\n +\t}\n +\n -+\tbitmap_git->map_pos = offset + sizeof(uint32_t) + sizeof(uint8_t);\n ++\tbitmap_git->map_pos = offset;\n ++\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n ++\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n ++\t\t\toid_to_hex(oid));\n ++\t\treturn NULL;\n ++\t}\n ++\n ++\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n +\tbitmap = read_bitmap_1(bitmap_git);\n +\n @@ pack-bitmap.c: struct include_data {\n +\t\tif (!bitmap_git->table_lookup)\n +\t\t\treturn NULL;\n +\n ++\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n +\t\t/* NEEDSWORK: cache misses aren't recorded */\n +\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n -+\t\tif(!bitmap)\n ++\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n ++\t\tif (!bitmap)\n +\t\t\treturn NULL;\n +\t\treturn lookup_stored_bitmap(bitmap);\n +\t}\n @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n -\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n -+\tfprintf(stderr, \"Bitmap v%d test (%d entries)\\n\",\n - \t\tbitmap_git->version, bitmap_git->entry_count);\n +-\t\tbitmap_git->version, bitmap_git->entry_count);\n ++\tfprintf(stderr, \"Bitmap v%d test (%d entries%s)\",\n ++\t\tbitmap_git->version,\n ++\t\tbitmap_git->entry_count,\n ++\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n \n -+\tif (!bitmap_git->table_lookup)\n -+\t\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n -+\t\t\tbitmap_git->version, bitmap_git->entry_count);\n -+\n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n - \n @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \n int test_bitmap_commits(struct repository *r)\n {\n -\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n -+\tstruct bitmap_index *bitmap_git = NULL;\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n - \n -+\t/* As this function is only used to print bitmap selected\n ++\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n ++\n ++\t/*\n ++\t * As this function is only used to print bitmap selected\n +\t * commits, we don't have to read the commit table.\n +\t */\n -+\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"0\", 1);\n -+\n -+\tbitmap_git = prepare_bitmap_git(r);\n + \n \tif (!bitmap_git)\n \t\tdie(\"failed to load bitmap indexes\");\n \n -@@ pack-bitmap.c: int test_bitmap_commits(struct repository *r)\n ++\tif (bitmap_git->table_lookup) {\n ++\t\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n ++\t\t\tdie(_(\"failed to load bitmap indexes\"));\n ++\t}\n ++\n + \tkh_foreach(bitmap_git->bitmaps, oid, value, {\n \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n \t});\n +\n + ## pack-bitmap.h ##\n +@@ pack-bitmap.h: struct bitmap_disk_header {\n \n -+\tsetenv(\"GIT_TEST_READ_COMMIT_TABLE\", \"1\", 1);\n - \tfree_bitmap_index(bitmap_git);\n + #define NEEDS_BITMAP (1u<<22)\n \n - \treturn 0;\n ++/*\n ++ * The width in bytes of a single triplet in the lookup table\n ++ * extension:\n ++ * (commit_pos, offset, xor_row)\n ++ *\n ++ * whose fields ar 32-, 64-, 32- bits wide, respectively.\n ++ */\n ++#define BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH (16)\n ++\n + enum pack_bitmap_opts {\n + \tBITMAP_OPT_FULL_DAG = 0x1,\n + \tBITMAP_OPT_HASH_CACHE = 0x4,\n \n ## t/t5310-pack-bitmaps.sh ##\n -@@ t/t5310-pack-bitmaps.sh: test_expect_success 'full repack creates bitmaps' '\n - \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n +@@ t/t5310-pack-bitmaps.sh: test_bitmap_cases () {\n + \n + \ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n + \t\ttest_config pack.writebitmaphashcache false &&\n ++\t\ttest_config pack.writebitmaplookuptable false &&\n + \t\tgit repack -ad &&\n + \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n + \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n +@@ t/t5310-pack-bitmaps.sh: test_bitmap_cases () {\n + \t'\n + \n + \ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n ++\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n + \t\tgit repack -ad &&\n + \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n + \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'verify writing bitmap lookup table when enabled' '\n + \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n -+test_expect_success 'using lookup table loads only necessary bitmaps' '\n -+\tgit rev-list --test-bitmap HEAD 2>out &&\n -+\t! grep \"Bitmap v1 test (106 entries loaded)\" out &&\n -+\tgrep \"Found bitmap for\" out\n ++test_expect_success 'lookup table is actually used to traverse objects' '\n ++\tgit repack -adb &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n ++\t\tgit rev-list --use-bitmap-index --count --all &&\n ++\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n +'\n +\n - basic_bitmap_tests\n - \n - test_expect_success 'incremental repack fails when bitmaps are requested' '\n -@@ t/t5310-pack-bitmaps.sh: test_expect_success 'pack reuse respects --incremental' '\n - \n - test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n - \ttest_config pack.writebitmaphashcache false &&\n -+\ttest_config pack.writebitmaplookuptable false &&\n - \tgit repack -ad &&\n - \tgit rev-list --use-bitmap-index --count --all >expect &&\n - \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n -\n - ## t/t5326-multi-pack-bitmaps.sh ##\n -@@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n - \t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n - \t)\n - '\n ++test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n ++\ttest_config pack.writebitmaphashcache false &&\n ++\tgit repack -adb &&\n ++\tgit rev-list --use-bitmap-index --count --all >expect &&\n ++\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n ++\ttest_when_finished \"rm -f $bitmap\" &&\n ++\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n ++\tmv -f $bitmap.tmp $bitmap &&\n ++\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n ++\ttest_cmp expect actual &&\n ++\ttest_i18ngrep corrupted.bitmap.index stderr\n ++'\n +\n test_done\n 5: 96c0041688f ! 5: a155c1e2eba bitmap-lookup-table: add performance tests for lookup table\n @@ Metadata\n ## Commit message ##\n bitmap-lookup-table: add performance tests for lookup table\n \n - Add performance tests to verify the performance of lookup table.\n + Add performance tests to verify the performance of lookup table with\n + `pack.writeReverseIndex` enabled. This is to check the performance\n + when the above configuration is set.\n \n Lookup table makes Git run faster in most of the cases. Below is the\n result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\n gives similar result. The repository used in the test is linux kernel.\n \n Test this tree\n - --------------------------------------------------------------------------\n - 5310.4: repack to disk (lookup=false) 295.94(250.45+15.24)\n - 5310.5: simulated clone 12.52(5.07+1.40)\n - 5310.6: simulated fetch 1.89(2.94+0.24)\n - 5310.7: pack to file (bitmap) 41.39(20.33+7.20)\n - 5310.8: rev-list (commits) 0.98(0.59+0.12)\n - 5310.9: rev-list (objects) 3.40(3.27+0.10)\n + ---------------------------------------------------------------------------\n + 5310.4: repack to disk (lookup=false) 296.55(256.53+14.52)\n + 5310.5: simulated clone 15.64(8.88+1.39)\n + 5310.6: simulated fetch 1.65(2.75+0.20)\n + 5310.7: pack to file (bitmap) 48.71(30.20+7.58)\n + 5310.8: rev-list (commits) 0.61(0.41+0.08)\n + 5310.9: rev-list (objects) 4.38(4.26+0.09)\n 5310.10: rev-list with tag negated via --not 0.07(0.02+0.04)\n --all (objects)\n - 5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n - 5310.12: rev-list count with blob:none 0.26(0.18+0.07)\n - 5310.13: rev-list count with blob:limit=1k 6.45(5.94+0.37)\n - 5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n - 5310.15: simulated partial clone 4.99(3.19+0.45)\n - 5310.19: repack to disk (lookup=true) 269.67(174.70+21.33)\n - 5310.20: simulated clone 11.03(5.07+1.11)\n - 5310.21: simulated fetch 0.79(0.79+0.17)\n - 5310.22: pack to file (bitmap) 43.03(20.28+7.43)\n - 5310.23: rev-list (commits) 0.86(0.54+0.09)\n - 5310.24: rev-list (objects) 3.35(3.26+0.07)\n - 5310.25: rev-list with tag negated via --not 0.05(0.00+0.03)\n + 5310.11: rev-list with negative tag (objects) 0.05(0.01+0.03)\n + 5310.12: rev-list count with blob:none 0.08(0.03+0.04)\n + 5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n + 5310.14: rev-list count with tree:0 0.08(0.03+0.04)\n + 5310.15: simulated partial clone 9.45(8.12+0.41)\n + 5310.19: repack to disk (lookup=true) 255.92(188.13+20.47)\n + 5310.20: simulated clone 13.78(8.84+1.09)\n + 5310.21: simulated fetch 0.52(0.63+0.14)\n + 5310.22: pack to file (bitmap) 44.34(28.94+6.84)\n + 5310.23: rev-list (commits) 0.48(0.31+0.06)\n + 5310.24: rev-list (objects) 4.02(3.93+0.07)\n + 5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n - 5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n - 5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n - 5310.28: rev-list count with blob:limit=1k 6.45(5.87+0.31)\n - 5310.29: rev-list count with tree:0 0.22(0.16+0.05)\n - 5310.30: simulated partial clone 5.17(3.12+0.48)\n + 5310.26: rev-list with negative tag (objects) 0.04(0.00+0.03)\n + 5310.27: rev-list count with blob:none 0.04(0.01+0.03)\n + 5310.28: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n + 5310.29: rev-list count with tree:0 0.04(0.01+0.03)\n + 5310.30: simulated partial clone 8.30(7.21+0.36)\n \n Test 4-15 are tested without using lookup table. Same tests are\n repeated in 16-30 (using lookup table).\n \n - Signed-off-by: Abhradeep Chakraborty \n Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n + Signed-off-by: Abhradeep Chakraborty \n \n ## t/perf/p5310-pack-bitmaps.sh ##\n -@@ t/perf/p5310-pack-bitmaps.sh: test_expect_success 'setup bitmap config' '\n - \tgit config pack.writebitmaps true\n +@@ t/perf/p5310-pack-bitmaps.sh: test_perf_large_repo\n + # We intentionally use the deprecated pack.writebitmaps\n + # config so that we can test against older versions of git.\n + test_expect_success 'setup bitmap config' '\n +-\tgit config pack.writebitmaps true\n ++\tgit config pack.writebitmaps true &&\n ++\tgit config pack.writeReverseIndex true\n '\n \n -# we need to create the tag up front such that it is covered by the repack and\n @@ t/perf/p5310-pack-bitmaps.sh: test_expect_success 'setup bitmap config' '\n -test_expect_success 'create tags' '\n -\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n -'\n --\n ++test_bitmap () {\n ++\tlocal enabled=\"$1\"\n + \n -test_perf 'repack to disk' '\n -\tgit repack -ad\n -'\n --\n ++\t# we need to create the tag up front such that it is covered by the repack and\n ++\t# thus by generated bitmaps.\n ++\ttest_expect_success 'create tags' '\n ++\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n ++\t'\n + \n -test_full_bitmap\n --\n ++\ttest_expect_success \"use lookup table: $enabled\" '\n ++\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n ++\t'\n + \n -test_expect_success 'create partial bitmap state' '\n -\t# pick a commit to represent the repo tip in the past\n -\tcutoff=$(git rev-list HEAD~100 -1) &&\n -\torig_tip=$(git rev-parse HEAD) &&\n --\n ++\ttest_perf \"repack to disk (lookup=$enabled)\" '\n ++\t\tgit repack -ad\n ++\t'\n + \n -\t# now kill off all of the refs and pretend we had\n -\t# just the one tip\n -\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n -\tgit update-ref HEAD $cutoff &&\n --\n ++\ttest_full_bitmap\n + \n -\t# and then repack, which will leave us with a nice\n -\t# big bitmap pack of the \"old\" history, and all of\n -\t# the new history will be loose, as if it had been pushed\n -\t# up incrementally and exploded via unpack-objects\n -\tgit repack -Ad &&\n --\n ++\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n ++\t\t# pick a commit to represent the repo tip in the past\n ++\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n ++\t\torig_tip=$(git rev-parse HEAD) &&\n + \n -\t# and now restore our original tip, as if the pushes\n -\t# had happened\n -\tgit update-ref HEAD $orig_tip\n -'\n --\n --test_partial_bitmap\n -+test_bitmap () {\n -+ local enabled=\"$1\"\n -+\n -+\t# we need to create the tag up front such that it is covered by the repack and\n -+\t# thus by generated bitmaps.\n -+\ttest_expect_success 'create tags' '\n -+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n -+\t'\n -+\n -+\ttest_expect_success \"use lookup table: $enabled\" '\n -+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n -+\t'\n -+\n -+\ttest_perf \"repack to disk (lookup=$enabled)\" '\n -+\t\tgit repack -ad\n -+\t'\n -+\n -+\ttest_full_bitmap\n -+\n -+ test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n -+\t\t# pick a commit to represent the repo tip in the past\n -+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n -+\t\torig_tip=$(git rev-parse HEAD) &&\n -+\n +\t\t# now kill off all of the refs and pretend we had\n +\t\t# just the one tip\n +\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n @@ t/perf/p5310-pack-bitmaps.sh: test_expect_success 'setup bitmap config' '\n +\t\t# and now restore our original tip, as if the pushes\n +\t\t# had happened\n +\t\tgit update-ref HEAD $orig_tip\n -+ '\n ++\t'\n +}\n -+\n + \n +-test_partial_bitmap\n +test_bitmap false\n +test_bitmap true\n \n @@ t/perf/p5326-multi-pack-bitmaps.sh: test_description='Tests performance using mi\n -\n -test_partial_bitmap\n +test_bitmap () {\n -+ local enabled=\"$1\"\n ++\tlocal enabled=\"$1\"\n +\n +\t# we need to create the tag up front such that it is covered by the repack and\n +\t# thus by generated bitmaps.\n @@ t/perf/p5326-multi-pack-bitmaps.sh: test_description='Tests performance using mi\n +\n +\ttest_full_bitmap\n +\n -+ test_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n ++\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n +\t\t# pick a commit to represent the repo tip in the past\n +\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n +\t\torig_tip=$(git rev-parse HEAD) &&\n @@ t/perf/p5326-multi-pack-bitmaps.sh: test_description='Tests performance using mi\n +\t\t# and now restore our original tip, as if the pushes\n +\t\t# had happened\n +\t\tgit update-ref HEAD $orig_tip\n -+ '\n ++\t'\n +}\n +\n +test_bitmap false\n 6: fe556b58814 ! 6: 4f9f1049485 p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing\n @@ Metadata\n Author: Abhradeep Chakraborty \n \n ## Commit message ##\n - p5310-pack-bitmaps.sh: enable pack.writeReverseIndex for testing\n + p5310-pack-bitmaps.sh: remove pack.writeReverseIndex\n \n - Enable pack.writeReverseIndex to true to see the effect of writing\n - the reverse index in the existing bitmap tests (with and without\n - lookup table).\n + The previous change enables the `pack.writereverseindex` to see\n + the effect of writing reverse index in the performance test.\n +\n + Remove the `pack.writeReverseIndex` configuration.\n \n Below is the result of performance test. Output format is in\n seconds.\n \n - Test this tree\n - -------------------------------------------------------------------\n - 5310.4: repack to disk (lookup=false) 294.92(257.60+14.29)\n - 5310.5: simulated clone 14.97(8.95+1.31)\n - 5310.6: simulated fetch 1.64(2.77+0.20)\n - 5310.7: pack to file (bitmap) 41.76(29.33+6.77)\n - 5310.8: rev-list (commits) 0.71(0.49+0.09)\n - 5310.9: rev-list (objects) 4.65(4.55+0.09)\n - 5310.10: rev-list with tag negated via --not 0.08(0.02+0.05)\n + Test this tree\n + ------------------------------------------------------------------------\n + 5310.4: repack to disk (lookup=false) 293.80(251.30+14.30)\n + 5310.5: simulated clone 12.50(5.15+1.36)\n + 5310.6: simulated fetch 1.83(2.90+0.23)\n + 5310.7: pack to file (bitmap) 39.70(20.25+7.14)\n + 5310.8: rev-list (commits) 1.00(0.60+0.13)\n + 5310.9: rev-list (objects) 4.11(4.00+0.10)\n + 5310.10: rev-list with tag negated via --not 0.07(0.02+0.05)\n --all (objects)\n - 5310.11: rev-list with negative tag (objects) 0.06(0.01+0.04)\n - 5310.12: rev-list count with blob:none 0.09(0.03+0.05)\n - 5310.13: rev-list count with blob:limit=1k 7.58(7.06+0.33)\n - 5310.14: rev-list count with tree:0 0.09(0.03+0.06)\n - 5310.15: simulated partial clone 8.64(8.04+0.35)\n - 5310.19: repack to disk (lookup=true) 249.86(191.57+19.50)\n - 5310.20: simulated clone 13.67(8.83+1.06)\n - 5310.21: simulated fetch 0.50(0.63+0.13)\n - 5310.22: pack to file (bitmap) 41.24(28.99+6.67)\n - 5310.23: rev-list (commits) 0.67(0.50+0.07)\n - 5310.24: rev-list (objects) 4.88(4.79+0.08)\n - 5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n + 5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n + 5310.12: rev-list count with blob:none 0.27(0.18+0.08)\n + 5310.13: rev-list count with blob:limit=1k 6.41(5.98+0.41)\n + 5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n + 5310.15: simulated partial clone 4.34(3.29+0.37)\n + 5310.19: repack to disk (lookup=true) 250.93(171.97+20.78)\n + 5310.20: simulated clone 10.80(5.14+1.06)\n + 5310.21: simulated fetch 0.71(0.79+0.16)\n + 5310.22: pack to file (bitmap) 39.49(20.19+6.98)\n + 5310.23: rev-list (commits) 0.81(0.48+0.09)\n + 5310.24: rev-list (objects) 3.48(3.38+0.09)\n + 5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n - 5310.26: rev-list with negative tag (objects) 0.05(0.00+0.04)\n - 5310.27: rev-list count with blob:none 0.05(0.01+0.03)\n - 5310.28: rev-list count with blob:limit=1k 8.02(7.16+0.34)\n - 5310.29: rev-list count with tree:0 0.05(0.01+0.04)\n - 5310.30: simulated partial clone 8.57(8.16+0.32)\n + 5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n + 5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n + 5310.28: rev-list count with blob:limit=1k 6.21(5.76+0.29)\n + 5310.29: rev-list count with tree:0 0.23(0.16+0.06)\n + 5310.30: simulated partial clone 4.53(3.14+0.39)\n \n Tests 4-15 are without the use of lookup table. The rests are\n repeatation of the previous tests but using lookup table.\n \n - Signed-off-by: Abhradeep Chakraborty \n Mentored-by: Taylor Blau \n Co-Mentored-by: Kaartic Sivaraam \n + Signed-off-by: Abhradeep Chakraborty \n \n ## t/perf/p5310-pack-bitmaps.sh ##\n @@ t/perf/p5310-pack-bitmaps.sh: test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n --\tgit config pack.writebitmaps true\n -+\tgit config pack.writebitmaps true &&\n -+\tgit config pack.writeReverseIndex true\n +-\tgit config pack.writebitmaps true &&\n +-\tgit config pack.writeReverseIndex true\n ++\tgit config pack.writebitmaps true\n '\n \n test_bitmap () {\n\n-- \ngitgitgadget\n"},{"id":"458439","messageId":"3dc40cc7f735401b6f4f7c307bfd18fede051609.1656924376.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v3 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:13Z","receivedAt":"2022-07-04T08:46:40Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nTeach Git to provide a way for users to enable/disable bitmap lookup\ntable extension by providing a config option named 'writeBitmapLookupTable'.\nDefault is false.\n\nAlso add test to verify writting of lookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/config/pack.txt | 7 +\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n t/t5310-pack-bitmaps.sh | 792 ++++++++++++++++--------------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++++-------\n t/t5327-multi-pack-bitmaps-rev.sh | 9 +\n 9 files changed, 720 insertions(+), 581 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex ad7f73a1ead..b955ca572ec 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n computed; instead, any namehashes stored in an existing bitmap are\n permuted into their appropriate location when writing a new bitmap.\n \n+pack.writeBitmapLookupTable::\n+\tWhen true, Git will include a \"lookup table\" section in the\n+\tbitmap index (if one is written). This table is used to defer\n+\tloading individual bitmaps as late as possible. This can be\n+\tbeneficial in repositories that have relatively large bitmap\n+\tindexes. Defaults to false.\n+\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5edbb7fe86e..55402b46f41 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n \t}\n \n+\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(var, value))\n+\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t\telse\n+\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t}\n+\n \t/*\n \t * We should never make a fall-back call to 'git_default_config', since\n \t * this was already called in 'cmd_multi_pack_index()'.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39e28cfcafc..46e26774963 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n+\n+\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n+\t\telse\n+\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n+\t}\n+\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\ndiff --git a/midx.c b/midx.c\nindex 5f0dd386b02..9c26d04bfde 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1072,6 +1072,9 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n \n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n \tprepare_midx_packing_data(&pdata, ctx);\n \n \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\ndiff --git a/midx.h b/midx.h\nindex 22e8e53288e..5578cd7b835 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,6 +47,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n+#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f775fc1ce69..c0607172827 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -26,22 +26,413 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-setup_bitmap_history\n-\n-test_expect_success 'setup writing bitmaps during repack' '\n-\tgit config repack.writeBitmaps true\n-'\n-\n-test_expect_success 'full repack creates bitmaps' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+test_bitmap_cases () {\n+\twriteLookupTable=false\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase \"$i\" in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test repository' '\n+\t\trm -fr * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'setup writing bitmaps during repack' '\n+\t\tgit config repack.writeBitmaps true\n+\t'\n+\n+\ttest_expect_success 'full repack creates bitmaps' '\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit repack -ad &&\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\t\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n+\t'\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local loose)' '\n+\t\tgit init --bare alt.git &&\n+\t\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\t\techo content1 >file1 &&\n+\t\t# non-local loose object which is not present in bitmapped pack\n+\t\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t\t# non-local loose object which is also present in bitmapped pack\n+\t\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\t\tgit add file1 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file1 &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\t\tgit index-pack 1.pack &&\n+\t\tlist_packed_objects 1.idx >1.objects &&\n+\t\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t\t! has_any nonlocal-loose 1.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\t\techo content2 >file2 &&\n+\t\tblob2=$(git hash-object -w file2) &&\n+\t\tgit add file2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file2 &&\n+\t\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\t\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\t\trm $(objpath $blob2) &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\t\tgit index-pack 2a.pack &&\n+\t\tlist_packed_objects 2a.idx >2a.objects &&\n+\t\t! has_any keepobjects 2a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local pack)' '\n+\t\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\t\tgit index-pack 2b.pack &&\n+\t\tlist_packed_objects 2b.idx >2b.objects &&\n+\t\t! has_any keepobjects 2b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\t\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\t\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t\t>.git/objects/pack/$packbitmap.keep &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\t\tgit index-pack 3a.pack &&\n+\t\tlist_packed_objects 3a.idx >3a.objects &&\n+\t\t! has_any packbitmap.objects 3a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\t\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\t\trm -f .git/objects/pack/multi-pack-index &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\t\tgit index-pack 3b.pack &&\n+\t\tlist_packed_objects 3b.idx >3b.objects &&\n+\t\t! has_any packbitmap.objects 3b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects to file can use bitmap' '\n+\t\t# make sure we still have 1 bitmap index from previous tests\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\t# verify equivalent packs are generated with/without using bitmap index\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n+\t\ttest_cmp packa.objects packb.objects\n+\t'\n+\n+\ttest_expect_success 'full repack, reusing previous bitmaps' '\n \t\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n-'\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output\n+\t'\n+\n+\ttest_expect_success 'fetch (full bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'create objects for missing-HAVE tests' '\n+\t\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n+\t\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n+\t\tparent=$(echo parent | git commit-tree $tree) &&\n+\t\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n+\t\tcat >revs <<-EOF\n+\t\tHEAD\n+\t\t^HEAD^\n+\t\t^$commit\n+\t\tEOF\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --incremental' '\n+\t\tcat >revs2 <<-EOF &&\n+\t\tHEAD\n+\t\t$commit\n+\t\tEOF\n+\t\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\t\tgit index-pack 4.pack &&\n+\t\tlist_packed_objects 4.idx >4.objects &&\n+\t\ttest_line_count = 4 4.objects &&\n+\t\tgit rev-list --objects $commit >revlist &&\n+\t\tcut -d\" \" -f1 revlist |sort >objects &&\n+\t\ttest_cmp 4.objects objects\n+\t'\n+\n+\ttest_expect_success 'pack with missing blob' '\n+\t\trm $(objpath $blob) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing tree' '\n+\t\trm $(objpath $tree) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing parent' '\n+\t\trm $(objpath $parent) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n+\t\tgit clone --bare . compat-jgit.git &&\n+\t\t(\n+\t\t\tcd compat-jgit.git &&\n+\t\t\trm -f objects/pack/*.bitmap &&\n+\t\t\tjgit gc &&\n+\t\t\tgit rev-list --test-bitmap HEAD\n+\t\t)\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n+\t\tgit clone --bare . compat-us.git &&\n+\t\t(\n+\t\t\tcd compat-us.git &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\tgit repack -adb &&\n+\t\t\t# jgit gc will barf if it does not like our bitmaps\n+\t\t\tjgit gc\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'splitting packs does not generate bogus bitmaps' '\n+\t\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n+\t\tgit add rand &&\n+\t\tgit commit -m \"commit with big file\" &&\n+\t\tgit -c pack.packSizeLimit=500k repack -adb &&\n+\t\tgit init --bare no-bitmaps.git &&\n+\t\tgit -C no-bitmaps.git fetch .. HEAD\n+\t'\n+\n+\ttest_expect_success 'set up reusable pack' '\n+\t\trm -f .git/objects/pack/*.keep &&\n+\t\tgit repack -adb &&\n+\t\treusable_pack () {\n+\t\t\tgit for-each-ref --format=\"%(objectname)\" |\n+\t\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n+\t\t}\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --honor-pack-keep' '\n+\t\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n+\t\tfor i in .git/objects/pack/*.pack\n+\t\tdo\n+\t\t\t>${i%.pack}.keep || return 1\n+\t\tdone &&\n+\t\treusable_pack --honor-pack-keep >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --local' '\n+\t\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n+\t\treusable_pack --local >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --incremental' '\n+\t\treusable_pack --incremental >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\t\ttest_config pack.writebitmaphashcache false &&\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupted.bitmap.index stderr\n+\t'\n+\n+\t# Create a state of history with these properties:\n+\t#\n+\t# - refs that allow a client to fetch some new history, while sharing some old\n+\t# history with the server; we use branches delta-reuse-old and\n+\t# delta-reuse-new here\n+\t#\n+\t# - the new history contains an object that is stored on the server as a delta\n+\t# against a base that is in the old history\n+\t#\n+\t# - the base object is not immediately reachable from the tip of the old\n+\t# history; finding it would involve digging down through history we know the\n+\t# other side has\n+\t#\n+\t# This should result in a state where fetching from old->new would not\n+\t# traditionally reuse the on-disk delta (because we'd have to dig to realize\n+\t# that the client has it), but we will do so if bitmaps can tell us cheaply\n+\t# that the other side has it.\n+\ttest_expect_success 'set up thin delta-reuse parent' '\n+\t\t# This first commit contains the buried base object.\n+\t\ttest-tool genrandom delta 16384 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m \"delta base\" &&\n+\t\tbase=$(git rev-parse --verify HEAD:file) &&\n+\n+\t\t# These intermediate commits bury the base back in history.\n+\t\t# This becomes the \"old\" state.\n+\t\tfor i in 1 2 3 4 5\n+\t\tdo\n+\t\t\techo $i >file &&\n+\t\t\tgit commit -am \"intermediate $i\" || return 1\n+\t\tdone &&\n+\t\tgit branch delta-reuse-old &&\n+\n+\t\t# And now our new history has a delta against the buried base. Note\n+\t\t# that this must be smaller than the original file, since pack-objects\n+\t\t# prefers to create deltas from smaller objects to larger.\n+\t\ttest-tool genrandom delta 16300 >file &&\n+\t\tgit commit -am \"delta result\" &&\n+\t\tdelta=$(git rev-parse --verify HEAD:file) &&\n+\t\tgit branch delta-reuse-new &&\n+\n+\t\t# Repack with bitmaps and double check that we have the expected delta\n+\t\t# relationship.\n+\t\tgit repack -adb &&\n+\t\thave_delta $delta $base\n+\t'\n+\n+\t# Now we can sanity-check the non-bitmap behavior (that the server is not able\n+\t# to reuse the delta). This isn't strictly something we care about, so this\n+\t# test could be scrapped in the future. But it makes sure that the next test is\n+\t# actually triggering the feature we want.\n+\t#\n+\t# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n+\t# we actually perform the fetch, retain the resulting pack, and inspect the\n+\t# result.\n+\ttest_expect_success 'fetch without bitmaps ignores delta against old base' '\n+\t\ttest_config pack.usebitmaps false &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $ZERO_OID\n+\t\t)\n+\t'\n+\n+\t# And do the same for the bitmap case, where we do expect to find the delta.\n+\ttest_expect_success 'fetch with bitmaps can reuse old base' '\n+\t\ttest_config pack.usebitmaps true &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $base\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\t# create enough commits that not all are receive bitmap\n+\t\t\t# coverage even if they are all at the tip of some reference.\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\t\tgit rev-list HEAD >commits.raw &&\n+\t\t\tsort commits &&\n+\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin bitmaps &&\n+\n+\t\t\t# remember which commits did not receive bitmaps\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_file_not_empty before &&\n+\n+\t\t\t# mark the commits which did not receive bitmaps as preferred,\n+\t\t\t# and generate the bitmap again\n+\t\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'complains about multiple pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\ttest_commit base &&\n+\n+\t\t\tgit repack -adb &&\n+\t\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n+\t\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n+\n+\t\t\ttest_commit other &&\n+\t\t\tgit repack -ab &&\n+\n+\t\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n+\n+\t\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n+\t\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n+\t\t\ttest_line_count = 2 packs &&\n+\t\t\ttest_line_count = 2 bitmaps &&\n+\n+\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\tgrep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-basic_bitmap_tests\n+test_bitmap_cases\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -54,375 +445,12 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n-test_expect_success 'pack-objects respects --local (non-local loose)' '\n-\tgit init --bare alt.git &&\n-\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n-\techo content1 >file1 &&\n-\t# non-local loose object which is not present in bitmapped pack\n-\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n-\t# non-local loose object which is also present in bitmapped pack\n-\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n-\tgit add file1 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file1 &&\n-\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n-\tgit index-pack 1.pack &&\n-\tlist_packed_objects 1.idx >1.objects &&\n-\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n-\t! has_any nonlocal-loose 1.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n-\techo content2 >file2 &&\n-\tblob2=$(git hash-object -w file2) &&\n-\tgit add file2 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file2 &&\n-\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n-\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n-\trm $(objpath $blob2) &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n-\tgit index-pack 2a.pack &&\n-\tlist_packed_objects 2a.idx >2a.objects &&\n-\t! has_any keepobjects 2a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local pack)' '\n-\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n-\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n-\tgit index-pack 2b.pack &&\n-\tlist_packed_objects 2b.idx >2b.objects &&\n-\t! has_any keepobjects 2b.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tpackbitmap=$(basename $(cat output) .bitmap) &&\n-\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n-\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n-\t>.git/objects/pack/$packbitmap.keep &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n-\tgit index-pack 3a.pack &&\n-\tlist_packed_objects 3a.idx >3a.objects &&\n-\t! has_any packbitmap.objects 3a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n-\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n-\trm -f .git/objects/pack/multi-pack-index &&\n-\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n-\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n-\tgit index-pack 3b.pack &&\n-\tlist_packed_objects 3b.idx >3b.objects &&\n-\t! has_any packbitmap.objects 3b.objects\n-'\n-\n-test_expect_success 'pack-objects to file can use bitmap' '\n-\t# make sure we still have 1 bitmap index from previous tests\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\t# verify equivalent packs are generated with/without using bitmap index\n-\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n-\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n-\ttest_cmp packa.objects packb.objects\n-'\n-\n-test_expect_success 'full repack, reusing previous bitmaps' '\n-\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n-'\n-\n-test_expect_success 'fetch (full bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'create objects for missing-HAVE tests' '\n-\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n-\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n-\tparent=$(echo parent | git commit-tree $tree) &&\n-\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n-\tcat >revs <<-EOF\n-\tHEAD\n-\t^HEAD^\n-\t^$commit\n-\tEOF\n-'\n-\n-test_expect_success 'pack-objects respects --incremental' '\n-\tcat >revs2 <<-EOF &&\n-\tHEAD\n-\t$commit\n-\tEOF\n-\tgit pack-objects --incremental --stdout --revs 4.pack &&\n-\tgit index-pack 4.pack &&\n-\tlist_packed_objects 4.idx >4.objects &&\n-\ttest_line_count = 4 4.objects &&\n-\tgit rev-list --objects $commit >revlist &&\n-\tcut -d\" \" -f1 revlist |sort >objects &&\n-\ttest_cmp 4.objects objects\n-'\n-\n-test_expect_success 'pack with missing blob' '\n-\trm $(objpath $blob) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n+test_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'pack with missing tree' '\n-\trm $(objpath $tree) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success 'pack with missing parent' '\n-\trm $(objpath $parent) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n-\tgit clone --bare . compat-jgit.git &&\n-\t(\n-\t\tcd compat-jgit.git &&\n-\t\trm -f objects/pack/*.bitmap &&\n-\t\tjgit gc &&\n-\t\tgit rev-list --test-bitmap HEAD\n-\t)\n-'\n-\n-test_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n-\tgit clone --bare . compat-us.git &&\n-\t(\n-\t\tcd compat-us.git &&\n-\t\tgit repack -adb &&\n-\t\t# jgit gc will barf if it does not like our bitmaps\n-\t\tjgit gc\n-\t)\n-'\n-\n-test_expect_success 'splitting packs does not generate bogus bitmaps' '\n-\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n-\tgit add rand &&\n-\tgit commit -m \"commit with big file\" &&\n-\tgit -c pack.packSizeLimit=500k repack -adb &&\n-\tgit init --bare no-bitmaps.git &&\n-\tgit -C no-bitmaps.git fetch .. HEAD\n-'\n-\n-test_expect_success 'set up reusable pack' '\n-\trm -f .git/objects/pack/*.keep &&\n-\tgit repack -adb &&\n-\treusable_pack () {\n-\t\tgit for-each-ref --format=\"%(objectname)\" |\n-\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n-\t}\n-'\n-\n-test_expect_success 'pack reuse respects --honor-pack-keep' '\n-\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n-\tfor i in .git/objects/pack/*.pack\n-\tdo\n-\t\t>${i%.pack}.keep || return 1\n-\tdone &&\n-\treusable_pack --honor-pack-keep >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --local' '\n-\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n-\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n-\treusable_pack --local >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --incremental' '\n-\treusable_pack --incremental >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n-\ttest_config pack.writebitmaphashcache false &&\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt.ewah.bitmap stderr\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (cache)' '\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupted.bitmap.index stderr\n-'\n-\n-# Create a state of history with these properties:\n-#\n-# - refs that allow a client to fetch some new history, while sharing some old\n-# history with the server; we use branches delta-reuse-old and\n-# delta-reuse-new here\n-#\n-# - the new history contains an object that is stored on the server as a delta\n-# against a base that is in the old history\n-#\n-# - the base object is not immediately reachable from the tip of the old\n-# history; finding it would involve digging down through history we know the\n-# other side has\n-#\n-# This should result in a state where fetching from old->new would not\n-# traditionally reuse the on-disk delta (because we'd have to dig to realize\n-# that the client has it), but we will do so if bitmaps can tell us cheaply\n-# that the other side has it.\n-test_expect_success 'set up thin delta-reuse parent' '\n-\t# This first commit contains the buried base object.\n-\ttest-tool genrandom delta 16384 >file &&\n-\tgit add file &&\n-\tgit commit -m \"delta base\" &&\n-\tbase=$(git rev-parse --verify HEAD:file) &&\n-\n-\t# These intermediate commits bury the base back in history.\n-\t# This becomes the \"old\" state.\n-\tfor i in 1 2 3 4 5\n-\tdo\n-\t\techo $i >file &&\n-\t\tgit commit -am \"intermediate $i\" || return 1\n-\tdone &&\n-\tgit branch delta-reuse-old &&\n-\n-\t# And now our new history has a delta against the buried base. Note\n-\t# that this must be smaller than the original file, since pack-objects\n-\t# prefers to create deltas from smaller objects to larger.\n-\ttest-tool genrandom delta 16300 >file &&\n-\tgit commit -am \"delta result\" &&\n-\tdelta=$(git rev-parse --verify HEAD:file) &&\n-\tgit branch delta-reuse-new &&\n-\n-\t# Repack with bitmaps and double check that we have the expected delta\n-\t# relationship.\n-\tgit repack -adb &&\n-\thave_delta $delta $base\n-'\n-\n-# Now we can sanity-check the non-bitmap behavior (that the server is not able\n-# to reuse the delta). This isn't strictly something we care about, so this\n-# test could be scrapped in the future. But it makes sure that the next test is\n-# actually triggering the feature we want.\n-#\n-# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n-# we actually perform the fetch, retain the resulting pack, and inspect the\n-# result.\n-test_expect_success 'fetch without bitmaps ignores delta against old base' '\n-\ttest_config pack.usebitmaps false &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $ZERO_OID\n-\t)\n-'\n-\n-# And do the same for the bitmap case, where we do expect to find the delta.\n-test_expect_success 'fetch with bitmaps can reuse old base' '\n-\ttest_config pack.usebitmaps true &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $base\n-\t)\n-'\n-\n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\t# create enough commits that not all are receive bitmap\n-\t\t# coverage even if they are all at the tip of some reference.\n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n-\n-\t\tgit rev-list HEAD >commits.raw &&\n-\t\tsort commits &&\n-\n-\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n-\n-\t\t# remember which commits did not receive bitmaps\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_file_not_empty before &&\n-\n-\t\t# mark the commits which did not receive bitmaps as preferred,\n-\t\t# and generate the bitmap again\n-\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n-\n-\t\t! test_cmp before after\n-\t)\n-'\n-\n-test_expect_success 'complains about multiple pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit base &&\n-\n-\t\tgit repack -adb &&\n-\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n-\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n-\n-\t\ttest_commit other &&\n-\t\tgit repack -ab &&\n-\n-\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n-\n-\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n-\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n-\t\ttest_line_count = 2 packs &&\n-\t\ttest_line_count = 2 bitmaps &&\n-\n-\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\tgrep \"ignoring extra bitmap file\" err\n-\t)\n+test_expect_success 'verify writing bitmap lookup table when enabled' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2\" \\\n+\t\tgit repack -ad &&\n+\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n test_done\ndiff --git a/t/t5311-pack-bitmaps-shallow.sh b/t/t5311-pack-bitmaps-shallow.sh\nindex 872a95df338..f74c6a2da47 100755\n--- a/t/t5311-pack-bitmaps-shallow.sh\n+++ b/t/t5311-pack-bitmaps-shallow.sh\n@@ -17,23 +17,40 @@ test_description='check bitmap operation with shallow repositories'\n # the tree for A. But in a shallow one, we've grafted away\n # A, and fetching A to B requires that the other side send\n # us the tree for file=1.\n-test_expect_success 'setup shallow repo' '\n-\techo 1 >file &&\n-\tgit add file &&\n-\tgit commit -m orig &&\n-\techo 2 >file &&\n-\tgit commit -a -m update &&\n-\tgit clone --no-local --bare --depth=1 . shallow.git &&\n-\techo 1 >file &&\n-\tgit commit -a -m repeat\n-'\n-\n-test_expect_success 'turn on bitmaps in the parent' '\n-\tgit repack -adb\n-'\n-\n-test_expect_success 'shallow fetch from bitmapped repo' '\n-\t(cd shallow.git && git fetch)\n-'\n+test_shallow_bitmaps () {\n+\twriteLookupTable=false\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup shallow repo' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\techo 1 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m orig &&\n+\t\techo 2 >file &&\n+\t\tgit commit -a -m update &&\n+\t\tgit clone --no-local --bare --depth=1 . shallow.git &&\n+\t\techo 1 >file &&\n+\t\tgit commit -a -m repeat\n+\t'\n+\n+\ttest_expect_success 'turn on bitmaps in the parent' '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_expect_success 'shallow fetch from bitmapped repo' '\n+\t\t(cd shallow.git && git fetch)\n+\t'\n+}\n+\n+test_shallow_bitmaps\n+\n \n test_done\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4fe57414c13..3b206adcee6 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -15,17 +15,24 @@ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n sane_unset GIT_TEST_MIDX_WRITE_REV\n sane_unset GIT_TEST_MIDX_READ_RIDX\n \n-midx_bitmap_core\n-\n bitmap_reuse_tests() {\n \tfrom=$1\n \tto=$2\n+\twriteLookupTable=false\n+\n+\tfor i in $3-${$#}\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n \n \ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n \t\trm -fr repo &&\n \t\tgit init repo &&\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk 16 &&\n \t\t\tgit tag old-tip &&\n \n@@ -43,6 +50,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk --id=further 16 &&\n \t\t\tgit tag new-tip &&\n \n@@ -59,6 +67,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\tgit for-each-ref &&\n \t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n \t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n@@ -66,244 +75,294 @@ bitmap_reuse_tests() {\n \t'\n }\n \n-bitmap_reuse_tests 'pack' 'MIDX'\n-bitmap_reuse_tests 'MIDX' 'pack'\n-bitmap_reuse_tests 'MIDX' 'MIDX'\n+test_midx_bitmap_cases () {\n+\twriteLookupTable=false\n+\twriteBitmapLookupTable=\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\")\n+\t\t\twriteLookupTable=true\n+\t\t\twriteBitmapLookupTable=\"$i\"\n+\t\t\t;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test_repository' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n \n-test_expect_success 'missing object closure fails gracefully' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\tmidx_bitmap_core\n \n-\t\ttest_commit loose &&\n-\t\ttest_commit packed &&\n+\tbitmap_reuse_tests 'pack' 'MIDX' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'pack' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'MIDX' \"$writeBitmapLookupTable\"\n \n-\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n-\t\t# commit.\n-\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n-\t\t$(git rev-parse packed)\n-\t\tEOF\n+\ttest_expect_success 'missing object closure fails gracefully' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n-\t\tgrep \"doesn.t have full closure\" err &&\n-\t\ttest_path_is_missing $midx\n-\t)\n-'\n+\t\t\ttest_commit loose &&\n+\t\t\ttest_commit packed &&\n \n-midx_bitmap_partial_tests\n+\t\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t\t# commit.\n+\t\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t\t$(git rev-parse packed)\n+\t\t\tEOF\n \n-test_expect_success 'removing a MIDX clears stale bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\t\ttest_commit base &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\t\tgrep \"doesn.t have full closure\" err &&\n+\t\t\ttest_path_is_missing $midx\n+\t\t)\n+\t'\n \n-\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n-\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm $midx &&\n+\tmidx_bitmap_partial_tests\n \n-\t\t# Then write a new MIDX.\n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\ttest_expect_success 'removing a MIDX clears stale bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\ttest_commit base &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm $midx &&\n+\n+\t\t\t# Then write a new MIDX.\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest_path_is_missing $stale_bitmap\n+\t\t)\n+\t'\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_missing $stale_bitmap\n-\t)\n-'\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\t\tbitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n \n-\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n-\t\t\tmulti-pack-index write --bitmap &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\ttest_expect_success 'writing a bitmap with --refs-snapshot' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'writing a bitmap with --refs-snapshot' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\ttest_commit two &&\n \n-\t\ttest_commit one &&\n-\t\ttest_commit two &&\n+\t\t\tgit rev-parse one >snapshot &&\n \n-\t\tgit rev-parse one >snapshot &&\n+\t\t\tgit repack -ad &&\n \n-\t\tgit repack -ad &&\n+\t\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t\t# bitmaps).\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\t# First, write a MIDX which see both refs/tags/one and\n-\t\t# refs/tags/two (causing both of those commits to receive\n-\t\t# bitmaps).\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\t# Then again, but with a refs snapshot which only sees\n+\t\t\t# refs/tags/one.\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n \n-\t\t# Then again, but with a refs snapshot which only sees\n-\t\t# refs/tags/one.\n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\t! grep \"$(git rev-parse two)\" bitmaps\n-\t)\n-'\n+\ttest_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\t(\n+\t\t\t\tgrep -vf before commits.raw &&\n+\t\t\t\t# mark missing commits as preferred\n+\t\t\t\tsed \"s/^/+/\" before\n+\t\t\t) >snapshot &&\n \n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n+\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'hash-cache values are propagated from pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n \t\t(\n-\t\t\tgrep -vf before commits.raw &&\n-\t\t\t# mark missing commits as preferred\n-\t\t\tsed \"s/^/+/\" before\n-\t\t) >snapshot &&\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\ttest_commit base &&\n+\t\t\ttest_commit base2 &&\n+\t\t\tgit repack -adb &&\n \n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\ttest-tool bitmap dump-hashes >pack.raw &&\n+\t\t\ttest_file_not_empty pack.raw &&\n+\t\t\tsort pack.raw >pack.hashes &&\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest-tool bitmap dump-hashes >midx.raw &&\n+\t\t\tsort midx.raw >midx.hashes &&\n \n-\t\ttest_commit base &&\n-\t\ttest_commit base2 &&\n-\t\tgit repack -adb &&\n+\t\t\t# ensure that every namehash in the pack bitmap can be found in\n+\t\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n+\t\t\t# unique to the pack bitmap).\n+\t\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n+\t\t\ttest_must_be_empty dropped.hashes\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap dump-hashes >pack.raw &&\n-\t\ttest_file_not_empty pack.raw &&\n-\t\tsort pack.raw >pack.hashes &&\n+\ttest_expect_success 'no .bitmap is written without any objects' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n+\t\t\tpack-$empty.idx\n+\t\t\tEOF\n \n-\t\ttest-tool bitmap dump-hashes >midx.raw &&\n-\t\tsort midx.raw >midx.hashes &&\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\terr &&\n \n-\t\t# ensure that every namehash in the pack bitmap can be found in\n-\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n-\t\t# unique to the pack bitmap).\n-\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n-\t\ttest_must_be_empty dropped.hashes\n-\t)\n-'\n+\t\t\tgrep \"bitmap without any objects\" err &&\n \n-test_expect_success 'no .bitmap is written without any objects' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'graceful fallback when missing reverse index' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n-\t\tpack-$empty.idx\n-\t\tEOF\n+\t\t\ttest_commit base &&\n \n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\terr &&\n+\t\t\t# write a pack and MIDX bitmap containing base\n+\t\t\tgit repack -adb &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\tgrep \"bitmap without any objects\" err &&\n+\t\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n+\t\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\t! grep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n-\t)\n-'\n+test_midx_bitmap_cases\n+\n+test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'graceful fallback when missing reverse index' '\n+test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n \t(\n \t\tcd repo &&\n-\n \t\ttest_commit base &&\n-\n-\t\t# write a pack and MIDX bitmap containing base\n-\t\tgit repack -adb &&\n-\t\tgit multi-pack-index write --bitmap &&\n-\n-\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n-\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\t! grep \"ignoring extra bitmap file\" err\n+\t\tgit config pack.writeBitmapLookupTable true &&\n+\t\tgit repack -ad &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n \t)\n '\n \ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex d30ba632c87..d01c61c0c7e 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -20,4 +20,13 @@ export GIT_TEST_MIDX_READ_RIDX\n midx_bitmap_core rev\n midx_bitmap_partial_tests rev\n \n+test_expect_success 'reinitialize the repository with lookup table enabled' '\n+ rm -fr * .git &&\n+ git init &&\n+ git config pack.writeBitmapLookupTable true\n+'\n+\n+midx_bitmap_core rev\n+midx_bitmap_partial_tests rev\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"458440","messageId":"e64362621d235f2c79f52e984de7a2a2794e2842.1656924376.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:14Z","receivedAt":"2022-07-04T08:46:43Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEarlier change teaches Git to write bitmap lookup table. But Git\ndoes not know how to parse them.\n\nTeach Git to parse the existing bitmap lookup table. The older\nversions of Git are not affected by it. Those versions ignore the\nlookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap.c | 266 ++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 9 ++\n t/t5310-pack-bitmaps.sh | 22 ++++\n 3 files changed, 287 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 36134222d7a..e22bbbdc60e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -82,6 +82,12 @@ struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n+\t/*\n+\t * If not NULL, this point into the commit table extension\n+\t * (within the memory mapped region `map`).\n+\t */\n+\tunsigned char *table_lookup;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -185,6 +191,16 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex->hashes = (void *)(index_end - cache_size);\n \t\t\tindex_end -= cache_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE) {\n+\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n+\t\t\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit lookup table)\"));\n+\t\t\tif (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n+\t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n@@ -211,11 +227,13 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,\n \n \thash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret);\n \n-\t/* a 0 return code means the insertion succeeded with no changes,\n-\t * because the SHA1 already existed on the map. this is bad, there\n-\t * shouldn't be duplicated commits in the index */\n+\t/*\n+\t * A 0 return code means the insertion succeeded with no changes,\n+\t * because the SHA1 already existed on the map. This is bad, there\n+\t * shouldn't be duplicated commits in the index.\n+\t */\n \tif (ret == 0) {\n-\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n+\t\terror(_(\"duplicate entry in bitmap index: %s\"), oid_to_hex(oid));\n \t\treturn NULL;\n \t}\n \n@@ -470,7 +488,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n \n-\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n \treturn 0;\n@@ -557,13 +575,229 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct bitmap_lookup_table_triplet {\n+\tuint32_t commit_pos;\n+\tuint64_t offset;\n+\tuint32_t xor_row;\n+};\n+\n+struct bitmap_lookup_table_xor_item {\n+\tstruct object_id oid;\n+\tuint64_t offset;\n+};\n+\n+/*\n+ * This function gets the raw triplet from `row`'th row in the\n+ * lookup table and fills that data to the `triplet`.\n+ */\n+static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t pos,\n+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = NULL;\n+\tif (pos >= bitmap_git->entry_count)\n+\t\treturn error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n+\n+\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\n+\ttriplet->commit_pos = get_be32(p);\n+\tp += sizeof(uint32_t);\n+\ttriplet->offset = get_be64(p);\n+\tp += sizeof(uint64_t);\n+\ttriplet->xor_row = get_be32(p);\n+\treturn 0;\n+}\n+\n+/*\n+ * Searches for a matching triplet. `va` is a pointer\n+ * to the wanted commit position value. `vb` points to\n+ * a triplet in lookup table. The first 4 bytes of each\n+ * triplet (pointed by `vb`) are compared with `*va`.\n+ */\n+static int triplet_cmp(const void *va, const void *vb)\n+{\n+\n+\tuint32_t a = *(uint32_t *)va;\n+\tuint32_t b = get_be32(vb);\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n+\t\t\t struct object_id *oid,\n+\t\t\t uint32_t *result)\n+{\n+\tint found;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n+\telse\n+\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n+\n+\treturn found;\n+}\n+\n+/*\n+ * `bsearch_triplet` function searches for the raw triplet having\n+ * commit position same as `commit_pos` and fills `triplet`\n+ * object from the raw triplet. Returns 1 on success and 0\n+ * on failure.\n+ */\n+static int bsearch_triplet(uint32_t *commit_pos,\n+\t\t\t struct bitmap_index *bitmap_git,\n+\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = bsearch(commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n+\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n+\n+\tif (!p)\n+\t\treturn 0;\n+\ttriplet->commit_pos = get_be32(p);\n+\tp += sizeof(uint32_t);\n+\ttriplet->offset = get_be64(p);\n+\tp += sizeof(uint64_t);\n+\ttriplet->xor_row = get_be32(p);\n+\treturn 1;\n+}\n+\n+static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t struct commit *commit)\n+{\n+\tuint32_t commit_pos, xor_row;\n+\tuint64_t offset;\n+\tint flags;\n+\tstruct bitmap_lookup_table_triplet triplet;\n+\tstruct object_id *oid = &commit->object.oid;\n+\tstruct ewah_bitmap *bitmap;\n+\tstruct stored_bitmap *xor_bitmap = NULL;\n+\n+\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n+\n+\tif (!found)\n+\t\treturn NULL;\n+\n+\tif (!bsearch_triplet(&commit_pos, bitmap_git, &triplet))\n+\t\treturn NULL;\n+\n+\toffset = triplet.offset;\n+\txor_row = triplet.xor_row;\n+\n+\tif (xor_row != 0xffffffff) {\n+\t\tint xor_flags;\n+\t\tkhiter_t hash_pos;\n+\t\tuint64_t offset_xor;\n+\t\tstruct bitmap_lookup_table_xor_item *xor_items;\n+\t\tstruct bitmap_lookup_table_xor_item xor_item;\n+\t\tsize_t xor_items_nr = 0, xor_items_alloc = 64;\n+\n+\t\tALLOC_ARRAY(xor_items, xor_items_alloc);\n+\t\twhile (xor_row != 0xffffffff) {\n+\t\t\tstruct object_id xor_oid;\n+\n+\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n+\t\t\t\tfree(xor_items);\n+\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n+\t\t\t\treturn NULL;\n+\n+\t\t\toffset_xor = triplet.offset;\n+\n+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, triplet.commit_pos) < 0) {\n+\t\t\t\tfree(xor_items);\n+\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n+\t\t\t\t\ttriplet.commit_pos);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_oid);\n+\n+\t\t\t/*\n+\t\t\t * If desired bitmap is already stored, we don't need\n+\t\t\t * to iterate further. Because we know that bitmaps\n+\t\t\t * that are needed to be parsed to parse this bitmap\n+\t\t\t * has already been stored. So, assign this stored bitmap\n+\t\t\t * to the xor_bitmap.\n+\t\t\t */\n+\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n+\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n+\t\t\t\tbreak;\n+\n+\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n+\t\t\txor_items[xor_items_nr++] = (struct bitmap_lookup_table_xor_item) {.oid = xor_oid,\n+\t\t\t\t\t\t\t\t\t\t\t .offset = offset_xor};\n+\t\t\txor_row = triplet.xor_row;\n+\t\t}\n+\n+\t\twhile (xor_items_nr) {\n+\t\t\txor_item = xor_items[xor_items_nr - 1];\n+\t\t\toffset_xor = xor_item.offset;\n+\n+\t\t\tbitmap_git->map_pos = offset_xor;\n+\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n+\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\t\t\toid_to_hex(&xor_item.oid));\n+\t\t\t\tfree(xor_items);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n+\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\t\t\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\t\t\tif (!bitmap) {\n+\t\t\t\tfree(xor_items);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\n+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item.oid, xor_bitmap, xor_flags);\n+\t\t\txor_items_nr--;\n+\t\t}\n+\n+\t\tfree(xor_items);\n+\t}\n+\n+\tbitmap_git->map_pos = offset;\n+\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n+\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\toid_to_hex(oid));\n+\t\treturn NULL;\n+\t}\n+\n+\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n+\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\tif (!bitmap)\n+\t\treturn NULL;\n+\n+\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n+}\n+\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n-\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n-\t\treturn NULL;\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n+\t\tstruct stored_bitmap *bitmap = NULL;\n+\t\tif (!bitmap_git->table_lookup)\n+\t\t\treturn NULL;\n+\n+\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\t/* NEEDSWORK: cache misses aren't recorded */\n+\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n+\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\tif (!bitmap)\n+\t\t\treturn NULL;\n+\t\treturn lookup_stored_bitmap(bitmap);\n+\t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n@@ -1699,8 +1933,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n-\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n-\t\tbitmap_git->version, bitmap_git->entry_count);\n+\tfprintf(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\t\tbitmap_git->version,\n+\t\tbitmap_git->entry_count,\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n \n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n@@ -1753,13 +1989,23 @@ void test_bitmap_walk(struct rev_info *revs)\n \n int test_bitmap_commits(struct repository *r)\n {\n-\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n+\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n+\n+\t/*\n+\t * As this function is only used to print bitmap selected\n+\t * commits, we don't have to read the commit table.\n+\t */\n \n \tif (!bitmap_git)\n \t\tdie(\"failed to load bitmap indexes\");\n \n+\tif (bitmap_git->table_lookup) {\n+\t\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\t\t\tdie(_(\"failed to load bitmap indexes\"));\n+\t}\n+\n \tkh_foreach(bitmap_git->bitmaps, oid, value, {\n \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n \t});\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 67a9d0fc303..9278f71ac91 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -23,6 +23,15 @@ struct bitmap_disk_header {\n \n #define NEEDS_BITMAP (1u<<22)\n \n+/*\n+ * The width in bytes of a single triplet in the lookup table\n+ * extension:\n+ * (commit_pos, offset, xor_row)\n+ *\n+ * whose fields ar 32-, 64-, 32- bits wide, respectively.\n+ */\n+#define BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH (16)\n+\n enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c0607172827..7e50f8e7653 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -258,6 +258,7 @@ test_bitmap_cases () {\n \n \ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \t\ttest_config pack.writebitmaphashcache false &&\n+\t\ttest_config pack.writebitmaplookuptable false &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -270,6 +271,7 @@ test_bitmap_cases () {\n \t'\n \n \ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -453,4 +455,24 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n+test_expect_success 'lookup table is actually used to traverse objects' '\n+\tgit repack -adb &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n+\t\tgit rev-list --use-bitmap-index --count --all &&\n+\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n+\ttest_config pack.writebitmaphashcache false &&\n+\tgit repack -adb &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"458441","messageId":"a155c1e2ebacf54c451a069499325cdf280606fc.1656924376.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v3 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:15Z","receivedAt":"2022-07-04T08:46:44Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd performance tests to verify the performance of lookup table with\n`pack.writeReverseIndex` enabled. This is to check the performance\nwhen the above configuration is set.\n\nLookup table makes Git run faster in most of the cases. Below is the\nresult of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\ngives similar result. The repository used in the test is linux kernel.\n\nTest this tree\n---------------------------------------------------------------------------\n5310.4: repack to disk (lookup=false) 296.55(256.53+14.52)\n5310.5: simulated clone 15.64(8.88+1.39)\n5310.6: simulated fetch 1.65(2.75+0.20)\n5310.7: pack to file (bitmap) 48.71(30.20+7.58)\n5310.8: rev-list (commits) 0.61(0.41+0.08)\n5310.9: rev-list (objects) 4.38(4.26+0.09)\n5310.10: rev-list with tag negated via --not 0.07(0.02+0.04)\n --all (objects)\n5310.11: rev-list with negative tag (objects) 0.05(0.01+0.03)\n5310.12: rev-list count with blob:none 0.08(0.03+0.04)\n5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n5310.14: rev-list count with tree:0 0.08(0.03+0.04)\n5310.15: simulated partial clone 9.45(8.12+0.41)\n5310.19: repack to disk (lookup=true) 255.92(188.13+20.47)\n5310.20: simulated clone 13.78(8.84+1.09)\n5310.21: simulated fetch 0.52(0.63+0.14)\n5310.22: pack to file (bitmap) 44.34(28.94+6.84)\n5310.23: rev-list (commits) 0.48(0.31+0.06)\n5310.24: rev-list (objects) 4.02(3.93+0.07)\n5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n5310.26: rev-list with negative tag (objects) 0.04(0.00+0.03)\n5310.27: rev-list count with blob:none 0.04(0.01+0.03)\n5310.28: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n5310.29: rev-list count with tree:0 0.04(0.01+0.03)\n5310.30: simulated partial clone 8.30(7.21+0.36)\n\nTest 4-15 are tested without using lookup table. Same tests are\nrepeated in 16-30 (using lookup table).\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 66 ++++++++++++---------\n t/perf/p5326-multi-pack-bitmaps.sh | 93 ++++++++++++++++--------------\n 2 files changed, 89 insertions(+), 70 deletions(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 7ad4f237bc3..1ad3c3f14c6 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -13,42 +13,52 @@ test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true\n+\tgit config pack.writebitmaps true &&\n+\tgit config pack.writeReverseIndex true\n '\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n+test_bitmap () {\n+\tlocal enabled=\"$1\"\n \n-test_perf 'repack to disk' '\n-\tgit repack -ad\n-'\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n \n-test_full_bitmap\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n \n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n+\ttest_perf \"repack to disk (lookup=$enabled)\" '\n+\t\tgit repack -ad\n+\t'\n \n-\t# now kill off all of the refs and pretend we had\n-\t# just the one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n+\ttest_full_bitmap\n \n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n \n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n+\t\t# now kill off all of the refs and pretend we had\n+\t\t# just the one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+}\n \n-test_partial_bitmap\n+test_bitmap false\n+test_bitmap true\n \n test_done\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nindex f2fa228f16a..c8cc68185a1 100755\n--- a/t/perf/p5326-multi-pack-bitmaps.sh\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -6,47 +6,56 @@ test_description='Tests performance using midx bitmaps'\n \n test_perf_large_repo\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n-\n-test_expect_success 'start with bitmapped pack' '\n-\tgit repack -adb\n-'\n-\n-test_perf 'setup multi-pack index' '\n-\tgit multi-pack-index write --bitmap\n-'\n-\n-test_expect_success 'drop pack bitmap' '\n-\trm -f .git/objects/pack/pack-*.bitmap\n-'\n-\n-test_full_bitmap\n-\n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now pretend we have just one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n-\n-test_partial_bitmap\n+test_bitmap () {\n+\tlocal enabled=\"$1\"\n+\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n+\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n+\n+\ttest_expect_success \"start with bitmapped pack (lookup=$enabled)\" '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_perf \"setup multi-pack index (lookup=$enabled)\" '\n+\t\tgit multi-pack-index write --bitmap\n+\t'\n+\n+\ttest_expect_success \"drop pack bitmap (lookup=$enabled)\" '\n+\t\trm -f .git/objects/pack/pack-*.bitmap\n+\t'\n+\n+\ttest_full_bitmap\n+\n+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now pretend we have just one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+}\n+\n+test_bitmap false\n+test_bitmap true\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"458442","messageId":"4f9f10494855265133bc315fccedf81ced65ce83.1656924376.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v3 6/6] p5310-pack-bitmaps.sh: remove pack.writeReverseIndex","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-04T08:46:16Z","receivedAt":"2022-07-04T08:46:47Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe previous change enables the `pack.writereverseindex` to see\nthe effect of writing reverse index in the performance test.\n\nRemove the `pack.writeReverseIndex` configuration.\n\nBelow is the result of performance test. Output format is in\nseconds.\n\nTest this tree\n------------------------------------------------------------------------\n5310.4: repack to disk (lookup=false) 293.80(251.30+14.30)\n5310.5: simulated clone 12.50(5.15+1.36)\n5310.6: simulated fetch 1.83(2.90+0.23)\n5310.7: pack to file (bitmap) 39.70(20.25+7.14)\n5310.8: rev-list (commits) 1.00(0.60+0.13)\n5310.9: rev-list (objects) 4.11(4.00+0.10)\n5310.10: rev-list with tag negated via --not 0.07(0.02+0.05)\n --all (objects)\n5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n5310.12: rev-list count with blob:none 0.27(0.18+0.08)\n5310.13: rev-list count with blob:limit=1k 6.41(5.98+0.41)\n5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n5310.15: simulated partial clone 4.34(3.29+0.37)\n5310.19: repack to disk (lookup=true) 250.93(171.97+20.78)\n5310.20: simulated clone 10.80(5.14+1.06)\n5310.21: simulated fetch 0.71(0.79+0.16)\n5310.22: pack to file (bitmap) 39.49(20.19+6.98)\n5310.23: rev-list (commits) 0.81(0.48+0.09)\n5310.24: rev-list (objects) 3.48(3.38+0.09)\n5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n5310.28: rev-list count with blob:limit=1k 6.21(5.76+0.29)\n5310.29: rev-list count with tree:0 0.23(0.16+0.06)\n5310.30: simulated partial clone 4.53(3.14+0.39)\n\nTests 4-15 are without the use of lookup table. The rests are\nrepeatation of the previous tests but using lookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 3 +--\n 1 file changed, 1 insertion(+), 2 deletions(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 1ad3c3f14c6..ac5b7341e8e 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -13,8 +13,7 @@ test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true &&\n-\tgit config pack.writeReverseIndex true\n+\tgit config pack.writebitmaps true\n '\n \n test_bitmap () {\n-- \ngitgitgadget\n"},{"id":"458449","messageId":"20220704163506.76162-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-04T16:35:06Z","receivedAt":"2022-07-04T16:35:30Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nOops, I forgot to edit the PR cover letter!\n\nI am adding it here. Sorry for that :P\n\nChanges since v2:\n\n * Log messages related issues are fixed.\n * `pack.writeBitmapLookupTable` is now by default disabled.\n * Documentations are improved.\n * `xor_row` is used instead of `xor_pos` in triplets.\n * In `pack-bitmap-write.c`, `off_t *` is used for `offsets` array\n (Instead of `uint64_t *`).\n * `struct bitmap_lookup_table_triplet` is introduced and functions\n Like `triplet_get_offset()` and `triplet_get_xor_pos()` are removed.\n * `table_size` is getting subtracted from `index_end` irrespective of\n the value of `GIT_TEST_READ_COMMIT_TABLE`.\n * xor stack filling loop will stop iterating if a xor bitmap is already\n stored/parsed.\n * The stack will now store `bitmap_lookup_table_xor_item` items\n Of plain xor_row.\n * bitmap related test files are reformatted to allow repeating of tests\n with bitmap extension enabled.\n * comments are added.\n\nThanks :)\n"},{"id":"458539","messageId":"xmqqiloagi80.fsf@gitster.g","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-06T19:21:35Z","receivedAt":"2022-07-06T19:21:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Abhradeep Chakraborty via GitGitGadget\" \nwrites:\n\n> When parsing the .bitmap file, git loads all the bitmaps one by one even if\n> some of the bitmaps are not necessary. We can remove this overhead by\n> loading only the necessary bitmaps. A look up table extension can solve this\n> issue.\n>\n> Changes since v1:\n>\n> This is the second version which addressed all (I think) the reviews. Please\n> notify me if some reviews are not addressed :)\n\nIs this the second version that is labeled as \"v3\" ;-)?\n\n> Documentation/technical/bitmap-format.txt | 39 ++\n\nI haven't tried merging it yet, but doesn't [1/6] overlap with and\nsemantically depend on your other series that touch the formatting\nof this file?\n\nThanks.\n"},{"id":"458557","messageId":"20220707084818.79881-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"xmqqiloagi80.fsf@gitster.g","subject":"Re: [PATCH v3 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-07T08:48:18Z","receivedAt":"2022-07-07T08:48:56Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nJunio C Hamano wrote:\n\n> > This is the second version which addressed all (I think) the reviews. Please\n> > notify me if some reviews are not addressed :)\n>\n> Is this the second version that is labeled as \"v3\" ;-)?\n\nHi junio,\n\nNo, it is actually the third version. I forgot to update the cover\nletter :P.\n\nI am using Github's gitgitgadget to submit PRs and it\nuses PR description as the cover letter.\n\nSo before submitting a new version of patchset, PR description\nmust be updated which I missed this time.\n\nI wrote a reply comment[1] where you can find a summary of all the\nnew changes.\n\n[1] https://lore.kernel.org/git/20220704163506.76162-1-chakrabortyabhradeep79@gmail.com/\n\n> > Documentation/technical/bitmap-format.txt | 39 ++\n>\n> I haven't tried merging it yet, but doesn't [1/6] overlap with and\n> semantically depend on your other series that touch the formatting\n> of this file?\n\nCorrect, [1/6] indeed depends on my previous patch series[2] and it\nis assuming that that series has already been merged. As far as it seems,\nit will not create any merge conflicts while merging but I am not sure.\nThis would be interesting to see.\n\n[2] https://lore.kernel.org/git/pull.1246.v4.git.1655355834.gitgitgadget@gmail.com/\n\nThanks :)\n"},{"id":"458584","messageId":"7f7e8d91-47bc-ede4-a552-2ddc9fe98a1e@gmail.com","threadId":"58038","inReplyTo":"20220707084818.79881-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v3 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2022-07-07T18:09:09Z","receivedAt":"2022-07-07T18:09:17Z","isPatch":true,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On 07-07-2022 14:18, Abhradeep Chakraborty wrote:\n> \n> Junio C Hamano wrote:\n> \n>>> Documentation/technical/bitmap-format.txt | 39 ++\n>>\n>> I haven't tried merging it yet, but doesn't [1/6] overlap with and\n>> semantically depend on your other series that touch the formatting\n>> of this file?\n> \n> Correct, [1/6] indeed depends on my previous patch series[2] and it\n> is assuming that that series has already been merged.\n\nI suppose it's the opposite. A quick check shows that the patch applies\ncleanly over 'master' but fails to apply over 'next' which has the\nchanges from your other patch series. So, the base branch for [1/6]\nis 'master'. The other 5 patches clearly don't conflict.\n\n> As far as it seems,\n> it will not create any merge conflicts while merging but I am not sure.\n> This would be interesting to see.\n>\n\nSince the first hunk of 1/6 and your other series touch the same area\nof Documentation/technical/bitmap-format.txt, the changes conflict.\nJunio might be able to handle this one. If not, you would need to look\ninto separate 1/6 and based it over your other series to avoid the\nconflict.\n\n--\nSivaraam\n"},{"id":"458593","messageId":"20220707184233.80579-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"7f7e8d91-47bc-ede4-a552-2ddc9fe98a1e@gmail.com","subject":"Re: [PATCH v3 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-07T18:42:33Z","receivedAt":"2022-07-07T18:43:00Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nKaartic Sivaraam wrote:\n\n>> Correct, [1/6] indeed depends on my previous patch series[2] and it\n>> is assuming that that series has already been merged.\n>\n> I suppose it's the opposite. A quick check shows that the patch applies\n> cleanly over 'master' but fails to apply over 'next' which has the\n> changes from your other patch series. So, the base branch for [1/6]\n> is 'master'. The other 5 patches clearly don't conflict.\n\nActually by saying \"[1/6] indeed depends on my previous patch series[2]\nand it is assuming that that series has already been merged.\", I wanted\nto mean that the format followed in this patch (e.g. description list,\nindentation etc.) is dependent on the format changes introduced in that\nPatch series.\n\nIf you say about the base branch, yes, you're right. The base branch is\n'Master'.\n\n> Since the first hunk of 1/6 and your other series touch the same area\n> of Documentation/technical/bitmap-format.txt, the changes conflict.\n> Junio might be able to handle this one. If not, you would need to look\n> into separate 1/6 and based it over your other series to avoid the\n> conflict.\n\nOh, I see. I have no problem doing that :)\nLet me know if Junio face any problem fixing the conflict.\n\nThanks :)\n"},{"id":"458650","messageId":"ac52cfea-edb0-b68b-36e2-ab45d2959727@iee.email","threadId":"58038","inReplyTo":"f72bf11e6efb4690ae808c0b56c3991c2b1ef266.1656924376.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2022-07-08T16:38:28Z","receivedAt":"2022-07-08T16:38:37Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Abhradeep,\n\nOn 04/07/2022 09:46, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> When reading bitmap file, Git loads each and every bitmap one by one\n> even if all the bitmaps are not required. A \"bitmap lookup table\"\n> extension to the bitmap format can reduce the overhead of loading\n> bitmaps which stores a list of bitmapped commit id pos (in the midx\n> or pack, along with their offset and xor offset. This way git can\n> load only the necessary bitmaps without loading the previous bitmaps.\n>\n> Older versions of Git ignore the lookup table extension and don't\n> throw any kind of warning or error while parsing the bitmap file.\n>\n> Add some information for the new \"bitmap lookup table\" extension in the\n> bitmap-format documentation.\n\nNot sure if this is new in this extension, but should there be a link or\ntwo to the basics of XOR compression and some of the bitmap look up\ntechniques?\n\nIt's not always obvious if these techniques are 'heuristic' and only\nhave partial commit data, or they have all the commits listed, Nor\nhow/why they work. My point is more about giving new readers a hand-up\nin their understanding, rather than simple implementation details for\nthose who already know what is going on. For example, are there any\nexternal articles that you found helpful in getting started that could\nbe referenced somewhere in the docs?\n\nSeparately I'm preparing a short series on adding 'reachability bitmap'\nand 'commit graph' (among other stuff) to the glossary as part of giving\nfolks [0] stepping stones to cross the chasm of understanding\n\nPhilip\n\n[0] me included;-)\n>\n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n> Co-Authored-by: Taylor Blau \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> Documentation/technical/bitmap-format.txt | 39 +++++++++++++++++++++++\n> 1 file changed, 39 insertions(+)\n>\n> diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\n> index 04b3ec21785..c30dc177643 100644\n> --- a/Documentation/technical/bitmap-format.txt\n> +++ b/Documentation/technical/bitmap-format.txt\n> @@ -67,6 +67,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n> \t\t\tpack/MIDX. The format and meaning of the name-hash is\n> \t\t\tdescribed below.\n> \n> +\t\t\t** {empty}\n> +\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n> +\t\t\tIf present, the end of the bitmap file contains a table\n> +\t\t\tcontaining a list of `N` \n> +\t\t\ttriplets. The format and meaning of the table is described\n> +\t\t\tbelow.\n> ++\n> +NOTE: Unlike the xor_offset used to compress an individual bitmap,\n> +`xor_row` stores an *absolute* index into the lookup table, not a location\n> +relative to the current entry.\n> +\n> \t\t4-byte entry count (network byte order)\n> \n> \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n> @@ -205,3 +216,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n> If implementations want to choose a different hashing scheme, they are\n> free to do so, but MUST allocate a new header flag (because comparing\n> hashes made under two different schemes would be pointless).\n> +\n> +Commit lookup table\n> +-------------------\n> +\n> +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n> +bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`\n> +file contains a lookup table specifying the information needed to get\n> +the desired bitmap from the entries without parsing previous unnecessary\n> +bitmaps.\n> +\n> +For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n> +contains a list of `nr_entries` triplets\n> +(sorted in the ascending order of `commit_pos`). The content of i'th\n> +triplet is -\n> +\n> +\t* {empty}\n> +\tcommit_pos (4 byte integer, network byte order): ::\n> +\tIt stores the object position of a commit (in the midx or pack\n> +\tindex).\n> +\n> +\t* {empty}\n> +\toffset (8 byte integer, network byte order): ::\n> +\tThe offset from which that commit's bitmap can be read.\n> +\n> +\t* {empty}\n> +\txor_row (4 byte integer, network byte order): ::\n> +\tThe position of the triplet whose bitmap is used to compress\n> +\tthis one, or `0xffffffff` if no such bitmap exists.\n\n"},{"id":"458680","messageId":"20220709075310.83848-1-chakrabortyabhradeep79@gmail.com","threadId":"58038","inReplyTo":"ac52cfea-edb0-b68b-36e2-ab45d2959727@iee.email","subject":"Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-09T07:53:10Z","receivedAt":"2022-07-09T07:53:39Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"\nHello Philip,\n\nPhilip Oakley wrote:\n\n> Not sure if this is new in this extension, but should there be a link or\n> two to the basics of XOR compression and some of the bitmap look up\n> techniques?\n>\n> It's not always obvious if these techniques are 'heuristic' and only\n> have partial commit data, or they have all the commits listed, Nor\n> how/why they work. My point is more about giving new readers a hand-up\n> in their understanding, rather than simple implementation details for\n> those who already know what is going on. For example, are there any\n> external articles that you found helpful in getting started that could\n> be referenced somewhere in the docs?\n\nAs this series is only about adding a lookup-table extension (and not\nabout bitmap itself), I am not sure whether it's good to include those\nthings in this series. But I agree with your point that it should be\nable build a logical understanding among the new readers.\n\nThere are some external articles[1] which talk about bitmap internals.\nBut I think it would be better if we can make a new doc file (may be\n`Documentation/technical/reachability-bitmaps.txt` or similar) rather\nthan putting those details in the `bitmap-format.txt` (As the name \nsuggests, this file should only contain format details of bitmaps).\nThat file would provide the answers of \"Why bitmaps\", \"how they are\nstored\", \"How they are fetched\", \"how they work with pack-objects,\ngit-fetch, midx etc.\", \"Detailed explanation of each bitmap extension\"\n, and lastly \"what are the future works\" (if any).\n\nWhat do you think?\n\n> Separately I'm preparing a short series on adding 'reachability bitmap'\n> and 'commit graph' (among other stuff) to the glossary as part of giving\n> folks [0] stepping stones to cross the chasm of understanding\n\nGreat!\n\nThanks :)\n\n[1] https://github.blog/2015-09-22-counting-objects/, https://github.blog/2021-04-29-scaling-monorepo-maintenance/\n"},{"id":"458713","messageId":"d70a4505-60ef-82c4-5497-499ac788782a@iee.email","threadId":"58038","inReplyTo":"20220709075310.83848-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2022-07-10T15:01:11Z","receivedAt":"2022-07-10T15:01:18Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 09/07/2022 08:53, Abhradeep Chakraborty wrote:\n> Hello Philip,\n>\n> Philip Oakley wrote:\n>\n>> Not sure if this is new in this extension, but should there be a link or\n>> two to the basics of XOR compression and some of the bitmap look up\n>> techniques?\n>>\n>> It's not always obvious if these techniques are 'heuristic' and only\n>> have partial commit data, or they have all the commits listed, Nor\n>> how/why they work. My point is more about giving new readers a hand-up\n>> in their understanding, rather than simple implementation details for\n>> those who already know what is going on. For example, are there any\n>> external articles that you found helpful in getting started that could\n>> be referenced somewhere in the docs?\n> As this series is only about adding a lookup-table extension (and not\n> about bitmap itself), I am not sure whether it's good to include those\n> things in this series. \n\nThanks for the clarification. I must have slight misread some of the\ndiscussions and falsely thought it was the XOR compression (which is a\ntechnique I wasn't really aware of), that was being provided by the\nextension - Where would it be best for me to look up the background to\nyour \"extension\" project?\n\n\n> But I agree with your point that it should be\n> able build a logical understanding among the new readers.\n\n*nod*\n>\n> There are some external articles[1] which talk about bitmap internals.\n> But I think it would be better if we can make a new doc file (may be\n> `Documentation/technical/reachability-bitmaps.txt` or similar) rather\n> than putting those details in the `bitmap-format.txt` \n\nThanks for the two links. In general I agree about the format document.\n\n> (As the name \n> suggests, this file should only contain format details of bitmaps).\n> That file would provide the answers of \"Why bitmaps\", \"how they are\n> stored\", \"How they are fetched\", \"how they work with pack-objects,\n> git-fetch, midx etc.\", \"Detailed explanation of each bitmap extension\"\n> , and lastly \"what are the future works\" (if any).\n\nOne thing I've realised on reflection is that I'm unclear how the\n'reachability bitmaps' and the 'commit-graph file' techniques relate to\neach other (and to the ODB DAG), and what features they pick out within\ntheir heuristic, explained at just enough level to allow folks to\nappreciate what the options that select them will do for their use case.\n\n>\n> What do you think?\n\nI'd be happy to collate contributions, suggestions and thoughts.\n\nTrying to create these good introductory descriptions can be really\ndifficult, as you can only step into the same river once (the 'reading\nfor the first time problem' of not being able to un-hear the\nexplanations of others when reading a 2nd draft...)\n>\n>> Separately I'm preparing a short series on adding 'reachability bitmap'\n>> and 'commit graph' (among other stuff) to the glossary as part of giving\n>> folks [0] stepping stones to cross the chasm of understanding\n> Great!\n>\n> Thanks :)\n>\n> [1] https://github.blog/2015-09-22-counting-objects/, https://github.blog/2021-04-29-scaling-monorepo-maintenance/\nThank you.\n"},{"id":"459104","messageId":"YtCjlkPdA3CUn/Aw@nand.local","threadId":"58038","inReplyTo":"d70a4505-60ef-82c4-5497-499ac788782a@iee.email","subject":"Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-14T23:15:34Z","receivedAt":"2022-07-14T23:15:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jul 10, 2022 at 04:01:11PM +0100, Philip Oakley wrote:\n> >> Not sure if this is new in this extension, but should there be a link or\n> >> two to the basics of XOR compression and some of the bitmap look up\n> >> techniques?\n> >>\n> >> It's not always obvious if these techniques are 'heuristic' and only\n> >> have partial commit data, or they have all the commits listed, Nor\n> >> how/why they work. My point is more about giving new readers a hand-up\n> >> in their understanding, rather than simple implementation details for\n> >> those who already know what is going on. For example, are there any\n> >> external articles that you found helpful in getting started that could\n> >> be referenced somewhere in the docs?\n> > As this series is only about adding a lookup-table extension (and not\n> > about bitmap itself), I am not sure whether it's good to include those\n> > things in this series.\n>\n> Thanks for the clarification. I must have slight misread some of the\n> discussions and falsely thought it was the XOR compression (which is a\n> technique I wasn't really aware of), that was being provided by the\n> extension - Where would it be best for me to look up the background to\n> your \"extension\" project?\n\nYeah, Abhradeep is right that the XOR compression isn't new, we already\nserialize bitmaps with optional XOR offsets. The gist is that we give an\noffset of some previous bitmap that is used to compress the current one\nby XORing the bits in the current bitamp with the previous one. These\nXOR-compressed bitmaps are often sparse, so they compress well and\nreduce the overall size of the .bitmap.\n\nA slightly more detailed overview can be found in\nDocumentation/technical/bitmap-format.txt under the bullet point reading\n\"1-byte XOR-offset\".\n\nThanks,\nTaylor\n"},{"id":"459105","messageId":"YtCmESpC7DmNhAcm@nand.local","threadId":"58038","inReplyTo":"5e9b985e39b0b9edee7af55dd8b0698a20062cf7.1656924376.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-14T23:26:09Z","receivedAt":"2022-07-14T23:26:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 04, 2022 at 08:46:12AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> The bitmap lookup table extension was documented by an earlier\n> change, but Git does not yet know how to write that extension.\n\nThis and the first patch both look in great shape to me. I haven't had a\nchance to take a close look through the remaining four patches, but I\nanticipate that they are in similarly-good shape.\n\nI'll have some more time to finish reviewing this tomorrow morning. I\nwant to give it a closer inspection this round to make sure that\neverything is correct (and that we're assembling the various orderings\nthe right way by stepping through it in a debugger, etc., etc.).\n\nThanks for all of your patience :-).\n\nThanks,\nTaylor\n"},{"id":"459108","messageId":"YtDPePTo52A+Uo0p@nand.local","threadId":"58038","inReplyTo":"5e9b985e39b0b9edee7af55dd8b0698a20062cf7.1656924376.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-15T02:22:48Z","receivedAt":"2022-07-15T02:22:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Abhradeep,\n\nI just wanted to make absolutely sure that I understood what the\nimplementation in this patch was doing, since I think generating and\nconverting between all of these different orderings is by far the most\nconfusing component of this series.\n\nOn Mon, Jul 04, 2022 at 08:46:12AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> The bitmap lookup table extension was documented by an earlier\n> change, but Git does not yet know how to write that extension.\n> +static int table_cmp(const void *_va, const void *_vb, void *_data)\n> +{\n> +\tuint32_t *commit_positions = _data;\n> +\tuint32_t a = commit_positions[*(uint32_t *)_va];\n> +\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n> +\n> +\tif (a > b)\n> +\t\treturn 1;\n> +\telse if (a < b)\n> +\t\treturn -1;\n> +\n> +\treturn 0;\n> +}\n\nLet's skip the above part for now, and just look at the implementation\nof writing_lookup_table():\n\n> +static void write_lookup_table(struct hashfile *f,\n> +\t\t\t struct pack_idx_entry **index,\n> +\t\t\t uint32_t index_nr,\n> +\t\t\t off_t *offsets)\n> +{\n> +\tuint32_t i;\n> +\tuint32_t *table, *table_inv, *commit_positions;\n> +\n> +\tALLOC_ARRAY(table, writer.selected_nr);\n> +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n> +\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n\nMakes sense.\n\n> +\t/* store the index positions of the commits */\n> +\tfor (i = 0; i < writer.selected_nr; i++) {\n> +\t\tint pos = commit_bitmap_writer_pos(&writer.selected[i].commit->object.oid,\n> +\t\t\t\t\t\t index, index_nr);\n> +\t\tif (pos < 0)\n> +\t\t\tBUG(_(\"trying to write commit not in index\"));\n> +\n> +\t\tcommit_positions[i] = pos;\n> +\t}\n\nBy the end of this loop, we have an array `commit_positions` which maps\nthe ith selected commit to its lexical position among all objects in the\nbitmap. IOW, `commit_positions[i] = j` means the `i`th selected commit\ncan be found at index `j` among all objects in the pack/MIDX in their\nlexical order.\n\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable[i] = i;\n\nAt this point, table[i] = i.\n\n> +\t/*\n> +\t * At the end of this sort table[j] = i means that the i'th\n> +\t * bitmap corresponds to j'th bitmapped commit in lex order of\n> +\t * OIDs.\n> +\t */\n> +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n\nAnd then we sort table by treating its values as indexes into\n`commit_positions`. Here's where I'm not sure that I follow what's going\non. You say above that `table[j] = i`, where `i` corresponds to the\norder of selected commits, and `j` is in lexical order.\n\nIf that's the case, then I'd expect that printing `index[table[j]]` for\nincreasing `j` would output OIDs in increasing lexical order. But that\ndoesn't quite seem to be the case. From a debugger session that has a\nbreakpoint after computing and sorting table, along with building\n`table_inv`:\n\n (gdb) p oid_to_hex(&index[table[0]]->oid)\n $17 = 0x555555983ea0 \"0006763074748d43b539c1c8e8882c08034ab178\"\n (gdb) p oid_to_hex(&index[table[1]]->oid)\n $18 = 0x555555983ee1 \"001ce83dd43f03dcfc67f29d38922e4a9682aab0\"\n (gdb) p oid_to_hex(&index[table[2]]->oid)\n $19 = 0x555555983f22 \"002db882ece2ab6a240e495a169c6e06422289c8\"\n (gdb) p oid_to_hex(&index[table[3]]->oid)\n $20 = 0x555555983f63 \"0007a5feb040e1ff704f3ad636619ddca3e7382b\"\n\nthat doesn't look like the OIDs are increasing in lexical order.\n\nI'm not quite sure if I'm even looking at the right thing, or if this is\nto be expected, or if the comment isn't quite accurate. If you could\nhelp clarify what's going on here, that would be great.\n\n> +\t/* table_inv helps us discover that relationship (i'th bitmap\n> +\t * to j'th commit by j = table_inv[i])\n> +\t */\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable_inv[table[i]] = i;\n\nThis part makes sense, as does the rest of the implementation.\n\nThanks,\nTaylor\n"},{"id":"459115","messageId":"YtDVDu7VKgAcvRse@nand.local","threadId":"58038","inReplyTo":"e64362621d235f2c79f52e984de7a2a2794e2842.1656924376.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-15T02:46:38Z","receivedAt":"2022-07-15T02:46:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 04, 2022 at 08:46:14AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> +/*\n> + * Searches for a matching triplet. `va` is a pointer\n> + * to the wanted commit position value. `vb` points to\n> + * a triplet in lookup table. The first 4 bytes of each\n> + * triplet (pointed by `vb`) are compared with `*va`.\n> + */\n> +static int triplet_cmp(const void *va, const void *vb)\n> +{\n> +\n> +\tuint32_t a = *(uint32_t *)va;\n\nThe comment you added is definitely helpful, but I still think that this\nline is a little magical. `*va` isn't really a pointer to a `uint32_t`,\nbut a pointer to the start of a triplet, which just *happens* to have a\n4-byte integer at the beginning of it.\n\nI don't think there's a way to improve this much more than we already\nhave, though. Populating a triplet struct to just dereference the first\nfield feels wasteful and slow. So I think what you have here makes sense\nto me.\n\n> +static uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n> +\t\t\t struct object_id *oid,\n> +\t\t\t uint32_t *result)\n> +{\n> +\tint found;\n> +\n> +\tif (bitmap_is_midx(bitmap_git))\n> +\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n> +\telse\n> +\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n> +\n> +\treturn found;\n> +}\n> +\n> +/*\n> + * `bsearch_triplet` function searches for the raw triplet having\n> + * commit position same as `commit_pos` and fills `triplet`\n> + * object from the raw triplet. Returns 1 on success and 0\n> + * on failure.\n> + */\n> +static int bsearch_triplet(uint32_t *commit_pos,\n> +\t\t\t struct bitmap_index *bitmap_git,\n> +\t\t\t struct bitmap_lookup_table_triplet *triplet)\n> +{\n> +\tunsigned char *p = bsearch(commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n> +\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n> +\n> +\tif (!p)\n> +\t\treturn 0;\n> +\ttriplet->commit_pos = get_be32(p);\n> +\tp += sizeof(uint32_t);\n> +\ttriplet->offset = get_be64(p);\n> +\tp += sizeof(uint64_t);\n> +\ttriplet->xor_row = get_be32(p);\n> +\treturn 1;\n> +}\n\nThis implementation jumped out as being quite similar to\n`lookup_table_get_triplet()`. Ultimately they both end up filling a\ntriplet struct based on some position `p` within the bitmap. The main\ndifference being that in `lookup_table_get_triplet()`, `p` comes from a\nnumeric position which indexes into the table, while in\n`bsearch_triplet()` the position `p` is given to us by a call to\n`bsearch()`.\n\nI wonder if it would be worth extracting the common part of: given a\npointer `p` and a triplet struct, read the triplet beginning at `p` into\nthe struct.\n\n`lookup_table_get_triplet()` could compute `p` and then return the\nresult of calling the new auxiliary function with that `p`. Similarly\nfor `bsearch_triplet()`, it would call that auxiliary function with the\npointer it got from calling `bsearch()`, or return `0` if no match was\nfound.\n\nIt's a minor point, but I think it would help us clean up the\nimplementation a little bit.\n\n> +\n> +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t struct commit *commit)\n> +{\n> +\tuint32_t commit_pos, xor_row;\n> +\tuint64_t offset;\n> +\tint flags;\n> +\tstruct bitmap_lookup_table_triplet triplet;\n> +\tstruct object_id *oid = &commit->object.oid;\n> +\tstruct ewah_bitmap *bitmap;\n> +\tstruct stored_bitmap *xor_bitmap = NULL;\n> +\n> +\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n> +\n> +\tif (!found)\n> +\t\treturn NULL;\n> +\n> +\tif (!bsearch_triplet(&commit_pos, bitmap_git, &triplet))\n> +\t\treturn NULL;\n> +\n> +\toffset = triplet.offset;\n> +\txor_row = triplet.xor_row;\n> +\n> +\tif (xor_row != 0xffffffff) {\n> +\t\tint xor_flags;\n> +\t\tkhiter_t hash_pos;\n> +\t\tuint64_t offset_xor;\n> +\t\tstruct bitmap_lookup_table_xor_item *xor_items;\n> +\t\tstruct bitmap_lookup_table_xor_item xor_item;\n> +\t\tsize_t xor_items_nr = 0, xor_items_alloc = 64;\n> +\n> +\t\tALLOC_ARRAY(xor_items, xor_items_alloc);\n\nThis ALLOC_ARRAY() looks great to me. I wonder if we could amortize the\ncost of allocating in this (somewhat) hot function by treating the\n`xor_items` array as a reusable static buffer where we reset\nxor_items_nr to 0 when entering this function.\n\n> +\t\twhile (xor_row != 0xffffffff) {\n> +\t\t\tstruct object_id xor_oid;\n> +\n> +\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n> +\t\t\t\tfree(xor_items);\n> +\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n\nI think we can probably `die()` here, we're pretty much out of luck in\nthis case.\n\n> +\t\t\t\treturn NULL;\n> +\t\t\t}\n> +\n> +\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n> +\t\t\t\treturn NULL;\n> +\n> +\t\t\toffset_xor = triplet.offset;\n> +\n> +\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, triplet.commit_pos) < 0) {\n> +\t\t\t\tfree(xor_items);\n> +\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n> +\t\t\t\t\ttriplet.commit_pos);\n\nSame here.\n\n> +\t\t\t\treturn NULL;\n> +\t\t\t}\n> +\n> +\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_oid);\n> +\n> +\t\t\t/*\n> +\t\t\t * If desired bitmap is already stored, we don't need\n> +\t\t\t * to iterate further. Because we know that bitmaps\n> +\t\t\t * that are needed to be parsed to parse this bitmap\n> +\t\t\t * has already been stored. So, assign this stored bitmap\n> +\t\t\t * to the xor_bitmap.\n> +\t\t\t */\n> +\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n> +\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n> +\t\t\txor_items[xor_items_nr++] = (struct bitmap_lookup_table_xor_item) {.oid = xor_oid,\n> +\t\t\t\t\t\t\t\t\t\t\t .offset = offset_xor};\n\nThis style of initialization is somewhat uncommon for Git's codebase. It\nmight be a little more natural to write something like:\n\n xor_items[xor_items_nr].oid = xor_oid;\n xor_items[xor_items_nr].offset = offset_xor;\n xor_items_nr++;\n\nBut the struct-copying for `xor_oid` is definitely uncommon for us. We\nshould use the `oidcpy()` helper there instead. Or better yet, pass a\npointer to `&xor_items[xor_items_nr].oid` as the second argument to\n`nth_bitmap_object_oid()` to avoid the copy altogether.\n\n> +\t\t\txor_row = triplet.xor_row;\n> +\t\t}\n> +\n> +\t\twhile (xor_items_nr) {\n> +\t\t\txor_item = xor_items[xor_items_nr - 1];\n> +\t\t\toffset_xor = xor_item.offset;\n> +\n> +\t\t\tbitmap_git->map_pos = offset_xor;\n> +\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n\nShould we extract `6` out to a named constant?\n\nThanks,\nTaylor\n"},{"id":"459116","messageId":"YtDWmAg3R/eRpl0V@nand.local","threadId":"58038","inReplyTo":"a155c1e2ebacf54c451a069499325cdf280606fc.1656924376.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-15T02:53:12Z","receivedAt":"2022-07-15T02:53:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 04, 2022 at 08:46:15AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Add performance tests to verify the performance of lookup table with\n> `pack.writeReverseIndex` enabled. This is to check the performance\n> when the above configuration is set.\n>\n> Lookup table makes Git run faster in most of the cases. Below is the\n> result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\n> gives similar result. The repository used in the test is linux kernel.\n>\n> Test this tree\n> ---------------------------------------------------------------------------\n> 5310.4: repack to disk (lookup=false) 296.55(256.53+14.52)\n\nHaving \"lookup=false\" in this test definitely helps visually\ndifferentiate which tests have a bitmap with and without the lookup\ntable.\n\nI think we should take a slightly different approach for these\nperformance tests. I think the first change to the t/perf tests in this\nseries should only enable `pack.writeReverseIndex`. That patch would be\na good place to highlight the benefit of enabling the on-disk reverse\nindex by showing a before and after of running p5310 before and after\nthat commit.\n\nThen the patch after that should look like this one, which runs the\nsuite with and without the lookup table. That should give us a sense of:\n\n - bitmaps without a lookup table or reverse index\n - bitmaps without a lookup table, but with a reverse index\n - bitamps with a reverse index and a lookup table\n\n...which I think are the most interesting combinations (I wouldn't\nexpect many or any users to have lookup tables enabled without reverse\nindexes).\n\nI think that would allow us to drop the last patch in this version of\nthe series. But I'm definitely open to other testing strategies for the\nperformance tests (including this one!) if you have different thoughts\nabout what the best way to go about this is.\n\nThanks,\nTaylor\n"},{"id":"459135","messageId":"3320033a-54a3-ddbc-d03a-197f209541ec@iee.email","threadId":"58038","inReplyTo":"YtCjlkPdA3CUn/Aw@nand.local","subject":"Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2022-07-15T10:36:34Z","receivedAt":"2022-07-15T10:36:41Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 15/07/2022 00:15, Taylor Blau wrote:\n> On Sun, Jul 10, 2022 at 04:01:11PM +0100, Philip Oakley wrote:\n>>>> Not sure if this is new in this extension, but should there be a link or\n>>>> two to the basics of XOR compression and some of the bitmap look up\n>>>> techniques?\n>>>>\n>>>> It's not always obvious if these techniques are 'heuristic' and only\n>>>> have partial commit data, or they have all the commits listed, Nor\n>>>> how/why they work. My point is more about giving new readers a hand-up\n>>>> in their understanding, rather than simple implementation details for\n>>>> those who already know what is going on. For example, are there any\n>>>> external articles that you found helpful in getting started that could\n>>>> be referenced somewhere in the docs?\n>>> As this series is only about adding a lookup-table extension (and not\n>>> about bitmap itself), I am not sure whether it's good to include those\n>>> things in this series.\n>> Thanks for the clarification. I must have slight misread some of the\n>> discussions and falsely thought it was the XOR compression (which is a\n>> technique I wasn't really aware of), that was being provided by the\n>> extension - Where would it be best for me to look up the background to\n>> your \"extension\" project?\n> Yeah, Abhradeep is right that the XOR compression isn't new, we already\n> serialize bitmaps with optional XOR offsets. The gist is that we give an\n> offset of some previous bitmap that is used to compress the current one\n> by XORing the bits in the current bitamp with the previous one. These\n> XOR-compressed bitmaps are often sparse, so they compress well and\n> reduce the overall size of the .bitmap.\n\nI was thinking of a short paragraph that covers the broader 'why'\naspects, rather than the what/how. For me, XOR is a 'new' compression\nmethod that (IIUC) takes advantage of certain features of the way the\ndata is arranged, such that the XOR has lots of leading zeros, leading\nto the compression mentioned.\n\nI think it's that we sort on oid name, so that despite the oid being\nlong, we have (typically) sufficient oids that the leading XOR bits of\nadjacent pairs allows effective compression. But I could have guessed\nwildly wrong.\n\nI'd been looking at\nhttps://www.timescale.com/blog/time-series-compression-algorithms-explained/\nwhich gave an overview for sorted floats.\nI'd not had time to review the paper \"Gorilla: A Fast, Scalable,\nIn-Memory Time Series Database\"\nhttp://www.vldb.org/pvldb/vol8/p1816-teller.pdf\n>\n> A slightly more detailed overview can be found in\n> Documentation/technical/bitmap-format.txt under the bullet point reading\n> \"1-byte XOR-offset\".\n>\nA separate point is the linkage (or not) between the older reachability\nbit maps, and the commit graph, which sound to be independent options\nand features, yet appear rather interrelated.\n\nThanks\n\nPhilip\n\n\n"},{"id":"459152","messageId":"CAPOJW5x8Vf2qJ-109UH=gvy2i7HdfbFH84hb6fD+YUBN4-GkRg@mail.gmail.com","threadId":"58038","inReplyTo":"YtDPePTo52A+Uo0p@nand.local","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-15T15:58:25Z","receivedAt":"2022-07-15T15:58:52Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Fri, Jul 15, 2022 at 7:52 AM Taylor Blau wrote:\n>\n> By the end of this loop, we have an array `commit_positions` which maps\n> the ith selected commit to its lexical position among all objects in the\n> bitmap. IOW, `commit_positions[i] = j` means the `i`th selected commit\n> can be found at index `j` among all objects in the pack/MIDX in their\n> lexical order.\n\nRight.\n\n> > + /*\n> > + * At the end of this sort table[j] = i means that the i'th\n> > + * bitmap corresponds to j'th bitmapped commit in lex order of\n> > + * OIDs.\n> > + */\n> > + QSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n>\n> And then we sort table by treating its values as indexes into\n> `commit_positions`. Here's where I'm not sure that I follow what's going\n> on. You say above that `table[j] = i`, where `i` corresponds to the\n> order of selected commits, and `j` is in lexical order.\n\nCorrect.\n\n> If that's the case, then I'd expect that printing `index[table[j]]` for\n> increasing `j` would output OIDs in increasing lexical order. But that\n> doesn't quite seem to be the case. From a debugger session that has a\n> breakpoint after computing and sorting table, along with building\n> `table_inv`:\n>\n> (gdb) p oid_to_hex(&index[table[0]]->oid)\n> $17 = 0x555555983ea0 \"0006763074748d43b539c1c8e8882c08034ab178\"\n> (gdb) p oid_to_hex(&index[table[1]]->oid)\n> $18 = 0x555555983ee1 \"001ce83dd43f03dcfc67f29d38922e4a9682aab0\"\n> (gdb) p oid_to_hex(&index[table[2]]->oid)\n> $19 = 0x555555983f22 \"002db882ece2ab6a240e495a169c6e06422289c8\"\n> (gdb) p oid_to_hex(&index[table[3]]->oid)\n> $20 = 0x555555983f63 \"0007a5feb040e1ff704f3ad636619ddca3e7382b\"\n>\n> that doesn't look like the OIDs are increasing in lexical order.\n>\n> I'm not quite sure if I'm even looking at the right thing, or if this is\n> to be expected, or if the comment isn't quite accurate. If you could\n> help clarify what's going on here, that would be great.\n\nI think you're not looking at the right thing. you should look at\n`writer.selected[table[i]].commit->object.oid` instead. I think the\norder of `index[]`\nis not the same as the pack index (or midx).\n\nI am saying this because if we use the `pos` variable (that we get\nfrom `commit_bitmap_writer_pos(&writer.selected[table[i]].commit->object.oid,\nindex, index_nr)`) in `fprintf(stderr, \"commit hex: %s\\n\",\n&index[pos]->oid);`, you'll see that `&index[pos]->oid` and\n`&writer.selected[table[i]].commit->object.oid` are not same. So, If\nyou do -\n\n int spos = commit_bitmap_writer_pos(&index[pos]->oid, index, index_nr);\n\nyou'll see `spos` is not equal to `pos`.\n\n> > + /* table_inv helps us discover that relationship (i'th bitmap\n> > + * to j'th commit by j = table_inv[i])\n> > + */\n> > + for (i = 0; i < writer.selected_nr; i++)\n> > + table_inv[table[i]] = i;\n>\n> This part makes sense, as does the rest of the implementation.\n>\n> Thanks,\n> Taylor\n"},{"id":"459154","messageId":"CAPOJW5y+ywbiT2XBYYNN+y73+V98Ro33D1bgZQveQLTPfrgE_g@mail.gmail.com","threadId":"58038","inReplyTo":"YtDVDu7VKgAcvRse@nand.local","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-15T16:38:17Z","receivedAt":"2022-07-15T16:38:33Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Fri, Jul 15, 2022 at 8:16 AM Taylor Blau wrote:\n>\n> On Mon, Jul 04, 2022 at 08:46:14AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> > +/*\n> > + * Searches for a matching triplet. `va` is a pointer\n> > + * to the wanted commit position value. `vb` points to\n> > + * a triplet in lookup table. The first 4 bytes of each\n> > + * triplet (pointed by `vb`) are compared with `*va`.\n> > + */\n> > +static int triplet_cmp(const void *va, const void *vb)\n> > +{\n> > +\n> > + uint32_t a = *(uint32_t *)va;\n>\n> The comment you added is definitely helpful, but I still think that this\n> line is a little magical. `*va` isn't really a pointer to a `uint32_t`,\n> but a pointer to the start of a triplet, which just *happens* to have a\n> 4-byte integer at the beginning of it.\n\nAre you sure about this? As far as I know, the first parameter of such\ncomparing functions is always a pointer to the given key that we need\nto search for and the second parameter points to each element of an\narray.\n\nI think \"`va is a pointer to the wanted commit position value\" is not\nthat descriptive. Maybe \"`va` is a pointer to the given key\" is\nbetter. What do you think?\n\n> > + * `bsearch_triplet` function searches for the raw triplet having\n> > + * commit position same as `commit_pos` and fills `triplet`\n> > + * object from the raw triplet. Returns 1 on success and 0\n> > + * on failure.\n> > + */\n> > +static int bsearch_triplet(uint32_t *commit_pos,\n> > + struct bitmap_index *bitmap_git,\n> > + struct bitmap_lookup_table_triplet *triplet)\n> > +{\n> > + unsigned char *p = bsearch(commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n> > + BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n> > +\n> > + if (!p)\n> > + return 0;\n> > + triplet->commit_pos = get_be32(p);\n> > + p += sizeof(uint32_t);\n> > + triplet->offset = get_be64(p);\n> > + p += sizeof(uint64_t);\n> > + triplet->xor_row = get_be32(p);\n> > + return 1;\n> > +}\n>\n> This implementation jumped out as being quite similar to\n> `lookup_table_get_triplet()`. Ultimately they both end up filling a\n> triplet struct based on some position `p` within the bitmap. The main\n> difference being that in `lookup_table_get_triplet()`, `p` comes from a\n> numeric position which indexes into the table, while in\n> `bsearch_triplet()` the position `p` is given to us by a call to\n> `bsearch()`.\n>\n> I wonder if it would be worth extracting the common part of: given a\n> pointer `p` and a triplet struct, read the triplet beginning at `p` into\n> the struct.\n>\n> `lookup_table_get_triplet()` could compute `p` and then return the\n> result of calling the new auxiliary function with that `p`. Similarly\n> for `bsearch_triplet()`, it would call that auxiliary function with the\n> pointer it got from calling `bsearch()`, or return `0` if no match was\n> found.\n>\n> It's a minor point, but I think it would help us clean up the\n> implementation a little bit.\n\nSure! That would be a great idea!\n\n> > + ALLOC_ARRAY(xor_items, xor_items_alloc);\n>\n> This ALLOC_ARRAY() looks great to me. I wonder if we could amortize the\n> cost of allocating in this (somewhat) hot function by treating the\n> `xor_items` array as a reusable static buffer where we reset\n> xor_items_nr to 0 when entering this function.\n>\n> > + while (xor_row != 0xffffffff) {\n> > + struct object_id xor_oid;\n> > +\n> > + if (xor_items_nr + 1 >= bitmap_git->entry_count) {\n> > + free(xor_items);\n> > + error(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n>\n> I think we can probably `die()` here, we're pretty much out of luck in\n> this case.\n> ...\n> > + error(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n> > + triplet.commit_pos);\n>\n> Same here.\n\nI didn't use `die()` here because I thought returning NULL would be a\nbetter idea. In that case, Git can still do its job by using the\ntraditional approach - traversing between objects.\n`load_bitmap_entries_v1` also returns NULL if an error occurs. What do\nyou think?\n\n> > + ALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n> > + xor_items[xor_items_nr++] = (struct bitmap_lookup_table_xor_item) {.oid = xor_oid,\n> > + .offset = offset_xor};\n>\n> This style of initialization is somewhat uncommon for Git's codebase. It\n> might be a little more natural to write something like:\n>\n> xor_items[xor_items_nr].oid = xor_oid;\n> xor_items[xor_items_nr].offset = offset_xor;\n> xor_items_nr++;\n>\n> But the struct-copying for `xor_oid` is definitely uncommon for us. We\n> should use the `oidcpy()` helper there instead. Or better yet, pass a\n> pointer to `&xor_items[xor_items_nr].oid` as the second argument to\n> `nth_bitmap_object_oid()` to avoid the copy altogether.\n\nOk, got it.\n\n> > + bitmap_git->map_pos = offset_xor;\n> > + if (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n>\n> Should we extract `6` out to a named constant?\n\nOk, sure!\n\nThanks :)\n\n>\n> Thanks,\n> Taylor\n"},{"id":"459163","messageId":"CAPOJW5zkUDo7C7knyQWJCpMowWEbKd0ea=MP67L1R4VkDqH17A@mail.gmail.com","threadId":"58038","inReplyTo":"YtDWmAg3R/eRpl0V@nand.local","subject":"Re: [PATCH v3 5/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-15T18:23:25Z","receivedAt":"2022-07-15T18:24:22Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Fri, Jul 15, 2022 at 8:23 AM Taylor Blau wrote:\n>\n> Having \"lookup=false\" in this test definitely helps visually\n> differentiate which tests have a bitmap with and without the lookup\n> table.\n>\n> I think we should take a slightly different approach for these\n> performance tests. I think the first change to the t/perf tests in this\n> series should only enable `pack.writeReverseIndex`. That patch would be\n> a good place to highlight the benefit of enabling the on-disk reverse\n> index by showing a before and after of running p5310 before and after\n> that commit.\n>\n> Then the patch after that should look like this one, which runs the\n> suite with and without the lookup table. That should give us a sense of:\n>\n> - bitmaps without a lookup table or reverse index\n> - bitmaps without a lookup table, but with a reverse index\n> - bitamps with a reverse index and a lookup table\n>\n> ...which I think are the most interesting combinations (I wouldn't\n> expect many or any users to have lookup tables enabled without reverse\n> indexes).\n>\n> I think that would allow us to drop the last patch in this version of\n> the series. But I'm definitely open to other testing strategies for the\n> performance tests (including this one!) if you have different thoughts\n> about what the best way to go about this is.\n\nGot it. Thanks !\n\n> Thanks,\n> Taylor\n"},{"id":"459166","messageId":"CAPOJW5wPuWR3wgLk3Svo2mNgSNr4R0D9Y_ygSAG9OTb1F3WyTg@mail.gmail.com","threadId":"58038","inReplyTo":"d70a4505-60ef-82c4-5497-499ac788782a@iee.email","subject":"Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-15T18:48:19Z","receivedAt":"2022-07-15T18:48:35Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Sun, Jul 10, 2022 at 8:31 PM Philip Oakley wrote:\n>\n> On 09/07/2022 08:53, Abhradeep Chakraborty wrote:\n> > Hello Philip,\n> >\n> > Philip Oakley wrote:\n> >\n> >> Not sure if this is new in this extension, but should there be a link or\n> >> two to the basics of XOR compression and some of the bitmap look up\n> >> techniques?\n> >>\n> >> It's not always obvious if these techniques are 'heuristic' and only\n> >> have partial commit data, or they have all the commits listed, Nor\n> >> how/why they work. My point is more about giving new readers a hand-up\n> >> in their understanding, rather than simple implementation details for\n> >> those who already know what is going on. For example, are there any\n> >> external articles that you found helpful in getting started that could\n> >> be referenced somewhere in the docs?\n> > As this series is only about adding a lookup-table extension (and not\n> > about bitmap itself), I am not sure whether it's good to include those\n> > things in this series.\n>\n> Thanks for the clarification. I must have slight misread some of the\n> discussions and falsely thought it was the XOR compression (which is a\n> technique I wasn't really aware of), that was being provided by the\n> extension - Where would it be best for me to look up the background to\n> your \"extension\" project?\n\nSorry that I missed this message. I got the information related to\nthis project from the gsoc project ideas[1] page, additionally you can\nsee the comments[2].\n\n[1] https://git.github.io/SoC-2022-Ideas/\n[2] https://lore.kernel.org/git/YNovuzAsaEb2uIaa@nand.local/\n\n> > (As the name\n> > suggests, this file should only contain format details of bitmaps).\n> > That file would provide the answers of \"Why bitmaps\", \"how they are\n> > stored\", \"How they are fetched\", \"how they work with pack-objects,\n> > git-fetch, midx etc.\", \"Detailed explanation of each bitmap extension\"\n> > , and lastly \"what are the future works\" (if any).\n>\n> One thing I've realised on reflection is that I'm unclear how the\n> 'reachability bitmaps' and the 'commit-graph file' techniques relate to\n> each other (and to the ODB DAG), and what features they pick out within\n> their heuristic, explained at just enough level to allow folks to\n> appreciate what the options that select them will do for their use case.\n\nI am not familiar with 'commit-graph file', so I can't tell you about\nthat. But for bitmaps, you can look at the introductory patches[3].\nAfter that, if you wish, you can also inspect the code related to\nbitmaps.\n\n[3] https://github.com/gitster/git/commit/e127310\n\n> > What do you think?\n>\n> I'd be happy to collate contributions, suggestions and thoughts.\n>\n> Trying to create these good introductory descriptions can be really\n> difficult, as you can only step into the same river once (the 'reading\n> for the first time problem' of not being able to un-hear the\n> explanations of others when reading a 2nd draft...)\n\nI agree ;-)\n\nThanks :)\n"},{"id":"459174","messageId":"YtHm+Dv0lN3Ktibx@nand.local","threadId":"58038","inReplyTo":"CAPOJW5x8Vf2qJ-109UH=gvy2i7HdfbFH84hb6fD+YUBN4-GkRg@mail.gmail.com","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-15T22:15:20Z","receivedAt":"2022-07-15T22:16:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 15, 2022 at 09:28:25PM +0530, Abhradeep Chakraborty wrote:\n> > If that's the case, then I'd expect that printing `index[table[j]]` for\n> > increasing `j` would output OIDs in increasing lexical order. But that\n> > doesn't quite seem to be the case. From a debugger session that has a\n> > breakpoint after computing and sorting table, along with building\n> > `table_inv`:\n> >\n> > (gdb) p oid_to_hex(&index[table[0]]->oid)\n> > $17 = 0x555555983ea0 \"0006763074748d43b539c1c8e8882c08034ab178\"\n> > (gdb) p oid_to_hex(&index[table[1]]->oid)\n> > $18 = 0x555555983ee1 \"001ce83dd43f03dcfc67f29d38922e4a9682aab0\"\n> > (gdb) p oid_to_hex(&index[table[2]]->oid)\n> > $19 = 0x555555983f22 \"002db882ece2ab6a240e495a169c6e06422289c8\"\n> > (gdb) p oid_to_hex(&index[table[3]]->oid)\n> > $20 = 0x555555983f63 \"0007a5feb040e1ff704f3ad636619ddca3e7382b\"\n> >\n> > that doesn't look like the OIDs are increasing in lexical order.\n> >\n> > I'm not quite sure if I'm even looking at the right thing, or if this is\n> > to be expected, or if the comment isn't quite accurate. If you could\n> > help clarify what's going on here, that would be great.\n>\n> I think you're not looking at the right thing. you should look at\n> `writer.selected[table[i]].commit->object.oid` instead. I think the\n> order of `index[]`\n> is not the same as the pack index (or midx).\n>\n> I am saying this because if we use the `pos` variable (that we get\n> from `commit_bitmap_writer_pos(&writer.selected[table[i]].commit->object.oid,\n> index, index_nr)`) in `fprintf(stderr, \"commit hex: %s\\n\",\n> &index[pos]->oid);`, you'll see that `&index[pos]->oid` and\n> `&writer.selected[table[i]].commit->object.oid` are not same. So, If\n> you do -\n>\n> int spos = commit_bitmap_writer_pos(&index[pos]->oid, index, index_nr);\n>\n> you'll see `spos` is not equal to `pos`.\n\n`index` there comes from the list of objects that `pack-objects` or the\nMIDX told us about, and it's sorted in lexical order (via\n`write_pack_file()` -> `stage_tmp_packfiles()` -> `write_idx_file()`).\n\nSo I think this implementation is indexing the commits by the order they\nappearn in the `writer.selected` array, *not* by the order they appear\nin the index.\n\nFor what it's worth, I think the latter ordering makes more sense to use\nto refer to individual objects. But we should be consistent with our\nchoice here and what's in the documentation. And right now I think we're\nnot, since the documentation change in the first patch says we write the\n`commit_pos` field in order of the index:\n\n * {empty}\n commit_pos (4 byte integer, network byte order): ::\n It stores the object position of a commit (in the midx or pack\n index).\n\nThanks,\nTaylor\n"},{"id":"459175","messageId":"YtHoJ90N6rmDmn6M@nand.local","threadId":"58038","inReplyTo":"CAPOJW5y+ywbiT2XBYYNN+y73+V98Ro33D1bgZQveQLTPfrgE_g@mail.gmail.com","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-15T22:20:23Z","receivedAt":"2022-07-15T22:20:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 15, 2022 at 10:08:17PM +0530, Abhradeep Chakraborty wrote:\n> On Fri, Jul 15, 2022 at 8:16 AM Taylor Blau wrote:\n> >\n> > On Mon, Jul 04, 2022 at 08:46:14AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> > > +/*\n> > > + * Searches for a matching triplet. `va` is a pointer\n> > > + * to the wanted commit position value. `vb` points to\n> > > + * a triplet in lookup table. The first 4 bytes of each\n> > > + * triplet (pointed by `vb`) are compared with `*va`.\n> > > + */\n> > > +static int triplet_cmp(const void *va, const void *vb)\n> > > +{\n> > > +\n> > > + uint32_t a = *(uint32_t *)va;\n> >\n> > The comment you added is definitely helpful, but I still think that this\n> > line is a little magical. `*va` isn't really a pointer to a `uint32_t`,\n> > but a pointer to the start of a triplet, which just *happens* to have a\n> > 4-byte integer at the beginning of it.\n>\n> Are you sure about this? As far as I know, the first parameter of such\n> comparing functions is always a pointer to the given key that we need\n> to search for and the second parameter points to each element of an\n> array.\n>\n> I think \"`va is a pointer to the wanted commit position value\" is not\n> that descriptive. Maybe \"`va` is a pointer to the given key\" is\n> better. What do you think?\n\nYes, the first argument to the comparison function used in bsearch() is\na pointer to some element in the array. I just meant that that array is\nthe bitmap_git->table_lookup region, so each element isn't actually a\nuint32_t array, but the whole thing is an array of (uint32_t, uint64_t,\nuint32_t) triplets.\n\nWhat you wrote here is fine, and I don't even think that the comment\nneeds updating. If you did want to clarify, I think you could say\nsomething along the lines of what you wrote above (\"`va` is a pointer to\nan array element\") and add something along the lines of \"where the array\nis the lookup table region of the .bitmap\".\n\n> > > + ALLOC_ARRAY(xor_items, xor_items_alloc);\n> >\n> > This ALLOC_ARRAY() looks great to me. I wonder if we could amortize the\n> > cost of allocating in this (somewhat) hot function by treating the\n> > `xor_items` array as a reusable static buffer where we reset\n> > xor_items_nr to 0 when entering this function.\n> >\n> > > + while (xor_row != 0xffffffff) {\n> > > + struct object_id xor_oid;\n> > > +\n> > > + if (xor_items_nr + 1 >= bitmap_git->entry_count) {\n> > > + free(xor_items);\n> > > + error(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n> >\n> > I think we can probably `die()` here, we're pretty much out of luck in\n> > this case.\n> > ...\n> > > + error(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n> > > + triplet.commit_pos);\n> >\n> > Same here.\n>\n> I didn't use `die()` here because I thought returning NULL would be a\n> better idea. In that case, Git can still do its job by using the\n> traditional approach - traversing between objects.\n> `load_bitmap_entries_v1` also returns NULL if an error occurs. What do\n> you think?\n\nAh, I wasn't aware that our callers are graceful enough to handle this\nlike that. Yes, if we can fallback gracefully, we should, so I think\njust error()-ing here (and above) is the right choice. Thanks for saying\nso.\n\nThanks,\nTaylor\n"},{"id":"459190","messageId":"CAPOJW5yH=Xywqos2tPS4Cn7dAdDqymPVbb6tn_XoAz0ofsACAA@mail.gmail.com","threadId":"58038","inReplyTo":"YtHm+Dv0lN3Ktibx@nand.local","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-16T11:50:57Z","receivedAt":"2022-07-16T11:51:13Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Sat, Jul 16, 2022 at 3:45 AM Taylor Blau wrote:\n>\n> On Fri, Jul 15, 2022 at 09:28:25PM +0530, Abhradeep Chakraborty wrote:\n> > > If that's the case, then I'd expect that printing `index[table[j]]` for\n> > > increasing `j` would output OIDs in increasing lexical order. But that\n> > > doesn't quite seem to be the case. From a debugger session that has a\n> > > breakpoint after computing and sorting table, along with building\n> > > `table_inv`:\n> > >\n> > > (gdb) p oid_to_hex(&index[table[0]]->oid)\n> > > $17 = 0x555555983ea0 \"0006763074748d43b539c1c8e8882c08034ab178\"\n> > > (gdb) p oid_to_hex(&index[table[1]]->oid)\n> > > $18 = 0x555555983ee1 \"001ce83dd43f03dcfc67f29d38922e4a9682aab0\"\n> > > (gdb) p oid_to_hex(&index[table[2]]->oid)\n> > > $19 = 0x555555983f22 \"002db882ece2ab6a240e495a169c6e06422289c8\"\n> > > (gdb) p oid_to_hex(&index[table[3]]->oid)\n> > > $20 = 0x555555983f63 \"0007a5feb040e1ff704f3ad636619ddca3e7382b\"\n> > >\n> > > that doesn't look like the OIDs are increasing in lexical order.\n> > >\n> > > I'm not quite sure if I'm even looking at the right thing, or if this is\n> > > to be expected, or if the comment isn't quite accurate. If you could\n> > > help clarify what's going on here, that would be great.\n> >\n> > I think you're not looking at the right thing. you should look at\n> > `writer.selected[table[i]].commit->object.oid` instead. I think the\n> > order of `index[]`\n> > is not the same as the pack index (or midx).\n> >\n> > I am saying this because if we use the `pos` variable (that we get\n> > from `commit_bitmap_writer_pos(&writer.selected[table[i]].commit->object.oid,\n> > index, index_nr)`) in `fprintf(stderr, \"commit hex: %s\\n\",\n> > &index[pos]->oid);`, you'll see that `&index[pos]->oid` and\n> > `&writer.selected[table[i]].commit->object.oid` are not same. So, If\n> > you do -\n> >\n> > int spos = commit_bitmap_writer_pos(&index[pos]->oid, index, index_nr);\n> >\n> > you'll see `spos` is not equal to `pos`.\n>\n> `index` there comes from the list of objects that `pack-objects` or the\n> MIDX told us about, and it's sorted in lexical order (via\n> `write_pack_file()` -> `stage_tmp_packfiles()` -> `write_idx_file()`).\n\nThis was a bit strange for me because all the tests were passing. But\nnow I find the reason why your results were not in lexical order. you\nwere doing `oid_to_hex(&index[table[i]]->oid)` which is not what you\nintended to do. Let me explain it with a simple workflow -\n\nSuppose 12 commits are selected for bitmaps and are sorted by their\ndate. I will now use their index numbers to denote those commits\n(i.e. `0` denotes the most recent commit, `1` denotes the second\ncommit in this order and so on..).\nSo, before that quick sort, `table` = {0,1, 2, 3, 4, ...,11}. Now\nsuppose, `11`th commit is lexically smallest among all the selected\ncommits, `5`th commit is the second smallest commit and so on. So,\nafter that quick sort, `table` array now contains the following - {11,\n5, 9, 4,0, 3, ...}.\n\nSo, when you do `&index[table[0]]->oid`, it becomes `&index[11]->oid`.\nSimilarly, `&index[table[1]]->oid` becomes `&index[5]->oid` and so on.\nThat's why you're not getting the oids in lexical order -\n`&index[11]->oid` gives the 11th oid in the pack-index and\n`&index[5]->oid` gives the 5th oid in the pack-index.\n\nSo, the right thing would be to do\n`&index[commit_positions[table[0]]]->oid`,\n`&index[commit_positions[table[1]]]->oid` ...\n\nHere `&index[commit_positions[table[0]]]->oid` becomes\n`&index[commit_positions[11]]->oid` =>\n`&index[pos_of_11_commit_with_respect_to_pack_index]->oid` which\nultimately prints the oid of 11th commit ( among the selected bitmap\ncommits IN THE SELECTED BITMAP COMMIT ORDER) .\n\nI think the comment I added is not that good. The following might be better -\n\n At the end of this sort table[j] = i means that the i'th\n bitmap corresponds to j'th bitmapped commit (among the selected commits)\n in lex order of OIDs.\n\n> Thanks,\n> Taylor\n"},{"id":"459246","messageId":"CAN0heSoBca4BcBgR01cwE567NHQyjO+gRkY1V_6Z0nEd-_tW4g@mail.gmail.com","threadId":"58038","inReplyTo":"5e9b985e39b0b9edee7af55dd8b0698a20062cf7.1656924376.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2022-07-18T08:59:27Z","receivedAt":"2022-07-18T08:59:43Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Mon, 4 Jul 2022 at 10:48, Abhradeep Chakraborty via GitGitGadget\n wrote:\n>\n> +static int table_cmp(const void *_va, const void *_vb, void *_data)\n> +{\n> + uint32_t *commit_positions = _data;\n> + uint32_t a = commit_positions[*(uint32_t *)_va];\n> + uint32_t b = commit_positions[*(uint32_t *)_vb];\n\nThis casting and dereferencing are ok because ...\n\n> +static void write_lookup_table(struct hashfile *f,\n> + struct pack_idx_entry **index,\n> + uint32_t index_nr,\n> + off_t *offsets)\n> +{\n> + uint32_t i;\n> + uint32_t *table, *table_inv, *commit_positions;\n> +\n> + ALLOC_ARRAY(table, writer.selected_nr);\n> + ALLOC_ARRAY(table_inv, writer.selected_nr);\n> + ALLOC_ARRAY(commit_positions, writer.selected_nr);\n\n... `table` is where `_va` and `_vb` will be pointing into.\n\n> + QSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n\nI started looking at this casting because of something similar in\n\"pack-bitmap: prepare to read lookup table extension\". I'm pointing out\nthis instance just to say that it looks ok to me.\n\n\nMartin\n"},{"id":"459248","messageId":"CAN0heSoA=wv4syJ3VOe92QPpjPHyqUPJ8+Pv+mbB0-TiiieVmw@mail.gmail.com","threadId":"58038","inReplyTo":"YtHoJ90N6rmDmn6M@nand.local","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2022-07-18T09:06:53Z","receivedAt":"2022-07-18T09:07:07Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Hi Abhradeep and Taylor,\n\nI very much enjoy following from a distance Abhradeep's work on this\nseries and all the reviewing and mentoring. I don't grasp anywhere near\nall the details, but I've looked into this a bit:\n\nOn Sat, 16 Jul 2022 at 00:37, Taylor Blau wrote:\n>\n> On Fri, Jul 15, 2022 at 10:08:17PM +0530, Abhradeep Chakraborty wrote:\n> > On Fri, Jul 15, 2022 at 8:16 AM Taylor Blau wrote:\n> > >\n> > > On Mon, Jul 04, 2022 at 08:46:14AM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> > > > +/*\n> > > > + * Searches for a matching triplet. `va` is a pointer\n> > > > + * to the wanted commit position value. `vb` points to\n> > > > + * a triplet in lookup table. The first 4 bytes of each\n> > > > + * triplet (pointed by `vb`) are compared with `*va`.\n> > > > + */\n> > > > +static int triplet_cmp(const void *va, const void *vb)\n> > > > +{\n> > > > +\n> > > > + uint32_t a = *(uint32_t *)va;\n> > >\n> > > The comment you added is definitely helpful, but I still think that this\n> > > line is a little magical. `*va` isn't really a pointer to a `uint32_t`,\n> > > but a pointer to the start of a triplet, which just *happens* to have a\n> > > 4-byte integer at the beginning of it.\n\nYeah, this all looks quite magical with the casting, and with the\nasymmetric handling of `va` and `vb`.\n\n> > Are you sure about this? As far as I know, the first parameter of such\n> > comparing functions is always a pointer to the given key that we need\n> > to search for and the second parameter points to each element of an\n> > array.\n\nYes, that matches my understanding and the man-page for bsearch(3):\n\n \"The compar routine is expected to have two arguments which point to\n the key object and to an array member, in that order, [...]\"\n\nI think it would help to make this something like\n\n static int triplet_cmp(const void *key, const void *array_item)\n\nto really highlight this asymmetric nature of this function, or to make\nclear how the values flow through our call-chain through something like\n\n static int triplet_cmp(const void *commit_pos, const void *table_entry)\n\nBecause we really do rely on this promise of bsearch(3) -- if we would\ninstantiate a 'dummy' triplet carrying the key, we wouldn't need to (but\nwe would instead need to have our `cmp` function constantly re-read the\nsame value, including doing the byteswap).\n\nWould it make sense to let the `const void *key` directly carry the\n32-bit value and hope that `sizeof(key) >= sizeof(uint32_t)`? That's\nprobably too magical, \"just\" to save on dereferencing.\n\nOne thing that could perhaps make things clearer is if\n`bsearch_triplet()` did take the position directly, rather than as a\npointer:\n\n-static int bsearch_triplet(uint32_t *commit_pos,\n+static int bsearch_triplet(uint32_t commit_pos,\n struct bitmap_index *bitmap_git,\n struct bitmap_lookup_table_triplet *triplet)\n {\n- unsigned char *p = bsearch(commit_pos,\nbitmap_git->table_lookup, bitmap_git->entry_count,\n+ unsigned char *p = bsearch(&commit_pos,\nbitmap_git->table_lookup, bitmap_git->entry_count,\n BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH,\ntriplet_cmp);\n\n\nAlso, maybe s/bsearch_triplet/&_by_pos/ could clarify the intent of this\nfunction?\n\n> > I think \"`va is a pointer to the wanted commit position value\" is not\n> > that descriptive. Maybe \"`va` is a pointer to the given key\" is\n> > better. What do you think?\n>\n> Yes, the first argument to the comparison function used in bsearch() is\n\ns/first/second/\n\n> a pointer to some element in the array. I just meant that that array is\n> the bitmap_git->table_lookup region, so each element isn't actually a\n> uint32_t array, but the whole thing is an array of (uint32_t, uint64_t,\n> uint32_t) triplets.\n>\n> What you wrote here is fine, and I don't even think that the comment\n> needs updating. If you did want to clarify, I think you could say\n> something along the lines of what you wrote above (\"`va` is a pointer to\n> an array element\") and add something along the lines of \"where the array\n> is the lookup table region of the .bitmap\".\n\nI mentioned a few ideas for clarifying things above. I do think it would\nbe a good idea to differentiate the names of `va` and `vb` to make the\nfundamental asymmetry between them clearer. The rest of my comments are\nreally just musings.\n\nI originally started looking at this because I wanted to see why the\ncasting to a `uint32_t *` and dereferencing it was safe. The reason is,\nwe're always handling the same pointer to a `uint32_t` on the stack, so\nalignment is guaranteed.\n\n\nMartin\n"},{"id":"459315","messageId":"CAPOJW5zEJJwx+1_4MrpwPVnpV=i_82obO-uqAcYJGJDS6y=31w@mail.gmail.com","threadId":"58038","inReplyTo":"CAN0heSoA=wv4syJ3VOe92QPpjPHyqUPJ8+Pv+mbB0-TiiieVmw@mail.gmail.com","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-18T19:25:54Z","receivedAt":"2022-07-18T19:26:10Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Mon, Jul 18, 2022 at 2:37 PM Martin Ågren wrote:\n>\n> Hi Abhradeep and Taylor,\n>\n> I very much enjoy following from a distance Abhradeep's work on this\n> series and all the reviewing and mentoring. I don't grasp anywhere near\n> all the details, but I've looked into this a bit:\n\nThanks!\n\n> \"The compar routine is expected to have two arguments which point to\n> the key object and to an array member, in that order, [...]\"\n>\n> I think it would help to make this something like\n>\n> static int triplet_cmp(const void *key, const void *array_item)\n>\n> to really highlight this asymmetric nature of this function, or to make\n> clear how the values flow through our call-chain through something like\n>\n> static int triplet_cmp(const void *commit_pos, const void *table_entry)\n\nNice. Will update it.\n\n> Would it make sense to let the `const void *key` directly carry the\n> 32-bit value and hope that `sizeof(key) >= sizeof(uint32_t)`? That's\n> probably too magical, \"just\" to save on dereferencing.\n\nI do not have any particular opinion here. I will do whatever you think is best.\n\n> One thing that could perhaps make things clearer is if\n> `bsearch_triplet()` did take the position directly, rather than as a\n> pointer:\n>\n> -static int bsearch_triplet(uint32_t *commit_pos,\n> +static int bsearch_triplet(uint32_t commit_pos,\n> struct bitmap_index *bitmap_git,\n> struct bitmap_lookup_table_triplet *triplet)\n> {\n> - unsigned char *p = bsearch(commit_pos,\n> bitmap_git->table_lookup, bitmap_git->entry_count,\n> + unsigned char *p = bsearch(&commit_pos,\n> bitmap_git->table_lookup, bitmap_git->entry_count,\n> BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH,\n> triplet_cmp);\n>\n>\n> Also, maybe s/bsearch_triplet/&_by_pos/ could clarify the intent of this\n> function?\n\nOk, sure!\n\nThanks :)\n"},{"id":"459338","messageId":"CAN0heSo-uo10XN-3c0jYpULQW+h2ykS=Pp2Xv4=cOrXnnyzYNA@mail.gmail.com","threadId":"58038","inReplyTo":"CAPOJW5zEJJwx+1_4MrpwPVnpV=i_82obO-uqAcYJGJDS6y=31w@mail.gmail.com","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2022-07-18T23:26:02Z","receivedAt":"2022-07-18T23:26:18Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Mon, 18 Jul 2022 at 21:26, Abhradeep Chakraborty\n wrote:\n>\n> On Mon, Jul 18, 2022 at 2:37 PM Martin Ågren wrote:\n> >\n> > Would it make sense to let the `const void *key` directly carry the\n> > 32-bit value and hope that `sizeof(key) >= sizeof(uint32_t)`? That's\n> > probably too magical, \"just\" to save on dereferencing.\n>\n> I do not have any particular opinion here. I will do whatever you think is best.\n\nTo be honest, I think it would be better not to do that. I floated it as\na random idea, but it's somewhere in the vicinity of undefined behavior,\nand in any case, it might be a bit too tricky. If we're doing a byteswap\nanyway (on virtually all platforms) and doing a bunch of comparisons,\ntrying to save on a dereference doesn't seem worth the increased \"huh\"\nfactor.\n\nMartin\n"},{"id":"459503","messageId":"f72bf11e6efb4690ae808c0b56c3991c2b1ef266.1658325914.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v4 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:08Z","receivedAt":"2022-07-20T14:05:21Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nWhen reading bitmap file, Git loads each and every bitmap one by one\neven if all the bitmaps are not required. A \"bitmap lookup table\"\nextension to the bitmap format can reduce the overhead of loading\nbitmaps which stores a list of bitmapped commit id pos (in the midx\nor pack, along with their offset and xor offset. This way git can\nload only the necessary bitmaps without loading the previous bitmaps.\n\nOlder versions of Git ignore the lookup table extension and don't\nthrow any kind of warning or error while parsing the bitmap file.\n\nAdd some information for the new \"bitmap lookup table\" extension in the\nbitmap-format documentation.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/technical/bitmap-format.txt | 39 +++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 04b3ec21785..c30dc177643 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -67,6 +67,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n \t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n+\t\t\t** {empty}\n+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n+\t\t\tIf present, the end of the bitmap file contains a table\n+\t\t\tcontaining a list of `N` \n+\t\t\ttriplets. The format and meaning of the table is described\n+\t\t\tbelow.\n++\n+NOTE: Unlike the xor_offset used to compress an individual bitmap,\n+`xor_row` stores an *absolute* index into the lookup table, not a location\n+relative to the current entry.\n+\n \t\t4-byte entry count (network byte order)\n \n \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n@@ -205,3 +216,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n hashes made under two different schemes would be pointless).\n+\n+Commit lookup table\n+-------------------\n+\n+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n+bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`\n+file contains a lookup table specifying the information needed to get\n+the desired bitmap from the entries without parsing previous unnecessary\n+bitmaps.\n+\n+For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n+contains a list of `nr_entries` triplets\n+(sorted in the ascending order of `commit_pos`). The content of i'th\n+triplet is -\n+\n+\t* {empty}\n+\tcommit_pos (4 byte integer, network byte order): ::\n+\tIt stores the object position of a commit (in the midx or pack\n+\tindex).\n+\n+\t* {empty}\n+\toffset (8 byte integer, network byte order): ::\n+\tThe offset from which that commit's bitmap can be read.\n+\n+\t* {empty}\n+\txor_row (4 byte integer, network byte order): ::\n+\tThe position of the triplet whose bitmap is used to compress\n+\tthis one, or `0xffffffff` if no such bitmap exists.\n-- \ngitgitgadget\n\n"},{"id":"459504","messageId":"04244fadf5cd1948885a29decb137ff6e88dc3c2.1658325914.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v4 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:09Z","receivedAt":"2022-07-20T14:05:26Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe bitmap lookup table extension was documented by an earlier\nchange, but Git does not yet know how to write that extension.\n\nTeach Git to write bitmap lookup table extension. The table contains\nthe list of `N` ` triplets. These\ntriplets are sorted according to their commit pos (ascending order).\nThe meaning of each data in the i'th triplet is given below:\n\n - commit_pos stores commit position (in the pack-index or midx).\n It is a 4 byte network byte order unsigned integer.\n\n - offset is the position (in the bitmap file) from which that\n commit's bitmap can be read.\n\n - xor_row is the position of the triplet in the lookup table\n whose bitmap is used to compress this bitmap, or `0xffffffff`\n if no such bitmap exists.\n\nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nCo-authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 112 ++++++++++++++++++++++++++++++++++++++++----\n pack-bitmap.h | 5 +-\n 2 files changed, 107 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c43375bd344..9843790cb60 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -650,20 +650,19 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t off_t *offsets,\n+\t\t\t\t uint32_t *commit_positions)\n {\n \tint i;\n \n \tfor (i = 0; i < writer.selected_nr; ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n-\t\tint commit_pos =\n-\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\t\tif (offsets)\n+\t\t\toffsets[i] = hashfile_total(f);\n \n-\t\tif (commit_pos < 0)\n-\t\t\tBUG(\"trying to write commit not in index\");\n-\n-\t\thashwrite_be32(f, commit_pos);\n+\t\thashwrite_be32(f, commit_positions[i]);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n \n@@ -671,6 +670,81 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static int table_cmp(const void *_va, const void *_vb, void *_data)\n+{\n+\tuint32_t *commit_positions = _data;\n+\tuint32_t a = commit_positions[*(uint32_t *)_va];\n+\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n+\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static void write_lookup_table(struct hashfile *f,\n+\t\t\t struct pack_idx_entry **index,\n+\t\t\t uint32_t index_nr,\n+\t\t\t off_t *offsets,\n+\t\t\t uint32_t *commit_positions)\n+{\n+\tuint32_t i;\n+\tuint32_t *table, *table_inv;\n+\n+\tALLOC_ARRAY(table, writer.selected_nr);\n+\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable[i] = i;\n+\n+\t/*\n+\t * At the end of this sort table[j] = i means that the i'th\n+\t * bitmap corresponds to j'th bitmapped commit (among the selected\n+\t * commits) in lex order of OIDs.\n+\t */\n+\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\n+\t/* table_inv helps us discover that relationship (i'th bitmap\n+\t * to j'th commit by j = table_inv[i])\n+\t */\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable_inv[table[i]] = i;\n+\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\t\tuint32_t xor_offset = selected->xor_offset;\n+\t\tuint32_t xor_row;\n+\n+\t\tif (xor_offset) {\n+\t\t\t/*\n+\t\t\t * xor_index stores the index (in the bitmap entries)\n+\t\t\t * of the corresponding xor bitmap. But we need to convert\n+\t\t\t * this index into lookup table's index. So, table_inv[xor_index]\n+\t\t\t * gives us the index position w.r.t. the lookup table.\n+\t\t\t *\n+\t\t\t * If \"k = table[i] - xor_offset\" then the xor base is the k'th\n+\t\t\t * bitmap. `table_inv[k]` gives us the position of that bitmap\n+\t\t\t * in the lookup table.\n+\t\t\t */\n+\t\t\tuint32_t xor_index = table[i] - xor_offset;\n+\t\t\txor_row = table_inv[xor_index];\n+\t\t} else {\n+\t\t\txor_row = 0xffffffff;\n+\t\t}\n+\n+\t\thashwrite_be32(f, commit_positions[table[i]]);\n+\t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n+\t\thashwrite_be32(f, xor_row);\n+\t}\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\n+\tfree(table);\n+\tfree(table_inv);\n+}\n+\n static void write_hash_cache(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n@@ -695,8 +769,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n+\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n+\tuint32_t *commit_positions = NULL;\n \n \tstruct bitmap_disk_header header;\n \n@@ -715,7 +791,25 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n-\twrite_selected_commits_v1(f, index, index_nr);\n+\n+\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n+\tfor (uint32_t i = 0; i < writer.selected_nr; ++i) {\n+\t\tstruct bitmapped_commit *stored = &writer.selected[i];\n+\t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\n+\t\tif (commit_pos < 0)\n+\t\t\tBUG(_(\"trying to write commit not in index\"));\n+\n+\t\tcommit_positions[i] = commit_pos;\n+\t}\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\tCALLOC_ARRAY(offsets, index_nr);\n+\n+\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\twrite_lookup_table(f, index, index_nr, offsets, commit_positions);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n@@ -730,4 +824,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n+\tfree(offsets);\n+\tfree(commit_positions);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3d3ddd77345..67a9d0fc303 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -24,8 +24,9 @@ struct bitmap_disk_header {\n #define NEEDS_BITMAP (1u<<22)\n \n enum pack_bitmap_opts {\n-\tBITMAP_OPT_FULL_DAG = 1,\n-\tBITMAP_OPT_HASH_CACHE = 4,\n+\tBITMAP_OPT_FULL_DAG = 0x1,\n+\tBITMAP_OPT_HASH_CACHE = 0x4,\n+\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n };\n \n enum pack_bitmap_flags {\n-- \ngitgitgadget\n\n"},{"id":"459505","messageId":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v3.git.1656924376.gitgitgadget@gmail.com","subject":"[PATCH v4 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:07Z","receivedAt":"2022-07-20T14:05:28Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"When parsing the .bitmap file, git loads all the bitmaps one by one even if\nsome of the bitmaps are not necessary. We can remove this overhead by\nloading only the necessary bitmaps. A look up table extension can solve this\nissue.\n\nChanges since v3:\n\n * The common code from both lookup_table_get_triplet() and\n bsearch_triplet_by_pos are moved to lookup_table_get_triplet_by_pointer\n function\n * parameter names of triplet_cmp function is changes (as suggested by\n Martin)\n * xor_items array is now work as reusable static buffer.\n * I moved the filling commit_positions array part (from\n pack-bitmap-write.c) to bitmap_writer_finish function. Because we had to\n iterate two times for commit positions - one in write_selected_commits_v1\n and another in write_lookup_table function. Hope this is acceptable :)\n * changes in performance tests (as suggested by Taylor)\n\nChanges since v2:\n\n * Log messages related issues are fixed.\n * pack.writeBitmapLookupTable is now by default disabled.\n * Documentations are improved.\n * xor_row is used instead of xor_pos in triplets.\n * In pack-bitmap-write.c, off_t * is used for offsets array (Instead of\n uint64_t *).\n * struct bitmap_lookup_table_triplet is introduced and functions Like\n triplet_get_offset() and triplet_get_xor_pos() are removed.\n * table_size is getting subtracted from index_end irrespective of the value\n of GIT_TEST_READ_COMMIT_TABLE.\n * xor stack filling loop will stop iterating if a xor bitmap is already\n stored/parsed.\n * The stack will now store bitmap_lookup_table_xor_item items Of plain\n xor_row.\n * bitmap related test files are reformatted to allow repeating of tests\n with bitmap extension enabled.\n * comments are added.\n\nChanges since v1:\n\nThis is the second version which addressed all (I think) the reviews. Please\nnotify me if some reviews are not addressed :)\n\n * The table size is decreased and the format has also changed. It now\n contains nr_entries triplets of size 4+8+4 bytes. Each triplet contains\n the following things - (1) 4 byte commit position (in the pack-index or\n midx) (2) 8 byte offset and (3) 4 byte xor triplet (i.e. with whose\n bitmap the current triplet's bitmap has to xor) position.\n * Performance tests are splitted into two commits. First contains the\n actual performance tests and second enables the pack.writeReverseIndex\n (as suggested by Taylor).\n * st_*() functions are used.\n * commit order is changed according to Derrick's suggestion.\n * Iterative approach is used instead of recursive approach to parse xor\n bitmaps. (As suggested by Derrick).\n * Some minor bug fixes of previous version.\n\nInitial version:\n\nThe proposed table has:\n\n * a list of nr_entries object ids. These objects are commits that has\n bitmaps. Ids are stored in lexicographic order (for better searching).\n * a list of pairs (4-byte integers, network-byte\n order). The i'th pair denotes the offset and xor-offset(respectively) of\n the bitmap of i'th commit in the previous list. These two informations\n are necessary because only in this way bitmaps can be found without\n parsing all the bitmap.\n * a 4-byte integer for table specific flags (none exists currently).\n\nWhenever git want to parse the bitmap for a specific commit, it will first\nrefer to the table and will look for the offset and xor-offset for that\ncommit. Git will then try to parse the bitmap located at the offset\nposition. The xor-offset can be used to find the xor-bitmap for the\nbitmap(if any).\n\nAbhradeep Chakraborty (6):\n Documentation/technical: describe bitmap lookup table extension\n pack-bitmap-write.c: write lookup table extension\n pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n pack-bitmap: prepare to read lookup table extension\n p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`\n bitmap-lookup-table: add performance tests for lookup table\n\n Documentation/config/pack.txt | 7 +\n Documentation/technical/bitmap-format.txt | 39 ++\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n pack-bitmap-write.c | 112 ++-\n pack-bitmap.c | 278 +++++++-\n pack-bitmap.h | 14 +-\n t/perf/p5310-pack-bitmaps.sh | 68 +-\n t/perf/p5326-multi-pack-bitmaps.sh | 95 +--\n t/t5310-pack-bitmaps.sh | 786 ++++++++++++----------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++-----\n t/t5327-multi-pack-bitmaps-rev.sh | 9 +\n 15 files changed, 1244 insertions(+), 657 deletions(-)\n\n\nbase-commit: 39c15e485575089eb77c769f6da02f98a55905e0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1266%2FAbhra303%2Fbitmap-commit-table-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1266/Abhra303/bitmap-commit-table-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1266\n\nRange-diff vs v3:\n\n 1: f72bf11e6ef = 1: f72bf11e6ef Documentation/technical: describe bitmap lookup table extension\n 2: 5e9b985e39b ! 2: 04244fadf5c pack-bitmap-write.c: write lookup table extension\n @@ Commit message\n \n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: static const struct object_id *oid_access(size_t pos, const void *table)\n - \treturn &index[pos]->oid;\n - }\n \n -+static int commit_bitmap_writer_pos(struct object_id *oid,\n -+\t\t\t\t struct pack_idx_entry **index,\n -+\t\t\t\t uint32_t index_nr)\n -+{\n -+\treturn oid_pos(oid, index, index_nr, oid_access);\n -+}\n -+\n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n -\t\t\t\t uint32_t index_nr)\n +\t\t\t\t uint32_t index_nr,\n -+\t\t\t\t off_t *offsets)\n ++\t\t\t\t off_t *offsets,\n ++\t\t\t\t uint32_t *commit_positions)\n {\n \tint i;\n \n -@@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n + \tfor (i = 0; i < writer.selected_nr; ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n - \t\tint commit_pos =\n +-\t\tint commit_pos =\n -\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n -+\t\t\tcommit_bitmap_writer_pos(&stored->commit->object.oid, index, index_nr);\n - \n - \t\tif (commit_pos < 0)\n - \t\t\tBUG(\"trying to write commit not in index\");\n - \n +\t\tif (offsets)\n +\t\t\toffsets[i] = hashfile_total(f);\n -+\n - \t\thashwrite_be32(f, commit_pos);\n + \n +-\t\tif (commit_pos < 0)\n +-\t\t\tBUG(\"trying to write commit not in index\");\n +-\n +-\t\thashwrite_be32(f, commit_pos);\n ++\t\thashwrite_be32(f, commit_positions[i]);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n + \n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +static void write_lookup_table(struct hashfile *f,\n +\t\t\t struct pack_idx_entry **index,\n +\t\t\t uint32_t index_nr,\n -+\t\t\t off_t *offsets)\n ++\t\t\t off_t *offsets,\n ++\t\t\t uint32_t *commit_positions)\n +{\n +\tuint32_t i;\n -+\tuint32_t *table, *table_inv, *commit_positions;\n ++\tuint32_t *table, *table_inv;\n +\n +\tALLOC_ARRAY(table, writer.selected_nr);\n +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n -+\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n -+\n -+\t/* store the index positions of the commits */\n -+\tfor (i = 0; i < writer.selected_nr; i++) {\n -+\t\tint pos = commit_bitmap_writer_pos(&writer.selected[i].commit->object.oid,\n -+\t\t\t\t\t\t index, index_nr);\n -+\t\tif (pos < 0)\n -+\t\t\tBUG(_(\"trying to write commit not in index\"));\n -+\n -+\t\tcommit_positions[i] = pos;\n -+\t}\n +\n +\tfor (i = 0; i < writer.selected_nr; i++)\n +\t\ttable[i] = i;\n +\n +\t/*\n +\t * At the end of this sort table[j] = i means that the i'th\n -+\t * bitmap corresponds to j'th bitmapped commit in lex order of\n -+\t * OIDs.\n ++\t * bitmap corresponds to j'th bitmapped commit (among the selected\n ++\t * commits) in lex order of OIDs.\n +\t */\n +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n +\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\n +\tfree(table);\n +\tfree(table_inv);\n -+\tfree(commit_positions);\n +}\n +\n static void write_hash_cache(struct hashfile *f,\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n +\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n ++\tuint32_t *commit_positions = NULL;\n + \n + \tstruct bitmap_disk_header header;\n \n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.tags);\n -\twrite_selected_commits_v1(f, index, index_nr);\n +\n ++\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n ++\tfor (uint32_t i = 0; i < writer.selected_nr; ++i) {\n ++\t\tstruct bitmapped_commit *stored = &writer.selected[i];\n ++\t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n ++\n ++\t\tif (commit_pos < 0)\n ++\t\t\tBUG(_(\"trying to write commit not in index\"));\n ++\n ++\t\tcommit_positions[i] = commit_pos;\n ++\t}\n ++\n +\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n +\t\tCALLOC_ARRAY(offsets, index_nr);\n +\n -+\twrite_selected_commits_v1(f, index, index_nr, offsets);\n ++\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n +\n +\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n -+\t\twrite_lookup_table(f, index, index_nr, offsets);\n ++\t\twrite_lookup_table(f, index, index_nr, offsets, commit_positions);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \tstrbuf_release(&tmp_file);\n +\tfree(offsets);\n ++\tfree(commit_positions);\n }\n \n ## pack-bitmap.h ##\n 3: 3dc40cc7f73 = 3: 8bd7639e4b9 pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n 4: e64362621d2 ! 4: afc8c660ac1 pack-bitmap: prepare to read lookup table extension\n @@ pack-bitmap.c: struct include_data {\n +};\n +\n +/*\n ++ * Given a `triplet` struct pointer and pointer `p`, this\n ++ * function reads the triplet beginning at `p` into the struct.\n ++ * Note that this function assumes that there is enough memory\n ++ * left for filling the `triplet` struct from `p`.\n ++ */\n ++static int lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n ++\t\t\t\t\t const unsigned char *p)\n ++{\n ++\tif (!triplet)\n ++\t\treturn -1;\n ++\n ++\ttriplet->commit_pos = get_be32(p);\n ++\tp += sizeof(uint32_t);\n ++\ttriplet->offset = get_be64(p);\n ++\tp += sizeof(uint64_t);\n ++\ttriplet->xor_row = get_be32(p);\n ++\treturn 0;\n ++}\n ++\n ++/*\n + * This function gets the raw triplet from `row`'th row in the\n + * lookup table and fills that data to the `triplet`.\n + */\n @@ pack-bitmap.c: struct include_data {\n +\n +\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n +\n -+\ttriplet->commit_pos = get_be32(p);\n -+\tp += sizeof(uint32_t);\n -+\ttriplet->offset = get_be64(p);\n -+\tp += sizeof(uint64_t);\n -+\ttriplet->xor_row = get_be32(p);\n -+\treturn 0;\n ++\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n +}\n +\n +/*\n -+ * Searches for a matching triplet. `va` is a pointer\n -+ * to the wanted commit position value. `vb` points to\n ++ * Searches for a matching triplet. `commit_pos` is a pointer\n ++ * to the wanted commit position value. `table_entry` points to\n + * a triplet in lookup table. The first 4 bytes of each\n -+ * triplet (pointed by `vb`) are compared with `*va`.\n ++ * triplet (pointed by `table_entry`) are compared with `*commit_pos`.\n + */\n -+static int triplet_cmp(const void *va, const void *vb)\n ++static int triplet_cmp(const void *commit_pos, const void *table_entry)\n +{\n +\n -+\tuint32_t a = *(uint32_t *)va;\n -+\tuint32_t b = get_be32(vb);\n ++\tuint32_t a = *(uint32_t *)commit_pos;\n ++\tuint32_t b = get_be32(table_entry);\n +\tif (a > b)\n +\t\treturn 1;\n +\telse if (a < b)\n @@ pack-bitmap.c: struct include_data {\n +}\n +\n +/*\n -+ * `bsearch_triplet` function searches for the raw triplet having\n -+ * commit position same as `commit_pos` and fills `triplet`\n -+ * object from the raw triplet. Returns 1 on success and 0\n -+ * on failure.\n ++ * `bsearch_triplet_by_pos` function searches for the raw triplet\n ++ * having commit position same as `commit_pos` and fills `triplet`\n ++ * object from the raw triplet. Returns 1 on success and 0 on\n ++ * failure.\n + */\n -+static int bsearch_triplet(uint32_t *commit_pos,\n -+\t\t\t struct bitmap_index *bitmap_git,\n -+\t\t\t struct bitmap_lookup_table_triplet *triplet)\n ++static int bsearch_triplet_by_pos(uint32_t commit_pos,\n ++\t\t\t\t struct bitmap_index *bitmap_git,\n ++\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n +{\n -+\tunsigned char *p = bsearch(commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n ++\tunsigned char *p = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n +\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n +\n +\tif (!p)\n -+\t\treturn 0;\n -+\ttriplet->commit_pos = get_be32(p);\n -+\tp += sizeof(uint32_t);\n -+\ttriplet->offset = get_be64(p);\n -+\tp += sizeof(uint64_t);\n -+\ttriplet->xor_row = get_be32(p);\n -+\treturn 1;\n ++\t\treturn -1;\n ++\n ++\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n +}\n +\n +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n @@ pack-bitmap.c: struct include_data {\n +{\n +\tuint32_t commit_pos, xor_row;\n +\tuint64_t offset;\n -+\tint flags;\n ++\tint flags, found;\n +\tstruct bitmap_lookup_table_triplet triplet;\n +\tstruct object_id *oid = &commit->object.oid;\n +\tstruct ewah_bitmap *bitmap;\n +\tstruct stored_bitmap *xor_bitmap = NULL;\n ++\tconst int bitmap_header_size = 6;\n ++\tstatic struct bitmap_lookup_table_xor_item *xor_items = NULL;\n ++\tstatic size_t xor_items_nr = 0, xor_items_alloc = 0;\n ++\tstatic int is_corrupt = 0;\n ++\n ++\tif (is_corrupt)\n ++\t\treturn NULL;\n +\n -+\tint found = bsearch_pos(bitmap_git, oid, &commit_pos);\n ++\tfound = bsearch_pos(bitmap_git, oid, &commit_pos);\n +\n +\tif (!found)\n +\t\treturn NULL;\n +\n -+\tif (!bsearch_triplet(&commit_pos, bitmap_git, &triplet))\n ++\tif (bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n +\t\treturn NULL;\n +\n ++\txor_items_nr = 0;\n +\toffset = triplet.offset;\n +\txor_row = triplet.xor_row;\n +\n @@ pack-bitmap.c: struct include_data {\n +\t\tint xor_flags;\n +\t\tkhiter_t hash_pos;\n +\t\tuint64_t offset_xor;\n -+\t\tstruct bitmap_lookup_table_xor_item *xor_items;\n -+\t\tstruct bitmap_lookup_table_xor_item xor_item;\n -+\t\tsize_t xor_items_nr = 0, xor_items_alloc = 64;\n ++\t\tstruct bitmap_lookup_table_xor_item *xor_item;\n +\n -+\t\tALLOC_ARRAY(xor_items, xor_items_alloc);\n +\t\twhile (xor_row != 0xffffffff) {\n -+\t\t\tstruct object_id xor_oid;\n ++\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n +\n +\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n -+\t\t\t\tfree(xor_items);\n +\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n -+\t\t\t\treturn NULL;\n ++\t\t\t\tgoto corrupt;\n +\t\t\t}\n +\n +\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n -+\t\t\t\treturn NULL;\n ++\t\t\t\tgoto corrupt;\n +\n -+\t\t\toffset_xor = triplet.offset;\n ++\t\t\txor_item = &xor_items[xor_items_nr];\n ++\t\t\txor_item->offset = triplet.offset;\n +\n -+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_oid, triplet.commit_pos) < 0) {\n -+\t\t\t\tfree(xor_items);\n ++\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n +\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n +\t\t\t\t\ttriplet.commit_pos);\n -+\t\t\t\treturn NULL;\n ++\t\t\t\tgoto corrupt;\n +\t\t\t}\n +\n -+\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_oid);\n ++\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n +\n +\t\t\t/*\n +\t\t\t * If desired bitmap is already stored, we don't need\n @@ pack-bitmap.c: struct include_data {\n +\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n +\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n +\t\t\t\tbreak;\n -+\n -+\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n -+\t\t\txor_items[xor_items_nr++] = (struct bitmap_lookup_table_xor_item) {.oid = xor_oid,\n -+\t\t\t\t\t\t\t\t\t\t\t .offset = offset_xor};\n ++\t\t\txor_items_nr++;\n +\t\t\txor_row = triplet.xor_row;\n +\t\t}\n +\n +\t\twhile (xor_items_nr) {\n -+\t\t\txor_item = xor_items[xor_items_nr - 1];\n -+\t\t\toffset_xor = xor_item.offset;\n ++\t\t\txor_item = &xor_items[xor_items_nr - 1];\n ++\t\t\toffset_xor = xor_item->offset;\n +\n +\t\t\tbitmap_git->map_pos = offset_xor;\n -+\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n ++\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n +\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n -+\t\t\t\t\toid_to_hex(&xor_item.oid));\n -+\t\t\t\tfree(xor_items);\n -+\t\t\t\treturn NULL;\n ++\t\t\t\t\toid_to_hex(&xor_item->oid));\n ++\t\t\t\tgoto corrupt;\n +\t\t\t}\n +\n +\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n +\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n +\t\t\tbitmap = read_bitmap_1(bitmap_git);\n +\n -+\t\t\tif (!bitmap) {\n -+\t\t\t\tfree(xor_items);\n -+\t\t\t\treturn NULL;\n -+\t\t\t}\n ++\t\t\tif (!bitmap)\n ++\t\t\t\tgoto corrupt;\n +\n -+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item.oid, xor_bitmap, xor_flags);\n ++\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n +\t\t\txor_items_nr--;\n +\t\t}\n -+\n -+\t\tfree(xor_items);\n +\t}\n +\n +\tbitmap_git->map_pos = offset;\n -+\tif (bitmap_git->map_size - bitmap_git->map_pos < 6) {\n ++\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n +\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n +\t\t\toid_to_hex(oid));\n -+\t\treturn NULL;\n ++\t\tgoto corrupt;\n +\t}\n +\n +\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n @@ pack-bitmap.c: struct include_data {\n +\tbitmap = read_bitmap_1(bitmap_git);\n +\n +\tif (!bitmap)\n -+\t\treturn NULL;\n ++\t\tgoto corrupt;\n +\n +\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n ++\n ++corrupt:\n ++\tfree(xor_items);\n ++\tis_corrupt = 1;\n ++\treturn NULL;\n +}\n +\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n 6: 4f9f1049485 ! 5: fc69489e395 p5310-pack-bitmaps.sh: remove pack.writeReverseIndex\n @@ Metadata\n Author: Abhradeep Chakraborty \n \n ## Commit message ##\n - p5310-pack-bitmaps.sh: remove pack.writeReverseIndex\n + p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`\n \n - The previous change enables the `pack.writereverseindex` to see\n - the effect of writing reverse index in the performance test.\n + Enable `pack.writeReverseIndex` before running pack-bitmap related\n + performance tests.\n \n - Remove the `pack.writeReverseIndex` configuration.\n + The performance difference with `pack.writeReverseIndex` enabled and\n + with disabled are given below -\n \n - Below is the result of performance test. Output format is in\n - seconds.\n + With `pack.writeReverseIndex`\n + -------------------------------\n +\n + Test this tree\n + -------------------------------------------------------------------------\n + 5310.3: repack to disk 296.55(256.53+14.52)\n + 5310.4: simulated clone 15.64(8.88+1.39)\n + 5310.5: simulated fetch 1.65(2.75+0.20)\n + 5310.6: pack to file (bitmap) 48.71(30.20+7.58)\n + 5310.7: rev-list (commits) 0.61(0.41+0.08)\n + 5310.8: rev-list (objects) 4.38(4.26+0.09)\n + 5310.9: rev-list with tag negated via --not 0.07(0.02+0.04)\n + --all (objects)\n + 5310.10: rev-list with negative tag (objects) 0.05(0.01+0.03)\n + 5310.11: rev-list count with blob:none 0.08(0.03+0.04)\n + 5310.12: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n + 5310.13: rev-list count with tree:0 0.08(0.03+0.04)\n + 5310.14: simulated partial clone 9.45(8.12+0.41)\n + 5310.16: clone (partial bitmap) 17.02(10.61+2.67)\n + 5310.17: pack to file (partial bitmap) 51.91(28.57+7.48)\n + 5310.18: rev-list with tree filter (partial bitmap) 1.00(0.22+0.24)\n +\n + Without `pack.writeReverseIndex`:\n + -----------------------------\n \n Test this tree\n ------------------------------------------------------------------------\n - 5310.4: repack to disk (lookup=false) 293.80(251.30+14.30)\n - 5310.5: simulated clone 12.50(5.15+1.36)\n - 5310.6: simulated fetch 1.83(2.90+0.23)\n - 5310.7: pack to file (bitmap) 39.70(20.25+7.14)\n - 5310.8: rev-list (commits) 1.00(0.60+0.13)\n - 5310.9: rev-list (objects) 4.11(4.00+0.10)\n - 5310.10: rev-list with tag negated via --not 0.07(0.02+0.05)\n + 5310.3: repack to disk 293.80(251.30+14.30)\n + 5310.4: simulated clone 12.50(5.15+1.36)\n + 5310.5: simulated fetch 1.83(2.90+0.23)\n + 5310.6: pack to file (bitmap) 39.70(20.25+7.14)\n + 5310.7: rev-list (commits) 1.00(0.60+0.13)\n + 5310.8: rev-list (objects) 4.11(4.00+0.10)\n + 5310.9: rev-list with tag negated via --not 0.07(0.02+0.05)\n --all (objects)\n - 5310.11: rev-list with negative tag (objects) 0.23(0.16+0.06)\n - 5310.12: rev-list count with blob:none 0.27(0.18+0.08)\n - 5310.13: rev-list count with blob:limit=1k 6.41(5.98+0.41)\n - 5310.14: rev-list count with tree:0 0.26(0.18+0.07)\n - 5310.15: simulated partial clone 4.34(3.29+0.37)\n - 5310.19: repack to disk (lookup=true) 250.93(171.97+20.78)\n - 5310.20: simulated clone 10.80(5.14+1.06)\n - 5310.21: simulated fetch 0.71(0.79+0.16)\n - 5310.22: pack to file (bitmap) 39.49(20.19+6.98)\n - 5310.23: rev-list (commits) 0.81(0.48+0.09)\n - 5310.24: rev-list (objects) 3.48(3.38+0.09)\n - 5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n - --all (objects)\n - 5310.26: rev-list with negative tag (objects) 0.22(0.16+0.05)\n - 5310.27: rev-list count with blob:none 0.22(0.16+0.05)\n - 5310.28: rev-list count with blob:limit=1k 6.21(5.76+0.29)\n - 5310.29: rev-list count with tree:0 0.23(0.16+0.06)\n - 5310.30: simulated partial clone 4.53(3.14+0.39)\n -\n - Tests 4-15 are without the use of lookup table. The rests are\n - repeatation of the previous tests but using lookup table.\n + 5310.10: rev-list with negative tag (objects) 0.23(0.16+0.06)\n + 5310.11: rev-list count with blob:none 0.27(0.18+0.08)\n + 5310.12: rev-list count with blob:limit=1k 6.41(5.98+0.41)\n + 5310.13: rev-list count with tree:0 0.26(0.18+0.07)\n + 5310.14: simulated partial clone 4.34(3.29+0.37)\n + 5310.16: clone (partial bitmap) 21.48(15.12+2.42)\n + 5310.17: pack to file (partial bitmap) 47.35(37.80+4.84)\n + 5310.18: rev-list with tree filter (partial bitmap) 0.73(0.07+0.21)\n \n - Mentored-by: Taylor Blau \n - Co-Mentored-by: Kaartic Sivaraam \n Signed-off-by: Abhradeep Chakraborty \n \n ## t/perf/p5310-pack-bitmaps.sh ##\n @@ t/perf/p5310-pack-bitmaps.sh: test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n --\tgit config pack.writebitmaps true &&\n --\tgit config pack.writeReverseIndex true\n -+\tgit config pack.writebitmaps true\n +-\tgit config pack.writebitmaps true\n ++\tgit config pack.writebitmaps true &&\n ++\tgit config pack.writeReverseIndex true\n '\n \n - test_bitmap () {\n + # we need to create the tag up front such that it is covered by the repack and\n 5: a155c1e2eba ! 6: 52f7d8359ee bitmap-lookup-table: add performance tests for lookup table\n @@ Commit message\n 5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n 5310.14: rev-list count with tree:0 0.08(0.03+0.04)\n 5310.15: simulated partial clone 9.45(8.12+0.41)\n - 5310.19: repack to disk (lookup=true) 255.92(188.13+20.47)\n - 5310.20: simulated clone 13.78(8.84+1.09)\n - 5310.21: simulated fetch 0.52(0.63+0.14)\n - 5310.22: pack to file (bitmap) 44.34(28.94+6.84)\n - 5310.23: rev-list (commits) 0.48(0.31+0.06)\n - 5310.24: rev-list (objects) 4.02(3.93+0.07)\n - 5310.25: rev-list with tag negated via --not 0.04(0.00+0.03)\n + 5310.17: clone (partial bitmap) 21.00(15.04+2.39)\n + 5310.18: pack to file (partial bitmap) 47.98(38.13+5.23)\n + 5310.19: rev-list with tree filter (partial bitmap) 0.70(0.07+0.20)\n + 5310.22: repack to disk (lookup=true) 255.92(188.13+20.47)\n + 5310.23: simulated clone 13.78(8.84+1.09)\n + 5310.24: simulated fetch 0.52(0.63+0.14)\n + 5310.25: pack to file (bitmap) 44.34(28.94+6.84)\n + 5310.26: rev-list (commits) 0.48(0.31+0.06)\n + 5310.27: rev-list (objects) 4.02(3.93+0.07)\n + 5310.28: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n - 5310.26: rev-list with negative tag (objects) 0.04(0.00+0.03)\n - 5310.27: rev-list count with blob:none 0.04(0.01+0.03)\n - 5310.28: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n - 5310.29: rev-list count with tree:0 0.04(0.01+0.03)\n - 5310.30: simulated partial clone 8.30(7.21+0.36)\n + 5310.29: rev-list with negative tag (objects) 0.04(0.00+0.03)\n + 5310.30: rev-list count with blob:none 0.04(0.01+0.03)\n + 5310.31: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n + 5310.32: rev-list count with tree:0 0.04(0.01+0.03)\n + 5310.33: simulated partial clone 8.30(7.21+0.36)\n + 5310.35: clone (partial bitmap) 20.34(15.00+2.41)\n + 5310.36: pack to file (partial bitmap) 46.45(38.05+5.20)\n + 5310.37: rev-list with tree filter (partial bitmap) 0.61(0.06+0.20)\n \n Test 4-15 are tested without using lookup table. Same tests are\n repeated in 16-30 (using lookup table).\n @@ Commit message\n Signed-off-by: Abhradeep Chakraborty \n \n ## t/perf/p5310-pack-bitmaps.sh ##\n -@@ t/perf/p5310-pack-bitmaps.sh: test_perf_large_repo\n - # We intentionally use the deprecated pack.writebitmaps\n - # config so that we can test against older versions of git.\n - test_expect_success 'setup bitmap config' '\n --\tgit config pack.writebitmaps true\n -+\tgit config pack.writebitmaps true &&\n -+\tgit config pack.writeReverseIndex true\n +@@ t/perf/p5310-pack-bitmaps.sh: test_expect_success 'setup bitmap config' '\n + \tgit config pack.writeReverseIndex true\n '\n \n -# we need to create the tag up front such that it is covered by the repack and\n @@ t/perf/p5310-pack-bitmaps.sh: test_perf_large_repo\n +\t\t# had happened\n +\t\tgit update-ref HEAD $orig_tip\n +\t'\n ++\n ++\ttest_partial_bitmap\n +}\n \n -test_partial_bitmap\n @@ t/perf/p5326-multi-pack-bitmaps.sh: test_description='Tests performance using mi\n +\t\t# had happened\n +\t\tgit update-ref HEAD $orig_tip\n +\t'\n ++\n ++\ttest_partial_bitmap\n +}\n +\n +test_bitmap false\n\n-- \ngitgitgadget\n"},{"id":"459506","messageId":"afc8c660ac164a9446a1beeae6a5b595482ace55.1658325914.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v4 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:11Z","receivedAt":"2022-07-20T14:05:37Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEarlier change teaches Git to write bitmap lookup table. But Git\ndoes not know how to parse them.\n\nTeach Git to parse the existing bitmap lookup table. The older\nversions of Git are not affected by it. Those versions ignore the\nlookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap.c | 278 ++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 9 ++\n t/t5310-pack-bitmaps.sh | 22 ++++\n 3 files changed, 299 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 36134222d7a..7c66d4379f5 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -82,6 +82,12 @@ struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n+\t/*\n+\t * If not NULL, this point into the commit table extension\n+\t * (within the memory mapped region `map`).\n+\t */\n+\tunsigned char *table_lookup;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -185,6 +191,16 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex->hashes = (void *)(index_end - cache_size);\n \t\t\tindex_end -= cache_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE) {\n+\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n+\t\t\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit lookup table)\"));\n+\t\t\tif (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n+\t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n@@ -211,11 +227,13 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,\n \n \thash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret);\n \n-\t/* a 0 return code means the insertion succeeded with no changes,\n-\t * because the SHA1 already existed on the map. this is bad, there\n-\t * shouldn't be duplicated commits in the index */\n+\t/*\n+\t * A 0 return code means the insertion succeeded with no changes,\n+\t * because the SHA1 already existed on the map. This is bad, there\n+\t * shouldn't be duplicated commits in the index.\n+\t */\n \tif (ret == 0) {\n-\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n+\t\terror(_(\"duplicate entry in bitmap index: %s\"), oid_to_hex(oid));\n \t\treturn NULL;\n \t}\n \n@@ -470,7 +488,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n \n-\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n \treturn 0;\n@@ -557,13 +575,241 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct bitmap_lookup_table_triplet {\n+\tuint32_t commit_pos;\n+\tuint64_t offset;\n+\tuint32_t xor_row;\n+};\n+\n+struct bitmap_lookup_table_xor_item {\n+\tstruct object_id oid;\n+\tuint64_t offset;\n+};\n+\n+/*\n+ * Given a `triplet` struct pointer and pointer `p`, this\n+ * function reads the triplet beginning at `p` into the struct.\n+ * Note that this function assumes that there is enough memory\n+ * left for filling the `triplet` struct from `p`.\n+ */\n+static int lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n+\t\t\t\t\t const unsigned char *p)\n+{\n+\tif (!triplet)\n+\t\treturn -1;\n+\n+\ttriplet->commit_pos = get_be32(p);\n+\tp += sizeof(uint32_t);\n+\ttriplet->offset = get_be64(p);\n+\tp += sizeof(uint64_t);\n+\ttriplet->xor_row = get_be32(p);\n+\treturn 0;\n+}\n+\n+/*\n+ * This function gets the raw triplet from `row`'th row in the\n+ * lookup table and fills that data to the `triplet`.\n+ */\n+static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t pos,\n+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = NULL;\n+\tif (pos >= bitmap_git->entry_count)\n+\t\treturn error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n+\n+\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\n+\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n+}\n+\n+/*\n+ * Searches for a matching triplet. `commit_pos` is a pointer\n+ * to the wanted commit position value. `table_entry` points to\n+ * a triplet in lookup table. The first 4 bytes of each\n+ * triplet (pointed by `table_entry`) are compared with `*commit_pos`.\n+ */\n+static int triplet_cmp(const void *commit_pos, const void *table_entry)\n+{\n+\n+\tuint32_t a = *(uint32_t *)commit_pos;\n+\tuint32_t b = get_be32(table_entry);\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n+\t\t\t struct object_id *oid,\n+\t\t\t uint32_t *result)\n+{\n+\tint found;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n+\telse\n+\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n+\n+\treturn found;\n+}\n+\n+/*\n+ * `bsearch_triplet_by_pos` function searches for the raw triplet\n+ * having commit position same as `commit_pos` and fills `triplet`\n+ * object from the raw triplet. Returns 1 on success and 0 on\n+ * failure.\n+ */\n+static int bsearch_triplet_by_pos(uint32_t commit_pos,\n+\t\t\t\t struct bitmap_index *bitmap_git,\n+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n+\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n+\n+\tif (!p)\n+\t\treturn -1;\n+\n+\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n+}\n+\n+static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t struct commit *commit)\n+{\n+\tuint32_t commit_pos, xor_row;\n+\tuint64_t offset;\n+\tint flags, found;\n+\tstruct bitmap_lookup_table_triplet triplet;\n+\tstruct object_id *oid = &commit->object.oid;\n+\tstruct ewah_bitmap *bitmap;\n+\tstruct stored_bitmap *xor_bitmap = NULL;\n+\tconst int bitmap_header_size = 6;\n+\tstatic struct bitmap_lookup_table_xor_item *xor_items = NULL;\n+\tstatic size_t xor_items_nr = 0, xor_items_alloc = 0;\n+\tstatic int is_corrupt = 0;\n+\n+\tif (is_corrupt)\n+\t\treturn NULL;\n+\n+\tfound = bsearch_pos(bitmap_git, oid, &commit_pos);\n+\n+\tif (!found)\n+\t\treturn NULL;\n+\n+\tif (bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n+\t\treturn NULL;\n+\n+\txor_items_nr = 0;\n+\toffset = triplet.offset;\n+\txor_row = triplet.xor_row;\n+\n+\tif (xor_row != 0xffffffff) {\n+\t\tint xor_flags;\n+\t\tkhiter_t hash_pos;\n+\t\tuint64_t offset_xor;\n+\t\tstruct bitmap_lookup_table_xor_item *xor_item;\n+\n+\t\twhile (xor_row != 0xffffffff) {\n+\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n+\n+\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n+\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n+\t\t\t\tgoto corrupt;\n+\t\t\t}\n+\n+\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n+\t\t\t\tgoto corrupt;\n+\n+\t\t\txor_item = &xor_items[xor_items_nr];\n+\t\t\txor_item->offset = triplet.offset;\n+\n+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n+\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n+\t\t\t\t\ttriplet.commit_pos);\n+\t\t\t\tgoto corrupt;\n+\t\t\t}\n+\n+\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n+\n+\t\t\t/*\n+\t\t\t * If desired bitmap is already stored, we don't need\n+\t\t\t * to iterate further. Because we know that bitmaps\n+\t\t\t * that are needed to be parsed to parse this bitmap\n+\t\t\t * has already been stored. So, assign this stored bitmap\n+\t\t\t * to the xor_bitmap.\n+\t\t\t */\n+\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n+\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n+\t\t\t\tbreak;\n+\t\t\txor_items_nr++;\n+\t\t\txor_row = triplet.xor_row;\n+\t\t}\n+\n+\t\twhile (xor_items_nr) {\n+\t\t\txor_item = &xor_items[xor_items_nr - 1];\n+\t\t\toffset_xor = xor_item->offset;\n+\n+\t\t\tbitmap_git->map_pos = offset_xor;\n+\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n+\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\t\t\toid_to_hex(&xor_item->oid));\n+\t\t\t\tgoto corrupt;\n+\t\t\t}\n+\n+\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n+\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\t\t\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\t\t\tif (!bitmap)\n+\t\t\t\tgoto corrupt;\n+\n+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n+\t\t\txor_items_nr--;\n+\t\t}\n+\t}\n+\n+\tbitmap_git->map_pos = offset;\n+\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n+\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\toid_to_hex(oid));\n+\t\tgoto corrupt;\n+\t}\n+\n+\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n+\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\tif (!bitmap)\n+\t\tgoto corrupt;\n+\n+\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n+\n+corrupt:\n+\tfree(xor_items);\n+\tis_corrupt = 1;\n+\treturn NULL;\n+}\n+\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n-\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n-\t\treturn NULL;\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n+\t\tstruct stored_bitmap *bitmap = NULL;\n+\t\tif (!bitmap_git->table_lookup)\n+\t\t\treturn NULL;\n+\n+\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\t/* NEEDSWORK: cache misses aren't recorded */\n+\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n+\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\tif (!bitmap)\n+\t\t\treturn NULL;\n+\t\treturn lookup_stored_bitmap(bitmap);\n+\t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n@@ -1699,8 +1945,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n-\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n-\t\tbitmap_git->version, bitmap_git->entry_count);\n+\tfprintf(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\t\tbitmap_git->version,\n+\t\tbitmap_git->entry_count,\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n \n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n@@ -1753,13 +2001,23 @@ void test_bitmap_walk(struct rev_info *revs)\n \n int test_bitmap_commits(struct repository *r)\n {\n-\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n+\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n+\n+\t/*\n+\t * As this function is only used to print bitmap selected\n+\t * commits, we don't have to read the commit table.\n+\t */\n \n \tif (!bitmap_git)\n \t\tdie(\"failed to load bitmap indexes\");\n \n+\tif (bitmap_git->table_lookup) {\n+\t\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\t\t\tdie(_(\"failed to load bitmap indexes\"));\n+\t}\n+\n \tkh_foreach(bitmap_git->bitmaps, oid, value, {\n \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n \t});\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 67a9d0fc303..9278f71ac91 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -23,6 +23,15 @@ struct bitmap_disk_header {\n \n #define NEEDS_BITMAP (1u<<22)\n \n+/*\n+ * The width in bytes of a single triplet in the lookup table\n+ * extension:\n+ * (commit_pos, offset, xor_row)\n+ *\n+ * whose fields ar 32-, 64-, 32- bits wide, respectively.\n+ */\n+#define BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH (16)\n+\n enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c0607172827..7e50f8e7653 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -258,6 +258,7 @@ test_bitmap_cases () {\n \n \ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \t\ttest_config pack.writebitmaphashcache false &&\n+\t\ttest_config pack.writebitmaplookuptable false &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -270,6 +271,7 @@ test_bitmap_cases () {\n \t'\n \n \ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -453,4 +455,24 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n+test_expect_success 'lookup table is actually used to traverse objects' '\n+\tgit repack -adb &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n+\t\tgit rev-list --use-bitmap-index --count --all &&\n+\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n+\ttest_config pack.writebitmaphashcache false &&\n+\tgit repack -adb &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"459507","messageId":"fc69489e3956893d36394a29b2a9272ed343a2da.1658325914.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v4 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:12Z","receivedAt":"2022-07-20T14:05:38Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEnable `pack.writeReverseIndex` before running pack-bitmap related\nperformance tests.\n\nThe performance difference with `pack.writeReverseIndex` enabled and\nwith disabled are given below -\n\nWith `pack.writeReverseIndex`\n-------------------------------\n\nTest this tree\n-------------------------------------------------------------------------\n5310.3: repack to disk 296.55(256.53+14.52)\n5310.4: simulated clone 15.64(8.88+1.39)\n5310.5: simulated fetch 1.65(2.75+0.20)\n5310.6: pack to file (bitmap) 48.71(30.20+7.58)\n5310.7: rev-list (commits) 0.61(0.41+0.08)\n5310.8: rev-list (objects) 4.38(4.26+0.09)\n5310.9: rev-list with tag negated via --not 0.07(0.02+0.04)\n --all (objects)\n5310.10: rev-list with negative tag (objects) 0.05(0.01+0.03)\n5310.11: rev-list count with blob:none 0.08(0.03+0.04)\n5310.12: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n5310.13: rev-list count with tree:0 0.08(0.03+0.04)\n5310.14: simulated partial clone 9.45(8.12+0.41)\n5310.16: clone (partial bitmap) 17.02(10.61+2.67)\n5310.17: pack to file (partial bitmap) 51.91(28.57+7.48)\n5310.18: rev-list with tree filter (partial bitmap) 1.00(0.22+0.24)\n\nWithout `pack.writeReverseIndex`:\n-----------------------------\n\nTest this tree\n------------------------------------------------------------------------\n5310.3: repack to disk 293.80(251.30+14.30)\n5310.4: simulated clone 12.50(5.15+1.36)\n5310.5: simulated fetch 1.83(2.90+0.23)\n5310.6: pack to file (bitmap) 39.70(20.25+7.14)\n5310.7: rev-list (commits) 1.00(0.60+0.13)\n5310.8: rev-list (objects) 4.11(4.00+0.10)\n5310.9: rev-list with tag negated via --not 0.07(0.02+0.05)\n --all (objects)\n5310.10: rev-list with negative tag (objects) 0.23(0.16+0.06)\n5310.11: rev-list count with blob:none 0.27(0.18+0.08)\n5310.12: rev-list count with blob:limit=1k 6.41(5.98+0.41)\n5310.13: rev-list count with tree:0 0.26(0.18+0.07)\n5310.14: simulated partial clone 4.34(3.29+0.37)\n5310.16: clone (partial bitmap) 21.48(15.12+2.42)\n5310.17: pack to file (partial bitmap) 47.35(37.80+4.84)\n5310.18: rev-list with tree filter (partial bitmap) 0.73(0.07+0.21)\n\nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 7ad4f237bc3..6e8abcd5b21 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -13,7 +13,8 @@ test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true\n+\tgit config pack.writebitmaps true &&\n+\tgit config pack.writeReverseIndex true\n '\n \n # we need to create the tag up front such that it is covered by the repack and\n-- \ngitgitgadget\n\n"},{"id":"459508","messageId":"8bd7639e4b93c7e3263b036c2d2334c2a7c2e273.1658325914.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v4 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:10Z","receivedAt":"2022-07-20T14:05:41Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nTeach Git to provide a way for users to enable/disable bitmap lookup\ntable extension by providing a config option named 'writeBitmapLookupTable'.\nDefault is false.\n\nAlso add test to verify writting of lookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/config/pack.txt | 7 +\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n t/t5310-pack-bitmaps.sh | 792 ++++++++++++++++--------------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++++-------\n t/t5327-multi-pack-bitmaps-rev.sh | 9 +\n 9 files changed, 720 insertions(+), 581 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex ad7f73a1ead..b955ca572ec 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n computed; instead, any namehashes stored in an existing bitmap are\n permuted into their appropriate location when writing a new bitmap.\n \n+pack.writeBitmapLookupTable::\n+\tWhen true, Git will include a \"lookup table\" section in the\n+\tbitmap index (if one is written). This table is used to defer\n+\tloading individual bitmaps as late as possible. This can be\n+\tbeneficial in repositories that have relatively large bitmap\n+\tindexes. Defaults to false.\n+\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5edbb7fe86e..55402b46f41 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n \t}\n \n+\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(var, value))\n+\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t\telse\n+\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t}\n+\n \t/*\n \t * We should never make a fall-back call to 'git_default_config', since\n \t * this was already called in 'cmd_multi_pack_index()'.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39e28cfcafc..46e26774963 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n+\n+\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n+\t\telse\n+\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n+\t}\n+\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\ndiff --git a/midx.c b/midx.c\nindex 5f0dd386b02..9c26d04bfde 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1072,6 +1072,9 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n \n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n \tprepare_midx_packing_data(&pdata, ctx);\n \n \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\ndiff --git a/midx.h b/midx.h\nindex 22e8e53288e..5578cd7b835 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,6 +47,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n+#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f775fc1ce69..c0607172827 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -26,22 +26,413 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-setup_bitmap_history\n-\n-test_expect_success 'setup writing bitmaps during repack' '\n-\tgit config repack.writeBitmaps true\n-'\n-\n-test_expect_success 'full repack creates bitmaps' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+test_bitmap_cases () {\n+\twriteLookupTable=false\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase \"$i\" in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test repository' '\n+\t\trm -fr * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'setup writing bitmaps during repack' '\n+\t\tgit config repack.writeBitmaps true\n+\t'\n+\n+\ttest_expect_success 'full repack creates bitmaps' '\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit repack -ad &&\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\t\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n+\t'\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local loose)' '\n+\t\tgit init --bare alt.git &&\n+\t\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\t\techo content1 >file1 &&\n+\t\t# non-local loose object which is not present in bitmapped pack\n+\t\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t\t# non-local loose object which is also present in bitmapped pack\n+\t\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\t\tgit add file1 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file1 &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\t\tgit index-pack 1.pack &&\n+\t\tlist_packed_objects 1.idx >1.objects &&\n+\t\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t\t! has_any nonlocal-loose 1.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\t\techo content2 >file2 &&\n+\t\tblob2=$(git hash-object -w file2) &&\n+\t\tgit add file2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file2 &&\n+\t\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\t\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\t\trm $(objpath $blob2) &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\t\tgit index-pack 2a.pack &&\n+\t\tlist_packed_objects 2a.idx >2a.objects &&\n+\t\t! has_any keepobjects 2a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local pack)' '\n+\t\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\t\tgit index-pack 2b.pack &&\n+\t\tlist_packed_objects 2b.idx >2b.objects &&\n+\t\t! has_any keepobjects 2b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\t\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\t\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t\t>.git/objects/pack/$packbitmap.keep &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\t\tgit index-pack 3a.pack &&\n+\t\tlist_packed_objects 3a.idx >3a.objects &&\n+\t\t! has_any packbitmap.objects 3a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\t\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\t\trm -f .git/objects/pack/multi-pack-index &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\t\tgit index-pack 3b.pack &&\n+\t\tlist_packed_objects 3b.idx >3b.objects &&\n+\t\t! has_any packbitmap.objects 3b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects to file can use bitmap' '\n+\t\t# make sure we still have 1 bitmap index from previous tests\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\t# verify equivalent packs are generated with/without using bitmap index\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n+\t\ttest_cmp packa.objects packb.objects\n+\t'\n+\n+\ttest_expect_success 'full repack, reusing previous bitmaps' '\n \t\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n-'\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output\n+\t'\n+\n+\ttest_expect_success 'fetch (full bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'create objects for missing-HAVE tests' '\n+\t\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n+\t\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n+\t\tparent=$(echo parent | git commit-tree $tree) &&\n+\t\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n+\t\tcat >revs <<-EOF\n+\t\tHEAD\n+\t\t^HEAD^\n+\t\t^$commit\n+\t\tEOF\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --incremental' '\n+\t\tcat >revs2 <<-EOF &&\n+\t\tHEAD\n+\t\t$commit\n+\t\tEOF\n+\t\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\t\tgit index-pack 4.pack &&\n+\t\tlist_packed_objects 4.idx >4.objects &&\n+\t\ttest_line_count = 4 4.objects &&\n+\t\tgit rev-list --objects $commit >revlist &&\n+\t\tcut -d\" \" -f1 revlist |sort >objects &&\n+\t\ttest_cmp 4.objects objects\n+\t'\n+\n+\ttest_expect_success 'pack with missing blob' '\n+\t\trm $(objpath $blob) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing tree' '\n+\t\trm $(objpath $tree) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing parent' '\n+\t\trm $(objpath $parent) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n+\t\tgit clone --bare . compat-jgit.git &&\n+\t\t(\n+\t\t\tcd compat-jgit.git &&\n+\t\t\trm -f objects/pack/*.bitmap &&\n+\t\t\tjgit gc &&\n+\t\t\tgit rev-list --test-bitmap HEAD\n+\t\t)\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n+\t\tgit clone --bare . compat-us.git &&\n+\t\t(\n+\t\t\tcd compat-us.git &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\tgit repack -adb &&\n+\t\t\t# jgit gc will barf if it does not like our bitmaps\n+\t\t\tjgit gc\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'splitting packs does not generate bogus bitmaps' '\n+\t\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n+\t\tgit add rand &&\n+\t\tgit commit -m \"commit with big file\" &&\n+\t\tgit -c pack.packSizeLimit=500k repack -adb &&\n+\t\tgit init --bare no-bitmaps.git &&\n+\t\tgit -C no-bitmaps.git fetch .. HEAD\n+\t'\n+\n+\ttest_expect_success 'set up reusable pack' '\n+\t\trm -f .git/objects/pack/*.keep &&\n+\t\tgit repack -adb &&\n+\t\treusable_pack () {\n+\t\t\tgit for-each-ref --format=\"%(objectname)\" |\n+\t\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n+\t\t}\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --honor-pack-keep' '\n+\t\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n+\t\tfor i in .git/objects/pack/*.pack\n+\t\tdo\n+\t\t\t>${i%.pack}.keep || return 1\n+\t\tdone &&\n+\t\treusable_pack --honor-pack-keep >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --local' '\n+\t\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n+\t\treusable_pack --local >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --incremental' '\n+\t\treusable_pack --incremental >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\t\ttest_config pack.writebitmaphashcache false &&\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupted.bitmap.index stderr\n+\t'\n+\n+\t# Create a state of history with these properties:\n+\t#\n+\t# - refs that allow a client to fetch some new history, while sharing some old\n+\t# history with the server; we use branches delta-reuse-old and\n+\t# delta-reuse-new here\n+\t#\n+\t# - the new history contains an object that is stored on the server as a delta\n+\t# against a base that is in the old history\n+\t#\n+\t# - the base object is not immediately reachable from the tip of the old\n+\t# history; finding it would involve digging down through history we know the\n+\t# other side has\n+\t#\n+\t# This should result in a state where fetching from old->new would not\n+\t# traditionally reuse the on-disk delta (because we'd have to dig to realize\n+\t# that the client has it), but we will do so if bitmaps can tell us cheaply\n+\t# that the other side has it.\n+\ttest_expect_success 'set up thin delta-reuse parent' '\n+\t\t# This first commit contains the buried base object.\n+\t\ttest-tool genrandom delta 16384 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m \"delta base\" &&\n+\t\tbase=$(git rev-parse --verify HEAD:file) &&\n+\n+\t\t# These intermediate commits bury the base back in history.\n+\t\t# This becomes the \"old\" state.\n+\t\tfor i in 1 2 3 4 5\n+\t\tdo\n+\t\t\techo $i >file &&\n+\t\t\tgit commit -am \"intermediate $i\" || return 1\n+\t\tdone &&\n+\t\tgit branch delta-reuse-old &&\n+\n+\t\t# And now our new history has a delta against the buried base. Note\n+\t\t# that this must be smaller than the original file, since pack-objects\n+\t\t# prefers to create deltas from smaller objects to larger.\n+\t\ttest-tool genrandom delta 16300 >file &&\n+\t\tgit commit -am \"delta result\" &&\n+\t\tdelta=$(git rev-parse --verify HEAD:file) &&\n+\t\tgit branch delta-reuse-new &&\n+\n+\t\t# Repack with bitmaps and double check that we have the expected delta\n+\t\t# relationship.\n+\t\tgit repack -adb &&\n+\t\thave_delta $delta $base\n+\t'\n+\n+\t# Now we can sanity-check the non-bitmap behavior (that the server is not able\n+\t# to reuse the delta). This isn't strictly something we care about, so this\n+\t# test could be scrapped in the future. But it makes sure that the next test is\n+\t# actually triggering the feature we want.\n+\t#\n+\t# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n+\t# we actually perform the fetch, retain the resulting pack, and inspect the\n+\t# result.\n+\ttest_expect_success 'fetch without bitmaps ignores delta against old base' '\n+\t\ttest_config pack.usebitmaps false &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $ZERO_OID\n+\t\t)\n+\t'\n+\n+\t# And do the same for the bitmap case, where we do expect to find the delta.\n+\ttest_expect_success 'fetch with bitmaps can reuse old base' '\n+\t\ttest_config pack.usebitmaps true &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $base\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\t# create enough commits that not all are receive bitmap\n+\t\t\t# coverage even if they are all at the tip of some reference.\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\t\tgit rev-list HEAD >commits.raw &&\n+\t\t\tsort commits &&\n+\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin bitmaps &&\n+\n+\t\t\t# remember which commits did not receive bitmaps\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_file_not_empty before &&\n+\n+\t\t\t# mark the commits which did not receive bitmaps as preferred,\n+\t\t\t# and generate the bitmap again\n+\t\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'complains about multiple pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\ttest_commit base &&\n+\n+\t\t\tgit repack -adb &&\n+\t\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n+\t\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n+\n+\t\t\ttest_commit other &&\n+\t\t\tgit repack -ab &&\n+\n+\t\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n+\n+\t\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n+\t\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n+\t\t\ttest_line_count = 2 packs &&\n+\t\t\ttest_line_count = 2 bitmaps &&\n+\n+\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\tgrep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-basic_bitmap_tests\n+test_bitmap_cases\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -54,375 +445,12 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n-test_expect_success 'pack-objects respects --local (non-local loose)' '\n-\tgit init --bare alt.git &&\n-\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n-\techo content1 >file1 &&\n-\t# non-local loose object which is not present in bitmapped pack\n-\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n-\t# non-local loose object which is also present in bitmapped pack\n-\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n-\tgit add file1 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file1 &&\n-\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n-\tgit index-pack 1.pack &&\n-\tlist_packed_objects 1.idx >1.objects &&\n-\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n-\t! has_any nonlocal-loose 1.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n-\techo content2 >file2 &&\n-\tblob2=$(git hash-object -w file2) &&\n-\tgit add file2 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file2 &&\n-\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n-\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n-\trm $(objpath $blob2) &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n-\tgit index-pack 2a.pack &&\n-\tlist_packed_objects 2a.idx >2a.objects &&\n-\t! has_any keepobjects 2a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local pack)' '\n-\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n-\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n-\tgit index-pack 2b.pack &&\n-\tlist_packed_objects 2b.idx >2b.objects &&\n-\t! has_any keepobjects 2b.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tpackbitmap=$(basename $(cat output) .bitmap) &&\n-\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n-\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n-\t>.git/objects/pack/$packbitmap.keep &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n-\tgit index-pack 3a.pack &&\n-\tlist_packed_objects 3a.idx >3a.objects &&\n-\t! has_any packbitmap.objects 3a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n-\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n-\trm -f .git/objects/pack/multi-pack-index &&\n-\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n-\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n-\tgit index-pack 3b.pack &&\n-\tlist_packed_objects 3b.idx >3b.objects &&\n-\t! has_any packbitmap.objects 3b.objects\n-'\n-\n-test_expect_success 'pack-objects to file can use bitmap' '\n-\t# make sure we still have 1 bitmap index from previous tests\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\t# verify equivalent packs are generated with/without using bitmap index\n-\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n-\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n-\ttest_cmp packa.objects packb.objects\n-'\n-\n-test_expect_success 'full repack, reusing previous bitmaps' '\n-\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n-'\n-\n-test_expect_success 'fetch (full bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'create objects for missing-HAVE tests' '\n-\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n-\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n-\tparent=$(echo parent | git commit-tree $tree) &&\n-\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n-\tcat >revs <<-EOF\n-\tHEAD\n-\t^HEAD^\n-\t^$commit\n-\tEOF\n-'\n-\n-test_expect_success 'pack-objects respects --incremental' '\n-\tcat >revs2 <<-EOF &&\n-\tHEAD\n-\t$commit\n-\tEOF\n-\tgit pack-objects --incremental --stdout --revs 4.pack &&\n-\tgit index-pack 4.pack &&\n-\tlist_packed_objects 4.idx >4.objects &&\n-\ttest_line_count = 4 4.objects &&\n-\tgit rev-list --objects $commit >revlist &&\n-\tcut -d\" \" -f1 revlist |sort >objects &&\n-\ttest_cmp 4.objects objects\n-'\n-\n-test_expect_success 'pack with missing blob' '\n-\trm $(objpath $blob) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n+test_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'pack with missing tree' '\n-\trm $(objpath $tree) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success 'pack with missing parent' '\n-\trm $(objpath $parent) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n-\tgit clone --bare . compat-jgit.git &&\n-\t(\n-\t\tcd compat-jgit.git &&\n-\t\trm -f objects/pack/*.bitmap &&\n-\t\tjgit gc &&\n-\t\tgit rev-list --test-bitmap HEAD\n-\t)\n-'\n-\n-test_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n-\tgit clone --bare . compat-us.git &&\n-\t(\n-\t\tcd compat-us.git &&\n-\t\tgit repack -adb &&\n-\t\t# jgit gc will barf if it does not like our bitmaps\n-\t\tjgit gc\n-\t)\n-'\n-\n-test_expect_success 'splitting packs does not generate bogus bitmaps' '\n-\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n-\tgit add rand &&\n-\tgit commit -m \"commit with big file\" &&\n-\tgit -c pack.packSizeLimit=500k repack -adb &&\n-\tgit init --bare no-bitmaps.git &&\n-\tgit -C no-bitmaps.git fetch .. HEAD\n-'\n-\n-test_expect_success 'set up reusable pack' '\n-\trm -f .git/objects/pack/*.keep &&\n-\tgit repack -adb &&\n-\treusable_pack () {\n-\t\tgit for-each-ref --format=\"%(objectname)\" |\n-\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n-\t}\n-'\n-\n-test_expect_success 'pack reuse respects --honor-pack-keep' '\n-\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n-\tfor i in .git/objects/pack/*.pack\n-\tdo\n-\t\t>${i%.pack}.keep || return 1\n-\tdone &&\n-\treusable_pack --honor-pack-keep >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --local' '\n-\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n-\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n-\treusable_pack --local >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --incremental' '\n-\treusable_pack --incremental >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n-\ttest_config pack.writebitmaphashcache false &&\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt.ewah.bitmap stderr\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (cache)' '\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupted.bitmap.index stderr\n-'\n-\n-# Create a state of history with these properties:\n-#\n-# - refs that allow a client to fetch some new history, while sharing some old\n-# history with the server; we use branches delta-reuse-old and\n-# delta-reuse-new here\n-#\n-# - the new history contains an object that is stored on the server as a delta\n-# against a base that is in the old history\n-#\n-# - the base object is not immediately reachable from the tip of the old\n-# history; finding it would involve digging down through history we know the\n-# other side has\n-#\n-# This should result in a state where fetching from old->new would not\n-# traditionally reuse the on-disk delta (because we'd have to dig to realize\n-# that the client has it), but we will do so if bitmaps can tell us cheaply\n-# that the other side has it.\n-test_expect_success 'set up thin delta-reuse parent' '\n-\t# This first commit contains the buried base object.\n-\ttest-tool genrandom delta 16384 >file &&\n-\tgit add file &&\n-\tgit commit -m \"delta base\" &&\n-\tbase=$(git rev-parse --verify HEAD:file) &&\n-\n-\t# These intermediate commits bury the base back in history.\n-\t# This becomes the \"old\" state.\n-\tfor i in 1 2 3 4 5\n-\tdo\n-\t\techo $i >file &&\n-\t\tgit commit -am \"intermediate $i\" || return 1\n-\tdone &&\n-\tgit branch delta-reuse-old &&\n-\n-\t# And now our new history has a delta against the buried base. Note\n-\t# that this must be smaller than the original file, since pack-objects\n-\t# prefers to create deltas from smaller objects to larger.\n-\ttest-tool genrandom delta 16300 >file &&\n-\tgit commit -am \"delta result\" &&\n-\tdelta=$(git rev-parse --verify HEAD:file) &&\n-\tgit branch delta-reuse-new &&\n-\n-\t# Repack with bitmaps and double check that we have the expected delta\n-\t# relationship.\n-\tgit repack -adb &&\n-\thave_delta $delta $base\n-'\n-\n-# Now we can sanity-check the non-bitmap behavior (that the server is not able\n-# to reuse the delta). This isn't strictly something we care about, so this\n-# test could be scrapped in the future. But it makes sure that the next test is\n-# actually triggering the feature we want.\n-#\n-# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n-# we actually perform the fetch, retain the resulting pack, and inspect the\n-# result.\n-test_expect_success 'fetch without bitmaps ignores delta against old base' '\n-\ttest_config pack.usebitmaps false &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $ZERO_OID\n-\t)\n-'\n-\n-# And do the same for the bitmap case, where we do expect to find the delta.\n-test_expect_success 'fetch with bitmaps can reuse old base' '\n-\ttest_config pack.usebitmaps true &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $base\n-\t)\n-'\n-\n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\t# create enough commits that not all are receive bitmap\n-\t\t# coverage even if they are all at the tip of some reference.\n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n-\n-\t\tgit rev-list HEAD >commits.raw &&\n-\t\tsort commits &&\n-\n-\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n-\n-\t\t# remember which commits did not receive bitmaps\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_file_not_empty before &&\n-\n-\t\t# mark the commits which did not receive bitmaps as preferred,\n-\t\t# and generate the bitmap again\n-\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n-\n-\t\t! test_cmp before after\n-\t)\n-'\n-\n-test_expect_success 'complains about multiple pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit base &&\n-\n-\t\tgit repack -adb &&\n-\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n-\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n-\n-\t\ttest_commit other &&\n-\t\tgit repack -ab &&\n-\n-\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n-\n-\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n-\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n-\t\ttest_line_count = 2 packs &&\n-\t\ttest_line_count = 2 bitmaps &&\n-\n-\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\tgrep \"ignoring extra bitmap file\" err\n-\t)\n+test_expect_success 'verify writing bitmap lookup table when enabled' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2\" \\\n+\t\tgit repack -ad &&\n+\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n test_done\ndiff --git a/t/t5311-pack-bitmaps-shallow.sh b/t/t5311-pack-bitmaps-shallow.sh\nindex 872a95df338..f74c6a2da47 100755\n--- a/t/t5311-pack-bitmaps-shallow.sh\n+++ b/t/t5311-pack-bitmaps-shallow.sh\n@@ -17,23 +17,40 @@ test_description='check bitmap operation with shallow repositories'\n # the tree for A. But in a shallow one, we've grafted away\n # A, and fetching A to B requires that the other side send\n # us the tree for file=1.\n-test_expect_success 'setup shallow repo' '\n-\techo 1 >file &&\n-\tgit add file &&\n-\tgit commit -m orig &&\n-\techo 2 >file &&\n-\tgit commit -a -m update &&\n-\tgit clone --no-local --bare --depth=1 . shallow.git &&\n-\techo 1 >file &&\n-\tgit commit -a -m repeat\n-'\n-\n-test_expect_success 'turn on bitmaps in the parent' '\n-\tgit repack -adb\n-'\n-\n-test_expect_success 'shallow fetch from bitmapped repo' '\n-\t(cd shallow.git && git fetch)\n-'\n+test_shallow_bitmaps () {\n+\twriteLookupTable=false\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup shallow repo' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\techo 1 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m orig &&\n+\t\techo 2 >file &&\n+\t\tgit commit -a -m update &&\n+\t\tgit clone --no-local --bare --depth=1 . shallow.git &&\n+\t\techo 1 >file &&\n+\t\tgit commit -a -m repeat\n+\t'\n+\n+\ttest_expect_success 'turn on bitmaps in the parent' '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_expect_success 'shallow fetch from bitmapped repo' '\n+\t\t(cd shallow.git && git fetch)\n+\t'\n+}\n+\n+test_shallow_bitmaps\n+\n \n test_done\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4fe57414c13..3b206adcee6 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -15,17 +15,24 @@ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n sane_unset GIT_TEST_MIDX_WRITE_REV\n sane_unset GIT_TEST_MIDX_READ_RIDX\n \n-midx_bitmap_core\n-\n bitmap_reuse_tests() {\n \tfrom=$1\n \tto=$2\n+\twriteLookupTable=false\n+\n+\tfor i in $3-${$#}\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n \n \ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n \t\trm -fr repo &&\n \t\tgit init repo &&\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk 16 &&\n \t\t\tgit tag old-tip &&\n \n@@ -43,6 +50,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk --id=further 16 &&\n \t\t\tgit tag new-tip &&\n \n@@ -59,6 +67,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\tgit for-each-ref &&\n \t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n \t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n@@ -66,244 +75,294 @@ bitmap_reuse_tests() {\n \t'\n }\n \n-bitmap_reuse_tests 'pack' 'MIDX'\n-bitmap_reuse_tests 'MIDX' 'pack'\n-bitmap_reuse_tests 'MIDX' 'MIDX'\n+test_midx_bitmap_cases () {\n+\twriteLookupTable=false\n+\twriteBitmapLookupTable=\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\")\n+\t\t\twriteLookupTable=true\n+\t\t\twriteBitmapLookupTable=\"$i\"\n+\t\t\t;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test_repository' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n \n-test_expect_success 'missing object closure fails gracefully' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\tmidx_bitmap_core\n \n-\t\ttest_commit loose &&\n-\t\ttest_commit packed &&\n+\tbitmap_reuse_tests 'pack' 'MIDX' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'pack' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'MIDX' \"$writeBitmapLookupTable\"\n \n-\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n-\t\t# commit.\n-\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n-\t\t$(git rev-parse packed)\n-\t\tEOF\n+\ttest_expect_success 'missing object closure fails gracefully' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n-\t\tgrep \"doesn.t have full closure\" err &&\n-\t\ttest_path_is_missing $midx\n-\t)\n-'\n+\t\t\ttest_commit loose &&\n+\t\t\ttest_commit packed &&\n \n-midx_bitmap_partial_tests\n+\t\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t\t# commit.\n+\t\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t\t$(git rev-parse packed)\n+\t\t\tEOF\n \n-test_expect_success 'removing a MIDX clears stale bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\t\ttest_commit base &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\t\tgrep \"doesn.t have full closure\" err &&\n+\t\t\ttest_path_is_missing $midx\n+\t\t)\n+\t'\n \n-\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n-\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm $midx &&\n+\tmidx_bitmap_partial_tests\n \n-\t\t# Then write a new MIDX.\n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\ttest_expect_success 'removing a MIDX clears stale bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\ttest_commit base &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm $midx &&\n+\n+\t\t\t# Then write a new MIDX.\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest_path_is_missing $stale_bitmap\n+\t\t)\n+\t'\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_missing $stale_bitmap\n-\t)\n-'\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\t\tbitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n \n-\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n-\t\t\tmulti-pack-index write --bitmap &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\ttest_expect_success 'writing a bitmap with --refs-snapshot' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'writing a bitmap with --refs-snapshot' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\ttest_commit two &&\n \n-\t\ttest_commit one &&\n-\t\ttest_commit two &&\n+\t\t\tgit rev-parse one >snapshot &&\n \n-\t\tgit rev-parse one >snapshot &&\n+\t\t\tgit repack -ad &&\n \n-\t\tgit repack -ad &&\n+\t\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t\t# bitmaps).\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\t# First, write a MIDX which see both refs/tags/one and\n-\t\t# refs/tags/two (causing both of those commits to receive\n-\t\t# bitmaps).\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\t# Then again, but with a refs snapshot which only sees\n+\t\t\t# refs/tags/one.\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n \n-\t\t# Then again, but with a refs snapshot which only sees\n-\t\t# refs/tags/one.\n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\t! grep \"$(git rev-parse two)\" bitmaps\n-\t)\n-'\n+\ttest_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\t(\n+\t\t\t\tgrep -vf before commits.raw &&\n+\t\t\t\t# mark missing commits as preferred\n+\t\t\t\tsed \"s/^/+/\" before\n+\t\t\t) >snapshot &&\n \n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n+\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'hash-cache values are propagated from pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n \t\t(\n-\t\t\tgrep -vf before commits.raw &&\n-\t\t\t# mark missing commits as preferred\n-\t\t\tsed \"s/^/+/\" before\n-\t\t) >snapshot &&\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\ttest_commit base &&\n+\t\t\ttest_commit base2 &&\n+\t\t\tgit repack -adb &&\n \n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\ttest-tool bitmap dump-hashes >pack.raw &&\n+\t\t\ttest_file_not_empty pack.raw &&\n+\t\t\tsort pack.raw >pack.hashes &&\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest-tool bitmap dump-hashes >midx.raw &&\n+\t\t\tsort midx.raw >midx.hashes &&\n \n-\t\ttest_commit base &&\n-\t\ttest_commit base2 &&\n-\t\tgit repack -adb &&\n+\t\t\t# ensure that every namehash in the pack bitmap can be found in\n+\t\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n+\t\t\t# unique to the pack bitmap).\n+\t\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n+\t\t\ttest_must_be_empty dropped.hashes\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap dump-hashes >pack.raw &&\n-\t\ttest_file_not_empty pack.raw &&\n-\t\tsort pack.raw >pack.hashes &&\n+\ttest_expect_success 'no .bitmap is written without any objects' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n+\t\t\tpack-$empty.idx\n+\t\t\tEOF\n \n-\t\ttest-tool bitmap dump-hashes >midx.raw &&\n-\t\tsort midx.raw >midx.hashes &&\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\terr &&\n \n-\t\t# ensure that every namehash in the pack bitmap can be found in\n-\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n-\t\t# unique to the pack bitmap).\n-\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n-\t\ttest_must_be_empty dropped.hashes\n-\t)\n-'\n+\t\t\tgrep \"bitmap without any objects\" err &&\n \n-test_expect_success 'no .bitmap is written without any objects' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'graceful fallback when missing reverse index' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n-\t\tpack-$empty.idx\n-\t\tEOF\n+\t\t\ttest_commit base &&\n \n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\terr &&\n+\t\t\t# write a pack and MIDX bitmap containing base\n+\t\t\tgit repack -adb &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\tgrep \"bitmap without any objects\" err &&\n+\t\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n+\t\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\t! grep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n-\t)\n-'\n+test_midx_bitmap_cases\n+\n+test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'graceful fallback when missing reverse index' '\n+test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n \t(\n \t\tcd repo &&\n-\n \t\ttest_commit base &&\n-\n-\t\t# write a pack and MIDX bitmap containing base\n-\t\tgit repack -adb &&\n-\t\tgit multi-pack-index write --bitmap &&\n-\n-\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n-\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\t! grep \"ignoring extra bitmap file\" err\n+\t\tgit config pack.writeBitmapLookupTable true &&\n+\t\tgit repack -ad &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n \t)\n '\n \ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex d30ba632c87..d01c61c0c7e 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -20,4 +20,13 @@ export GIT_TEST_MIDX_READ_RIDX\n midx_bitmap_core rev\n midx_bitmap_partial_tests rev\n \n+test_expect_success 'reinitialize the repository with lookup table enabled' '\n+ rm -fr * .git &&\n+ git init &&\n+ git config pack.writeBitmapLookupTable true\n+'\n+\n+midx_bitmap_core rev\n+midx_bitmap_partial_tests rev\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"459509","messageId":"52f7d8359ee766442ca03f0b47a491bcb2fab81b.1658325914.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v4 6/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T14:05:13Z","receivedAt":"2022-07-20T14:05:43Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd performance tests to verify the performance of lookup table with\n`pack.writeReverseIndex` enabled. This is to check the performance\nwhen the above configuration is set.\n\nLookup table makes Git run faster in most of the cases. Below is the\nresult of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\ngives similar result. The repository used in the test is linux kernel.\n\nTest this tree\n---------------------------------------------------------------------------\n5310.4: repack to disk (lookup=false) 296.55(256.53+14.52)\n5310.5: simulated clone 15.64(8.88+1.39)\n5310.6: simulated fetch 1.65(2.75+0.20)\n5310.7: pack to file (bitmap) 48.71(30.20+7.58)\n5310.8: rev-list (commits) 0.61(0.41+0.08)\n5310.9: rev-list (objects) 4.38(4.26+0.09)\n5310.10: rev-list with tag negated via --not 0.07(0.02+0.04)\n --all (objects)\n5310.11: rev-list with negative tag (objects) 0.05(0.01+0.03)\n5310.12: rev-list count with blob:none 0.08(0.03+0.04)\n5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n5310.14: rev-list count with tree:0 0.08(0.03+0.04)\n5310.15: simulated partial clone 9.45(8.12+0.41)\n5310.17: clone (partial bitmap) 21.00(15.04+2.39)\n5310.18: pack to file (partial bitmap) 47.98(38.13+5.23)\n5310.19: rev-list with tree filter (partial bitmap) 0.70(0.07+0.20)\n5310.22: repack to disk (lookup=true) 255.92(188.13+20.47)\n5310.23: simulated clone 13.78(8.84+1.09)\n5310.24: simulated fetch 0.52(0.63+0.14)\n5310.25: pack to file (bitmap) 44.34(28.94+6.84)\n5310.26: rev-list (commits) 0.48(0.31+0.06)\n5310.27: rev-list (objects) 4.02(3.93+0.07)\n5310.28: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n5310.29: rev-list with negative tag (objects) 0.04(0.00+0.03)\n5310.30: rev-list count with blob:none 0.04(0.01+0.03)\n5310.31: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n5310.32: rev-list count with tree:0 0.04(0.01+0.03)\n5310.33: simulated partial clone 8.30(7.21+0.36)\n5310.35: clone (partial bitmap) 20.34(15.00+2.41)\n5310.36: pack to file (partial bitmap) 46.45(38.05+5.20)\n5310.37: rev-list with tree filter (partial bitmap) 0.61(0.06+0.20)\n\nTest 4-15 are tested without using lookup table. Same tests are\nrepeated in 16-30 (using lookup table).\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 65 +++++++++++---------\n t/perf/p5326-multi-pack-bitmaps.sh | 95 +++++++++++++++++-------------\n 2 files changed, 91 insertions(+), 69 deletions(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 6e8abcd5b21..adc753b6177 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -17,39 +17,50 @@ test_expect_success 'setup bitmap config' '\n \tgit config pack.writeReverseIndex true\n '\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n+test_bitmap () {\n+\tlocal enabled=\"$1\"\n \n-test_perf 'repack to disk' '\n-\tgit repack -ad\n-'\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n \n-test_full_bitmap\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n \n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n+\ttest_perf \"repack to disk (lookup=$enabled)\" '\n+\t\tgit repack -ad\n+\t'\n \n-\t# now kill off all of the refs and pretend we had\n-\t# just the one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n+\ttest_full_bitmap\n \n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n \n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n+\t\t# now kill off all of the refs and pretend we had\n+\t\t# just the one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+}\n \n-test_partial_bitmap\n+test_bitmap false\n+test_bitmap true\n \n test_done\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nindex f2fa228f16a..1f4c7103529 100755\n--- a/t/perf/p5326-multi-pack-bitmaps.sh\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -6,47 +6,58 @@ test_description='Tests performance using midx bitmaps'\n \n test_perf_large_repo\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n-\n-test_expect_success 'start with bitmapped pack' '\n-\tgit repack -adb\n-'\n-\n-test_perf 'setup multi-pack index' '\n-\tgit multi-pack-index write --bitmap\n-'\n-\n-test_expect_success 'drop pack bitmap' '\n-\trm -f .git/objects/pack/pack-*.bitmap\n-'\n-\n-test_full_bitmap\n-\n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now pretend we have just one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n-\n-test_partial_bitmap\n+test_bitmap () {\n+\tlocal enabled=\"$1\"\n+\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n+\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n+\n+\ttest_expect_success \"start with bitmapped pack (lookup=$enabled)\" '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_perf \"setup multi-pack index (lookup=$enabled)\" '\n+\t\tgit multi-pack-index write --bitmap\n+\t'\n+\n+\ttest_expect_success \"drop pack bitmap (lookup=$enabled)\" '\n+\t\trm -f .git/objects/pack/pack-*.bitmap\n+\t'\n+\n+\ttest_full_bitmap\n+\n+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now pretend we have just one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+}\n+\n+test_bitmap false\n+test_bitmap true\n \n test_done\n-- \ngitgitgadget\n"},{"id":"459524","messageId":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v4.git.1658325913.gitgitgadget@gmail.com","subject":"[PATCH v5 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:18Z","receivedAt":"2022-07-20T18:38:38Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"When parsing the .bitmap file, git loads all the bitmaps one by one even if\nsome of the bitmaps are not necessary. We can remove this overhead by\nloading only the necessary bitmaps. A look up table extension can solve this\nissue.\n\nChanges since v4:\n\n * There was a CI failing test for linux-sha256 in the previous version.\n Fixed now.\n\nChanges since v3:\n\n * The common code from both lookup_table_get_triplet() and\n bsearch_triplet_by_pos are moved to lookup_table_get_triplet_by_pointer\n function\n * parameter names of triplet_cmp function is changes (as suggested by\n Martin)\n * xor_items array is now work as reusable static buffer.\n * I moved the filling commit_positions array part (from\n pack-bitmap-write.c) to bitmap_writer_finish function. Because we had to\n iterate two times for commit positions - one in write_selected_commits_v1\n and another in write_lookup_table function. Hope this is acceptable :)\n * changes in performance tests (as suggested by Taylor)\n\nChanges since v2:\n\n * Log messages related issues are fixed.\n * pack.writeBitmapLookupTable is now by default disabled.\n * Documentations are improved.\n * xor_row is used instead of xor_pos in triplets.\n * In pack-bitmap-write.c, off_t * is used for offsets array (Instead of\n uint64_t *).\n * struct bitmap_lookup_table_triplet is introduced and functions Like\n triplet_get_offset() and triplet_get_xor_pos() are removed.\n * table_size is getting subtracted from index_end irrespective of the value\n of GIT_TEST_READ_COMMIT_TABLE.\n * xor stack filling loop will stop iterating if a xor bitmap is already\n stored/parsed.\n * The stack will now store bitmap_lookup_table_xor_item items Of plain\n xor_row.\n * bitmap related test files are reformatted to allow repeating of tests\n with bitmap extension enabled.\n * comments are added.\n\nChanges since v1:\n\nThis is the second version which addressed all (I think) the reviews. Please\nnotify me if some reviews are not addressed :)\n\n * The table size is decreased and the format has also changed. It now\n contains nr_entries triplets of size 4+8+4 bytes. Each triplet contains\n the following things - (1) 4 byte commit position (in the pack-index or\n midx) (2) 8 byte offset and (3) 4 byte xor triplet (i.e. with whose\n bitmap the current triplet's bitmap has to xor) position.\n * Performance tests are splitted into two commits. First contains the\n actual performance tests and second enables the pack.writeReverseIndex\n (as suggested by Taylor).\n * st_*() functions are used.\n * commit order is changed according to Derrick's suggestion.\n * Iterative approach is used instead of recursive approach to parse xor\n bitmaps. (As suggested by Derrick).\n * Some minor bug fixes of previous version.\n\nInitial version:\n\nThe proposed table has:\n\n * a list of nr_entries object ids. These objects are commits that has\n bitmaps. Ids are stored in lexicographic order (for better searching).\n * a list of pairs (4-byte integers, network-byte\n order). The i'th pair denotes the offset and xor-offset(respectively) of\n the bitmap of i'th commit in the previous list. These two informations\n are necessary because only in this way bitmaps can be found without\n parsing all the bitmap.\n * a 4-byte integer for table specific flags (none exists currently).\n\nWhenever git want to parse the bitmap for a specific commit, it will first\nrefer to the table and will look for the offset and xor-offset for that\ncommit. Git will then try to parse the bitmap located at the offset\nposition. The xor-offset can be used to find the xor-bitmap for the\nbitmap(if any).\n\nAbhradeep Chakraborty (6):\n Documentation/technical: describe bitmap lookup table extension\n pack-bitmap-write.c: write lookup table extension\n pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n pack-bitmap: prepare to read lookup table extension\n p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`\n bitmap-lookup-table: add performance tests for lookup table\n\n Documentation/config/pack.txt | 7 +\n Documentation/technical/bitmap-format.txt | 39 ++\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n pack-bitmap-write.c | 112 ++-\n pack-bitmap.c | 275 +++++++-\n pack-bitmap.h | 14 +-\n t/perf/p5310-pack-bitmaps.sh | 68 +-\n t/perf/p5326-multi-pack-bitmaps.sh | 95 +--\n t/t5310-pack-bitmaps.sh | 786 ++++++++++++----------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++-----\n t/t5327-multi-pack-bitmaps-rev.sh | 24 +-\n 15 files changed, 1254 insertions(+), 659 deletions(-)\n\n\nbase-commit: 39c15e485575089eb77c769f6da02f98a55905e0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1266%2FAbhra303%2Fbitmap-commit-table-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1266/Abhra303/bitmap-commit-table-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1266\n\nRange-diff vs v4:\n\n 1: f72bf11e6ef ! 1: 33aca8f3dc8 Documentation/technical: describe bitmap lookup table extension\n @@ Commit message\n even if all the bitmaps are not required. A \"bitmap lookup table\"\n extension to the bitmap format can reduce the overhead of loading\n bitmaps which stores a list of bitmapped commit id pos (in the midx\n - or pack, along with their offset and xor offset. This way git can\n + or pack, along with their offset and xor offset. This way Git can\n load only the necessary bitmaps without loading the previous bitmaps.\n \n Older versions of Git ignore the lookup table extension and don't\n 2: 04244fadf5c = 2: a913e6a2cb3 pack-bitmap-write.c: write lookup table extension\n 3: 8bd7639e4b9 ! 3: 59b465e5a78 pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n @@ t/t5311-pack-bitmaps-shallow.sh: test_description='check bitmap operation with s\n +}\n +\n +test_shallow_bitmaps\n -+\n ++test_shallow_bitmaps \"pack.writeBitmapLookupTable\"\n \n test_done\n \n @@ t/t5326-multi-pack-bitmaps.sh: bitmap_reuse_tests() {\n \n \n ## t/t5327-multi-pack-bitmaps-rev.sh ##\n -@@ t/t5327-multi-pack-bitmaps-rev.sh: export GIT_TEST_MIDX_READ_RIDX\n - midx_bitmap_core rev\n - midx_bitmap_partial_tests rev\n +@@ t/t5327-multi-pack-bitmaps-rev.sh: GIT_TEST_MIDX_READ_RIDX=0\n + export GIT_TEST_MIDX_WRITE_REV\n + export GIT_TEST_MIDX_READ_RIDX\n + \n +-midx_bitmap_core rev\n +-midx_bitmap_partial_tests rev\n ++test_midx_bitmap_rev () {\n ++ writeLookupTable=false\n ++\n ++ \tfor i in \"$@\"\n ++ \tdo\n ++ \t\tcase $i in\n ++ \t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n ++ \t\tesac\n ++ \tdone\n ++\n ++ test_expect_success 'setup bitmap config' '\n ++ rm -rf * .git &&\n ++ git init &&\n ++ git config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n ++ '\n ++\n ++ midx_bitmap_core rev\n ++ midx_bitmap_partial_tests rev\n ++ }\n ++\n ++ test_midx_bitmap_rev\n ++ test_midx_bitmap_rev \"pack.writeBitmapLookupTable\"\n \n -+test_expect_success 'reinitialize the repository with lookup table enabled' '\n -+ rm -fr * .git &&\n -+ git init &&\n -+ git config pack.writeBitmapLookupTable true\n -+'\n -+\n -+midx_bitmap_core rev\n -+midx_bitmap_partial_tests rev\n -+\n test_done\n 4: afc8c660ac1 ! 4: 6918f0860ad pack-bitmap: prepare to read lookup table extension\n @@ pack-bitmap.c: struct include_data {\n +\tif (xor_row != 0xffffffff) {\n +\t\tint xor_flags;\n +\t\tkhiter_t hash_pos;\n -+\t\tuint64_t offset_xor;\n +\t\tstruct bitmap_lookup_table_xor_item *xor_item;\n +\n +\t\twhile (xor_row != 0xffffffff) {\n @@ pack-bitmap.c: struct include_data {\n +\n +\t\twhile (xor_items_nr) {\n +\t\t\txor_item = &xor_items[xor_items_nr - 1];\n -+\t\t\toffset_xor = xor_item->offset;\n -+\n -+\t\t\tbitmap_git->map_pos = offset_xor;\n ++\t\t\tbitmap_git->map_pos = xor_item->offset;\n +\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n +\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n +\t\t\t\t\toid_to_hex(&xor_item->oid));\n 5: fc69489e395 = 5: e7ef420f321 p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`\n 6: 52f7d8359ee = 6: 6628001241d bitmap-lookup-table: add performance tests for lookup table\n\n-- \ngitgitgadget\n"},{"id":"459525","messageId":"33aca8f3dc8b12773f187fcb8e40060c2be4aabc.1658342304.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v5 1/6] Documentation/technical: describe bitmap lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:19Z","receivedAt":"2022-07-20T18:38:42Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nWhen reading bitmap file, Git loads each and every bitmap one by one\neven if all the bitmaps are not required. A \"bitmap lookup table\"\nextension to the bitmap format can reduce the overhead of loading\nbitmaps which stores a list of bitmapped commit id pos (in the midx\nor pack, along with their offset and xor offset. This way Git can\nload only the necessary bitmaps without loading the previous bitmaps.\n\nOlder versions of Git ignore the lookup table extension and don't\nthrow any kind of warning or error while parsing the bitmap file.\n\nAdd some information for the new \"bitmap lookup table\" extension in the\nbitmap-format documentation.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/technical/bitmap-format.txt | 39 +++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 04b3ec21785..c30dc177643 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -67,6 +67,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n \t\t\tpack/MIDX. The format and meaning of the name-hash is\n \t\t\tdescribed below.\n \n+\t\t\t** {empty}\n+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n+\t\t\tIf present, the end of the bitmap file contains a table\n+\t\t\tcontaining a list of `N` \n+\t\t\ttriplets. The format and meaning of the table is described\n+\t\t\tbelow.\n++\n+NOTE: Unlike the xor_offset used to compress an individual bitmap,\n+`xor_row` stores an *absolute* index into the lookup table, not a location\n+relative to the current entry.\n+\n \t\t4-byte entry count (network byte order)\n \n \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n@@ -205,3 +216,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n hashes made under two different schemes would be pointless).\n+\n+Commit lookup table\n+-------------------\n+\n+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n+bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`\n+file contains a lookup table specifying the information needed to get\n+the desired bitmap from the entries without parsing previous unnecessary\n+bitmaps.\n+\n+For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n+contains a list of `nr_entries` triplets\n+(sorted in the ascending order of `commit_pos`). The content of i'th\n+triplet is -\n+\n+\t* {empty}\n+\tcommit_pos (4 byte integer, network byte order): ::\n+\tIt stores the object position of a commit (in the midx or pack\n+\tindex).\n+\n+\t* {empty}\n+\toffset (8 byte integer, network byte order): ::\n+\tThe offset from which that commit's bitmap can be read.\n+\n+\t* {empty}\n+\txor_row (4 byte integer, network byte order): ::\n+\tThe position of the triplet whose bitmap is used to compress\n+\tthis one, or `0xffffffff` if no such bitmap exists.\n-- \ngitgitgadget\n\n"},{"id":"459526","messageId":"a913e6a2cb36d8ec7900b60820d8ab3c35f60164.1658342304.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v5 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:20Z","receivedAt":"2022-07-20T18:38:49Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe bitmap lookup table extension was documented by an earlier\nchange, but Git does not yet know how to write that extension.\n\nTeach Git to write bitmap lookup table extension. The table contains\nthe list of `N` ` triplets. These\ntriplets are sorted according to their commit pos (ascending order).\nThe meaning of each data in the i'th triplet is given below:\n\n - commit_pos stores commit position (in the pack-index or midx).\n It is a 4 byte network byte order unsigned integer.\n\n - offset is the position (in the bitmap file) from which that\n commit's bitmap can be read.\n\n - xor_row is the position of the triplet in the lookup table\n whose bitmap is used to compress this bitmap, or `0xffffffff`\n if no such bitmap exists.\n\nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nCo-authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 112 ++++++++++++++++++++++++++++++++++++++++----\n pack-bitmap.h | 5 +-\n 2 files changed, 107 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c43375bd344..9843790cb60 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -650,20 +650,19 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t off_t *offsets,\n+\t\t\t\t uint32_t *commit_positions)\n {\n \tint i;\n \n \tfor (i = 0; i < writer.selected_nr; ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n-\t\tint commit_pos =\n-\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\t\tif (offsets)\n+\t\t\toffsets[i] = hashfile_total(f);\n \n-\t\tif (commit_pos < 0)\n-\t\t\tBUG(\"trying to write commit not in index\");\n-\n-\t\thashwrite_be32(f, commit_pos);\n+\t\thashwrite_be32(f, commit_positions[i]);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n \n@@ -671,6 +670,81 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static int table_cmp(const void *_va, const void *_vb, void *_data)\n+{\n+\tuint32_t *commit_positions = _data;\n+\tuint32_t a = commit_positions[*(uint32_t *)_va];\n+\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n+\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static void write_lookup_table(struct hashfile *f,\n+\t\t\t struct pack_idx_entry **index,\n+\t\t\t uint32_t index_nr,\n+\t\t\t off_t *offsets,\n+\t\t\t uint32_t *commit_positions)\n+{\n+\tuint32_t i;\n+\tuint32_t *table, *table_inv;\n+\n+\tALLOC_ARRAY(table, writer.selected_nr);\n+\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable[i] = i;\n+\n+\t/*\n+\t * At the end of this sort table[j] = i means that the i'th\n+\t * bitmap corresponds to j'th bitmapped commit (among the selected\n+\t * commits) in lex order of OIDs.\n+\t */\n+\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\n+\t/* table_inv helps us discover that relationship (i'th bitmap\n+\t * to j'th commit by j = table_inv[i])\n+\t */\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable_inv[table[i]] = i;\n+\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\t\tuint32_t xor_offset = selected->xor_offset;\n+\t\tuint32_t xor_row;\n+\n+\t\tif (xor_offset) {\n+\t\t\t/*\n+\t\t\t * xor_index stores the index (in the bitmap entries)\n+\t\t\t * of the corresponding xor bitmap. But we need to convert\n+\t\t\t * this index into lookup table's index. So, table_inv[xor_index]\n+\t\t\t * gives us the index position w.r.t. the lookup table.\n+\t\t\t *\n+\t\t\t * If \"k = table[i] - xor_offset\" then the xor base is the k'th\n+\t\t\t * bitmap. `table_inv[k]` gives us the position of that bitmap\n+\t\t\t * in the lookup table.\n+\t\t\t */\n+\t\t\tuint32_t xor_index = table[i] - xor_offset;\n+\t\t\txor_row = table_inv[xor_index];\n+\t\t} else {\n+\t\t\txor_row = 0xffffffff;\n+\t\t}\n+\n+\t\thashwrite_be32(f, commit_positions[table[i]]);\n+\t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n+\t\thashwrite_be32(f, xor_row);\n+\t}\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\n+\tfree(table);\n+\tfree(table_inv);\n+}\n+\n static void write_hash_cache(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n@@ -695,8 +769,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n {\n \tstatic uint16_t default_version = 1;\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n+\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n+\tuint32_t *commit_positions = NULL;\n \n \tstruct bitmap_disk_header header;\n \n@@ -715,7 +791,25 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n-\twrite_selected_commits_v1(f, index, index_nr);\n+\n+\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n+\tfor (uint32_t i = 0; i < writer.selected_nr; ++i) {\n+\t\tstruct bitmapped_commit *stored = &writer.selected[i];\n+\t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\n+\t\tif (commit_pos < 0)\n+\t\t\tBUG(_(\"trying to write commit not in index\"));\n+\n+\t\tcommit_positions[i] = commit_pos;\n+\t}\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\tCALLOC_ARRAY(offsets, index_nr);\n+\n+\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\twrite_lookup_table(f, index, index_nr, offsets, commit_positions);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n@@ -730,4 +824,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n+\tfree(offsets);\n+\tfree(commit_positions);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3d3ddd77345..67a9d0fc303 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -24,8 +24,9 @@ struct bitmap_disk_header {\n #define NEEDS_BITMAP (1u<<22)\n \n enum pack_bitmap_opts {\n-\tBITMAP_OPT_FULL_DAG = 1,\n-\tBITMAP_OPT_HASH_CACHE = 4,\n+\tBITMAP_OPT_FULL_DAG = 0x1,\n+\tBITMAP_OPT_HASH_CACHE = 0x4,\n+\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n };\n \n enum pack_bitmap_flags {\n-- \ngitgitgadget\n\n"},{"id":"459527","messageId":"6918f0860adbea1156fb7a9f1a5aedd211872481.1658342304.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v5 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:22Z","receivedAt":"2022-07-20T18:39:00Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEarlier change teaches Git to write bitmap lookup table. But Git\ndoes not know how to parse them.\n\nTeach Git to parse the existing bitmap lookup table. The older\nversions of Git are not affected by it. Those versions ignore the\nlookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap.c | 275 ++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 9 ++\n t/t5310-pack-bitmaps.sh | 22 ++++\n 3 files changed, 296 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 36134222d7a..c7336397717 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -82,6 +82,12 @@ struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n+\t/*\n+\t * If not NULL, this point into the commit table extension\n+\t * (within the memory mapped region `map`).\n+\t */\n+\tunsigned char *table_lookup;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -185,6 +191,16 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex->hashes = (void *)(index_end - cache_size);\n \t\t\tindex_end -= cache_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE) {\n+\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n+\t\t\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit lookup table)\"));\n+\t\t\tif (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n+\t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n@@ -211,11 +227,13 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,\n \n \thash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret);\n \n-\t/* a 0 return code means the insertion succeeded with no changes,\n-\t * because the SHA1 already existed on the map. this is bad, there\n-\t * shouldn't be duplicated commits in the index */\n+\t/*\n+\t * A 0 return code means the insertion succeeded with no changes,\n+\t * because the SHA1 already existed on the map. This is bad, there\n+\t * shouldn't be duplicated commits in the index.\n+\t */\n \tif (ret == 0) {\n-\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n+\t\terror(_(\"duplicate entry in bitmap index: %s\"), oid_to_hex(oid));\n \t\treturn NULL;\n \t}\n \n@@ -470,7 +488,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n \n-\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n \treturn 0;\n@@ -557,13 +575,238 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct bitmap_lookup_table_triplet {\n+\tuint32_t commit_pos;\n+\tuint64_t offset;\n+\tuint32_t xor_row;\n+};\n+\n+struct bitmap_lookup_table_xor_item {\n+\tstruct object_id oid;\n+\tuint64_t offset;\n+};\n+\n+/*\n+ * Given a `triplet` struct pointer and pointer `p`, this\n+ * function reads the triplet beginning at `p` into the struct.\n+ * Note that this function assumes that there is enough memory\n+ * left for filling the `triplet` struct from `p`.\n+ */\n+static int lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n+\t\t\t\t\t const unsigned char *p)\n+{\n+\tif (!triplet)\n+\t\treturn -1;\n+\n+\ttriplet->commit_pos = get_be32(p);\n+\tp += sizeof(uint32_t);\n+\ttriplet->offset = get_be64(p);\n+\tp += sizeof(uint64_t);\n+\ttriplet->xor_row = get_be32(p);\n+\treturn 0;\n+}\n+\n+/*\n+ * This function gets the raw triplet from `row`'th row in the\n+ * lookup table and fills that data to the `triplet`.\n+ */\n+static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t pos,\n+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = NULL;\n+\tif (pos >= bitmap_git->entry_count)\n+\t\treturn error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n+\n+\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\n+\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n+}\n+\n+/*\n+ * Searches for a matching triplet. `commit_pos` is a pointer\n+ * to the wanted commit position value. `table_entry` points to\n+ * a triplet in lookup table. The first 4 bytes of each\n+ * triplet (pointed by `table_entry`) are compared with `*commit_pos`.\n+ */\n+static int triplet_cmp(const void *commit_pos, const void *table_entry)\n+{\n+\n+\tuint32_t a = *(uint32_t *)commit_pos;\n+\tuint32_t b = get_be32(table_entry);\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n+\t\t\t struct object_id *oid,\n+\t\t\t uint32_t *result)\n+{\n+\tint found;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n+\telse\n+\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n+\n+\treturn found;\n+}\n+\n+/*\n+ * `bsearch_triplet_by_pos` function searches for the raw triplet\n+ * having commit position same as `commit_pos` and fills `triplet`\n+ * object from the raw triplet. Returns 1 on success and 0 on\n+ * failure.\n+ */\n+static int bsearch_triplet_by_pos(uint32_t commit_pos,\n+\t\t\t\t struct bitmap_index *bitmap_git,\n+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n+\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n+\n+\tif (!p)\n+\t\treturn -1;\n+\n+\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n+}\n+\n+static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t struct commit *commit)\n+{\n+\tuint32_t commit_pos, xor_row;\n+\tuint64_t offset;\n+\tint flags, found;\n+\tstruct bitmap_lookup_table_triplet triplet;\n+\tstruct object_id *oid = &commit->object.oid;\n+\tstruct ewah_bitmap *bitmap;\n+\tstruct stored_bitmap *xor_bitmap = NULL;\n+\tconst int bitmap_header_size = 6;\n+\tstatic struct bitmap_lookup_table_xor_item *xor_items = NULL;\n+\tstatic size_t xor_items_nr = 0, xor_items_alloc = 0;\n+\tstatic int is_corrupt = 0;\n+\n+\tif (is_corrupt)\n+\t\treturn NULL;\n+\n+\tfound = bsearch_pos(bitmap_git, oid, &commit_pos);\n+\n+\tif (!found)\n+\t\treturn NULL;\n+\n+\tif (bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n+\t\treturn NULL;\n+\n+\txor_items_nr = 0;\n+\toffset = triplet.offset;\n+\txor_row = triplet.xor_row;\n+\n+\tif (xor_row != 0xffffffff) {\n+\t\tint xor_flags;\n+\t\tkhiter_t hash_pos;\n+\t\tstruct bitmap_lookup_table_xor_item *xor_item;\n+\n+\t\twhile (xor_row != 0xffffffff) {\n+\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n+\n+\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n+\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n+\t\t\t\tgoto corrupt;\n+\t\t\t}\n+\n+\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n+\t\t\t\tgoto corrupt;\n+\n+\t\t\txor_item = &xor_items[xor_items_nr];\n+\t\t\txor_item->offset = triplet.offset;\n+\n+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n+\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n+\t\t\t\t\ttriplet.commit_pos);\n+\t\t\t\tgoto corrupt;\n+\t\t\t}\n+\n+\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n+\n+\t\t\t/*\n+\t\t\t * If desired bitmap is already stored, we don't need\n+\t\t\t * to iterate further. Because we know that bitmaps\n+\t\t\t * that are needed to be parsed to parse this bitmap\n+\t\t\t * has already been stored. So, assign this stored bitmap\n+\t\t\t * to the xor_bitmap.\n+\t\t\t */\n+\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n+\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n+\t\t\t\tbreak;\n+\t\t\txor_items_nr++;\n+\t\t\txor_row = triplet.xor_row;\n+\t\t}\n+\n+\t\twhile (xor_items_nr) {\n+\t\t\txor_item = &xor_items[xor_items_nr - 1];\n+\t\t\tbitmap_git->map_pos = xor_item->offset;\n+\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n+\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\t\t\toid_to_hex(&xor_item->oid));\n+\t\t\t\tgoto corrupt;\n+\t\t\t}\n+\n+\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n+\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\t\t\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\t\t\tif (!bitmap)\n+\t\t\t\tgoto corrupt;\n+\n+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n+\t\t\txor_items_nr--;\n+\t\t}\n+\t}\n+\n+\tbitmap_git->map_pos = offset;\n+\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n+\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\toid_to_hex(oid));\n+\t\tgoto corrupt;\n+\t}\n+\n+\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n+\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\tif (!bitmap)\n+\t\tgoto corrupt;\n+\n+\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n+\n+corrupt:\n+\tfree(xor_items);\n+\tis_corrupt = 1;\n+\treturn NULL;\n+}\n+\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n-\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n-\t\treturn NULL;\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n+\t\tstruct stored_bitmap *bitmap = NULL;\n+\t\tif (!bitmap_git->table_lookup)\n+\t\t\treturn NULL;\n+\n+\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\t/* NEEDSWORK: cache misses aren't recorded */\n+\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n+\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\tif (!bitmap)\n+\t\t\treturn NULL;\n+\t\treturn lookup_stored_bitmap(bitmap);\n+\t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n@@ -1699,8 +1942,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(\"you must specify exactly one commit to test\");\n \n-\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n-\t\tbitmap_git->version, bitmap_git->entry_count);\n+\tfprintf(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\t\tbitmap_git->version,\n+\t\tbitmap_git->entry_count,\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n \n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n@@ -1753,13 +1998,23 @@ void test_bitmap_walk(struct rev_info *revs)\n \n int test_bitmap_commits(struct repository *r)\n {\n-\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n+\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n+\n+\t/*\n+\t * As this function is only used to print bitmap selected\n+\t * commits, we don't have to read the commit table.\n+\t */\n \n \tif (!bitmap_git)\n \t\tdie(\"failed to load bitmap indexes\");\n \n+\tif (bitmap_git->table_lookup) {\n+\t\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\t\t\tdie(_(\"failed to load bitmap indexes\"));\n+\t}\n+\n \tkh_foreach(bitmap_git->bitmaps, oid, value, {\n \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n \t});\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 67a9d0fc303..9278f71ac91 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -23,6 +23,15 @@ struct bitmap_disk_header {\n \n #define NEEDS_BITMAP (1u<<22)\n \n+/*\n+ * The width in bytes of a single triplet in the lookup table\n+ * extension:\n+ * (commit_pos, offset, xor_row)\n+ *\n+ * whose fields ar 32-, 64-, 32- bits wide, respectively.\n+ */\n+#define BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH (16)\n+\n enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c0607172827..7e50f8e7653 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -258,6 +258,7 @@ test_bitmap_cases () {\n \n \ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \t\ttest_config pack.writebitmaphashcache false &&\n+\t\ttest_config pack.writebitmaplookuptable false &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -270,6 +271,7 @@ test_bitmap_cases () {\n \t'\n \n \ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -453,4 +455,24 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n+test_expect_success 'lookup table is actually used to traverse objects' '\n+\tgit repack -adb &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n+\t\tgit rev-list --use-bitmap-index --count --all &&\n+\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n+\ttest_config pack.writebitmaphashcache false &&\n+\tgit repack -adb &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"459528","messageId":"e7ef420f321b3936185b2729460b1c28f5384438.1658342304.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:23Z","receivedAt":"2022-07-20T18:39:01Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEnable `pack.writeReverseIndex` before running pack-bitmap related\nperformance tests.\n\nThe performance difference with `pack.writeReverseIndex` enabled and\nwith disabled are given below -\n\nWith `pack.writeReverseIndex`\n-------------------------------\n\nTest this tree\n-------------------------------------------------------------------------\n5310.3: repack to disk 296.55(256.53+14.52)\n5310.4: simulated clone 15.64(8.88+1.39)\n5310.5: simulated fetch 1.65(2.75+0.20)\n5310.6: pack to file (bitmap) 48.71(30.20+7.58)\n5310.7: rev-list (commits) 0.61(0.41+0.08)\n5310.8: rev-list (objects) 4.38(4.26+0.09)\n5310.9: rev-list with tag negated via --not 0.07(0.02+0.04)\n --all (objects)\n5310.10: rev-list with negative tag (objects) 0.05(0.01+0.03)\n5310.11: rev-list count with blob:none 0.08(0.03+0.04)\n5310.12: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n5310.13: rev-list count with tree:0 0.08(0.03+0.04)\n5310.14: simulated partial clone 9.45(8.12+0.41)\n5310.16: clone (partial bitmap) 17.02(10.61+2.67)\n5310.17: pack to file (partial bitmap) 51.91(28.57+7.48)\n5310.18: rev-list with tree filter (partial bitmap) 1.00(0.22+0.24)\n\nWithout `pack.writeReverseIndex`:\n-----------------------------\n\nTest this tree\n------------------------------------------------------------------------\n5310.3: repack to disk 293.80(251.30+14.30)\n5310.4: simulated clone 12.50(5.15+1.36)\n5310.5: simulated fetch 1.83(2.90+0.23)\n5310.6: pack to file (bitmap) 39.70(20.25+7.14)\n5310.7: rev-list (commits) 1.00(0.60+0.13)\n5310.8: rev-list (objects) 4.11(4.00+0.10)\n5310.9: rev-list with tag negated via --not 0.07(0.02+0.05)\n --all (objects)\n5310.10: rev-list with negative tag (objects) 0.23(0.16+0.06)\n5310.11: rev-list count with blob:none 0.27(0.18+0.08)\n5310.12: rev-list count with blob:limit=1k 6.41(5.98+0.41)\n5310.13: rev-list count with tree:0 0.26(0.18+0.07)\n5310.14: simulated partial clone 4.34(3.29+0.37)\n5310.16: clone (partial bitmap) 21.48(15.12+2.42)\n5310.17: pack to file (partial bitmap) 47.35(37.80+4.84)\n5310.18: rev-list with tree filter (partial bitmap) 0.73(0.07+0.21)\n\nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 7ad4f237bc3..6e8abcd5b21 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -13,7 +13,8 @@ test_perf_large_repo\n # We intentionally use the deprecated pack.writebitmaps\n # config so that we can test against older versions of git.\n test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true\n+\tgit config pack.writebitmaps true &&\n+\tgit config pack.writeReverseIndex true\n '\n \n # we need to create the tag up front such that it is covered by the repack and\n-- \ngitgitgadget\n\n"},{"id":"459529","messageId":"6628001241d378c181564e6a27f7b6bc2fa5aa2e.1658342304.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v5 6/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:24Z","receivedAt":"2022-07-20T18:39:21Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd performance tests to verify the performance of lookup table with\n`pack.writeReverseIndex` enabled. This is to check the performance\nwhen the above configuration is set.\n\nLookup table makes Git run faster in most of the cases. Below is the\nresult of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\ngives similar result. The repository used in the test is linux kernel.\n\nTest this tree\n---------------------------------------------------------------------------\n5310.4: repack to disk (lookup=false) 296.55(256.53+14.52)\n5310.5: simulated clone 15.64(8.88+1.39)\n5310.6: simulated fetch 1.65(2.75+0.20)\n5310.7: pack to file (bitmap) 48.71(30.20+7.58)\n5310.8: rev-list (commits) 0.61(0.41+0.08)\n5310.9: rev-list (objects) 4.38(4.26+0.09)\n5310.10: rev-list with tag negated via --not 0.07(0.02+0.04)\n --all (objects)\n5310.11: rev-list with negative tag (objects) 0.05(0.01+0.03)\n5310.12: rev-list count with blob:none 0.08(0.03+0.04)\n5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n5310.14: rev-list count with tree:0 0.08(0.03+0.04)\n5310.15: simulated partial clone 9.45(8.12+0.41)\n5310.17: clone (partial bitmap) 21.00(15.04+2.39)\n5310.18: pack to file (partial bitmap) 47.98(38.13+5.23)\n5310.19: rev-list with tree filter (partial bitmap) 0.70(0.07+0.20)\n5310.22: repack to disk (lookup=true) 255.92(188.13+20.47)\n5310.23: simulated clone 13.78(8.84+1.09)\n5310.24: simulated fetch 0.52(0.63+0.14)\n5310.25: pack to file (bitmap) 44.34(28.94+6.84)\n5310.26: rev-list (commits) 0.48(0.31+0.06)\n5310.27: rev-list (objects) 4.02(3.93+0.07)\n5310.28: rev-list with tag negated via --not 0.04(0.00+0.03)\n --all (objects)\n5310.29: rev-list with negative tag (objects) 0.04(0.00+0.03)\n5310.30: rev-list count with blob:none 0.04(0.01+0.03)\n5310.31: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n5310.32: rev-list count with tree:0 0.04(0.01+0.03)\n5310.33: simulated partial clone 8.30(7.21+0.36)\n5310.35: clone (partial bitmap) 20.34(15.00+2.41)\n5310.36: pack to file (partial bitmap) 46.45(38.05+5.20)\n5310.37: rev-list with tree filter (partial bitmap) 0.61(0.06+0.20)\n\nTest 4-15 are tested without using lookup table. Same tests are\nrepeated in 16-30 (using lookup table).\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/p5310-pack-bitmaps.sh | 65 +++++++++++---------\n t/perf/p5326-multi-pack-bitmaps.sh | 95 +++++++++++++++++-------------\n 2 files changed, 91 insertions(+), 69 deletions(-)\n\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 6e8abcd5b21..adc753b6177 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -17,39 +17,50 @@ test_expect_success 'setup bitmap config' '\n \tgit config pack.writeReverseIndex true\n '\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n+test_bitmap () {\n+\tlocal enabled=\"$1\"\n \n-test_perf 'repack to disk' '\n-\tgit repack -ad\n-'\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n \n-test_full_bitmap\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n \n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n+\ttest_perf \"repack to disk (lookup=$enabled)\" '\n+\t\tgit repack -ad\n+\t'\n \n-\t# now kill off all of the refs and pretend we had\n-\t# just the one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n+\ttest_full_bitmap\n \n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n \n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n+\t\t# now kill off all of the refs and pretend we had\n+\t\t# just the one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+}\n \n-test_partial_bitmap\n+test_bitmap false\n+test_bitmap true\n \n test_done\ndiff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh\nindex f2fa228f16a..1f4c7103529 100755\n--- a/t/perf/p5326-multi-pack-bitmaps.sh\n+++ b/t/perf/p5326-multi-pack-bitmaps.sh\n@@ -6,47 +6,58 @@ test_description='Tests performance using midx bitmaps'\n \n test_perf_large_repo\n \n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n-\n-test_expect_success 'start with bitmapped pack' '\n-\tgit repack -adb\n-'\n-\n-test_perf 'setup multi-pack index' '\n-\tgit multi-pack-index write --bitmap\n-'\n-\n-test_expect_success 'drop pack bitmap' '\n-\trm -f .git/objects/pack/pack-*.bitmap\n-'\n-\n-test_full_bitmap\n-\n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now pretend we have just one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n-\n-test_partial_bitmap\n+test_bitmap () {\n+\tlocal enabled=\"$1\"\n+\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n+\n+\ttest_expect_success \"use lookup table: $enabled\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n+\t'\n+\n+\ttest_expect_success \"start with bitmapped pack (lookup=$enabled)\" '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_perf \"setup multi-pack index (lookup=$enabled)\" '\n+\t\tgit multi-pack-index write --bitmap\n+\t'\n+\n+\ttest_expect_success \"drop pack bitmap (lookup=$enabled)\" '\n+\t\trm -f .git/objects/pack/pack-*.bitmap\n+\t'\n+\n+\ttest_full_bitmap\n+\n+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now pretend we have just one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+}\n+\n+test_bitmap false\n+test_bitmap true\n \n test_done\n-- \ngitgitgadget\n"},{"id":"459530","messageId":"59b465e5a7817c145172f25e73ad807c7ba67e84.1658342304.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-20T18:38:21Z","receivedAt":"2022-07-20T18:39:21Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nTeach Git to provide a way for users to enable/disable bitmap lookup\ntable extension by providing a config option named 'writeBitmapLookupTable'.\nDefault is false.\n\nAlso add test to verify writting of lookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/config/pack.txt | 7 +\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n t/t5310-pack-bitmaps.sh | 792 ++++++++++++++++--------------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++++-------\n t/t5327-multi-pack-bitmaps-rev.sh | 24 +-\n 9 files changed, 733 insertions(+), 583 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex ad7f73a1ead..b955ca572ec 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n computed; instead, any namehashes stored in an existing bitmap are\n permuted into their appropriate location when writing a new bitmap.\n \n+pack.writeBitmapLookupTable::\n+\tWhen true, Git will include a \"lookup table\" section in the\n+\tbitmap index (if one is written). This table is used to defer\n+\tloading individual bitmaps as late as possible. This can be\n+\tbeneficial in repositories that have relatively large bitmap\n+\tindexes. Defaults to false.\n+\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5edbb7fe86e..55402b46f41 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n \t}\n \n+\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(var, value))\n+\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t\telse\n+\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t}\n+\n \t/*\n \t * We should never make a fall-back call to 'git_default_config', since\n \t * this was already called in 'cmd_multi_pack_index()'.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39e28cfcafc..46e26774963 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n+\n+\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n+\t\telse\n+\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n+\t}\n+\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\ndiff --git a/midx.c b/midx.c\nindex 5f0dd386b02..9c26d04bfde 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1072,6 +1072,9 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n \n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n \tprepare_midx_packing_data(&pdata, ctx);\n \n \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\ndiff --git a/midx.h b/midx.h\nindex 22e8e53288e..5578cd7b835 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,6 +47,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n+#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f775fc1ce69..c0607172827 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -26,22 +26,413 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-setup_bitmap_history\n-\n-test_expect_success 'setup writing bitmaps during repack' '\n-\tgit config repack.writeBitmaps true\n-'\n-\n-test_expect_success 'full repack creates bitmaps' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+test_bitmap_cases () {\n+\twriteLookupTable=false\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase \"$i\" in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test repository' '\n+\t\trm -fr * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'setup writing bitmaps during repack' '\n+\t\tgit config repack.writeBitmaps true\n+\t'\n+\n+\ttest_expect_success 'full repack creates bitmaps' '\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit repack -ad &&\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\t\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n+\t'\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local loose)' '\n+\t\tgit init --bare alt.git &&\n+\t\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\t\techo content1 >file1 &&\n+\t\t# non-local loose object which is not present in bitmapped pack\n+\t\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t\t# non-local loose object which is also present in bitmapped pack\n+\t\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\t\tgit add file1 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file1 &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\t\tgit index-pack 1.pack &&\n+\t\tlist_packed_objects 1.idx >1.objects &&\n+\t\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t\t! has_any nonlocal-loose 1.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\t\techo content2 >file2 &&\n+\t\tblob2=$(git hash-object -w file2) &&\n+\t\tgit add file2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file2 &&\n+\t\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\t\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\t\trm $(objpath $blob2) &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\t\tgit index-pack 2a.pack &&\n+\t\tlist_packed_objects 2a.idx >2a.objects &&\n+\t\t! has_any keepobjects 2a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local pack)' '\n+\t\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\t\tgit index-pack 2b.pack &&\n+\t\tlist_packed_objects 2b.idx >2b.objects &&\n+\t\t! has_any keepobjects 2b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\t\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\t\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t\t>.git/objects/pack/$packbitmap.keep &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\t\tgit index-pack 3a.pack &&\n+\t\tlist_packed_objects 3a.idx >3a.objects &&\n+\t\t! has_any packbitmap.objects 3a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\t\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\t\trm -f .git/objects/pack/multi-pack-index &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\t\tgit index-pack 3b.pack &&\n+\t\tlist_packed_objects 3b.idx >3b.objects &&\n+\t\t! has_any packbitmap.objects 3b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects to file can use bitmap' '\n+\t\t# make sure we still have 1 bitmap index from previous tests\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\t# verify equivalent packs are generated with/without using bitmap index\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n+\t\ttest_cmp packa.objects packb.objects\n+\t'\n+\n+\ttest_expect_success 'full repack, reusing previous bitmaps' '\n \t\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n-'\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output\n+\t'\n+\n+\ttest_expect_success 'fetch (full bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'create objects for missing-HAVE tests' '\n+\t\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n+\t\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n+\t\tparent=$(echo parent | git commit-tree $tree) &&\n+\t\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n+\t\tcat >revs <<-EOF\n+\t\tHEAD\n+\t\t^HEAD^\n+\t\t^$commit\n+\t\tEOF\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --incremental' '\n+\t\tcat >revs2 <<-EOF &&\n+\t\tHEAD\n+\t\t$commit\n+\t\tEOF\n+\t\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\t\tgit index-pack 4.pack &&\n+\t\tlist_packed_objects 4.idx >4.objects &&\n+\t\ttest_line_count = 4 4.objects &&\n+\t\tgit rev-list --objects $commit >revlist &&\n+\t\tcut -d\" \" -f1 revlist |sort >objects &&\n+\t\ttest_cmp 4.objects objects\n+\t'\n+\n+\ttest_expect_success 'pack with missing blob' '\n+\t\trm $(objpath $blob) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing tree' '\n+\t\trm $(objpath $tree) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing parent' '\n+\t\trm $(objpath $parent) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n+\t\tgit clone --bare . compat-jgit.git &&\n+\t\t(\n+\t\t\tcd compat-jgit.git &&\n+\t\t\trm -f objects/pack/*.bitmap &&\n+\t\t\tjgit gc &&\n+\t\t\tgit rev-list --test-bitmap HEAD\n+\t\t)\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n+\t\tgit clone --bare . compat-us.git &&\n+\t\t(\n+\t\t\tcd compat-us.git &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\tgit repack -adb &&\n+\t\t\t# jgit gc will barf if it does not like our bitmaps\n+\t\t\tjgit gc\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'splitting packs does not generate bogus bitmaps' '\n+\t\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n+\t\tgit add rand &&\n+\t\tgit commit -m \"commit with big file\" &&\n+\t\tgit -c pack.packSizeLimit=500k repack -adb &&\n+\t\tgit init --bare no-bitmaps.git &&\n+\t\tgit -C no-bitmaps.git fetch .. HEAD\n+\t'\n+\n+\ttest_expect_success 'set up reusable pack' '\n+\t\trm -f .git/objects/pack/*.keep &&\n+\t\tgit repack -adb &&\n+\t\treusable_pack () {\n+\t\t\tgit for-each-ref --format=\"%(objectname)\" |\n+\t\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n+\t\t}\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --honor-pack-keep' '\n+\t\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n+\t\tfor i in .git/objects/pack/*.pack\n+\t\tdo\n+\t\t\t>${i%.pack}.keep || return 1\n+\t\tdone &&\n+\t\treusable_pack --honor-pack-keep >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --local' '\n+\t\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n+\t\treusable_pack --local >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --incremental' '\n+\t\treusable_pack --incremental >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\t\ttest_config pack.writebitmaphashcache false &&\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupted.bitmap.index stderr\n+\t'\n+\n+\t# Create a state of history with these properties:\n+\t#\n+\t# - refs that allow a client to fetch some new history, while sharing some old\n+\t# history with the server; we use branches delta-reuse-old and\n+\t# delta-reuse-new here\n+\t#\n+\t# - the new history contains an object that is stored on the server as a delta\n+\t# against a base that is in the old history\n+\t#\n+\t# - the base object is not immediately reachable from the tip of the old\n+\t# history; finding it would involve digging down through history we know the\n+\t# other side has\n+\t#\n+\t# This should result in a state where fetching from old->new would not\n+\t# traditionally reuse the on-disk delta (because we'd have to dig to realize\n+\t# that the client has it), but we will do so if bitmaps can tell us cheaply\n+\t# that the other side has it.\n+\ttest_expect_success 'set up thin delta-reuse parent' '\n+\t\t# This first commit contains the buried base object.\n+\t\ttest-tool genrandom delta 16384 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m \"delta base\" &&\n+\t\tbase=$(git rev-parse --verify HEAD:file) &&\n+\n+\t\t# These intermediate commits bury the base back in history.\n+\t\t# This becomes the \"old\" state.\n+\t\tfor i in 1 2 3 4 5\n+\t\tdo\n+\t\t\techo $i >file &&\n+\t\t\tgit commit -am \"intermediate $i\" || return 1\n+\t\tdone &&\n+\t\tgit branch delta-reuse-old &&\n+\n+\t\t# And now our new history has a delta against the buried base. Note\n+\t\t# that this must be smaller than the original file, since pack-objects\n+\t\t# prefers to create deltas from smaller objects to larger.\n+\t\ttest-tool genrandom delta 16300 >file &&\n+\t\tgit commit -am \"delta result\" &&\n+\t\tdelta=$(git rev-parse --verify HEAD:file) &&\n+\t\tgit branch delta-reuse-new &&\n+\n+\t\t# Repack with bitmaps and double check that we have the expected delta\n+\t\t# relationship.\n+\t\tgit repack -adb &&\n+\t\thave_delta $delta $base\n+\t'\n+\n+\t# Now we can sanity-check the non-bitmap behavior (that the server is not able\n+\t# to reuse the delta). This isn't strictly something we care about, so this\n+\t# test could be scrapped in the future. But it makes sure that the next test is\n+\t# actually triggering the feature we want.\n+\t#\n+\t# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n+\t# we actually perform the fetch, retain the resulting pack, and inspect the\n+\t# result.\n+\ttest_expect_success 'fetch without bitmaps ignores delta against old base' '\n+\t\ttest_config pack.usebitmaps false &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $ZERO_OID\n+\t\t)\n+\t'\n+\n+\t# And do the same for the bitmap case, where we do expect to find the delta.\n+\ttest_expect_success 'fetch with bitmaps can reuse old base' '\n+\t\ttest_config pack.usebitmaps true &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $base\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\t# create enough commits that not all are receive bitmap\n+\t\t\t# coverage even if they are all at the tip of some reference.\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\t\tgit rev-list HEAD >commits.raw &&\n+\t\t\tsort commits &&\n+\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin bitmaps &&\n+\n+\t\t\t# remember which commits did not receive bitmaps\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_file_not_empty before &&\n+\n+\t\t\t# mark the commits which did not receive bitmaps as preferred,\n+\t\t\t# and generate the bitmap again\n+\t\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'complains about multiple pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\ttest_commit base &&\n+\n+\t\t\tgit repack -adb &&\n+\t\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n+\t\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n+\n+\t\t\ttest_commit other &&\n+\t\t\tgit repack -ab &&\n+\n+\t\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n+\n+\t\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n+\t\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n+\t\t\ttest_line_count = 2 packs &&\n+\t\t\ttest_line_count = 2 bitmaps &&\n+\n+\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\tgrep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-basic_bitmap_tests\n+test_bitmap_cases\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -54,375 +445,12 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n-test_expect_success 'pack-objects respects --local (non-local loose)' '\n-\tgit init --bare alt.git &&\n-\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n-\techo content1 >file1 &&\n-\t# non-local loose object which is not present in bitmapped pack\n-\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n-\t# non-local loose object which is also present in bitmapped pack\n-\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n-\tgit add file1 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file1 &&\n-\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n-\tgit index-pack 1.pack &&\n-\tlist_packed_objects 1.idx >1.objects &&\n-\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n-\t! has_any nonlocal-loose 1.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n-\techo content2 >file2 &&\n-\tblob2=$(git hash-object -w file2) &&\n-\tgit add file2 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file2 &&\n-\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n-\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n-\trm $(objpath $blob2) &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n-\tgit index-pack 2a.pack &&\n-\tlist_packed_objects 2a.idx >2a.objects &&\n-\t! has_any keepobjects 2a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local pack)' '\n-\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n-\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n-\tgit index-pack 2b.pack &&\n-\tlist_packed_objects 2b.idx >2b.objects &&\n-\t! has_any keepobjects 2b.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tpackbitmap=$(basename $(cat output) .bitmap) &&\n-\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n-\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n-\t>.git/objects/pack/$packbitmap.keep &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n-\tgit index-pack 3a.pack &&\n-\tlist_packed_objects 3a.idx >3a.objects &&\n-\t! has_any packbitmap.objects 3a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n-\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n-\trm -f .git/objects/pack/multi-pack-index &&\n-\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n-\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n-\tgit index-pack 3b.pack &&\n-\tlist_packed_objects 3b.idx >3b.objects &&\n-\t! has_any packbitmap.objects 3b.objects\n-'\n-\n-test_expect_success 'pack-objects to file can use bitmap' '\n-\t# make sure we still have 1 bitmap index from previous tests\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\t# verify equivalent packs are generated with/without using bitmap index\n-\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n-\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n-\ttest_cmp packa.objects packb.objects\n-'\n-\n-test_expect_success 'full repack, reusing previous bitmaps' '\n-\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n-'\n-\n-test_expect_success 'fetch (full bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'create objects for missing-HAVE tests' '\n-\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n-\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n-\tparent=$(echo parent | git commit-tree $tree) &&\n-\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n-\tcat >revs <<-EOF\n-\tHEAD\n-\t^HEAD^\n-\t^$commit\n-\tEOF\n-'\n-\n-test_expect_success 'pack-objects respects --incremental' '\n-\tcat >revs2 <<-EOF &&\n-\tHEAD\n-\t$commit\n-\tEOF\n-\tgit pack-objects --incremental --stdout --revs 4.pack &&\n-\tgit index-pack 4.pack &&\n-\tlist_packed_objects 4.idx >4.objects &&\n-\ttest_line_count = 4 4.objects &&\n-\tgit rev-list --objects $commit >revlist &&\n-\tcut -d\" \" -f1 revlist |sort >objects &&\n-\ttest_cmp 4.objects objects\n-'\n-\n-test_expect_success 'pack with missing blob' '\n-\trm $(objpath $blob) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n+test_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'pack with missing tree' '\n-\trm $(objpath $tree) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success 'pack with missing parent' '\n-\trm $(objpath $parent) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n-\tgit clone --bare . compat-jgit.git &&\n-\t(\n-\t\tcd compat-jgit.git &&\n-\t\trm -f objects/pack/*.bitmap &&\n-\t\tjgit gc &&\n-\t\tgit rev-list --test-bitmap HEAD\n-\t)\n-'\n-\n-test_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n-\tgit clone --bare . compat-us.git &&\n-\t(\n-\t\tcd compat-us.git &&\n-\t\tgit repack -adb &&\n-\t\t# jgit gc will barf if it does not like our bitmaps\n-\t\tjgit gc\n-\t)\n-'\n-\n-test_expect_success 'splitting packs does not generate bogus bitmaps' '\n-\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n-\tgit add rand &&\n-\tgit commit -m \"commit with big file\" &&\n-\tgit -c pack.packSizeLimit=500k repack -adb &&\n-\tgit init --bare no-bitmaps.git &&\n-\tgit -C no-bitmaps.git fetch .. HEAD\n-'\n-\n-test_expect_success 'set up reusable pack' '\n-\trm -f .git/objects/pack/*.keep &&\n-\tgit repack -adb &&\n-\treusable_pack () {\n-\t\tgit for-each-ref --format=\"%(objectname)\" |\n-\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n-\t}\n-'\n-\n-test_expect_success 'pack reuse respects --honor-pack-keep' '\n-\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n-\tfor i in .git/objects/pack/*.pack\n-\tdo\n-\t\t>${i%.pack}.keep || return 1\n-\tdone &&\n-\treusable_pack --honor-pack-keep >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --local' '\n-\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n-\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n-\treusable_pack --local >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --incremental' '\n-\treusable_pack --incremental >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n-\ttest_config pack.writebitmaphashcache false &&\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt.ewah.bitmap stderr\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (cache)' '\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupted.bitmap.index stderr\n-'\n-\n-# Create a state of history with these properties:\n-#\n-# - refs that allow a client to fetch some new history, while sharing some old\n-# history with the server; we use branches delta-reuse-old and\n-# delta-reuse-new here\n-#\n-# - the new history contains an object that is stored on the server as a delta\n-# against a base that is in the old history\n-#\n-# - the base object is not immediately reachable from the tip of the old\n-# history; finding it would involve digging down through history we know the\n-# other side has\n-#\n-# This should result in a state where fetching from old->new would not\n-# traditionally reuse the on-disk delta (because we'd have to dig to realize\n-# that the client has it), but we will do so if bitmaps can tell us cheaply\n-# that the other side has it.\n-test_expect_success 'set up thin delta-reuse parent' '\n-\t# This first commit contains the buried base object.\n-\ttest-tool genrandom delta 16384 >file &&\n-\tgit add file &&\n-\tgit commit -m \"delta base\" &&\n-\tbase=$(git rev-parse --verify HEAD:file) &&\n-\n-\t# These intermediate commits bury the base back in history.\n-\t# This becomes the \"old\" state.\n-\tfor i in 1 2 3 4 5\n-\tdo\n-\t\techo $i >file &&\n-\t\tgit commit -am \"intermediate $i\" || return 1\n-\tdone &&\n-\tgit branch delta-reuse-old &&\n-\n-\t# And now our new history has a delta against the buried base. Note\n-\t# that this must be smaller than the original file, since pack-objects\n-\t# prefers to create deltas from smaller objects to larger.\n-\ttest-tool genrandom delta 16300 >file &&\n-\tgit commit -am \"delta result\" &&\n-\tdelta=$(git rev-parse --verify HEAD:file) &&\n-\tgit branch delta-reuse-new &&\n-\n-\t# Repack with bitmaps and double check that we have the expected delta\n-\t# relationship.\n-\tgit repack -adb &&\n-\thave_delta $delta $base\n-'\n-\n-# Now we can sanity-check the non-bitmap behavior (that the server is not able\n-# to reuse the delta). This isn't strictly something we care about, so this\n-# test could be scrapped in the future. But it makes sure that the next test is\n-# actually triggering the feature we want.\n-#\n-# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n-# we actually perform the fetch, retain the resulting pack, and inspect the\n-# result.\n-test_expect_success 'fetch without bitmaps ignores delta against old base' '\n-\ttest_config pack.usebitmaps false &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $ZERO_OID\n-\t)\n-'\n-\n-# And do the same for the bitmap case, where we do expect to find the delta.\n-test_expect_success 'fetch with bitmaps can reuse old base' '\n-\ttest_config pack.usebitmaps true &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $base\n-\t)\n-'\n-\n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\t# create enough commits that not all are receive bitmap\n-\t\t# coverage even if they are all at the tip of some reference.\n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n-\n-\t\tgit rev-list HEAD >commits.raw &&\n-\t\tsort commits &&\n-\n-\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n-\n-\t\t# remember which commits did not receive bitmaps\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_file_not_empty before &&\n-\n-\t\t# mark the commits which did not receive bitmaps as preferred,\n-\t\t# and generate the bitmap again\n-\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n-\n-\t\t! test_cmp before after\n-\t)\n-'\n-\n-test_expect_success 'complains about multiple pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit base &&\n-\n-\t\tgit repack -adb &&\n-\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n-\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n-\n-\t\ttest_commit other &&\n-\t\tgit repack -ab &&\n-\n-\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n-\n-\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n-\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n-\t\ttest_line_count = 2 packs &&\n-\t\ttest_line_count = 2 bitmaps &&\n-\n-\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\tgrep \"ignoring extra bitmap file\" err\n-\t)\n+test_expect_success 'verify writing bitmap lookup table when enabled' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2\" \\\n+\t\tgit repack -ad &&\n+\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n test_done\ndiff --git a/t/t5311-pack-bitmaps-shallow.sh b/t/t5311-pack-bitmaps-shallow.sh\nindex 872a95df338..9dae60f73e3 100755\n--- a/t/t5311-pack-bitmaps-shallow.sh\n+++ b/t/t5311-pack-bitmaps-shallow.sh\n@@ -17,23 +17,40 @@ test_description='check bitmap operation with shallow repositories'\n # the tree for A. But in a shallow one, we've grafted away\n # A, and fetching A to B requires that the other side send\n # us the tree for file=1.\n-test_expect_success 'setup shallow repo' '\n-\techo 1 >file &&\n-\tgit add file &&\n-\tgit commit -m orig &&\n-\techo 2 >file &&\n-\tgit commit -a -m update &&\n-\tgit clone --no-local --bare --depth=1 . shallow.git &&\n-\techo 1 >file &&\n-\tgit commit -a -m repeat\n-'\n-\n-test_expect_success 'turn on bitmaps in the parent' '\n-\tgit repack -adb\n-'\n-\n-test_expect_success 'shallow fetch from bitmapped repo' '\n-\t(cd shallow.git && git fetch)\n-'\n+test_shallow_bitmaps () {\n+\twriteLookupTable=false\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup shallow repo' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\techo 1 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m orig &&\n+\t\techo 2 >file &&\n+\t\tgit commit -a -m update &&\n+\t\tgit clone --no-local --bare --depth=1 . shallow.git &&\n+\t\techo 1 >file &&\n+\t\tgit commit -a -m repeat\n+\t'\n+\n+\ttest_expect_success 'turn on bitmaps in the parent' '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_expect_success 'shallow fetch from bitmapped repo' '\n+\t\t(cd shallow.git && git fetch)\n+\t'\n+}\n+\n+test_shallow_bitmaps\n+test_shallow_bitmaps \"pack.writeBitmapLookupTable\"\n \n test_done\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4fe57414c13..3b206adcee6 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -15,17 +15,24 @@ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n sane_unset GIT_TEST_MIDX_WRITE_REV\n sane_unset GIT_TEST_MIDX_READ_RIDX\n \n-midx_bitmap_core\n-\n bitmap_reuse_tests() {\n \tfrom=$1\n \tto=$2\n+\twriteLookupTable=false\n+\n+\tfor i in $3-${$#}\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n \n \ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n \t\trm -fr repo &&\n \t\tgit init repo &&\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk 16 &&\n \t\t\tgit tag old-tip &&\n \n@@ -43,6 +50,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk --id=further 16 &&\n \t\t\tgit tag new-tip &&\n \n@@ -59,6 +67,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\tgit for-each-ref &&\n \t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n \t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n@@ -66,244 +75,294 @@ bitmap_reuse_tests() {\n \t'\n }\n \n-bitmap_reuse_tests 'pack' 'MIDX'\n-bitmap_reuse_tests 'MIDX' 'pack'\n-bitmap_reuse_tests 'MIDX' 'MIDX'\n+test_midx_bitmap_cases () {\n+\twriteLookupTable=false\n+\twriteBitmapLookupTable=\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\")\n+\t\t\twriteLookupTable=true\n+\t\t\twriteBitmapLookupTable=\"$i\"\n+\t\t\t;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test_repository' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n \n-test_expect_success 'missing object closure fails gracefully' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\tmidx_bitmap_core\n \n-\t\ttest_commit loose &&\n-\t\ttest_commit packed &&\n+\tbitmap_reuse_tests 'pack' 'MIDX' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'pack' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'MIDX' \"$writeBitmapLookupTable\"\n \n-\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n-\t\t# commit.\n-\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n-\t\t$(git rev-parse packed)\n-\t\tEOF\n+\ttest_expect_success 'missing object closure fails gracefully' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n-\t\tgrep \"doesn.t have full closure\" err &&\n-\t\ttest_path_is_missing $midx\n-\t)\n-'\n+\t\t\ttest_commit loose &&\n+\t\t\ttest_commit packed &&\n \n-midx_bitmap_partial_tests\n+\t\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t\t# commit.\n+\t\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t\t$(git rev-parse packed)\n+\t\t\tEOF\n \n-test_expect_success 'removing a MIDX clears stale bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\t\ttest_commit base &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\t\tgrep \"doesn.t have full closure\" err &&\n+\t\t\ttest_path_is_missing $midx\n+\t\t)\n+\t'\n \n-\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n-\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm $midx &&\n+\tmidx_bitmap_partial_tests\n \n-\t\t# Then write a new MIDX.\n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\ttest_expect_success 'removing a MIDX clears stale bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\ttest_commit base &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm $midx &&\n+\n+\t\t\t# Then write a new MIDX.\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest_path_is_missing $stale_bitmap\n+\t\t)\n+\t'\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_missing $stale_bitmap\n-\t)\n-'\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\t\tbitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n \n-\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n-\t\t\tmulti-pack-index write --bitmap &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\ttest_expect_success 'writing a bitmap with --refs-snapshot' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'writing a bitmap with --refs-snapshot' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\ttest_commit two &&\n \n-\t\ttest_commit one &&\n-\t\ttest_commit two &&\n+\t\t\tgit rev-parse one >snapshot &&\n \n-\t\tgit rev-parse one >snapshot &&\n+\t\t\tgit repack -ad &&\n \n-\t\tgit repack -ad &&\n+\t\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t\t# bitmaps).\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\t# First, write a MIDX which see both refs/tags/one and\n-\t\t# refs/tags/two (causing both of those commits to receive\n-\t\t# bitmaps).\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\t# Then again, but with a refs snapshot which only sees\n+\t\t\t# refs/tags/one.\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n \n-\t\t# Then again, but with a refs snapshot which only sees\n-\t\t# refs/tags/one.\n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\t! grep \"$(git rev-parse two)\" bitmaps\n-\t)\n-'\n+\ttest_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\t(\n+\t\t\t\tgrep -vf before commits.raw &&\n+\t\t\t\t# mark missing commits as preferred\n+\t\t\t\tsed \"s/^/+/\" before\n+\t\t\t) >snapshot &&\n \n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n+\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'hash-cache values are propagated from pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n \t\t(\n-\t\t\tgrep -vf before commits.raw &&\n-\t\t\t# mark missing commits as preferred\n-\t\t\tsed \"s/^/+/\" before\n-\t\t) >snapshot &&\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\ttest_commit base &&\n+\t\t\ttest_commit base2 &&\n+\t\t\tgit repack -adb &&\n \n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\ttest-tool bitmap dump-hashes >pack.raw &&\n+\t\t\ttest_file_not_empty pack.raw &&\n+\t\t\tsort pack.raw >pack.hashes &&\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest-tool bitmap dump-hashes >midx.raw &&\n+\t\t\tsort midx.raw >midx.hashes &&\n \n-\t\ttest_commit base &&\n-\t\ttest_commit base2 &&\n-\t\tgit repack -adb &&\n+\t\t\t# ensure that every namehash in the pack bitmap can be found in\n+\t\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n+\t\t\t# unique to the pack bitmap).\n+\t\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n+\t\t\ttest_must_be_empty dropped.hashes\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap dump-hashes >pack.raw &&\n-\t\ttest_file_not_empty pack.raw &&\n-\t\tsort pack.raw >pack.hashes &&\n+\ttest_expect_success 'no .bitmap is written without any objects' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n+\t\t\tpack-$empty.idx\n+\t\t\tEOF\n \n-\t\ttest-tool bitmap dump-hashes >midx.raw &&\n-\t\tsort midx.raw >midx.hashes &&\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\terr &&\n \n-\t\t# ensure that every namehash in the pack bitmap can be found in\n-\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n-\t\t# unique to the pack bitmap).\n-\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n-\t\ttest_must_be_empty dropped.hashes\n-\t)\n-'\n+\t\t\tgrep \"bitmap without any objects\" err &&\n \n-test_expect_success 'no .bitmap is written without any objects' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'graceful fallback when missing reverse index' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n-\t\tpack-$empty.idx\n-\t\tEOF\n+\t\t\ttest_commit base &&\n \n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\terr &&\n+\t\t\t# write a pack and MIDX bitmap containing base\n+\t\t\tgit repack -adb &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\tgrep \"bitmap without any objects\" err &&\n+\t\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n+\t\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\t! grep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n-\t)\n-'\n+test_midx_bitmap_cases\n+\n+test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'graceful fallback when missing reverse index' '\n+test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n \t(\n \t\tcd repo &&\n-\n \t\ttest_commit base &&\n-\n-\t\t# write a pack and MIDX bitmap containing base\n-\t\tgit repack -adb &&\n-\t\tgit multi-pack-index write --bitmap &&\n-\n-\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n-\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\t! grep \"ignoring extra bitmap file\" err\n+\t\tgit config pack.writeBitmapLookupTable true &&\n+\t\tgit repack -ad &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n \t)\n '\n \ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex d30ba632c87..5ed16a820d1 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -17,7 +17,27 @@ GIT_TEST_MIDX_READ_RIDX=0\n export GIT_TEST_MIDX_WRITE_REV\n export GIT_TEST_MIDX_READ_RIDX\n \n-midx_bitmap_core rev\n-midx_bitmap_partial_tests rev\n+test_midx_bitmap_rev () {\n+ writeLookupTable=false\n+\n+ \tfor i in \"$@\"\n+ \tdo\n+ \t\tcase $i in\n+ \t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+ \t\tesac\n+ \tdone\n+\n+ test_expect_success 'setup bitmap config' '\n+ rm -rf * .git &&\n+ git init &&\n+ git config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+ '\n+\n+ midx_bitmap_core rev\n+ midx_bitmap_partial_tests rev\n+ }\n+\n+ test_midx_bitmap_rev\n+ test_midx_bitmap_rev \"pack.writeBitmapLookupTable\"\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"459945","messageId":"Yt82kskifigs9kAf@nand.local","threadId":"58038","inReplyTo":"CAPOJW5yH=Xywqos2tPS4Cn7dAdDqymPVbb6tn_XoAz0ofsACAA@mail.gmail.com","subject":"Re: [PATCH v3 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-26T00:34:26Z","receivedAt":"2022-07-26T00:34:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Jul 16, 2022 at 05:20:57PM +0530, Abhradeep Chakraborty wrote:\n> I think the comment I added is not that good. The following might be better -\n>\n> At the end of this sort table[j] = i means that the i'th\n> bitmap corresponds to j'th bitmapped commit (among the selected commits)\n> in lex order of OIDs.\n\nMakes sense, I think that version of the comment is more helpful. I\nappreciate your attention to detail on getting these things right!\n\nThanks,\nTaylor\n"},{"id":"459946","messageId":"Yt85P8p3ePW5WWWc@nand.local","threadId":"58038","inReplyTo":"CAN0heSoA=wv4syJ3VOe92QPpjPHyqUPJ8+Pv+mbB0-TiiieVmw@mail.gmail.com","subject":"Re: [PATCH v3 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-26T00:45:51Z","receivedAt":"2022-07-26T00:46:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Martin,\n\n> > > > The comment you added is definitely helpful, but I still think that this\n> > > > line is a little magical. `*va` isn't really a pointer to a `uint32_t`,\n> > > > but a pointer to the start of a triplet, which just *happens* to have a\n> > > > 4-byte integer at the beginning of it.\n>\n> Yeah, this all looks quite magical with the casting, and with the\n> asymmetric handling of `va` and `vb`.\n\nYeah, this was my main point (which I didn't intend to create as much of\na digression with as I appear to have!).\n\nThe handling here is all correct, but what I was saying was that even\nthough we're treating `*vb` as a pointer to a `uint32_t`, reading vb[1]\nis bogus, since there isn't another 32-bit value there.\n\nSo I was saying that you *could* initialize a triplet struct, assign its\nfields appropriately, and then compare `*va` to `triplet->foo`. But I\nthink setting up a struct to only bother reading the first field is\nprobably wasteful, hence my suggestion for a clarifying comment.\n\n> > > Are you sure about this? As far as I know, the first parameter of such\n> > > comparing functions is always a pointer to the given key that we need\n> > > to search for and the second parameter points to each element of an\n> > > array.\n>\n> Yes, that matches my understanding and the man-page for bsearch(3):\n>\n> \"The compar routine is expected to have two arguments which point to\n> the key object and to an array member, in that order, [...]\"\n>\n> I think it would help to make this something like\n>\n> static int triplet_cmp(const void *key, const void *array_item)\n>\n> to really highlight this asymmetric nature of this function, or to make\n> clear how the values flow through our call-chain through something like\n>\n> static int triplet_cmp(const void *commit_pos, const void *table_entry)\n\nYeah, that makes sense to me. I'm not too attached to either name, both\nseem OK to me.\n\nThanks,\nTaylor\n"},{"id":"459947","messageId":"Yt8650eWLfm5VlLe@nand.local","threadId":"58038","inReplyTo":"a913e6a2cb36d8ec7900b60820d8ab3c35f60164.1658342304.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-26T00:52:55Z","receivedAt":"2022-07-26T00:53:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 20, 2022 at 06:38:20PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> The bitmap lookup table extension was documented by an earlier\n> change, but Git does not yet know how to write that extension.\n>\n> Teach Git to write bitmap lookup table extension. The table contains\n> the list of `N` ` triplets. These\n> triplets are sorted according to their commit pos (ascending order).\n> The meaning of each data in the i'th triplet is given below:\n>\n> - commit_pos stores commit position (in the pack-index or midx).\n> It is a 4 byte network byte order unsigned integer.\n>\n> - offset is the position (in the bitmap file) from which that\n> commit's bitmap can be read.\n>\n> - xor_row is the position of the triplet in the lookup table\n> whose bitmap is used to compress this bitmap, or `0xffffffff`\n> if no such bitmap exists.\n>\n> Mentored-by: Taylor Blau \n> Co-mentored-by: Kaartic Sivaraam \n> Co-authored-by: Taylor Blau \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> pack-bitmap-write.c | 112 ++++++++++++++++++++++++++++++++++++++++----\n> pack-bitmap.h | 5 +-\n> 2 files changed, 107 insertions(+), 10 deletions(-)\n>\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index c43375bd344..9843790cb60 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -650,20 +650,19 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n>\n> static void write_selected_commits_v1(struct hashfile *f,\n> \t\t\t\t struct pack_idx_entry **index,\n> -\t\t\t\t uint32_t index_nr)\n> +\t\t\t\t uint32_t index_nr,\n> +\t\t\t\t off_t *offsets,\n> +\t\t\t\t uint32_t *commit_positions)\n> {\n> \tint i;\n>\n> \tfor (i = 0; i < writer.selected_nr; ++i) {\n> \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n>\n> -\t\tint commit_pos =\n> -\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n> +\t\tif (offsets)\n> +\t\t\toffsets[i] = hashfile_total(f);\n>\n> -\t\tif (commit_pos < 0)\n> -\t\t\tBUG(\"trying to write commit not in index\");\n> -\n> -\t\thashwrite_be32(f, commit_pos);\n> +\t\thashwrite_be32(f, commit_positions[i]);\n\nI wonder if it would make this patch a little more readable to construct\nand use the commit_positions array as a single preparatory step before\nthis commit.\n\nWhat do you think?\n\n> +static void write_lookup_table(struct hashfile *f,\n> +\t\t\t struct pack_idx_entry **index,\n> +\t\t\t uint32_t index_nr,\n> +\t\t\t off_t *offsets,\n> +\t\t\t uint32_t *commit_positions)\n> +{\n> +\tuint32_t i;\n> +\tuint32_t *table, *table_inv;\n> +\n> +\tALLOC_ARRAY(table, writer.selected_nr);\n> +\tALLOC_ARRAY(table_inv, writer.selected_nr);\n> +\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable[i] = i;\n> +\n> +\t/*\n> +\t * At the end of this sort table[j] = i means that the i'th\n> +\t * bitmap corresponds to j'th bitmapped commit (among the selected\n> +\t * commits) in lex order of OIDs.\n> +\t */\n> +\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n> +\n> +\t/* table_inv helps us discover that relationship (i'th bitmap\n> +\t * to j'th commit by j = table_inv[i])\n> +\t */\n> +\tfor (i = 0; i < writer.selected_nr; i++)\n> +\t\ttable_inv[table[i]] = i;\n> +\n> +\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n> +\tfor (i = 0; i < writer.selected_nr; i++) {\n> +\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n> +\t\tuint32_t xor_offset = selected->xor_offset;\n> +\t\tuint32_t xor_row;\n> +\n> +\t\tif (xor_offset) {\n> +\t\t\t/*\n> +\t\t\t * xor_index stores the index (in the bitmap entries)\n> +\t\t\t * of the corresponding xor bitmap. But we need to convert\n> +\t\t\t * this index into lookup table's index. So, table_inv[xor_index]\n> +\t\t\t * gives us the index position w.r.t. the lookup table.\n> +\t\t\t *\n> +\t\t\t * If \"k = table[i] - xor_offset\" then the xor base is the k'th\n> +\t\t\t * bitmap. `table_inv[k]` gives us the position of that bitmap\n> +\t\t\t * in the lookup table.\n> +\t\t\t */\n> +\t\t\tuint32_t xor_index = table[i] - xor_offset;\n> +\t\t\txor_row = table_inv[xor_index];\n> +\t\t} else {\n> +\t\t\txor_row = 0xffffffff;\n> +\t\t}\n> +\n> +\t\thashwrite_be32(f, commit_positions[table[i]]);\n> +\t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n> +\t\thashwrite_be32(f, xor_row);\n> +\t}\n> +\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n> +\n> +\tfree(table);\n> +\tfree(table_inv);\n> +}\n\n\n> @@ -715,7 +791,25 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n> \tdump_bitmap(f, writer.trees);\n> \tdump_bitmap(f, writer.blobs);\n> \tdump_bitmap(f, writer.tags);\n> -\twrite_selected_commits_v1(f, index, index_nr);\n> +\n> +\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n> +\tfor (uint32_t i = 0; i < writer.selected_nr; ++i) {\n\nNit; we don't typically write for-loop expressions with variable\ndeclarations inside of them. Make sure to declare i outside of the loop,\nand then this becomes:\n\n for (i = 0; i < writer.selected_nr; i++)\n\n(also, we typically use the postfix ++ operator, that is \"i++\" instead\nof \"++i\" unless there is a reason to prefer the latter over the former).\n\nThanks,\nTaylor\n"},{"id":"459949","messageId":"Yt8/zl/uOLRBRS4h@nand.local","threadId":"58038","inReplyTo":"6918f0860adbea1156fb7a9f1a5aedd211872481.1658342304.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-26T01:13:50Z","receivedAt":"2022-07-26T01:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 20, 2022 at 06:38:22PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n\n> @@ -557,13 +575,238 @@ struct include_data {\n> \tstruct bitmap *seen;\n> };\n>\n> +struct bitmap_lookup_table_triplet {\n> +\tuint32_t commit_pos;\n> +\tuint64_t offset;\n> +\tuint32_t xor_row;\n> +};\n> +\n> +struct bitmap_lookup_table_xor_item {\n> +\tstruct object_id oid;\n> +\tuint64_t offset;\n> +};\n> +\n> +/*\n> + * Given a `triplet` struct pointer and pointer `p`, this\n> + * function reads the triplet beginning at `p` into the struct.\n> + * Note that this function assumes that there is enough memory\n> + * left for filling the `triplet` struct from `p`.\n> + */\n> +static int lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n> +\t\t\t\t\t const unsigned char *p)\n> +{\n> +\tif (!triplet)\n> +\t\treturn -1;\n> +\n> +\ttriplet->commit_pos = get_be32(p);\n> +\tp += sizeof(uint32_t);\n> +\ttriplet->offset = get_be64(p);\n> +\tp += sizeof(uint64_t);\n> +\ttriplet->xor_row = get_be32(p);\n> +\treturn 0;\n\nJust noticing this now, but I wonder if we could avoid incrementing `p`\nhere and instead write something like:\n\n triplet->commit_pos = get_be32(p);\n triplet->offset = get_be64(p + sizeof(uint32_t));\n triplet->xor_row = get_be64(p + sizeof(uint64_t) + sizeof(uint32_t));\n\nI don't have a strong feeling about it, though, it just seems to read a\nlittle more directly to me and avoid modifying a variable that is only\ngoing to live as long as the function executes (p).\n\n> +/*\n> + * This function gets the raw triplet from `row`'th row in the\n> + * lookup table and fills that data to the `triplet`.\n> + */\n> +static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n> +\t\t\t\t uint32_t pos,\n> +\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n> +{\n> +\tunsigned char *p = NULL;\n> +\tif (pos >= bitmap_git->entry_count)\n> +\t\treturn error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n> +\n> +\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n> +\n> +\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n> +}\n\nVery nice. This cleans things up nicely by being able to call\nlookup_table_get_triplet_by_pointer().\n\nSince these are static functions, it doesn't really matter whether or\nnot they are prefixed with 'bitmap_', since they won't be visible\noutside of pack-bitmap.c's compilation unit. But it may be nice to\nprefix them with 'bitmap_' just to make it extra clear that these are\ninternal functions meant to be used within the bitmap machinery only.\n\n> +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t struct commit *commit)\n> +{\n> +\tuint32_t commit_pos, xor_row;\n> +\tuint64_t offset;\n> +\tint flags, found;\n> +\tstruct bitmap_lookup_table_triplet triplet;\n> +\tstruct object_id *oid = &commit->object.oid;\n> +\tstruct ewah_bitmap *bitmap;\n> +\tstruct stored_bitmap *xor_bitmap = NULL;\n> +\tconst int bitmap_header_size = 6;\n> +\tstatic struct bitmap_lookup_table_xor_item *xor_items = NULL;\n> +\tstatic size_t xor_items_nr = 0, xor_items_alloc = 0;\n\nI had to double check, but xor_items_alloc inherits the same storage\nclass at xor_items_nr, so they are both static size_t's.\n\n> +\tstatic int is_corrupt = 0;\n> +\n> +\tif (is_corrupt)\n> +\t\treturn NULL;\n\nWhat is the purpose of this conditional? We don't modify `is_corrupt`\nbefore reading it here, so this should be dead code, unless I'm missing\nsomething.\n\n> +\tfound = bsearch_pos(bitmap_git, oid, &commit_pos);\n> +\n> +\tif (!found)\n> +\t\treturn NULL;\n\nFWIW, we could eliminate this variable from a particularly long list of\nstack variables above by just writing:\n\n if (!bsearch_pos(bitmap_git, oid, &commit_pos))\n return NULL;\n\nand I think that would be OK.\n\n> +\tif (bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n> +\t\treturn NULL;\n> +\n> +\txor_items_nr = 0;\n\nThis initialization *is* necessary, since the xor_items_nr and\nxor_items_alloc variable are statically allocated.\n\n> +\toffset = triplet.offset;\n> +\txor_row = triplet.xor_row;\n> +\n> +\tif (xor_row != 0xffffffff) {\n\nIs this outer conditional needed? I don't think it is. If xor_row is\n0xffffffff, then the while loop below won't be entered, and\nxor_items_nr will be zero, meaning that the second while loop will also\nbe skipped.\n\nSo I think we can just as easily get rid of this outermost if-statement\nand de-dent the main part of this function's body.\n\n> +\t\tint xor_flags;\n> +\t\tkhiter_t hash_pos;\n> +\t\tstruct bitmap_lookup_table_xor_item *xor_item;\n> +\n> +\t\twhile (xor_row != 0xffffffff) {\n> +\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n> +\n> +\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n> +\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n> +\t\t\t\tgoto corrupt;\n> +\t\t\t}\n> +\n> +\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n> +\t\t\t\tgoto corrupt;\n> +\n> +\t\t\txor_item = &xor_items[xor_items_nr];\n> +\t\t\txor_item->offset = triplet.offset;\n> +\n> +\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n> +\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n> +\t\t\t\t\ttriplet.commit_pos);\n> +\t\t\t\tgoto corrupt;\n> +\t\t\t}\n> +\n> +\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n> +\n> +\t\t\t/*\n> +\t\t\t * If desired bitmap is already stored, we don't need\n> +\t\t\t * to iterate further. Because we know that bitmaps\n> +\t\t\t * that are needed to be parsed to parse this bitmap\n> +\t\t\t * has already been stored. So, assign this stored bitmap\n> +\t\t\t * to the xor_bitmap.\n> +\t\t\t */\n> +\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n> +\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n> +\t\t\t\tbreak;\n> +\t\t\txor_items_nr++;\n> +\t\t\txor_row = triplet.xor_row;\n> +\t\t}\n> +\n> +\t\twhile (xor_items_nr) {\n> +\t\t\txor_item = &xor_items[xor_items_nr - 1];\n> +\t\t\tbitmap_git->map_pos = xor_item->offset;\n> +\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n> +\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n> +\t\t\t\t\toid_to_hex(&xor_item->oid));\n> +\t\t\t\tgoto corrupt;\n> +\t\t\t}\n> +\n> +\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n\nCould we write:\n\n bitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t)\n\nor similar? Also, a clarifying comment would help explain why we are\nadvancing, what we're skipping past, and why it's OK.\n\n> +\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\t\t\tbitmap = read_bitmap_1(bitmap_git);\n> +\n> +\t\t\tif (!bitmap)\n> +\t\t\t\tgoto corrupt;\n> +\n> +\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n> +\t\t\txor_items_nr--;\n> +\t\t}\n> +\t}\n> +\n> +\tbitmap_git->map_pos = offset;\n> +\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n> +\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n> +\t\t\toid_to_hex(oid));\n> +\t\tgoto corrupt;\n> +\t}\n> +\n> +\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n> +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n> +\tbitmap = read_bitmap_1(bitmap_git);\n> +\n> +\tif (!bitmap)\n> +\t\tgoto corrupt;\n> +\n> +\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n> +\n> +corrupt:\n> +\tfree(xor_items);\n> +\tis_corrupt = 1;\n> +\treturn NULL;\n> +}\n\nThanks,\nTaylor\n"},{"id":"459950","messageId":"Yt9A4Lh5MzHigeVe@nand.local","threadId":"58038","inReplyTo":"e7ef420f321b3936185b2729460b1c28f5384438.1658342304.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-26T01:18:24Z","receivedAt":"2022-07-26T01:18:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 20, 2022 at 06:38:23PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> From: Abhradeep Chakraborty \n>\n> Enable `pack.writeReverseIndex` before running pack-bitmap related\n> performance tests.\n>\n> The performance difference with `pack.writeReverseIndex` enabled and\n> with disabled are given below -\n\nThanks; this order of changes in the t/perf suite makes sense to me. One\nnote, this sort of change where we're comparing all of the tests in a\nsingle t/perf file against themselves before and after some change, it\nis helpful to do (in t/perf)\n\n ./run HEAD . p5310-pack-bitmaps.sh\n\nwhich compares HEAD to what's in the current tree. You'll get the\nresults side-by-side, which makes them a little easier to scan. You can\nalso aggregate results together from multiple runs with the\nt/perf/aggregate.perl script.\n\nOne gotcha (that has often bitten me in the past) is that when running\nthe perf suite with `.` as your build target, it uses whatever git\nbinary is sitting in your tree. So make sure that it is both (a)\nup-to-date, ie., that it is the result of compiling what's currently in\nyour tree, and (b) that it is compiled with the same settings as what\nyou built HEAD with.\n\nI have often scratched my head at why the result of running some perf\nsuite on '.' seems much slower than it should be, only to realize that\nthe \"git\" binary sitting in my tree was built with -O0 or something.\n\nThanks,\nTaylor\n"},{"id":"459962","messageId":"220726.86bktcny14.gmgdl@evledraar.gmail.com","threadId":"58038","inReplyTo":"Yt9A4Lh5MzHigeVe@nand.local","subject":"Re: [PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-26T07:15:15Z","receivedAt":"2022-07-26T07:17:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 25 2022, Taylor Blau wrote:\n\n> On Wed, Jul 20, 2022 at 06:38:23PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n>> From: Abhradeep Chakraborty \n>>\n>> Enable `pack.writeReverseIndex` before running pack-bitmap related\n>> performance tests.\n>>\n>> The performance difference with `pack.writeReverseIndex` enabled and\n>> with disabled are given below -\n>\n> Thanks; this order of changes in the t/perf suite makes sense to me. One\n> note, this sort of change where we're comparing all of the tests in a\n> single t/perf file against themselves before and after some change, it\n> is helpful to do (in t/perf)\n>\n> ./run HEAD . p5310-pack-bitmaps.sh\n>\n> which compares HEAD to what's in the current tree. You'll get the\n> results side-by-side, which makes them a little easier to scan. You can\n> also aggregate results together from multiple runs with the\n> t/perf/aggregate.perl script.\n>\n> One gotcha (that has often bitten me in the past) is that when running\n> the perf suite with `.` as your build target, it uses whatever git\n> binary is sitting in your tree. So make sure that it is both (a)\n> up-to-date, ie., that it is the result of compiling what's currently in\n> your tree, and (b) that it is compiled with the same settings as what\n> you built HEAD with.\n>\n> I have often scratched my head at why the result of running some perf\n> suite on '.' seems much slower than it should be, only to realize that\n> the \"git\" binary sitting in my tree was built with -O0 or something.\n\nRather than comparing HEAD to your current tree it's generally better\nto do something like:\n\n\tGIT_PERF_MAKE_OPTS='-j3' ./run HEAD~ HEAD [...]\n"},{"id":"459969","messageId":"733f2432-00ab-f0c2-269f-90af02b2105c@github.com","threadId":"58038","inReplyTo":"220726.86bktcny14.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-26T13:32:44Z","receivedAt":"2022-07-26T13:32:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/26/2022 3:15 AM, Ævar Arnfjörð Bjarmason wrote:\n> Rather than comparing HEAD to your current tree it's generally better\n> to do something like:\n> \n> \tGIT_PERF_MAKE_OPTS='-j3' ./run HEAD~ HEAD [...]\n\nUsing the 'run' script fixes the perf test in the worktree and tests\ndifferent versions of the 'git' executable.\n\nThat doesn't work when the change is in the performance test itself.\n\nThanks,\n-Stolee\n"},{"id":"459971","messageId":"220726.8635eonfjp.gmgdl@evledraar.gmail.com","threadId":"58038","inReplyTo":"733f2432-00ab-f0c2-269f-90af02b2105c@github.com","subject":"Re: [PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-26T13:54:04Z","receivedAt":"2022-07-26T13:57:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 26 2022, Derrick Stolee wrote:\n\n> On 7/26/2022 3:15 AM, Ævar Arnfjörð Bjarmason wrote:\n>> Rather than comparing HEAD to your current tree it's generally better\n>> to do something like:\n>> \n>> \tGIT_PERF_MAKE_OPTS='-j3' ./run HEAD~ HEAD [...]\n>\n> Using the 'run' script fixes the perf test in the worktree and tests\n> different versions of the 'git' executable.\n>\n> That doesn't work when the change is in the performance test itself.\n\nThanks, I'm clearly wrong about that. I didn't look enough at the\ncontext.\n\nBut then we're losing the perf test coverage for the case where we don't\nhave the *.rev files. Isn't it better to run both with & without *.rev,\nperhaps by splitting up the test file? We could make it a function in\nperf/lib-bitmap.sh that we call both with & without the wanted *.rev\nrepack config.\n\nI suspect that's also subtly broken, in that t/perf assumes that it can\nre-use the repo for a given , but this is modifying that repo, so\nif you run e.g. test Y after this Y, that Y will unexpectedly get a\nrepack'd repo ...\n\nBut we could just start the test with a git clone . \"$TEST_NAME\" or\nwhatever, then repack that with whatever options we want...\n\n"},{"id":"459992","messageId":"CAPOJW5zS2yhbmmfZXR3yz=CqzgccTQHEV5A2bjj52ZLcQCWfsQ@mail.gmail.com","threadId":"58038","inReplyTo":"220726.8635eonfjp.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-26T18:17:06Z","receivedAt":"2022-07-26T18:17:22Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Tue, Jul 26, 2022 at 7:26 PM Ævar Arnfjörð Bjarmason\n wrote:\n> But then we're losing the perf test coverage for the case where we don't\n> have the *.rev files. Isn't it better to run both with & without *.rev,\n> perhaps by splitting up the test file? We could make it a function in\n> perf/lib-bitmap.sh that we call both with & without the wanted *.rev\n> repack config.\n\nOk.\n\n> I suspect that's also subtly broken, in that t/perf assumes that it can\n> re-use the repo for a given , but this is modifying that repo, so\n> if you run e.g. test Y after this Y, that Y will unexpectedly get a\n> repack'd repo ...\n\nThanks Ævar! This is the problem that I informed Taylor off-list. Will\nupdate it.\n\n> But we could just start the test with a git clone . \"$TEST_NAME\" or\n> whatever, then repack that with whatever options we want...\n\nThanks :)\n"},{"id":"459994","messageId":"CAPOJW5zYngCgab80yFsBrFg0ENiWcGMn71R=QH1k=kfZ+aPkaQ@mail.gmail.com","threadId":"58038","inReplyTo":"Yt8650eWLfm5VlLe@nand.local","subject":"Re: [PATCH v5 2/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-26T18:22:54Z","receivedAt":"2022-07-26T18:23:13Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Tue, Jul 26, 2022 at 6:22 AM Taylor Blau wrote:\n> I wonder if it would make this patch a little more readable to construct\n> and use the commit_positions array as a single preparatory step before\n> this commit.\n>\n> What do you think?\n\nYeah, sure! I have no problem with that.\n> > +\n> > + ALLOC_ARRAY(commit_positions, writer.selected_nr);\n> > + for (uint32_t i = 0; i < writer.selected_nr; ++i) {\n>\n> Nit; we don't typically write for-loop expressions with variable\n> declarations inside of them. Make sure to declare i outside of the loop,\n> and then this becomes:\n>\n> for (i = 0; i < writer.selected_nr; i++)\n>\n> (also, we typically use the postfix ++ operator, that is \"i++\" instead\n> of \"++i\" unless there is a reason to prefer the latter over the former).\n\nGot it. Thanks :)\n"},{"id":"459996","messageId":"CAPOJW5y74tj89OxJg0zSY90J2K5TG57ztGQq_8_LQ_DDxMQBGA@mail.gmail.com","threadId":"58038","inReplyTo":"Yt8/zl/uOLRBRS4h@nand.local","subject":"Re: [PATCH v5 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-26T18:56:44Z","receivedAt":"2022-07-26T18:57:02Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Tue, Jul 26, 2022 at 6:43 AM Taylor Blau wrote:\n> Just noticing this now, but I wonder if we could avoid incrementing `p`\n> here and instead write something like:\n>\n> triplet->commit_pos = get_be32(p);\n> triplet->offset = get_be64(p + sizeof(uint32_t));\n> triplet->xor_row = get_be64(p + sizeof(uint64_t) + sizeof(uint32_t));\n>\n> I don't have a strong feeling about it, though, it just seems to read a\n> little more directly to me and avoid modifying a variable that is only\n> going to live as long as the function executes (p).\n\nOk, will update.\n\n> > +/*\n> > + * This function gets the raw triplet from `row`'th row in the\n> > + * lookup table and fills that data to the `triplet`.\n> > + */\n> > +static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n> > + uint32_t pos,\n> > + struct bitmap_lookup_table_triplet *triplet)\n> > +{\n> > + unsigned char *p = NULL;\n> > + if (pos >= bitmap_git->entry_count)\n> > + return error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n> > +\n> > + p = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n> > +\n> > + return lookup_table_get_triplet_by_pointer(triplet, p);\n> > +}\n>\n> Very nice. This cleans things up nicely by being able to call\n> lookup_table_get_triplet_by_pointer().\n>\n> Since these are static functions, it doesn't really matter whether or\n> not they are prefixed with 'bitmap_', since they won't be visible\n> outside of pack-bitmap.c's compilation unit. But it may be nice to\n> prefix them with 'bitmap_' just to make it extra clear that these are\n> internal functions meant to be used within the bitmap machinery only.\n\nYeah, sure!\n\n> > + static int is_corrupt = 0;\n> > +\n> > + if (is_corrupt)\n> > + return NULL;\n>\n> What is the purpose of this conditional? We don't modify `is_corrupt`\n> before reading it here, so this should be dead code, unless I'm missing\n> something.\n\nMy intention behind this code was -\nInitially `is_corrupt` is 0, so the above code will not execute for\nthe first `lazy_bitmap...()` call. Now, for some reason, if we get to\nknow that the `.bitmap` file is corrupted, the function will `goto\ncorrupt` and `is_corrupt` will be set to 1.\nAs `is_corrupt` is a static variable, its value will be preserved. So,\nwhenr we call `lazy_bitmap...()` function for the second time (or\nthird time etc.; i.e. `bitmap_for_commit` under a for loop), we\ninstantly know that `.bitmap` file is corrupt (by seeing `is_corrupt`)\nand we will not do all the computations any more.\n\n>\n> > + offset = triplet.offset;\n> > + xor_row = triplet.xor_row;\n> > +\n> > + if (xor_row != 0xffffffff) {\n>\n> Is this outer conditional needed? I don't think it is. If xor_row is\n> 0xffffffff, then the while loop below won't be entered, and\n> xor_items_nr will be zero, meaning that the second while loop will also\n> be skipped.\n\nYes, you're right - it is not needed. But it guarantees that all the\ncode inside its braces will be run only if has a `xor offset` causing\nthe allocation of `xor_items` array as lazy as possible.\n\nShould I remove it?\n\n> > + bitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n>\n> Could we write:\n>\n> bitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t)\n\nSure!\n\nThanks :)\n"},{"id":"459998","messageId":"CAPig+cTEUsebnsAQxunnonpWodUC_9kPAhy8uHor9HNn9-AZEA@mail.gmail.com","threadId":"58038","inReplyTo":"Yt8/zl/uOLRBRS4h@nand.local","subject":"Re: [PATCH v5 4/6] pack-bitmap: prepare to read lookup table extension","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-07-26T19:36:37Z","receivedAt":"2022-07-26T19:36:51Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Jul 25, 2022 at 9:21 PM Taylor Blau wrote:\n> On Wed, Jul 20, 2022 at 06:38:22PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> > + triplet->commit_pos = get_be32(p);\n> > + p += sizeof(uint32_t);\n> > + triplet->offset = get_be64(p);\n> > + p += sizeof(uint64_t);\n> > + triplet->xor_row = get_be32(p);\n> > + return 0;\n>\n> Just noticing this now, but I wonder if we could avoid incrementing `p`\n> here and instead write something like:\n>\n> triplet->commit_pos = get_be32(p);\n> triplet->offset = get_be64(p + sizeof(uint32_t));\n> triplet->xor_row = get_be64(p + sizeof(uint64_t) + sizeof(uint32_t));\n>\n> I don't have a strong feeling about it, though, it just seems to read a\n> little more directly to me and avoid modifying a variable that is only\n> going to live as long as the function executes (p).\n\nWhile it may not matter much in this tiny function, the code in the\npatch sets a better precedent by conforming to a pattern which is more\nmaintainable in situations involving more pieces of data which need to\nbe decoded. It's also easier to reason about than the suggested\nreplacement since you don't have to spend extra cycles double-checking\nif it's adding the correct number of and correctly-sized offsets at\neach get_be*() invocation. IMHO, the way the patch already handles\nthis seems preferable.\n"},{"id":"460142","messageId":"p3r70610-8n52-s8q0-n641-onp4ps01330n@tzk.qr","threadId":"58038","inReplyTo":"59b465e5a7817c145172f25e73ad807c7ba67e84.1658342304.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-07-28T19:22:53Z","receivedAt":"2022-07-28T19:23:04Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Abhradeep,\n\nOn Wed, 20 Jul 2022, Abhradeep Chakraborty via GitGitGadget wrote:\n\n> From: Abhradeep Chakraborty \n>\n> Teach Git to provide a way for users to enable/disable bitmap lookup\n> table extension by providing a config option named 'writeBitmapLookupTable'.\n> Default is false.\n>\n> Also add test to verify writting of lookup table.\n>\n> Mentored-by: Taylor Blau \n> Co-Mentored-by: Kaartic Sivaraam \n> Co-Authored-by: Taylor Blau \n> Signed-off-by: Abhradeep Chakraborty \n> ---\n> Documentation/config/pack.txt | 7 +\n> builtin/multi-pack-index.c | 7 +\n> builtin/pack-objects.c | 8 +\n> midx.c | 3 +\n> midx.h | 1 +\n> t/t5310-pack-bitmaps.sh | 792 ++++++++++++++++--------------\n> t/t5311-pack-bitmaps-shallow.sh | 53 +-\n> t/t5326-multi-pack-bitmaps.sh | 421 +++++++++-------\n\nThat's quite a large a change, and unfortunately I pinpointed a flake to\nthis patch when running with GIT_TEST_DEFAULT_HASH=sha256. The symptom is\nthis:\n\n-- snip --\n[...]\n+ diff -u expect.normalized actual.normalized\n+ rm -f expect.normalized actual.normalized\nok 317 - enumerate --objects (full bitmap, other)\n\nexpecting success of 5326.318 'bitmap --objects handles non-commit objects (full bitmap, other)':\n git rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n grep $blob actual\n\n+ git rev-list --objects --use-bitmap-index other tagged-blob\n+ grep bff4ed5e839bd73e821f78b45a7fa34208aa85596535ec8e9ac5eab477ca6f81 actual\nbff4ed5e839bd73e821f78b45a7fa34208aa85596535ec8e9ac5eab477ca6f81\nok 318 - bitmap --objects handles non-commit objects (full bitmap, other)\n\nexpecting success of 5326.319 'clone from bitmapped repository':\n rm -fr clone.git &&\n git clone --no-local --bare . clone.git &&\n git rev-parse HEAD >expect &&\n git --git-dir=clone.git rev-parse HEAD >actual &&\n test_cmp expect actual\n\n+ rm -fr clone.git\n+ git clone --no-local --bare . clone.git\nCloning into bare repository 'clone.git'...\nremote: Enumerating objects: 756, done.\nremote: Counting objects: 100% (754/754), done.\nremote: Compressing objects: 100% (281/281), done.\nremote: Total 756 (delta 245), reused 740 (delta 234), pack-reused 2\nReceiving objects: 100% (756/756), 77.50 KiB | 8.61 MiB/s, done.\nfatal: REF_DELTA at offset 221 already resolved (duplicate base 4d332072f161629ffe4652ecd3ce377ef88447bec73f05ab0f3515f98bd061cf?)\nfatal: fetch-pack: invalid index-pack output\nerror: last command exited with $?=128\nnot ok 319 - clone from bitmapped repository\n#\n# rm -fr clone.git &&\n# git clone --no-local --bare . clone.git &&\n# git rev-parse HEAD >expect &&\n# git --git-dir=clone.git rev-parse HEAD >actual &&\n# test_cmp expect actual\n#\n1..319\n-- snap --\n\nOn a hunch, I ran this through valgrind (took a while) but it did not\npoint out the problem.\n\nAgain, this is only with SHA-256 (and somewhat flaky), it passes every\ntime with SHA-1. Maybe you can reproduce on your side with that\ninformation?\n\nSadly, this patch is way too large for me to do a drive-by debugging\nsession, so I will have to leave it to you to investigate further.\n\nCiao,\nDscho\n\n> t/t5327-multi-pack-bitmaps-rev.sh | 24 +-\n> 9 files changed, 733 insertions(+), 583 deletions(-)\n>\n> diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\n> index ad7f73a1ead..b955ca572ec 100644\n> --- a/Documentation/config/pack.txt\n> +++ b/Documentation/config/pack.txt\n> @@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n> computed; instead, any namehashes stored in an existing bitmap are\n> permuted into their appropriate location when writing a new bitmap.\n>\n> +pack.writeBitmapLookupTable::\n> +\tWhen true, Git will include a \"lookup table\" section in the\n> +\tbitmap index (if one is written). This table is used to defer\n> +\tloading individual bitmaps as late as possible. This can be\n> +\tbeneficial in repositories that have relatively large bitmap\n> +\tindexes. Defaults to false.\n> +\n> pack.writeReverseIndex::\n> \tWhen true, git will write a corresponding .rev file (see:\n> \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 5edbb7fe86e..55402b46f41 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n> \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n> \t}\n>\n> +\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n> +\t\tif (git_config_bool(var, value))\n> +\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n> +\t\telse\n> +\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n> +\t}\n> +\n> \t/*\n> \t * We should never make a fall-back call to 'git_default_config', since\n> \t * this was already called in 'cmd_multi_pack_index()'.\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 39e28cfcafc..46e26774963 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\telse\n> \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n> \t}\n> +\n> +\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n> +\t\tif (git_config_bool(k, v))\n> +\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n> +\t\telse\n> +\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n> +\t}\n> +\n> \tif (!strcmp(k, \"pack.usebitmaps\")) {\n> \t\tuse_bitmap_index_default = git_config_bool(k, v);\n> \t\treturn 0;\n> diff --git a/midx.c b/midx.c\n> index 5f0dd386b02..9c26d04bfde 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1072,6 +1072,9 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n> \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n> \t\toptions |= BITMAP_OPT_HASH_CACHE;\n>\n> +\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n> +\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n> +\n> \tprepare_midx_packing_data(&pdata, ctx);\n>\n> \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\n> diff --git a/midx.h b/midx.h\n> index 22e8e53288e..5578cd7b835 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -47,6 +47,7 @@ struct multi_pack_index {\n> #define MIDX_WRITE_REV_INDEX (1 << 1)\n> #define MIDX_WRITE_BITMAP (1 << 2)\n> #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n> +#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n>\n> const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n> void get_midx_filename(struct strbuf *out, const char *object_dir);\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index f775fc1ce69..c0607172827 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -26,22 +26,413 @@ has_any () {\n> \tgrep -Ff \"$1\" \"$2\"\n> }\n>\n> -setup_bitmap_history\n> -\n> -test_expect_success 'setup writing bitmaps during repack' '\n> -\tgit config repack.writeBitmaps true\n> -'\n> -\n> -test_expect_success 'full repack creates bitmaps' '\n> -\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n> +test_bitmap_cases () {\n> +\twriteLookupTable=false\n> +\tfor i in \"$@\"\n> +\tdo\n> +\t\tcase \"$i\" in\n> +\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n> +\t\tesac\n> +\tdone\n> +\n> +\ttest_expect_success 'setup test repository' '\n> +\t\trm -fr * .git &&\n> +\t\tgit init &&\n> +\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n> +\t'\n> +\tsetup_bitmap_history\n> +\n> +\ttest_expect_success 'setup writing bitmaps during repack' '\n> +\t\tgit config repack.writeBitmaps true\n> +\t'\n> +\n> +\ttest_expect_success 'full repack creates bitmaps' '\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n> +\t\t\tgit repack -ad &&\n> +\t\tls .git/objects/pack/ | grep bitmap >output &&\n> +\t\ttest_line_count = 1 output &&\n> +\t\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n> +\t\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n> +\t'\n> +\n> +\tbasic_bitmap_tests\n> +\n> +\ttest_expect_success 'pack-objects respects --local (non-local loose)' '\n> +\t\tgit init --bare alt.git &&\n> +\t\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n> +\t\techo content1 >file1 &&\n> +\t\t# non-local loose object which is not present in bitmapped pack\n> +\t\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n> +\t\t# non-local loose object which is also present in bitmapped pack\n> +\t\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n> +\t\tgit add file1 &&\n> +\t\ttest_tick &&\n> +\t\tgit commit -m commit_file1 &&\n> +\t\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n> +\t\tgit index-pack 1.pack &&\n> +\t\tlist_packed_objects 1.idx >1.objects &&\n> +\t\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n> +\t\t! has_any nonlocal-loose 1.objects\n> +\t'\n> +\n> +\ttest_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n> +\t\techo content2 >file2 &&\n> +\t\tblob2=$(git hash-object -w file2) &&\n> +\t\tgit add file2 &&\n> +\t\ttest_tick &&\n> +\t\tgit commit -m commit_file2 &&\n> +\t\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n> +\t\tpack2=$(git pack-objects pack2 +\t\tmv pack2-$pack2.* .git/objects/pack/ &&\n> +\t\t>.git/objects/pack/pack2-$pack2.keep &&\n> +\t\trm $(objpath $blob2) &&\n> +\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n> +\t\tgit index-pack 2a.pack &&\n> +\t\tlist_packed_objects 2a.idx >2a.objects &&\n> +\t\t! has_any keepobjects 2a.objects\n> +\t'\n> +\n> +\ttest_expect_success 'pack-objects respects --local (non-local pack)' '\n> +\t\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n> +\t\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n> +\t\tgit index-pack 2b.pack &&\n> +\t\tlist_packed_objects 2b.idx >2b.objects &&\n> +\t\t! has_any keepobjects 2b.objects\n> +\t'\n> +\n> +\ttest_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n> +\t\tls .git/objects/pack/ | grep bitmap >output &&\n> +\t\ttest_line_count = 1 output &&\n> +\t\tpackbitmap=$(basename $(cat output) .bitmap) &&\n> +\t\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n> +\t\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n> +\t\t>.git/objects/pack/$packbitmap.keep &&\n> +\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n> +\t\tgit index-pack 3a.pack &&\n> +\t\tlist_packed_objects 3a.idx >3a.objects &&\n> +\t\t! has_any packbitmap.objects 3a.objects\n> +\t'\n> +\n> +\ttest_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n> +\t\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n> +\t\trm -f .git/objects/pack/multi-pack-index &&\n> +\t\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n> +\t\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n> +\t\tgit index-pack 3b.pack &&\n> +\t\tlist_packed_objects 3b.idx >3b.objects &&\n> +\t\t! has_any packbitmap.objects 3b.objects\n> +\t'\n> +\n> +\ttest_expect_success 'pack-objects to file can use bitmap' '\n> +\t\t# make sure we still have 1 bitmap index from previous tests\n> +\t\tls .git/objects/pack/ | grep bitmap >output &&\n> +\t\ttest_line_count = 1 output &&\n> +\t\t# verify equivalent packs are generated with/without using bitmap index\n> +\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa +\t\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb +\t\tlist_packed_objects packa-$packasha1.idx >packa.objects &&\n> +\t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n> +\t\ttest_cmp packa.objects packb.objects\n> +\t'\n> +\n> +\ttest_expect_success 'full repack, reusing previous bitmaps' '\n> \t\tgit repack -ad &&\n> -\tls .git/objects/pack/ | grep bitmap >output &&\n> -\ttest_line_count = 1 output &&\n> -\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n> -\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n> -'\n> +\t\tls .git/objects/pack/ | grep bitmap >output &&\n> +\t\ttest_line_count = 1 output\n> +\t'\n> +\n> +\ttest_expect_success 'fetch (full bitmap)' '\n> +\t\tgit --git-dir=clone.git fetch origin second:second &&\n> +\t\tgit rev-parse HEAD >expect &&\n> +\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n> +\t\ttest_cmp expect actual\n> +\t'\n> +\n> +\ttest_expect_success 'create objects for missing-HAVE tests' '\n> +\t\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n> +\t\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n> +\t\tparent=$(echo parent | git commit-tree $tree) &&\n> +\t\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n> +\t\tcat >revs <<-EOF\n> +\t\tHEAD\n> +\t\t^HEAD^\n> +\t\t^$commit\n> +\t\tEOF\n> +\t'\n> +\n> +\ttest_expect_success 'pack-objects respects --incremental' '\n> +\t\tcat >revs2 <<-EOF &&\n> +\t\tHEAD\n> +\t\t$commit\n> +\t\tEOF\n> +\t\tgit pack-objects --incremental --stdout --revs 4.pack &&\n> +\t\tgit index-pack 4.pack &&\n> +\t\tlist_packed_objects 4.idx >4.objects &&\n> +\t\ttest_line_count = 4 4.objects &&\n> +\t\tgit rev-list --objects $commit >revlist &&\n> +\t\tcut -d\" \" -f1 revlist |sort >objects &&\n> +\t\ttest_cmp 4.objects objects\n> +\t'\n> +\n> +\ttest_expect_success 'pack with missing blob' '\n> +\t\trm $(objpath $blob) &&\n> +\t\tgit pack-objects --stdout --revs /dev/null\n> +\t'\n> +\n> +\ttest_expect_success 'pack with missing tree' '\n> +\t\trm $(objpath $tree) &&\n> +\t\tgit pack-objects --stdout --revs /dev/null\n> +\t'\n> +\n> +\ttest_expect_success 'pack with missing parent' '\n> +\t\trm $(objpath $parent) &&\n> +\t\tgit pack-objects --stdout --revs /dev/null\n> +\t'\n> +\n> +\ttest_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n> +\t\tgit clone --bare . compat-jgit.git &&\n> +\t\t(\n> +\t\t\tcd compat-jgit.git &&\n> +\t\t\trm -f objects/pack/*.bitmap &&\n> +\t\t\tjgit gc &&\n> +\t\t\tgit rev-list --test-bitmap HEAD\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n> +\t\tgit clone --bare . compat-us.git &&\n> +\t\t(\n> +\t\t\tcd compat-us.git &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> +\t\t\tgit repack -adb &&\n> +\t\t\t# jgit gc will barf if it does not like our bitmaps\n> +\t\t\tjgit gc\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success 'splitting packs does not generate bogus bitmaps' '\n> +\t\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n> +\t\tgit add rand &&\n> +\t\tgit commit -m \"commit with big file\" &&\n> +\t\tgit -c pack.packSizeLimit=500k repack -adb &&\n> +\t\tgit init --bare no-bitmaps.git &&\n> +\t\tgit -C no-bitmaps.git fetch .. HEAD\n> +\t'\n> +\n> +\ttest_expect_success 'set up reusable pack' '\n> +\t\trm -f .git/objects/pack/*.keep &&\n> +\t\tgit repack -adb &&\n> +\t\treusable_pack () {\n> +\t\t\tgit for-each-ref --format=\"%(objectname)\" |\n> +\t\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n> +\t\t}\n> +\t'\n> +\n> +\ttest_expect_success 'pack reuse respects --honor-pack-keep' '\n> +\t\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n> +\t\tfor i in .git/objects/pack/*.pack\n> +\t\tdo\n> +\t\t\t>${i%.pack}.keep || return 1\n> +\t\tdone &&\n> +\t\treusable_pack --honor-pack-keep >empty.pack &&\n> +\t\tgit index-pack empty.pack &&\n> +\t\tgit show-index actual &&\n> +\t\ttest_must_be_empty actual\n> +\t'\n> +\n> +\ttest_expect_success 'pack reuse respects --local' '\n> +\t\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n> +\t\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n> +\t\treusable_pack --local >empty.pack &&\n> +\t\tgit index-pack empty.pack &&\n> +\t\tgit show-index actual &&\n> +\t\ttest_must_be_empty actual\n> +\t'\n> +\n> +\ttest_expect_success 'pack reuse respects --incremental' '\n> +\t\treusable_pack --incremental >empty.pack &&\n> +\t\tgit index-pack empty.pack &&\n> +\t\tgit show-index actual &&\n> +\t\ttest_must_be_empty actual\n> +\t'\n> +\n> +\ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n> +\t\ttest_config pack.writebitmaphashcache false &&\n> +\t\tgit repack -ad &&\n> +\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n> +\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n> +\t\ttest_when_finished \"rm -f $bitmap\" &&\n> +\t\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n> +\t\tmv -f $bitmap.tmp $bitmap &&\n> +\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n> +\t\ttest_cmp expect actual &&\n> +\t\ttest_i18ngrep corrupt.ewah.bitmap stderr\n> +\t'\n> +\n> +\ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n> +\t\tgit repack -ad &&\n> +\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n> +\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n> +\t\ttest_when_finished \"rm -f $bitmap\" &&\n> +\t\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n> +\t\tmv -f $bitmap.tmp $bitmap &&\n> +\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n> +\t\ttest_cmp expect actual &&\n> +\t\ttest_i18ngrep corrupted.bitmap.index stderr\n> +\t'\n> +\n> +\t# Create a state of history with these properties:\n> +\t#\n> +\t# - refs that allow a client to fetch some new history, while sharing some old\n> +\t# history with the server; we use branches delta-reuse-old and\n> +\t# delta-reuse-new here\n> +\t#\n> +\t# - the new history contains an object that is stored on the server as a delta\n> +\t# against a base that is in the old history\n> +\t#\n> +\t# - the base object is not immediately reachable from the tip of the old\n> +\t# history; finding it would involve digging down through history we know the\n> +\t# other side has\n> +\t#\n> +\t# This should result in a state where fetching from old->new would not\n> +\t# traditionally reuse the on-disk delta (because we'd have to dig to realize\n> +\t# that the client has it), but we will do so if bitmaps can tell us cheaply\n> +\t# that the other side has it.\n> +\ttest_expect_success 'set up thin delta-reuse parent' '\n> +\t\t# This first commit contains the buried base object.\n> +\t\ttest-tool genrandom delta 16384 >file &&\n> +\t\tgit add file &&\n> +\t\tgit commit -m \"delta base\" &&\n> +\t\tbase=$(git rev-parse --verify HEAD:file) &&\n> +\n> +\t\t# These intermediate commits bury the base back in history.\n> +\t\t# This becomes the \"old\" state.\n> +\t\tfor i in 1 2 3 4 5\n> +\t\tdo\n> +\t\t\techo $i >file &&\n> +\t\t\tgit commit -am \"intermediate $i\" || return 1\n> +\t\tdone &&\n> +\t\tgit branch delta-reuse-old &&\n> +\n> +\t\t# And now our new history has a delta against the buried base. Note\n> +\t\t# that this must be smaller than the original file, since pack-objects\n> +\t\t# prefers to create deltas from smaller objects to larger.\n> +\t\ttest-tool genrandom delta 16300 >file &&\n> +\t\tgit commit -am \"delta result\" &&\n> +\t\tdelta=$(git rev-parse --verify HEAD:file) &&\n> +\t\tgit branch delta-reuse-new &&\n> +\n> +\t\t# Repack with bitmaps and double check that we have the expected delta\n> +\t\t# relationship.\n> +\t\tgit repack -adb &&\n> +\t\thave_delta $delta $base\n> +\t'\n> +\n> +\t# Now we can sanity-check the non-bitmap behavior (that the server is not able\n> +\t# to reuse the delta). This isn't strictly something we care about, so this\n> +\t# test could be scrapped in the future. But it makes sure that the next test is\n> +\t# actually triggering the feature we want.\n> +\t#\n> +\t# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n> +\t# we actually perform the fetch, retain the resulting pack, and inspect the\n> +\t# result.\n> +\ttest_expect_success 'fetch without bitmaps ignores delta against old base' '\n> +\t\ttest_config pack.usebitmaps false &&\n> +\t\ttest_when_finished \"rm -rf client.git\" &&\n> +\t\tgit init --bare client.git &&\n> +\t\t(\n> +\t\t\tcd client.git &&\n> +\t\t\tgit config transfer.unpackLimit 1 &&\n> +\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n> +\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n> +\t\t\thave_delta $delta $ZERO_OID\n> +\t\t)\n> +\t'\n> +\n> +\t# And do the same for the bitmap case, where we do expect to find the delta.\n> +\ttest_expect_success 'fetch with bitmaps can reuse old base' '\n> +\t\ttest_config pack.usebitmaps true &&\n> +\t\ttest_when_finished \"rm -rf client.git\" &&\n> +\t\tgit init --bare client.git &&\n> +\t\t(\n> +\t\t\tcd client.git &&\n> +\t\t\tgit config transfer.unpackLimit 1 &&\n> +\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n> +\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n> +\t\t\thave_delta $delta $base\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success 'pack.preferBitmapTips' '\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> +\n> +\t\t\t# create enough commits that not all are receive bitmap\n> +\t\t\t# coverage even if they are all at the tip of some reference.\n> +\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n> +\n> +\t\t\tgit rev-list HEAD >commits.raw &&\n> +\t\t\tsort commits &&\n> +\n> +\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> +\t\t\tgit update-ref --stdin +\n> +\t\t\tgit repack -adb &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\n> +\t\t\t# remember which commits did not receive bitmaps\n> +\t\t\tcomm -13 bitmaps commits >before &&\n> +\t\t\ttest_file_not_empty before &&\n> +\n> +\t\t\t# mark the commits which did not receive bitmaps as preferred,\n> +\t\t\t# and generate the bitmap again\n> +\t\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" +\t\t\t\tgit update-ref --stdin &&\n> +\t\t\tgit -c pack.preferBitmapTips=refs/tags/include repack -adb &&\n> +\n> +\t\t\t# finally, check that the commit(s) without bitmap coverage\n> +\t\t\t# are not the same ones as before\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tcomm -13 bitmaps commits >after &&\n> +\n> +\t\t\t! test_cmp before after\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success 'complains about multiple pack bitmaps' '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> +\n> +\t\t\ttest_commit base &&\n> +\n> +\t\t\tgit repack -adb &&\n> +\t\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n> +\t\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n> +\n> +\t\t\ttest_commit other &&\n> +\t\t\tgit repack -ab &&\n> +\n> +\t\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n> +\n> +\t\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n> +\t\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n> +\t\t\ttest_line_count = 2 packs &&\n> +\t\t\ttest_line_count = 2 bitmaps &&\n> +\n> +\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n> +\t\t\tgrep \"ignoring extra bitmap file\" err\n> +\t\t)\n> +\t'\n> +}\n>\n> -basic_bitmap_tests\n> +test_bitmap_cases\n>\n> test_expect_success 'incremental repack fails when bitmaps are requested' '\n> \ttest_commit more-1 &&\n> @@ -54,375 +445,12 @@ test_expect_success 'incremental repack can disable bitmaps' '\n> \tgit repack -d --no-write-bitmap-index\n> '\n>\n> -test_expect_success 'pack-objects respects --local (non-local loose)' '\n> -\tgit init --bare alt.git &&\n> -\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n> -\techo content1 >file1 &&\n> -\t# non-local loose object which is not present in bitmapped pack\n> -\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n> -\t# non-local loose object which is also present in bitmapped pack\n> -\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n> -\tgit add file1 &&\n> -\ttest_tick &&\n> -\tgit commit -m commit_file1 &&\n> -\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n> -\tgit index-pack 1.pack &&\n> -\tlist_packed_objects 1.idx >1.objects &&\n> -\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n> -\t! has_any nonlocal-loose 1.objects\n> -'\n> -\n> -test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n> -\techo content2 >file2 &&\n> -\tblob2=$(git hash-object -w file2) &&\n> -\tgit add file2 &&\n> -\ttest_tick &&\n> -\tgit commit -m commit_file2 &&\n> -\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n> -\tpack2=$(git pack-objects pack2 -\tmv pack2-$pack2.* .git/objects/pack/ &&\n> -\t>.git/objects/pack/pack2-$pack2.keep &&\n> -\trm $(objpath $blob2) &&\n> -\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n> -\tgit index-pack 2a.pack &&\n> -\tlist_packed_objects 2a.idx >2a.objects &&\n> -\t! has_any keepobjects 2a.objects\n> -'\n> -\n> -test_expect_success 'pack-objects respects --local (non-local pack)' '\n> -\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n> -\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n> -\tgit index-pack 2b.pack &&\n> -\tlist_packed_objects 2b.idx >2b.objects &&\n> -\t! has_any keepobjects 2b.objects\n> -'\n> -\n> -test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n> -\tls .git/objects/pack/ | grep bitmap >output &&\n> -\ttest_line_count = 1 output &&\n> -\tpackbitmap=$(basename $(cat output) .bitmap) &&\n> -\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n> -\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n> -\t>.git/objects/pack/$packbitmap.keep &&\n> -\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n> -\tgit index-pack 3a.pack &&\n> -\tlist_packed_objects 3a.idx >3a.objects &&\n> -\t! has_any packbitmap.objects 3a.objects\n> -'\n> -\n> -test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n> -\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n> -\trm -f .git/objects/pack/multi-pack-index &&\n> -\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n> -\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n> -\tgit index-pack 3b.pack &&\n> -\tlist_packed_objects 3b.idx >3b.objects &&\n> -\t! has_any packbitmap.objects 3b.objects\n> -'\n> -\n> -test_expect_success 'pack-objects to file can use bitmap' '\n> -\t# make sure we still have 1 bitmap index from previous tests\n> -\tls .git/objects/pack/ | grep bitmap >output &&\n> -\ttest_line_count = 1 output &&\n> -\t# verify equivalent packs are generated with/without using bitmap index\n> -\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa -\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb -\tlist_packed_objects packa-$packasha1.idx >packa.objects &&\n> -\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n> -\ttest_cmp packa.objects packb.objects\n> -'\n> -\n> -test_expect_success 'full repack, reusing previous bitmaps' '\n> -\tgit repack -ad &&\n> -\tls .git/objects/pack/ | grep bitmap >output &&\n> -\ttest_line_count = 1 output\n> -'\n> -\n> -test_expect_success 'fetch (full bitmap)' '\n> -\tgit --git-dir=clone.git fetch origin second:second &&\n> -\tgit rev-parse HEAD >expect &&\n> -\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n> -\ttest_cmp expect actual\n> -'\n> -\n> -test_expect_success 'create objects for missing-HAVE tests' '\n> -\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n> -\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n> -\tparent=$(echo parent | git commit-tree $tree) &&\n> -\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n> -\tcat >revs <<-EOF\n> -\tHEAD\n> -\t^HEAD^\n> -\t^$commit\n> -\tEOF\n> -'\n> -\n> -test_expect_success 'pack-objects respects --incremental' '\n> -\tcat >revs2 <<-EOF &&\n> -\tHEAD\n> -\t$commit\n> -\tEOF\n> -\tgit pack-objects --incremental --stdout --revs 4.pack &&\n> -\tgit index-pack 4.pack &&\n> -\tlist_packed_objects 4.idx >4.objects &&\n> -\ttest_line_count = 4 4.objects &&\n> -\tgit rev-list --objects $commit >revlist &&\n> -\tcut -d\" \" -f1 revlist |sort >objects &&\n> -\ttest_cmp 4.objects objects\n> -'\n> -\n> -test_expect_success 'pack with missing blob' '\n> -\trm $(objpath $blob) &&\n> -\tgit pack-objects --stdout --revs /dev/null\n> -'\n> +test_bitmap_cases \"pack.writeBitmapLookupTable\"\n>\n> -test_expect_success 'pack with missing tree' '\n> -\trm $(objpath $tree) &&\n> -\tgit pack-objects --stdout --revs /dev/null\n> -'\n> -\n> -test_expect_success 'pack with missing parent' '\n> -\trm $(objpath $parent) &&\n> -\tgit pack-objects --stdout --revs /dev/null\n> -'\n> -\n> -test_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n> -\tgit clone --bare . compat-jgit.git &&\n> -\t(\n> -\t\tcd compat-jgit.git &&\n> -\t\trm -f objects/pack/*.bitmap &&\n> -\t\tjgit gc &&\n> -\t\tgit rev-list --test-bitmap HEAD\n> -\t)\n> -'\n> -\n> -test_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n> -\tgit clone --bare . compat-us.git &&\n> -\t(\n> -\t\tcd compat-us.git &&\n> -\t\tgit repack -adb &&\n> -\t\t# jgit gc will barf if it does not like our bitmaps\n> -\t\tjgit gc\n> -\t)\n> -'\n> -\n> -test_expect_success 'splitting packs does not generate bogus bitmaps' '\n> -\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n> -\tgit add rand &&\n> -\tgit commit -m \"commit with big file\" &&\n> -\tgit -c pack.packSizeLimit=500k repack -adb &&\n> -\tgit init --bare no-bitmaps.git &&\n> -\tgit -C no-bitmaps.git fetch .. HEAD\n> -'\n> -\n> -test_expect_success 'set up reusable pack' '\n> -\trm -f .git/objects/pack/*.keep &&\n> -\tgit repack -adb &&\n> -\treusable_pack () {\n> -\t\tgit for-each-ref --format=\"%(objectname)\" |\n> -\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n> -\t}\n> -'\n> -\n> -test_expect_success 'pack reuse respects --honor-pack-keep' '\n> -\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n> -\tfor i in .git/objects/pack/*.pack\n> -\tdo\n> -\t\t>${i%.pack}.keep || return 1\n> -\tdone &&\n> -\treusable_pack --honor-pack-keep >empty.pack &&\n> -\tgit index-pack empty.pack &&\n> -\tgit show-index actual &&\n> -\ttest_must_be_empty actual\n> -'\n> -\n> -test_expect_success 'pack reuse respects --local' '\n> -\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n> -\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n> -\treusable_pack --local >empty.pack &&\n> -\tgit index-pack empty.pack &&\n> -\tgit show-index actual &&\n> -\ttest_must_be_empty actual\n> -'\n> -\n> -test_expect_success 'pack reuse respects --incremental' '\n> -\treusable_pack --incremental >empty.pack &&\n> -\tgit index-pack empty.pack &&\n> -\tgit show-index actual &&\n> -\ttest_must_be_empty actual\n> -'\n> -\n> -test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n> -\ttest_config pack.writebitmaphashcache false &&\n> -\tgit repack -ad &&\n> -\tgit rev-list --use-bitmap-index --count --all >expect &&\n> -\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n> -\ttest_when_finished \"rm -f $bitmap\" &&\n> -\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n> -\tmv -f $bitmap.tmp $bitmap &&\n> -\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n> -\ttest_cmp expect actual &&\n> -\ttest_i18ngrep corrupt.ewah.bitmap stderr\n> -'\n> -\n> -test_expect_success 'truncated bitmap fails gracefully (cache)' '\n> -\tgit repack -ad &&\n> -\tgit rev-list --use-bitmap-index --count --all >expect &&\n> -\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n> -\ttest_when_finished \"rm -f $bitmap\" &&\n> -\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n> -\tmv -f $bitmap.tmp $bitmap &&\n> -\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n> -\ttest_cmp expect actual &&\n> -\ttest_i18ngrep corrupted.bitmap.index stderr\n> -'\n> -\n> -# Create a state of history with these properties:\n> -#\n> -# - refs that allow a client to fetch some new history, while sharing some old\n> -# history with the server; we use branches delta-reuse-old and\n> -# delta-reuse-new here\n> -#\n> -# - the new history contains an object that is stored on the server as a delta\n> -# against a base that is in the old history\n> -#\n> -# - the base object is not immediately reachable from the tip of the old\n> -# history; finding it would involve digging down through history we know the\n> -# other side has\n> -#\n> -# This should result in a state where fetching from old->new would not\n> -# traditionally reuse the on-disk delta (because we'd have to dig to realize\n> -# that the client has it), but we will do so if bitmaps can tell us cheaply\n> -# that the other side has it.\n> -test_expect_success 'set up thin delta-reuse parent' '\n> -\t# This first commit contains the buried base object.\n> -\ttest-tool genrandom delta 16384 >file &&\n> -\tgit add file &&\n> -\tgit commit -m \"delta base\" &&\n> -\tbase=$(git rev-parse --verify HEAD:file) &&\n> -\n> -\t# These intermediate commits bury the base back in history.\n> -\t# This becomes the \"old\" state.\n> -\tfor i in 1 2 3 4 5\n> -\tdo\n> -\t\techo $i >file &&\n> -\t\tgit commit -am \"intermediate $i\" || return 1\n> -\tdone &&\n> -\tgit branch delta-reuse-old &&\n> -\n> -\t# And now our new history has a delta against the buried base. Note\n> -\t# that this must be smaller than the original file, since pack-objects\n> -\t# prefers to create deltas from smaller objects to larger.\n> -\ttest-tool genrandom delta 16300 >file &&\n> -\tgit commit -am \"delta result\" &&\n> -\tdelta=$(git rev-parse --verify HEAD:file) &&\n> -\tgit branch delta-reuse-new &&\n> -\n> -\t# Repack with bitmaps and double check that we have the expected delta\n> -\t# relationship.\n> -\tgit repack -adb &&\n> -\thave_delta $delta $base\n> -'\n> -\n> -# Now we can sanity-check the non-bitmap behavior (that the server is not able\n> -# to reuse the delta). This isn't strictly something we care about, so this\n> -# test could be scrapped in the future. But it makes sure that the next test is\n> -# actually triggering the feature we want.\n> -#\n> -# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n> -# we actually perform the fetch, retain the resulting pack, and inspect the\n> -# result.\n> -test_expect_success 'fetch without bitmaps ignores delta against old base' '\n> -\ttest_config pack.usebitmaps false &&\n> -\ttest_when_finished \"rm -rf client.git\" &&\n> -\tgit init --bare client.git &&\n> -\t(\n> -\t\tcd client.git &&\n> -\t\tgit config transfer.unpackLimit 1 &&\n> -\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n> -\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n> -\t\thave_delta $delta $ZERO_OID\n> -\t)\n> -'\n> -\n> -# And do the same for the bitmap case, where we do expect to find the delta.\n> -test_expect_success 'fetch with bitmaps can reuse old base' '\n> -\ttest_config pack.usebitmaps true &&\n> -\ttest_when_finished \"rm -rf client.git\" &&\n> -\tgit init --bare client.git &&\n> -\t(\n> -\t\tcd client.git &&\n> -\t\tgit config transfer.unpackLimit 1 &&\n> -\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n> -\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n> -\t\thave_delta $delta $base\n> -\t)\n> -'\n> -\n> -test_expect_success 'pack.preferBitmapTips' '\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> -\n> -\t\t# create enough commits that not all are receive bitmap\n> -\t\t# coverage even if they are all at the tip of some reference.\n> -\t\ttest_commit_bulk --message=\"%s\" 103 &&\n> -\n> -\t\tgit rev-list HEAD >commits.raw &&\n> -\t\tsort commits &&\n> -\n> -\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> -\t\tgit update-ref --stdin -\n> -\t\tgit repack -adb &&\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\n> -\t\t# remember which commits did not receive bitmaps\n> -\t\tcomm -13 bitmaps commits >before &&\n> -\t\ttest_file_not_empty before &&\n> -\n> -\t\t# mark the commits which did not receive bitmaps as preferred,\n> -\t\t# and generate the bitmap again\n> -\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" -\t\t\tgit update-ref --stdin &&\n> -\t\tgit -c pack.preferBitmapTips=refs/tags/include repack -adb &&\n> -\n> -\t\t# finally, check that the commit(s) without bitmap coverage\n> -\t\t# are not the same ones as before\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tcomm -13 bitmaps commits >after &&\n> -\n> -\t\t! test_cmp before after\n> -\t)\n> -'\n> -\n> -test_expect_success 'complains about multiple pack bitmaps' '\n> -\trm -fr repo &&\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> -\n> -\t\ttest_commit base &&\n> -\n> -\t\tgit repack -adb &&\n> -\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n> -\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n> -\n> -\t\ttest_commit other &&\n> -\t\tgit repack -ab &&\n> -\n> -\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n> -\n> -\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n> -\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n> -\t\ttest_line_count = 2 packs &&\n> -\t\ttest_line_count = 2 bitmaps &&\n> -\n> -\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n> -\t\tgrep \"ignoring extra bitmap file\" err\n> -\t)\n> +test_expect_success 'verify writing bitmap lookup table when enabled' '\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2\" \\\n> +\t\tgit repack -ad &&\n> +\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n> '\n>\n> test_done\n> diff --git a/t/t5311-pack-bitmaps-shallow.sh b/t/t5311-pack-bitmaps-shallow.sh\n> index 872a95df338..9dae60f73e3 100755\n> --- a/t/t5311-pack-bitmaps-shallow.sh\n> +++ b/t/t5311-pack-bitmaps-shallow.sh\n> @@ -17,23 +17,40 @@ test_description='check bitmap operation with shallow repositories'\n> # the tree for A. But in a shallow one, we've grafted away\n> # A, and fetching A to B requires that the other side send\n> # us the tree for file=1.\n> -test_expect_success 'setup shallow repo' '\n> -\techo 1 >file &&\n> -\tgit add file &&\n> -\tgit commit -m orig &&\n> -\techo 2 >file &&\n> -\tgit commit -a -m update &&\n> -\tgit clone --no-local --bare --depth=1 . shallow.git &&\n> -\techo 1 >file &&\n> -\tgit commit -a -m repeat\n> -'\n> -\n> -test_expect_success 'turn on bitmaps in the parent' '\n> -\tgit repack -adb\n> -'\n> -\n> -test_expect_success 'shallow fetch from bitmapped repo' '\n> -\t(cd shallow.git && git fetch)\n> -'\n> +test_shallow_bitmaps () {\n> +\twriteLookupTable=false\n> +\n> +\tfor i in \"$@\"\n> +\tdo\n> +\t\tcase $i in\n> +\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n> +\t\tesac\n> +\tdone\n> +\n> +\ttest_expect_success 'setup shallow repo' '\n> +\t\trm -rf * .git &&\n> +\t\tgit init &&\n> +\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> +\t\techo 1 >file &&\n> +\t\tgit add file &&\n> +\t\tgit commit -m orig &&\n> +\t\techo 2 >file &&\n> +\t\tgit commit -a -m update &&\n> +\t\tgit clone --no-local --bare --depth=1 . shallow.git &&\n> +\t\techo 1 >file &&\n> +\t\tgit commit -a -m repeat\n> +\t'\n> +\n> +\ttest_expect_success 'turn on bitmaps in the parent' '\n> +\t\tgit repack -adb\n> +\t'\n> +\n> +\ttest_expect_success 'shallow fetch from bitmapped repo' '\n> +\t\t(cd shallow.git && git fetch)\n> +\t'\n> +}\n> +\n> +test_shallow_bitmaps\n> +test_shallow_bitmaps \"pack.writeBitmapLookupTable\"\n>\n> test_done\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> index 4fe57414c13..3b206adcee6 100755\n> --- a/t/t5326-multi-pack-bitmaps.sh\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -15,17 +15,24 @@ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> sane_unset GIT_TEST_MIDX_WRITE_REV\n> sane_unset GIT_TEST_MIDX_READ_RIDX\n>\n> -midx_bitmap_core\n> -\n> bitmap_reuse_tests() {\n> \tfrom=$1\n> \tto=$2\n> +\twriteLookupTable=false\n> +\n> +\tfor i in $3-${$#}\n> +\tdo\n> +\t\tcase $i in\n> +\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n> +\t\tesac\n> +\tdone\n>\n> \ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n> \t\trm -fr repo &&\n> \t\tgit init repo &&\n> \t\t(\n> \t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> \t\t\ttest_commit_bulk 16 &&\n> \t\t\tgit tag old-tip &&\n>\n> @@ -43,6 +50,7 @@ bitmap_reuse_tests() {\n> \ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n> \t\t(\n> \t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> \t\t\ttest_commit_bulk --id=further 16 &&\n> \t\t\tgit tag new-tip &&\n>\n> @@ -59,6 +67,7 @@ bitmap_reuse_tests() {\n> \ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n> \t\t(\n> \t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> \t\t\tgit for-each-ref &&\n> \t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n> \t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n> @@ -66,244 +75,294 @@ bitmap_reuse_tests() {\n> \t'\n> }\n>\n> -bitmap_reuse_tests 'pack' 'MIDX'\n> -bitmap_reuse_tests 'MIDX' 'pack'\n> -bitmap_reuse_tests 'MIDX' 'MIDX'\n> +test_midx_bitmap_cases () {\n> +\twriteLookupTable=false\n> +\twriteBitmapLookupTable=\n> +\n> +\tfor i in \"$@\"\n> +\tdo\n> +\t\tcase $i in\n> +\t\t\"pack.writeBitmapLookupTable\")\n> +\t\t\twriteLookupTable=true\n> +\t\t\twriteBitmapLookupTable=\"$i\"\n> +\t\t\t;;\n> +\t\tesac\n> +\tdone\n> +\n> +\ttest_expect_success 'setup test_repository' '\n> +\t\trm -rf * .git &&\n> +\t\tgit init &&\n> +\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n> +\t'\n>\n> -test_expect_success 'missing object closure fails gracefully' '\n> -\trm -fr repo &&\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> +\tmidx_bitmap_core\n>\n> -\t\ttest_commit loose &&\n> -\t\ttest_commit packed &&\n> +\tbitmap_reuse_tests 'pack' 'MIDX' \"$writeBitmapLookupTable\"\n> +\tbitmap_reuse_tests 'MIDX' 'pack' \"$writeBitmapLookupTable\"\n> +\tbitmap_reuse_tests 'MIDX' 'MIDX' \"$writeBitmapLookupTable\"\n>\n> -\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n> -\t\t# commit.\n> -\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n> -\t\t$(git rev-parse packed)\n> -\t\tEOF\n> +\ttest_expect_success 'missing object closure fails gracefully' '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n> -\t\tgrep \"doesn.t have full closure\" err &&\n> -\t\ttest_path_is_missing $midx\n> -\t)\n> -'\n> +\t\t\ttest_commit loose &&\n> +\t\t\ttest_commit packed &&\n>\n> -midx_bitmap_partial_tests\n> +\t\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n> +\t\t\t# commit.\n> +\t\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n> +\t\t\t$(git rev-parse packed)\n> +\t\t\tEOF\n>\n> -test_expect_success 'removing a MIDX clears stale bitmaps' '\n> -\trm -fr repo &&\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> -\t\ttest_commit base &&\n> -\t\tgit repack &&\n> -\t\tgit multi-pack-index write --bitmap &&\n> +\t\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n> +\t\t\tgrep \"doesn.t have full closure\" err &&\n> +\t\t\ttest_path_is_missing $midx\n> +\t\t)\n> +\t'\n>\n> -\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n> -\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n> -\t\trm $midx &&\n> +\tmidx_bitmap_partial_tests\n>\n> -\t\t# Then write a new MIDX.\n> -\t\ttest_commit new &&\n> -\t\tgit repack &&\n> -\t\tgit multi-pack-index write --bitmap &&\n> +\ttest_expect_success 'removing a MIDX clears stale bitmaps' '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n> +\t\t\ttest_commit base &&\n> +\t\t\tgit repack &&\n> +\t\t\tgit multi-pack-index write --bitmap &&\n> +\n> +\t\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n> +\t\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\trm $midx &&\n> +\n> +\t\t\t# Then write a new MIDX.\n> +\t\t\ttest_commit new &&\n> +\t\t\tgit repack &&\n> +\t\t\tgit multi-pack-index write --bitmap &&\n> +\n> +\t\t\ttest_path_is_file $midx &&\n> +\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\ttest_path_is_missing $stale_bitmap\n> +\t\t)\n> +\t'\n>\n> -\t\ttest_path_is_file $midx &&\n> -\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> -\t\ttest_path_is_missing $stale_bitmap\n> -\t)\n> -'\n> +\ttest_expect_success 'pack.preferBitmapTips' '\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -test_expect_success 'pack.preferBitmapTips' '\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> +\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n>\n> -\t\ttest_commit_bulk --message=\"%s\" 103 &&\n> +\t\t\tgit log --format=\"%H\" >commits.raw &&\n> +\t\t\tsort commits &&\n>\n> -\t\tgit log --format=\"%H\" >commits.raw &&\n> -\t\tsort commits &&\n> +\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> +\t\t\tgit update-ref --stdin \n> -\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> -\t\tgit update-ref --stdin +\t\t\tgit multi-pack-index write --bitmap &&\n> +\t\t\ttest_path_is_file $midx &&\n> +\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n>\n> -\t\tgit multi-pack-index write --bitmap &&\n> -\t\ttest_path_is_file $midx &&\n> -\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tcomm -13 bitmaps commits >before &&\n> +\t\t\ttest_line_count = 1 before &&\n>\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tcomm -13 bitmaps commits >before &&\n> -\t\ttest_line_count = 1 before &&\n> +\t\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n> +\t\t\t\t\n> -\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n> -\t\t\t +\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\trm -fr $midx &&\n>\n> -\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> -\t\trm -fr $midx &&\n> +\t\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n> +\t\t\t\tmulti-pack-index write --bitmap &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tcomm -13 bitmaps commits >after &&\n>\n> -\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n> -\t\t\tmulti-pack-index write --bitmap &&\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tcomm -13 bitmaps commits >after &&\n> +\t\t\t! test_cmp before after\n> +\t\t)\n> +\t'\n>\n> -\t\t! test_cmp before after\n> -\t)\n> -'\n> +\ttest_expect_success 'writing a bitmap with --refs-snapshot' '\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -test_expect_success 'writing a bitmap with --refs-snapshot' '\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> +\t\t\ttest_commit one &&\n> +\t\t\ttest_commit two &&\n>\n> -\t\ttest_commit one &&\n> -\t\ttest_commit two &&\n> +\t\t\tgit rev-parse one >snapshot &&\n>\n> -\t\tgit rev-parse one >snapshot &&\n> +\t\t\tgit repack -ad &&\n>\n> -\t\tgit repack -ad &&\n> +\t\t\t# First, write a MIDX which see both refs/tags/one and\n> +\t\t\t# refs/tags/two (causing both of those commits to receive\n> +\t\t\t# bitmaps).\n> +\t\t\tgit multi-pack-index write --bitmap &&\n>\n> -\t\t# First, write a MIDX which see both refs/tags/one and\n> -\t\t# refs/tags/two (causing both of those commits to receive\n> -\t\t# bitmaps).\n> -\t\tgit multi-pack-index write --bitmap &&\n> +\t\t\ttest_path_is_file $midx &&\n> +\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n>\n> -\t\ttest_path_is_file $midx &&\n> -\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n> +\t\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n>\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n> -\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n> +\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\trm -fr $midx &&\n>\n> -\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> -\t\trm -fr $midx &&\n> +\t\t\t# Then again, but with a refs snapshot which only sees\n> +\t\t\t# refs/tags/one.\n> +\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n>\n> -\t\t# Then again, but with a refs snapshot which only sees\n> -\t\t# refs/tags/one.\n> -\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n> +\t\t\ttest_path_is_file $midx &&\n> +\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n>\n> -\t\ttest_path_is_file $midx &&\n> -\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n> +\t\t\t! grep \"$(git rev-parse two)\" bitmaps\n> +\t\t)\n> +\t'\n>\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n> -\t\t! grep \"$(git rev-parse two)\" bitmaps\n> -\t)\n> -'\n> +\ttest_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> +\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n>\n> -\t\ttest_commit_bulk --message=\"%s\" 103 &&\n> +\t\t\tgit log --format=\"%H\" >commits.raw &&\n> +\t\t\tsort commits &&\n>\n> -\t\tgit log --format=\"%H\" >commits.raw &&\n> -\t\tsort commits &&\n> +\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> +\t\t\tgit update-ref --stdin \n> -\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n> -\t\tgit update-ref --stdin +\t\t\tgit multi-pack-index write --bitmap &&\n> +\t\t\ttest_path_is_file $midx &&\n> +\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n>\n> -\t\tgit multi-pack-index write --bitmap &&\n> -\t\ttest_path_is_file $midx &&\n> -\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tcomm -13 bitmaps commits >before &&\n> +\t\t\ttest_line_count = 1 before &&\n>\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tcomm -13 bitmaps commits >before &&\n> -\t\ttest_line_count = 1 before &&\n> +\t\t\t(\n> +\t\t\t\tgrep -vf before commits.raw &&\n> +\t\t\t\t# mark missing commits as preferred\n> +\t\t\t\tsed \"s/^/+/\" before\n> +\t\t\t) >snapshot &&\n>\n> +\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> +\t\t\trm -fr $midx &&\n> +\n> +\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n> +\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> +\t\t\tcomm -13 bitmaps commits >after &&\n> +\n> +\t\t\t! test_cmp before after\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success 'hash-cache values are propagated from pack bitmaps' '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> \t\t(\n> -\t\t\tgrep -vf before commits.raw &&\n> -\t\t\t# mark missing commits as preferred\n> -\t\t\tsed \"s/^/+/\" before\n> -\t\t) >snapshot &&\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n> -\t\trm -fr $midx &&\n> +\t\t\ttest_commit base &&\n> +\t\t\ttest_commit base2 &&\n> +\t\t\tgit repack -adb &&\n>\n> -\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n> -\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n> -\t\tcomm -13 bitmaps commits >after &&\n> +\t\t\ttest-tool bitmap dump-hashes >pack.raw &&\n> +\t\t\ttest_file_not_empty pack.raw &&\n> +\t\t\tsort pack.raw >pack.hashes &&\n>\n> -\t\t! test_cmp before after\n> -\t)\n> -'\n> +\t\t\ttest_commit new &&\n> +\t\t\tgit repack &&\n> +\t\t\tgit multi-pack-index write --bitmap &&\n>\n> -test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n> -\trm -fr repo &&\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> +\t\t\ttest-tool bitmap dump-hashes >midx.raw &&\n> +\t\t\tsort midx.raw >midx.hashes &&\n>\n> -\t\ttest_commit base &&\n> -\t\ttest_commit base2 &&\n> -\t\tgit repack -adb &&\n> +\t\t\t# ensure that every namehash in the pack bitmap can be found in\n> +\t\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n> +\t\t\t# unique to the pack bitmap).\n> +\t\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n> +\t\t\ttest_must_be_empty dropped.hashes\n> +\t\t)\n> +\t'\n>\n> -\t\ttest-tool bitmap dump-hashes >pack.raw &&\n> -\t\ttest_file_not_empty pack.raw &&\n> -\t\tsort pack.raw >pack.hashes &&\n> +\ttest_expect_success 'no .bitmap is written without any objects' '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -\t\ttest_commit new &&\n> -\t\tgit repack &&\n> -\t\tgit multi-pack-index write --bitmap &&\n> +\t\t\tempty=\"$(git pack-objects $objdir/pack/pack +\t\t\tcat >packs <<-EOF &&\n> +\t\t\tpack-$empty.idx\n> +\t\t\tEOF\n>\n> -\t\ttest-tool bitmap dump-hashes >midx.raw &&\n> -\t\tsort midx.raw >midx.hashes &&\n> +\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n> +\t\t\t\terr &&\n>\n> -\t\t# ensure that every namehash in the pack bitmap can be found in\n> -\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n> -\t\t# unique to the pack bitmap).\n> -\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n> -\t\ttest_must_be_empty dropped.hashes\n> -\t)\n> -'\n> +\t\t\tgrep \"bitmap without any objects\" err &&\n>\n> -test_expect_success 'no .bitmap is written without any objects' '\n> -\trm -fr repo &&\n> -\tgit init repo &&\n> -\ttest_when_finished \"rm -fr repo\" &&\n> -\t(\n> -\t\tcd repo &&\n> +\t\t\ttest_path_is_file $midx &&\n> +\t\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n> +\t\t)\n> +\t'\n> +\n> +\ttest_expect_success 'graceful fallback when missing reverse index' '\n> +\t\trm -fr repo &&\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n>\n> -\t\tempty=\"$(git pack-objects $objdir/pack/pack -\t\tcat >packs <<-EOF &&\n> -\t\tpack-$empty.idx\n> -\t\tEOF\n> +\t\t\ttest_commit base &&\n>\n> -\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n> -\t\t\terr &&\n> +\t\t\t# write a pack and MIDX bitmap containing base\n> +\t\t\tgit repack -adb &&\n> +\t\t\tgit multi-pack-index write --bitmap &&\n>\n> -\t\tgrep \"bitmap without any objects\" err &&\n> +\t\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n> +\t\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n> +\t\t\t! grep \"ignoring extra bitmap file\" err\n> +\t\t)\n> +\t'\n> +}\n>\n> -\t\ttest_path_is_file $midx &&\n> -\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n> -\t)\n> -'\n> +test_midx_bitmap_cases\n> +\n> +test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n>\n> -test_expect_success 'graceful fallback when missing reverse index' '\n> +test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n> \trm -fr repo &&\n> \tgit init repo &&\n> \ttest_when_finished \"rm -fr repo\" &&\n> \t(\n> \t\tcd repo &&\n> -\n> \t\ttest_commit base &&\n> -\n> -\t\t# write a pack and MIDX bitmap containing base\n> -\t\tgit repack -adb &&\n> -\t\tgit multi-pack-index write --bitmap &&\n> -\n> -\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n> -\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n> -\t\t! grep \"ignoring extra bitmap file\" err\n> +\t\tgit config pack.writeBitmapLookupTable true &&\n> +\t\tgit repack -ad &&\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n> +\t\t\tgit multi-pack-index write --bitmap &&\n> +\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n> \t)\n> '\n>\n> diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\n> index d30ba632c87..5ed16a820d1 100755\n> --- a/t/t5327-multi-pack-bitmaps-rev.sh\n> +++ b/t/t5327-multi-pack-bitmaps-rev.sh\n> @@ -17,7 +17,27 @@ GIT_TEST_MIDX_READ_RIDX=0\n> export GIT_TEST_MIDX_WRITE_REV\n> export GIT_TEST_MIDX_READ_RIDX\n>\n> -midx_bitmap_core rev\n> -midx_bitmap_partial_tests rev\n> +test_midx_bitmap_rev () {\n> + writeLookupTable=false\n> +\n> + \tfor i in \"$@\"\n> + \tdo\n> + \t\tcase $i in\n> + \t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n> + \t\tesac\n> + \tdone\n> +\n> + test_expect_success 'setup bitmap config' '\n> + rm -rf * .git &&\n> + git init &&\n> + git config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n> + '\n> +\n> + midx_bitmap_core rev\n> + midx_bitmap_partial_tests rev\n> + }\n> +\n> + test_midx_bitmap_rev\n> + test_midx_bitmap_rev \"pack.writeBitmapLookupTable\"\n>\n> test_done\n> --\n> gitgitgadget\n>\n>\n>\n"},{"id":"460417","messageId":"CAPOJW5xBUaAJtOvrefwbXv_WDTLa=6PTL5kEoOpRQfqqFAx3oA@mail.gmail.com","threadId":"58038","inReplyTo":"p3r70610-8n52-s8q0-n641-onp4ps01330n@tzk.qr","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-02T12:40:38Z","receivedAt":"2022-08-02T12:40:55Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Fri, Jul 29, 2022 at 12:52 AM Johannes Schindelin\n wrote:\n> That's quite a large a change, and unfortunately I pinpointed a flake to\n> this patch when running with GIT_TEST_DEFAULT_HASH=sha256. The symptom is\n> this:\n\nHi Dscho, sorry for this long delay in response. I was quite busy for\n3-4 days in hostel room shifting. So, I couldn't work properly during\nthis time.\n\n> -- snip --\n> [...]\n> + diff -u expect.normalized actual.normalized\n> + rm -f expect.normalized actual.normalized\n> ok 317 - enumerate --objects (full bitmap, other)\n>\n> expecting success of 5326.318 'bitmap --objects handles non-commit objects (full bitmap, other)':\n> git rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n> grep $blob actual\n>\n> + git rev-list --objects --use-bitmap-index other tagged-blob\n> + grep bff4ed5e839bd73e821f78b45a7fa34208aa85596535ec8e9ac5eab477ca6f81 actual\n> bff4ed5e839bd73e821f78b45a7fa34208aa85596535ec8e9ac5eab477ca6f81\n> ok 318 - bitmap --objects handles non-commit objects (full bitmap, other)\n>\n> expecting success of 5326.319 'clone from bitmapped repository':\n> rm -fr clone.git &&\n> git clone --no-local --bare . clone.git &&\n> git rev-parse HEAD >expect &&\n> git --git-dir=clone.git rev-parse HEAD >actual &&\n> test_cmp expect actual\n>\n> + rm -fr clone.git\n> + git clone --no-local --bare . clone.git\n> Cloning into bare repository 'clone.git'...\n> remote: Enumerating objects: 756, done.\n> remote: Counting objects: 100% (754/754), done.\n> remote: Compressing objects: 100% (281/281), done.\n> remote: Total 756 (delta 245), reused 740 (delta 234), pack-reused 2\n> Receiving objects: 100% (756/756), 77.50 KiB | 8.61 MiB/s, done.\n> fatal: REF_DELTA at offset 221 already resolved (duplicate base 4d332072f161629ffe4652ecd3ce377ef88447bec73f05ab0f3515f98bd061cf?)\n> fatal: fetch-pack: invalid index-pack output\n> error: last command exited with $?=128\n> not ok 319 - clone from bitmapped repository\n> #\n> # rm -fr clone.git &&\n> # git clone --no-local --bare . clone.git &&\n> # git rev-parse HEAD >expect &&\n> # git --git-dir=clone.git rev-parse HEAD >actual &&\n> # test_cmp expect actual\n> #\n> 1..319\n> -- snap --\n>\n> On a hunch, I ran this through valgrind (took a while) but it did not\n> point out the problem.\n>\n> Again, this is only with SHA-256 (and somewhat flaky), it passes every\n> time with SHA-1. Maybe you can reproduce on your side with that\n> information?\n\nYeah, I can reproduce it on my side. But I am sure it is not related\nto the lookup table implementation code. Because when I swap the order\nof calling `test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"` and\n`test_midx_bitmap_cases` (in t5326-multi-pack-bitmaps.sh), in that\ncase, the error is being generated in `test_midx_bitmap_cases` call.\nGenerally speaking, the error is always being generated in the second\ncall.\n\nFor now, my understanding says that there is something fishy in the\ntest script. I am still not able to figure out the problem here. But\nlet me further investigate.\n\nIf anyone has some idea about what could be the culprit, I will be\nvery happy to know.\n\nThanks :)\n"},{"id":"460442","messageId":"6s4n3600-q5p7-92sr-4206-non3s8rr3n46@tzk.qr","threadId":"58038","inReplyTo":"CAPOJW5xBUaAJtOvrefwbXv_WDTLa=6PTL5kEoOpRQfqqFAx3oA@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-02T15:35:02Z","receivedAt":"2022-08-02T15:35:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Abhradeep,\n\nOn Tue, 2 Aug 2022, Abhradeep Chakraborty wrote:\n\n> On Fri, Jul 29, 2022 at 12:52 AM Johannes Schindelin\n> wrote:\n> > That's quite a large a change, and unfortunately I pinpointed a flake to\n> > this patch when running with GIT_TEST_DEFAULT_HASH=sha256. The symptom is\n> > this:\n>\n> Hi Dscho, sorry for this long delay in response. I was quite busy for\n> 3-4 days in hostel room shifting. So, I couldn't work properly during\n> this time.\n>\n> > -- snip --\n> > [...]\n> > + diff -u expect.normalized actual.normalized\n> > + rm -f expect.normalized actual.normalized\n> > ok 317 - enumerate --objects (full bitmap, other)\n> >\n> > expecting success of 5326.318 'bitmap --objects handles non-commit objects (full bitmap, other)':\n> > git rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n> > grep $blob actual\n> >\n> > + git rev-list --objects --use-bitmap-index other tagged-blob\n> > + grep bff4ed5e839bd73e821f78b45a7fa34208aa85596535ec8e9ac5eab477ca6f81 actual\n> > bff4ed5e839bd73e821f78b45a7fa34208aa85596535ec8e9ac5eab477ca6f81\n> > ok 318 - bitmap --objects handles non-commit objects (full bitmap, other)\n> >\n> > expecting success of 5326.319 'clone from bitmapped repository':\n> > rm -fr clone.git &&\n> > git clone --no-local --bare . clone.git &&\n> > git rev-parse HEAD >expect &&\n> > git --git-dir=clone.git rev-parse HEAD >actual &&\n> > test_cmp expect actual\n> >\n> > + rm -fr clone.git\n> > + git clone --no-local --bare . clone.git\n> > Cloning into bare repository 'clone.git'...\n> > remote: Enumerating objects: 756, done.\n> > remote: Counting objects: 100% (754/754), done.\n> > remote: Compressing objects: 100% (281/281), done.\n> > remote: Total 756 (delta 245), reused 740 (delta 234), pack-reused 2\n> > Receiving objects: 100% (756/756), 77.50 KiB | 8.61 MiB/s, done.\n> > fatal: REF_DELTA at offset 221 already resolved (duplicate base 4d332072f161629ffe4652ecd3ce377ef88447bec73f05ab0f3515f98bd061cf?)\n> > fatal: fetch-pack: invalid index-pack output\n> > error: last command exited with $?=128\n> > not ok 319 - clone from bitmapped repository\n> > #\n> > # rm -fr clone.git &&\n> > # git clone --no-local --bare . clone.git &&\n> > # git rev-parse HEAD >expect &&\n> > # git --git-dir=clone.git rev-parse HEAD >actual &&\n> > # test_cmp expect actual\n> > #\n> > 1..319\n> > -- snap --\n> >\n> > On a hunch, I ran this through valgrind (took a while) but it did not\n> > point out the problem.\n> >\n> > Again, this is only with SHA-256 (and somewhat flaky), it passes every\n> > time with SHA-1. Maybe you can reproduce on your side with that\n> > information?\n>\n> Yeah, I can reproduce it on my side.\n\nGood.\n\n> But I am sure it is not related to the lookup table implementation code.\n> Because when I swap the order of calling `test_midx_bitmap_cases\n> \"pack.writeBitmapLookupTable\"` and `test_midx_bitmap_cases` (in\n> t5326-multi-pack-bitmaps.sh), in that case, the error is being generated\n> in `test_midx_bitmap_cases` call. Generally speaking, the error is\n> always being generated in the second call.\n\nIndeed, it probably has something to do with the test tick (which gives\nrise to the author/committer date of the commits that are generated, and\nhence with the SHA order of said commits).\n\nWith this patch:\n\n-- snip --\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 3b206adcee6..a340f005b89 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -347,7 +347,11 @@ test_midx_bitmap_cases () {\n \t'\n }\n\n-test_midx_bitmap_cases\n+# test_midx_bitmap_cases\n+\n+GIT_COMMITTER_DATE='1112928553 -0700'\n+GIT_AUTHOR_DATE='1112928553 -0700'\n+test_tick='1112928553'\n\n test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n\n-- snap --\n\nI can reproduce it quicker via\n\n\tGIT_TEST_DEFAULT_HASH=sha256 sh t5326-*.sh --run=1,71,91,92,93,124,145\n\nWithout setting those variables, I cannot skip the first\n`test_midx_bitmap_cases` invocation _and_ reproduce the failure.\n\nFor shiggles, I now also ran this command-line after deleting the\n`\"pack.writeBitmapLookupTable\"` argument, and it fails in the exact same\nway. So you're correct: this has nothing to do with the\n`writeBitmapLookupTable` code, it's just a failure that is triggered by\nthose patches.\n\n> For now, my understanding says that there is something fishy in the\n> test script.\n\nI do not actually think so. I believe that this just points out a bug in\nthe MIDX bitmap code.\n\n> I am still not able to figure out the problem here. But let me further\n> investigate.\n>\n> If anyone has some idea about what could be the culprit, I will be\n> very happy to know.\n\nSo I noticed that the test will pass every 4th to 5th time over here,\nwhich means that it is a racy condition that is the culprit.\n\nI dug a bit deeper and reduced the reproducer even further, by running\nthis command with a trash directory just after above test script\ninvocation failed:\n\n\tbin-wrappers/git -C t/trash\\ directory.t5326-multi-pack-bitmaps/ \\\n\t\t-c pack.threads=1 pack-objects --revs --thin --stdout \\\n\t\t--progress --delta-base-offset wrote:\n> > But I am sure it is not related to the lookup table implementation code.\n> > Because when I swap the order of calling `test_midx_bitmap_cases\n> > \"pack.writeBitmapLookupTable\"` and `test_midx_bitmap_cases` (in\n> > t5326-multi-pack-bitmaps.sh), in that case, the error is being generated\n> > in `test_midx_bitmap_cases` call. Generally speaking, the error is\n> > always being generated in the second call.\n>\n> Indeed, it probably has something to do with the test tick (which gives\n> rise to the author/committer date of the commits that are generated, and\n> hence with the SHA order of said commits).\n>\n> With this patch:\n>\n> -- snip --\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> index 3b206adcee6..a340f005b89 100755\n> --- a/t/t5326-multi-pack-bitmaps.sh\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -347,7 +347,11 @@ test_midx_bitmap_cases () {\n> '\n> }\n>\n> -test_midx_bitmap_cases\n> +# test_midx_bitmap_cases\n> +\n> +GIT_COMMITTER_DATE='1112928553 -0700'\n> +GIT_AUTHOR_DATE='1112928553 -0700'\n> +test_tick='1112928553'\n>\n> test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n>\n> -- snap --\n>\n> I can reproduce it quicker via\n>\n> GIT_TEST_DEFAULT_HASH=sha256 sh t5326-*.sh --run=1,71,91,92,93,124,145\n>\n> Without setting those variables, I cannot skip the first\n> `test_midx_bitmap_cases` invocation _and_ reproduce the failure.\n\nYeah, this works for me also.\n\n> > I am still not able to figure out the problem here. But let me further\n> > investigate.\n> >\n> > If anyone has some idea about what could be the culprit, I will be\n> > very happy to know.\n>\n> So I noticed that the test will pass every 4th to 5th time over here,\n> which means that it is a racy condition that is the culprit.\n\nI also encountered the same and it blew my mind at first (because it\nis the first race condition that I faced in my life) :)\n\n> I dug a bit deeper and reduced the reproducer even further, by running\n> this command with a trash directory just after above test script\n> invocation failed:\n>\n> bin-wrappers/git -C t/trash\\ directory.t5326-multi-pack-bitmaps/ \\\n> -c pack.threads=1 pack-objects --revs --thin --stdout \\\n> --progress --delta-base-offset bin-wrappers/git -C t/trash\\ directory.t5326-multi-pack-bitmaps/ \\\n> -c pack.threads=1 index-pack --stdin -v --fix-thin \\\n> '--keep=fetch-pack 12345 on labtop' \\\n> --check-self-contained-and-connected\n>\n> where `/tmp/a5` contains these lines:\n>\n> -- snip --\n> 0ae5a358dcea86d81c0903aaec1e21857688cdb36c7fd89b04bd293fb2cceaa6\n> 67df8a01ac84cf5f028855c48384eac3336bb02a52603bac285c4b31d66b3ab5\n> 098a57f7753320c8a37cf0cb84526a9e50439d9f70fb673c91436a5283a7efe8\n> --not\n> -- snap --\n>\n> This allowed me to instrument the code with _many_ debug printf statements\n> (I actually use `error(\"%s:d: ...\", __FILE__, __LINE__, ...)` calls) to\n> dive deeper into the weeds.\n>\n> One relatively obvious difference I can see is that when the code reaches\n> builtin/pack-objects.c:1198, in the passing case after writing the reused\n> pack we're at offset 900 in the written pack file, but in the failing case\n> we're at offset 269.\n>\n> Another difference I first saw was that the mtime of\n> `.git/objects/pack/multi-pack-index` was identical to the mtime of\n> `.git/objects/pack/multi-pack-index-2ec3c30357d2fff78db9b36cc749b393087e989bffdd278771d6f62089406061.bitmap`\n> in the failing case, while the mtimes of the corresponding files were\n> different in the passing case.\n>\n> But in another failing run, the mtimes were also non-identical. Meaning:\n> the race cannot be caused by identical or non-identical timestamps there.\n>\n> One consistent difference, however, was the SHA-256 in that `.bitmap` file\n> name: In the failing case it was always\n> 2ec3c30357d2fff78db9b36cc749b393087e989bffdd278771d6f62089406061, while in\n> the succeeding case it was always\n> 0c275657a915eeff1f2a1c17e5ded43cc3b232b0e178923e44fc15c1970516fb.\n>\n> My suspicion is that this `.bitmap` file is written out in an earlier test\n> case, and is already incorrect at that stage. Maybe it should have been\n> updated, but isn't, and the result is an incorrectly-reused partial pack\n> file.\n\nI agree with you.\n\n> I also noticed that deleting the `multi-pack-index-*.bitmap` file in the\n> failing case will \"fix\" the `pack-objects | index-pack` command I showed\n> above.\n>\n> Hopefully this will help you dig in further because even if the bug is not\n> in your code, it needs to be fixed. And I suspect that it is a bug in the\n> code we already have in the main branch, so that fix is really, really\n> needed, now.\n\nYeah, definitely! Thanks for all the information! It will truly help\nme to identify the problem.\n\n> Since you are very familiar with the details of bitmaps now, I would like\n> to encourage you to work on some kind of validator/inspector, e.g.\n> something along the lines of a `test-tool midx-bitmap dump` (and later\n> `... verify`) that would help future you (and future me) investigate\n> similar breakages. Ideally, that tool will not only parse the `.bitmap`\n> file but immediately print out everything in a human-readable form.\n>\n> The reason I suggest this: I got a bit tired of staring at the output of\n> `hexdump -C` and comparing it to the documentation in\n> https://git-scm.com/docs/pack-format, so I had to stop after looking too\n> long at one broken pack file (i.e. the output of the `pack-objects`\n> command I showed above, where already the first entry seems to have an\n> infinite delta chain that pretends that\n> 4d332072f161629ffe4652ecd3ce377ef88447bec73f05ab0f3515f98bd061cf has\n> itself as delta base) before I even could analyze the MIDX bitmap files.\n>\n> The proposed tool would make analyzing MIDX bitmaps substantially more\n> fun, and would also help stave off future breakages if it was taught some\n> `verify` mode that would essentially automate what right now has to be\n> done manually: to verify that the MIDX bitmap file contents are sound and\n> consistent with the contents of the pack files.\n>\n> Obviously, this `verify` command should be called in strategic places of\n> t5326.\n\nOk, sure!\n\nThanks :)\n"},{"id":"460815","messageId":"p69r38sn-1ppn-q66q-9089-59394pq78772@tzk.qr","threadId":"58038","inReplyTo":"CAPOJW5yUi471cfAXuXaM4BCzVsfZ15J1Era4NuEpxEnmY6md9Q@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-08T13:06:31Z","receivedAt":"2022-08-08T13:06:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Abhradeep,\n\nOn Tue, 2 Aug 2022, Abhradeep Chakraborty wrote:\n\n> On Tue, Aug 2, 2022 at 9:05 PM Johannes Schindelin\n> wrote:\n>\n> > Since you are very familiar with the details of bitmaps now, I would\n> > like to encourage you to work on some kind of validator/inspector,\n> > e.g. something along the lines of a `test-tool midx-bitmap dump` (and\n> > later `... verify`) that would help future you (and future me)\n> > investigate similar breakages. Ideally, that tool will not only parse\n> > the `.bitmap` file but immediately print out everything in a\n> > human-readable form.\n\nHave you made progress on this? I am interested mostly because I am trying\nvery hard to maintain passing CI runs of Git for Windows' `shears/seen`\nbranch (which essentially tries to rebase all of Git for Windows' patches\non top of `seen`), and this failure is consistently causing said CI runs\nto fail for a while already.\n\nCiao,\nDscho\n"},{"id":"460827","messageId":"CAPOJW5zYndyqwyN8xOcRQnwebqXciY-25hNL3fU=V5ac8fCpNA@mail.gmail.com","threadId":"58038","inReplyTo":"p69r38sn-1ppn-q66q-9089-59394pq78772@tzk.qr","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-08T13:58:07Z","receivedAt":"2022-08-08T13:58:38Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Mon, Aug 8, 2022 at 6:36 PM Johannes Schindelin\n wrote:\n>\n> Hi Abhradeep,\n>\n> On Tue, 2 Aug 2022, Abhradeep Chakraborty wrote:\n>\n> > On Tue, Aug 2, 2022 at 9:05 PM Johannes Schindelin\n> > wrote:\n> >\n> > > Since you are very familiar with the details of bitmaps now, I would\n> > > like to encourage you to work on some kind of validator/inspector,\n> > > e.g. something along the lines of a `test-tool midx-bitmap dump` (and\n> > > later `... verify`) that would help future you (and future me)\n> > > investigate similar breakages. Ideally, that tool will not only parse\n> > > the `.bitmap` file but immediately print out everything in a\n> > > human-readable form.\n>\n> Have you made progress on this? I am interested mostly because I am trying\n> very hard to maintain passing CI runs of Git for Windows' `shears/seen`\n> branch (which essentially tries to rebase all of Git for Windows' patches\n> on top of `seen`), and this failure is consistently causing said CI runs\n> to fail for a while already.\n\nHey Dscho, I am trying hard to solve the issue but unfortunately I\nhaven't found the key yet.\nI investigated the bitmap code-base and used debug lines but didn't\nfind a way to fix it. Sorry for that :|\nI am still trying it.\n\nHope I will be able to share the good news soon. Thanks :)\n> Ciao,\n> Dscho\n"},{"id":"460881","messageId":"s714sq49-o13q-5417-0o21-6397s3646q9o@tzk.qr","threadId":"58038","inReplyTo":"CAPOJW5zYndyqwyN8xOcRQnwebqXciY-25hNL3fU=V5ac8fCpNA@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-09T09:03:02Z","receivedAt":"2022-08-09T09:03:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Abhradeep,\n\nOn Mon, 8 Aug 2022, Abhradeep Chakraborty wrote:\n\n> On Mon, Aug 8, 2022 at 6:36 PM Johannes Schindelin\n> wrote:\n> >\n> > On Tue, 2 Aug 2022, Abhradeep Chakraborty wrote:\n> >\n> > > On Tue, Aug 2, 2022 at 9:05 PM Johannes Schindelin\n> > > wrote:\n> > >\n> > > > Since you are very familiar with the details of bitmaps now, I would\n> > > > like to encourage you to work on some kind of validator/inspector,\n> > > > e.g. something along the lines of a `test-tool midx-bitmap dump` (and\n> > > > later `... verify`) that would help future you (and future me)\n> > > > investigate similar breakages. Ideally, that tool will not only parse\n> > > > the `.bitmap` file but immediately print out everything in a\n> > > > human-readable form.\n> >\n> > Have you made progress on this? I am interested mostly because I am trying\n> > very hard to maintain passing CI runs of Git for Windows' `shears/seen`\n> > branch (which essentially tries to rebase all of Git for Windows' patches\n> > on top of `seen`), and this failure is consistently causing said CI runs\n> > to fail for a while already.\n>\n> Hey Dscho, I am trying hard to solve the issue but unfortunately I\n> haven't found the key yet.\n\nThe tool I proposed could potentially help, in particular with\ndistributing the burden of the investigation on more shoulders than just\nyours.\n\n> I investigated the bitmap code-base and used debug lines but didn't\n> find a way to fix it.\n\nHave you investigated whether the `.bitmap` file was produced for the\nlatest set of pack files? It should be relatively quick to investigate\nthat, and if it turns out not to be the case, the fix should be quick,\ntoo.\n\nThanks,\nDscho\n"},{"id":"460890","messageId":"CAPOJW5yNQvO3quG91jjC9pT-+NNhJta+H_E2R9-1wUzR+rPXnw@mail.gmail.com","threadId":"58038","inReplyTo":"s714sq49-o13q-5417-0o21-6397s3646q9o@tzk.qr","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-09T12:03:50Z","receivedAt":"2022-08-09T12:04:06Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Tue, Aug 9, 2022 at 2:33 PM Johannes Schindelin\n wrote:\n>\n> Hi Abhradeep,\n>\n> On Mon, 8 Aug 2022, Abhradeep Chakraborty wrote:\n>\n> > On Mon, Aug 8, 2022 at 6:36 PM Johannes Schindelin\n> > wrote:\n> > >\n> > > On Tue, 2 Aug 2022, Abhradeep Chakraborty wrote:\n> > >\n> > > > On Tue, Aug 2, 2022 at 9:05 PM Johannes Schindelin\n> > > > wrote:\n> > > >\n> > > > > Since you are very familiar with the details of bitmaps now, I would\n> > > > > like to encourage you to work on some kind of validator/inspector,\n> > > > > e.g. something along the lines of a `test-tool midx-bitmap dump` (and\n> > > > > later `... verify`) that would help future you (and future me)\n> > > > > investigate similar breakages. Ideally, that tool will not only parse\n> > > > > the `.bitmap` file but immediately print out everything in a\n> > > > > human-readable form.\n> > >\n> > > Have you made progress on this? I am interested mostly because I am trying\n> > > very hard to maintain passing CI runs of Git for Windows' `shears/seen`\n> > > branch (which essentially tries to rebase all of Git for Windows' patches\n> > > on top of `seen`), and this failure is consistently causing said CI runs\n> > > to fail for a while already.\n> >\n> > Hey Dscho, I am trying hard to solve the issue but unfortunately I\n> > haven't found the key yet.\n>\n> The tool I proposed could potentially help, in particular with\n> distributing the burden of the investigation on more shoulders than just\n> yours.\n\nYeah, it should. I thought that I would do that after fixing the bug.\nNow I think I was wrong.\n\n> > I investigated the bitmap code-base and used debug lines but didn't\n> > find a way to fix it.\n>\n> Have you investigated whether the `.bitmap` file was produced for the\n> latest set of pack files? It should be relatively quick to investigate\n> that, and if it turns out not to be the case, the fix should be quick,\n> too.\n\nFrankly speaking, I doubt that the generated multi-pack-index file is\nfaulty. The first reason is the `.bitmap` filename. As you said before\n(and as I noticed here), `.bitmap` filenames in failing case and in\npassing case are different. As far as I know the hash value in the\nfilename depends on the content of its respective midx file. So, if\nthe midx contents were the same in both cases, `.bitmap` filename\nshould not differ.\n\nI compared both the multi-pack-index files (i.e. passing case and\nfailing case) using `cmp ./trash\\\ndirectory.t5326-multi-pack-bitmaps/.git/objects/pack/multi-pack-index\n../tmp/trash\\ directory.t5326-multi-pack-bitmaps/.git/objects/pack/multi-pack-index`\nand found that these both defers -\n\n differ: char 3124, line 10\n\nI also checked whether the `packing_data->in_pack_by_idx` contained\nall the packs. For this I wrote a debug error message in\n`prepare_in_pack_by_idx()[1]` function and found that `packing_data`\nis using the latest packs.\n\n I noticed in the 'setup partial bitmaps' test case that if we comment\nout the line `git repack &&` , it runs successfully.\n\n test_expect_success 'setup partial bitmaps' '\n test_commit packed &&\n # git repack &&\n test_commit loose &&\n git multi-pack-index write --bitmap 2>err &&\n ...\n '\n"},{"id":"460891","messageId":"CAPOJW5wKnpTpQTbNY2cGt87ZLb9=2i+=ZdzH9XBiuQHtmMcn+Q@mail.gmail.com","threadId":"58038","inReplyTo":"CAPOJW5yNQvO3quG91jjC9pT-+NNhJta+H_E2R9-1wUzR+rPXnw@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-09T12:07:08Z","receivedAt":"2022-08-09T12:07:25Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Tue, Aug 9, 2022 at 5:33 PM Abhradeep Chakraborty\n wrote:\n> I also checked whether the `packing_data->in_pack_by_idx` contained\n> all the packs. For this I wrote a debug error message in\n> `prepare_in_pack_by_idx()[1]` function and found that `packing_data`\n> is using the latest packs.\n\nLooks like I forgot to specify the link -\nhttps://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/pack-objects.c#L87\n"},{"id":"460968","messageId":"68r08n47-9o07-351s-710q-786q69429q86@tzk.qr","threadId":"58038","inReplyTo":"CAPOJW5yNQvO3quG91jjC9pT-+NNhJta+H_E2R9-1wUzR+rPXnw@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-10T09:09:40Z","receivedAt":"2022-08-10T09:09:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Abhradeep,\n\nOn Tue, 9 Aug 2022, Abhradeep Chakraborty wrote:\n\n> I noticed in the 'setup partial bitmaps' test case that if we comment\n> out the line `git repack &&` , it runs successfully.\n>\n> test_expect_success 'setup partial bitmaps' '\n> test_commit packed &&\n> # git repack &&\n> test_commit loose &&\n> git multi-pack-index write --bitmap 2>err &&\n> ...\n> '\n\nThat's interesting. Are the `.bitmap` and `.midx` files updated as part of\nthat `repack`?\n\nCiao,\nDscho\n"},{"id":"460969","messageId":"4rs1s351-73np-4sq8-p6o8-r7178rp0p0n0@tzk.qr","threadId":"58038","inReplyTo":"68r08n47-9o07-351s-710q-786q69429q86@tzk.qr","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-10T09:20:29Z","receivedAt":"2022-08-10T09:21:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Abhradeep,\n\nOn Wed, 10 Aug 2022, Johannes Schindelin wrote:\n\n> On Tue, 9 Aug 2022, Abhradeep Chakraborty wrote:\n>\n> > I noticed in the 'setup partial bitmaps' test case that if we comment\n> > out the line `git repack &&` , it runs successfully.\n> >\n> > test_expect_success 'setup partial bitmaps' '\n> > test_commit packed &&\n> > # git repack &&\n> > test_commit loose &&\n> > git multi-pack-index write --bitmap 2>err &&\n> > ...\n> > '\n>\n> That's interesting. Are the `.bitmap` and `.midx` files updated as part of\n> that `repack`?\n\nI instrumented this, and saw that the `multi-pack-index` and\n`multi-pack-index*.bitmap` files were unchanged by the `git repack`\ninvocation.\n\nRe-generating the MIDX bitmap forcefully after the repack seems to fix\nthings over here:\n\n-- snip --\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex a95537e759b..564124bda27 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -438,7 +438,10 @@ midx_bitmap_partial_tests () {\n\n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\n+ls -l .git/objects/pack/ &&\n \t\tgit repack &&\n+git multi-pack-index write --bitmap &&\n+ls -l .git/objects/pack/ &&\n \t\ttest_commit loose &&\n \t\tgit multi-pack-index write --bitmap 2>err &&\n \t\ttest_path_is_file $midx &&\n-- snap --\n\nThis suggests to me that the `multi-pack-index write --bitmap 2>err` call\nin this hunk might reuse a stale MIDX bitmap, and that _that_ might be\nthe root cause of this breakage.\n\nWhat do you think?\n\nCiao,\nDscho\n"},{"id":"460975","messageId":"CAPOJW5w2NYbRkFOaqrNYVFkp5ud=aAxhGGV6gpdDPwnyx5TAVw@mail.gmail.com","threadId":"58038","inReplyTo":"4rs1s351-73np-4sq8-p6o8-r7178rp0p0n0@tzk.qr","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-10T10:04:46Z","receivedAt":"2022-08-10T10:05:02Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Wed, Aug 10, 2022 at 2:50 PM Johannes Schindelin\n wrote:\n>\n> Hi Abhradeep,\n> I instrumented this, and saw that the `multi-pack-index` and\n> `multi-pack-index*.bitmap` files were unchanged by the `git repack`\n> invocation.\n\nYeah, those two files remain unchanged here.\n\n> Re-generating the MIDX bitmap forcefully after the repack seems to fix\n> things over here:\n>\n> -- snip --\n> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> index a95537e759b..564124bda27 100644\n> --- a/t/lib-bitmap.sh\n> +++ b/t/lib-bitmap.sh\n> @@ -438,7 +438,10 @@ midx_bitmap_partial_tests () {\n>\n> test_expect_success 'setup partial bitmaps' '\n> test_commit packed &&\n> +ls -l .git/objects/pack/ &&\n> git repack &&\n> +git multi-pack-index write --bitmap &&\n> +ls -l .git/objects/pack/ &&\n> test_commit loose &&\n> git multi-pack-index write --bitmap 2>err &&\n> test_path_is_file $midx &&\n> -- snap --\n>\n> This suggests to me that the `multi-pack-index write --bitmap 2>err` call\n> in this hunk might reuse a stale MIDX bitmap, and that _that_ might be\n> the root cause of this breakage.\n\nYeah, the `multi-pack-index write --bitmap 2>err` is creating the\nproblem. More specifically the `multi-pack-index write` part. As you\ncan see in my previous comment (if you get the comment), I shared a\nscreenshot there which pointed out that the multi-pack-index files in\nboth cases are different. The portion from which it started to differ\nbelongs to the `RIDX` chunk.\n\nSo, I used some debug lines in `midx_pack_order()` function[1] and\nfound that the objects are sorted differently in those cases (i.e.\npassing case and failing case). For passing case, the RIDX chunk\ncontents are like below -\n\npack_order = [ 1, 36, 11, 6, 18, 3, 19, 12, 5, 31, 27, 23, 29, 8, 38,\n22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17,\n32, 0, 21, 16, 25, 13, 40, 20,]\n\nAnd in the failing case, this is -\n\npack_order = [ 12, 18, 3, 19, 1, 36, 11, 6, 5, 31, 27, 23, 29, 8, 38,\n22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17,\n32, 0, 21, 16, 25, 13, 40, 20,]\n\nI went further and realized that this is due to the line[2] -\n\n if (!e->preferred)\n data[i].pack |= (1U << 31);\n\nI.e. 4- 5 `pack_midx_entry` objects have different `preferred` values\nin those cases. For example,\n\"46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b\"\n(with pack order 12) is `preferred` in failing case (that's why it is\nin the first position) and the same is `not preferred` in the passing\ncase.\n\nIt may be because of reusing a stale midx bitmap (as you said). But I\nam not sure. Just to ensure myself, I compared all the other\npackfiles, idx files and a pack `.bitmap` file (which you can see\nusing ls command) of failing and passing cases and found that they are\nthe same.\n\nThanks :)\n\n[1] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/midx.c#L861\n[2] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/midx.c#L872\n"},{"id":"461012","messageId":"805fb0df-45ab-7edd-8787-662b84201e2b@github.com","threadId":"58038","inReplyTo":"CAPOJW5w2NYbRkFOaqrNYVFkp5ud=aAxhGGV6gpdDPwnyx5TAVw@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-10T17:51:27Z","receivedAt":"2022-08-10T17:51:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/10/2022 6:04 AM, Abhradeep Chakraborty wrote:\n> On Wed, Aug 10, 2022 at 2:50 PM Johannes Schindelin\n> wrote:\n>>\n>> Hi Abhradeep,\n>> I instrumented this, and saw that the `multi-pack-index` and\n>> `multi-pack-index*.bitmap` files were unchanged by the `git repack`\n>> invocation.\n> \n> Yeah, those two files remain unchanged here.\n> \n>> Re-generating the MIDX bitmap forcefully after the repack seems to fix\n>> things over here:\n>>\n>> -- snip --\n>> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n>> index a95537e759b..564124bda27 100644\n>> --- a/t/lib-bitmap.sh\n>> +++ b/t/lib-bitmap.sh\n>> @@ -438,7 +438,10 @@ midx_bitmap_partial_tests () {\n>>\n>> test_expect_success 'setup partial bitmaps' '\n>> test_commit packed &&\n>> +ls -l .git/objects/pack/ &&\n>> git repack &&\n>> +git multi-pack-index write --bitmap &&\n>> +ls -l .git/objects/pack/ &&\n>> test_commit loose &&\n>> git multi-pack-index write --bitmap 2>err &&\n>> test_path_is_file $midx &&\n>> -- snap --\n>>\n>> This suggests to me that the `multi-pack-index write --bitmap 2>err` call\n>> in this hunk might reuse a stale MIDX bitmap, and that _that_ might be\n>> the root cause of this breakage.\n> \n> Yeah, the `multi-pack-index write --bitmap 2>err` is creating the\n> problem. More specifically the `multi-pack-index write` part. As you\n> can see in my previous comment (if you get the comment), I shared a\n> screenshot there which pointed out that the multi-pack-index files in\n> both cases are different. The portion from which it started to differ\n> belongs to the `RIDX` chunk.\n> \n> So, I used some debug lines in `midx_pack_order()` function[1] and\n> found that the objects are sorted differently in those cases (i.e.\n> passing case and failing case). For passing case, the RIDX chunk\n> contents are like below -\n> \n> pack_order = [ 1, 36, 11, 6, 18, 3, 19, 12, 5, 31, 27, 23, 29, 8, 38,\n> 22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17,\n> 32, 0, 21, 16, 25, 13, 40, 20,]\n> \n> And in the failing case, this is -\n> \n> pack_order = [ 12, 18, 3, 19, 1, 36, 11, 6, 5, 31, 27, 23, 29, 8, 38,\n> 22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17,\n> 32, 0, 21, 16, 25, 13, 40, 20,]\n> \n> I went further and realized that this is due to the line[2] -\n> \n> if (!e->preferred)\n> data[i].pack |= (1U << 31);\n> \n> I.e. 4- 5 `pack_midx_entry` objects have different `preferred` values\n> in those cases. For example,\n> \"46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b\"\n> (with pack order 12) is `preferred` in failing case (that's why it is\n> in the first position) and the same is `not preferred` in the passing\n> case.\n> \n> It may be because of reusing a stale midx bitmap (as you said). But I\n> am not sure. Just to ensure myself, I compared all the other\n> packfiles, idx files and a pack `.bitmap` file (which you can see\n> using ls command) of failing and passing cases and found that they are\n> the same.\n\nYou are right that this choice of a 'preferred' pack is part of the\nroot cause for this flake. This choice is not deterministic if the\nmtime of some pack-files are within the same second.\n\nI can make the flake go away with this change:\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex a95537e759b0..30347285f10f 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -438,7 +438,9 @@ midx_bitmap_partial_tests () {\n \n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\n+\t\ttest_tick &&\n \t\tgit repack &&\n+\t\ttest_tick &&\n \t\ttest_commit loose &&\n \t\tgit multi-pack-index write --bitmap 2>err &&\n \t\ttest_path_is_file $midx &&\n\n\nHowever, that doesn't help us actually find out what the problem is\nin our case.\n\nI've tried exploring other considerations, resulting in this diff:\n\ndiff --git a/midx.c b/midx.c\nindex 9c26d04bfded..3b9094d55ae5 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -921,8 +921,9 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n+\t\t/* Why does removing the permutation here not change the outcome? */\n \t\toe_set_in_pack(pdata, to,\n-\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t\t\t ctx->info[from->pack_int_id].p);\n \t}\n }\n \nThis method is setting up some important information, supposedly, and\nin the failing case I see that the ctx->pack_perm performs a 5-cycle\n( 0->1, 1->2, 2->3, 3->4, 4->0 ) but this removal does not affect _any\nexisting test cases_!\n\nTurns out that the packfile sent here goes through this very trivial\npath in oe_set_in_pack() every time we are writing a multi-pack-index:\n\nstatic inline void oe_set_in_pack(struct packing_data *pack,\n\t\t\t\t struct object_entry *e,\n\t\t\t\t struct packed_git *p)\n{\n\tif (pack->in_pack_by_idx) {\n\t\tif (p->index) {\n\t\t\te->in_pack_idx = p->index;\n\t\t\treturn;\n\t\t}\n\t\t/*\n\t\t * We're accessing packs by index, but this pack doesn't have\n\t\t * an index (e.g., because it was added since we created the\n\t\t * in_pack_by_idx array). Bail to oe_map_new_pack(), which\n\t\t * will convert us to using the full in_pack array, and then\n\t\t * fall through to our in_pack handling.\n\t\t */\n\t\toe_map_new_pack(pack);\n\t}\n\tpack->in_pack[e - pack->objects] = p;\n}\n\nBy debugging, I discovered we are hitting the case that calls\noe_map_new_pack(pack). The documentation for that method provides\nthe following (**emphasis mine**):\n\n/*\n * A new pack appears after prepare_in_pack_by_idx() has been\n * run. **This is likely a race.**\n *\n * We could map this new pack to in_pack_by_idx[] array, but then we\n * have to deal with full array anyway. And since it's hard to test\n * this fall back code, just stay simple and fall back to using\n * in_pack[] array.\n */\nvoid oe_map_new_pack(struct packing_data *pack)\n\nThe issue being that prepare_packing_data() uses get_all_packs() to\nget the list of pack-files, but that list is stale for some reason.\nAdding a reprepare_packed_git() in advance of that call also removes\nthe flake (with always passing):\n\ndiff --git a/midx.c b/midx.c\nindex 9c26d04bfded..48db91d2728a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -915,6 +915,7 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \tuint32_t i;\n \n \tmemset(pdata, 0, sizeof(struct packing_data));\n+\treprepare_packed_git(the_repository);\n \tprepare_packing_data(the_repository, pdata);\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n\nBut this still appears like it is just a band-aid over a trickier\nunderlying issue.\n\nHopefully my rambling helps push you in a helpful direction to find\na more complete fix.\n\nThanks,\n-Stolee\n"},{"id":"461132","messageId":"CAPOJW5x0coFREUPjFbF_zzQYbfEjOrL-j-G4N7MBUN4N6uS2jw@mail.gmail.com","threadId":"58038","inReplyTo":"805fb0df-45ab-7edd-8787-662b84201e2b@github.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-12T18:51:42Z","receivedAt":"2022-08-12T18:52:00Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Hello,\n\nI think I have found the problem. Derrick was right that `mtime` part\nis not the culprit. I tried to understand the whole midx workflow and\nsome questions were raised in my mind. I don't know whether those are\nfeatures or bugs (because I do not have much experience in\nmulti-pack-index code).\n\nI am writing a brief description for the context of the issue and the\nquestions I have.\n\nLet us start from the `write_midx_internal()` function. As\n`packs_to_include` is null in our case, We can use the old midx to\nwrite a new midx file. The line `ctx.m =\nlookup_multi_pack_index(the_repository, object_dir)`[1] does this. It\nalso loads packs that do not belong to any multi-pack-indexes. It also\nsets `the_repository->objects->packed_git_intialized` to 1. If we\nlook at our test case (`setup partial bitmap`) the last `.pack` file\n(generated by `git repack &&` ) does not belong to any midx. So, that\npack will be loaded in this step.\n\n[1] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L1169\n\nNext let us move to the `if (ctx.m)`[2] block. As we will be writing a\nbitmap, `if (flags & MIDX_WRITE_REV_INDEX)` is true. Thus all packs\nrelated to the old midx are loaded and `ctx.info[ctx.nr].p` stores the\npointers of these packs.\n\n[2] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L1182\n\nAfter that we come to the `for_each_file_in_pack_dir(object_dir,\nadd_pack_to_midx, &ctx);` line[3] . The `add_pack_to_midx`[4] function\nadds packs (that are not in the old midx) to `ctx.info`. Now I have a\nquestion here - Why are we using the `add_packed_git()`[5] function\nprovided we already loaded those packs in the\n`lookup_multi_pack_index` step (i.e. 1st step)? These packs are not\nadded in `r->objects->packed_git`. This question is related to our\ncurrent issue.\n\nI.e. instead of this -\n\n ctx->info[ctx->nr].p = add_packed_git(full_path,\n\nfull_path_len, 0);\n\nWhy not this (or similar) -\n\n for (cp = the_repository->objects->packed_git; cp; cp = cp->next)\n if (!cmp_idx_or_pack_name(cp->pack_name, full_path))\n ctx->info[ctx->nr].p = cp;\n\n[3] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L1221\n[4] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L462\n[5] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L492\n\n `write_midx_bitmap()` function is where bitmap related code starts.\nlet us directly jump into the `prepare_packed_git()` function (called\nby `get_all_packs()`[6]). As I said previously,\n`r->objects->packed_git_initialized` is already enabled so this\nfunction becomes a no-op function. Which means it does not load the\nnewly written midx (by calling `prepare_multi_pack_index_one`\nfunction) and uses old midx to write the bitmap (though we still have\nnew packs and they can be used with the old midx to generate the\nbitmap, maybe?) . Here comes my second question - Is this the desired\ncase? or should we use the new midx to write the bitmaps?\n\nOne important point to note is that `get_all_packs()` returns\n`r->objects->packed_git` which now stores pointers of all the packs\nand only these packfiles have their `->index` set.\n\n[6] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/packfile.c#L1043\n\nNow let us move to the last function - `oe_set_in_pack()` (called by\n`prepare_midx_packing_data()`). Note that, we are passing\n`ctx->info[ctx->pack_perm[from->pack_int_id]].p` along with other\nparameters. As I have said in an earlier para (containing my first\nquestion), `ctx->info` has some packs (i.e. newer packs that are not\nrelated to the old midx) that are not installed in\n`r->objects->packed_git` . In other words, we have two instances of\nthe same pack file - one in `r->objects->packed_git` list and another\nin `ctx->info[id].p`. As `prepare_in_pack_by_idx` function only sets\n`->index` for `r->objects->packed_git` packs, these packs (i.e.\n`ctx.info[id].p`) do not have their p->index set and thus end up\ncalling the `oe_map_new_pack` function.\n"},{"id":"461134","messageId":"179c0d30-ccb1-36cf-f783-814c9c8d84c2@github.com","threadId":"58038","inReplyTo":"CAPOJW5x0coFREUPjFbF_zzQYbfEjOrL-j-G4N7MBUN4N6uS2jw@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-12T19:22:40Z","receivedAt":"2022-08-12T19:22:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/12/2022 2:51 PM, Abhradeep Chakraborty wrote:\n\n> I think I have found the problem. Derrick was right that `mtime` part\n> is not the culprit. I tried to understand the whole midx workflow and\n> some questions were raised in my mind. I don't know whether those are\n> features or bugs (because I do not have much experience in\n> multi-pack-index code).\n> \n> I am writing a brief description for the context of the issue and the\n> questions I have.\n\nThanks for the detailed writeup.\n\n> Let us start from the `write_midx_internal()` function. As\n> `packs_to_include` is null in our case, We can use the old midx to\n> write a new midx file. The line `ctx.m =\n> lookup_multi_pack_index(the_repository, object_dir)`[1] does this. It\n> also loads packs that do not belong to any multi-pack-indexes. It also\n> sets `the_repository->objects->packed_git_intialized` to 1. If we\n> look at our test case (`setup partial bitmap`) the last `.pack` file\n> (generated by `git repack &&` ) does not belong to any midx. So, that\n> pack will be loaded in this step.\n> \n> [1] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L1169\n> \n> Next let us move to the `if (ctx.m)`[2] block. As we will be writing a\n> bitmap, `if (flags & MIDX_WRITE_REV_INDEX)` is true. Thus all packs\n> related to the old midx are loaded and `ctx.info[ctx.nr].p` stores the\n> pointers of these packs.\n> \n> [2] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L1182\n> \n> After that we come to the `for_each_file_in_pack_dir(object_dir,\n> add_pack_to_midx, &ctx);` line[3] . The `add_pack_to_midx`[4] function\n> adds packs (that are not in the old midx) to `ctx.info`. Now I have a\n> question here - Why are we using the `add_packed_git()`[5] function\n> provided we already loaded those packs in the\n> `lookup_multi_pack_index` step (i.e. 1st step)? These packs are not\n> added in `r->objects->packed_git`. This question is related to our\n> current issue.\n> \n> I.e. instead of this -\n> \n> ctx->info[ctx->nr].p = add_packed_git(full_path,\n> \n> full_path_len, 0);\n> \n> Why not this (or similar) -\n> \n> for (cp = the_repository->objects->packed_git; cp; cp = cp->next)\n> if (!cmp_idx_or_pack_name(cp->pack_name, full_path))\n> ctx->info[ctx->nr].p = cp;\n> \n> [3] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L1221\n> [4] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L462\n> [5] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/midx.c#L492\n> \n> `write_midx_bitmap()` function is where bitmap related code starts.\n> let us directly jump into the `prepare_packed_git()` function (called\n> by `get_all_packs()`[6]). As I said previously,\n> `r->objects->packed_git_initialized` is already enabled so this\n> function becomes a no-op function. Which means it does not load the\n> newly written midx (by calling `prepare_multi_pack_index_one`\n> function) and uses old midx to write the bitmap (though we still have\n> new packs and they can be used with the old midx to generate the\n> bitmap, maybe?) . Here comes my second question - Is this the desired\n> case? or should we use the new midx to write the bitmaps?\n\nThe confusing part of all this is that the bitmaps are being written\nwhile the \"new\" midx is written only to \"multi-pack-index.lock\" and\nhas not been renamed to \"multi-pack-index\". If we renamed first, then\nthe old .bitmap file would not match the new midx and all Git commands\nwould act as if there was no .bitmap file.\n \n> One important point to note is that `get_all_packs()` returns\n> `r->objects->packed_git` which now stores pointers of all the packs\n> and only these packfiles have their `->index` set.\n> \n> [6] https://github.com/git/git/blob/5502f77b6944eda8e26813d8f542cffe7d110aea/packfile.c#L1043\n> \n> Now let us move to the last function - `oe_set_in_pack()` (called by\n> `prepare_midx_packing_data()`). Note that, we are passing\n> `ctx->info[ctx->pack_perm[from->pack_int_id]].p` along with other\n> parameters. As I have said in an earlier para (containing my first\n> question), `ctx->info` has some packs (i.e. newer packs that are not\n> related to the old midx) that are not installed in\n> `r->objects->packed_git` . In other words, we have two instances of\n> the same pack file - one in `r->objects->packed_git` list and another\n> in `ctx->info[id].p`. As `prepare_in_pack_by_idx` function only sets\n> `->index` for `r->objects->packed_git` packs, these packs (i.e.\n> `ctx.info[id].p`) do not have their p->index set and thus end up\n> calling the `oe_map_new_pack` function.\n\nSo really, the problem is that we are handling the r->objects->packed_git\nlist instead of an array of packs that are under the control of the new\nmidx. This assumption is baked deep in the pack-objects flow, so it\nwould be hard to separate this idea.\n\nPerhaps doing the reprepare_packed_git() to regenerate the list would be\nsufficient as a band-aid for now, but we would want to later do the big\ndig of focusing the pack_data struct to a specific list of pack-files\n(by default the set from get_all_packs(), but for midx bitmaps we can\nsupply a specific set of packs).\n\nThanks,\n-Stolee\n"},{"id":"461170","messageId":"CAPOJW5z99b0_NGBYDbZUvmzbWECJKxGvB4RffoPJYszfFB0cEg@mail.gmail.com","threadId":"58038","inReplyTo":"179c0d30-ccb1-36cf-f783-814c9c8d84c2@github.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-13T10:59:32Z","receivedAt":"2022-08-13T10:59:52Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Sat, Aug 13, 2022 at 12:52 AM Derrick Stolee\n wrote:\n>\n> So really, the problem is that we are handling the r->objects->packed_git\n> list instead of an array of packs that are under the control of the new\n> midx. This assumption is baked deep in the pack-objects flow, so it\n> would be hard to separate this idea.\n>\n> Perhaps doing the reprepare_packed_git() to regenerate the list would be\n> sufficient as a band-aid for now, but we would want to later do the big\n> dig of focusing the pack_data struct to a specific list of pack-files\n> (by default the set from get_all_packs(), but for midx bitmaps we can\n> supply a specific set of packs).\n\n`reprepare_packed_git()` can not stop it. Because this function\nupdates `r->objects->packed_git` list (i.e. it reloads packs that are\nnot in the old midx) and as I said before, we are setting `->index`\nfor only r->objects->packed_git not ctx.info[id].p. So, it will call\nthe `oe_map_new_pack()` function in either way. I have tested it.\n\nOne thing that really worries me is what if the failure is not related\nto calling `oe_map_new_pack()? I did all my work assuming that this\nfunction is the culprit. But I don't know if it is.\n"},{"id":"461171","messageId":"CAPOJW5yLQykokmg-MtQDwu69PtEWq+6BqiaHcP4AMn3f5E9WtQ@mail.gmail.com","threadId":"58038","inReplyTo":"4rs1s351-73np-4sq8-p6o8-r7178rp0p0n0@tzk.qr","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-13T11:05:22Z","receivedAt":"2022-08-13T11:05:45Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Wed, Aug 10, 2022 at 2:50 PM Johannes Schindelin\n wrote:\n>\n> Hi Abhradeep,\n>\n> On Wed, 10 Aug 2022, Johannes Schindelin wrote:\n>\n> > On Tue, 9 Aug 2022, Abhradeep Chakraborty wrote:\n> >\n> > > I noticed in the 'setup partial bitmaps' test case that if we comment\n> > > out the line `git repack &&` , it runs successfully.\n> > >\n> > > test_expect_success 'setup partial bitmaps' '\n> > > test_commit packed &&\n> > > # git repack &&\n> > > test_commit loose &&\n> > > git multi-pack-index write --bitmap 2>err &&\n> > > ...\n> > > '\n> >\n> > That's interesting. Are the `.bitmap` and `.midx` files updated as part of\n> > that `repack`?\n>\n> I instrumented this, and saw that the `multi-pack-index` and\n> `multi-pack-index*.bitmap` files were unchanged by the `git repack`\n> invocation.\n>\n> Re-generating the MIDX bitmap forcefully after the repack seems to fix\n> things over here:\n>\n> -- snip --\n> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> index a95537e759b..564124bda27 100644\n> --- a/t/lib-bitmap.sh\n> +++ b/t/lib-bitmap.sh\n> @@ -438,7 +438,10 @@ midx_bitmap_partial_tests () {\n>\n> test_expect_success 'setup partial bitmaps' '\n> test_commit packed &&\n> +ls -l .git/objects/pack/ &&\n> git repack &&\n> +git multi-pack-index write --bitmap &&\n> +ls -l .git/objects/pack/ &&\n> test_commit loose &&\n> git multi-pack-index write --bitmap 2>err &&\n> test_path_is_file $midx &&\n> -- snap --\n>\n> This suggests to me that the `multi-pack-index write --bitmap 2>err` call\n> in this hunk might reuse a stale MIDX bitmap, and that _that_ might be\n> the root cause of this breakage.\n>\n> What do you think?\n\nHi Dscho,\nI used your code to see if it is the case but it doesn't affect the\nresult (at least in my laptop).\n"},{"id":"461192","messageId":"92ca58fbeeb0ac74a411fc2e67fcbceccc819883.1660496112.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"[PATCH v6 2/6] bitmap: move `get commit positions` code to `bitmap_writer_finish`","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-14T16:55:07Z","receivedAt":"2022-08-14T17:00:10Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe `write_selected_commits_v1` function takes care of writing commit\npositions along with their corresponding bitmaps in the disk. It is\nOK because this `search commit position of a given commit` algorithm\nis needed only once here. But in later changes of the `lookup table\nextension series`, we need same commit positions which means we have\nto run the above mentioned algorithm one more time.\n\nMove the `search commit position of a given commit` algorithm to\n`bitmap_writer_finish()` and use the `commit_positions` array\nto get commit positions of their corresponding bitmaps.\n\nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 29 ++++++++++++++++++++---------\n 1 file changed, 20 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 4fcfaed428f..9b1be59f6d3 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -650,20 +650,15 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n-\t\t\t\t uint32_t index_nr)\n+\t\t\t\t uint32_t index_nr,\n+\t\t\t\t uint32_t *commit_positions)\n {\n \tint i;\n \n \tfor (i = 0; i < writer.selected_nr; ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n-\t\tint commit_pos =\n-\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n-\n-\t\tif (commit_pos < 0)\n-\t\t\tBUG(\"trying to write commit not in index\");\n-\n-\t\thashwrite_be32(f, commit_pos);\n+\t\thashwrite_be32(f, commit_positions[i]);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n \n@@ -697,6 +692,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n+\tuint32_t *commit_positions = NULL;\n+\tuint32_t i;\n \n \tstruct bitmap_disk_header header;\n \n@@ -715,7 +712,20 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n-\twrite_selected_commits_v1(f, index, index_nr);\n+\n+\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *stored = &writer.selected[i];\n+\t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\n+\t\tif (commit_pos < 0)\n+\t\t\tBUG(_(\"trying to write commit not in index\"));\n+\n+\t\tcommit_positions[i] = commit_pos;\n+\t}\n+\n+\twrite_selected_commits_v1(f, index, index_nr, commit_positions);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n@@ -730,4 +740,5 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n+\tfree(commit_positions);\n }\n-- \ngitgitgadget\n\n"},{"id":"461193","messageId":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v5.git.1658342304.gitgitgadget@gmail.com","subject":"[PATCH v6 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-14T16:55:05Z","receivedAt":"2022-08-14T17:00:12Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"When parsing the .bitmap file, git loads all the bitmaps one by one even if\nsome of the bitmaps are not necessary. We can remove this overhead by\nloading only the necessary bitmaps. A look up table extension can solve this\nissue.\n\nChanges since v5:\n\nAs the failure in the test case is not due to this code, I think it makes no\nsense to delay the patch further.\n\n * The performance test changes were not accurate as the second\n test_bitmap_cases call using the repo built for the previous call. This\n version fixes that.\n * Taylor suggested some minor changes. Those are addressed in this version.\n\nChanges since v4:\n\n * There was a CI failing test for linux-sha256 in the previous version.\n Fixed now.\n\nChanges since v3:\n\n * The common code from both lookup_table_get_triplet() and\n bsearch_triplet_by_pos are moved to lookup_table_get_triplet_by_pointer\n function\n * parameter names of triplet_cmp function is changes (as suggested by\n Martin)\n * xor_items array is now work as reusable static buffer.\n * I moved the filling commit_positions array part (from\n pack-bitmap-write.c) to bitmap_writer_finish function. Because we had to\n iterate two times for commit positions - one in write_selected_commits_v1\n and another in write_lookup_table function. Hope this is acceptable :)\n * changes in performance tests (as suggested by Taylor)\n\nChanges since v2:\n\n * Log messages related issues are fixed.\n * pack.writeBitmapLookupTable is now by default disabled.\n * Documentations are improved.\n * xor_row is used instead of xor_pos in triplets.\n * In pack-bitmap-write.c, off_t * is used for offsets array (Instead of\n uint64_t *).\n * struct bitmap_lookup_table_triplet is introduced and functions Like\n triplet_get_offset() and triplet_get_xor_pos() are removed.\n * table_size is getting subtracted from index_end irrespective of the value\n of GIT_TEST_READ_COMMIT_TABLE.\n * xor stack filling loop will stop iterating if a xor bitmap is already\n stored/parsed.\n * The stack will now store bitmap_lookup_table_xor_item items Of plain\n xor_row.\n * bitmap related test files are reformatted to allow repeating of tests\n with bitmap extension enabled.\n * comments are added.\n\nChanges since v1:\n\nThis is the second version which addressed all (I think) the reviews. Please\nnotify me if some reviews are not addressed :)\n\n * The table size is decreased and the format has also changed. It now\n contains nr_entries triplets of size 4+8+4 bytes. Each triplet contains\n the following things - (1) 4 byte commit position (in the pack-index or\n midx) (2) 8 byte offset and (3) 4 byte xor triplet (i.e. with whose\n bitmap the current triplet's bitmap has to xor) position.\n * Performance tests are splitted into two commits. First contains the\n actual performance tests and second enables the pack.writeReverseIndex\n (as suggested by Taylor).\n * st_*() functions are used.\n * commit order is changed according to Derrick's suggestion.\n * Iterative approach is used instead of recursive approach to parse xor\n bitmaps. (As suggested by Derrick).\n * Some minor bug fixes of previous version.\n\nInitial version:\n\nThe proposed table has:\n\n * a list of nr_entries object ids. These objects are commits that has\n bitmaps. Ids are stored in lexicographic order (for better searching).\n * a list of pairs (4-byte integers, network-byte\n order). The i'th pair denotes the offset and xor-offset(respectively) of\n the bitmap of i'th commit in the previous list. These two informations\n are necessary because only in this way bitmaps can be found without\n parsing all the bitmap.\n * a 4-byte integer for table specific flags (none exists currently).\n\nWhenever git want to parse the bitmap for a specific commit, it will first\nrefer to the table and will look for the offset and xor-offset for that\ncommit. Git will then try to parse the bitmap located at the offset\nposition. The xor-offset can be used to find the xor-bitmap for the\nbitmap(if any).\n\nAbhradeep Chakraborty (6):\n Documentation/technical: describe bitmap lookup table extension\n bitmap: move `get commit positions` code to `bitmap_writer_finish`\n pack-bitmap-write.c: write lookup table extension\n pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n pack-bitmap: prepare to read lookup table extension\n bitmap-lookup-table: add performance tests for lookup table\n\n Documentation/config/pack.txt | 7 +\n Documentation/technical/bitmap-format.txt | 39 ++\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n pack-bitmap-write.c | 114 +++-\n pack-bitmap.c | 290 +++++++-\n pack-bitmap.h | 14 +-\n t/perf/lib-bitmap.sh | 31 +\n t/perf/p5310-pack-bitmaps.sh | 78 +--\n t/perf/p5311-pack-bitmaps-fetch.sh | 74 +-\n t/perf/p5312-pack-bitmaps-revs.sh | 35 +\n t/perf/p5326-multi-pack-bitmaps.sh | 103 +--\n t/t5310-pack-bitmaps.sh | 786 ++++++++++++----------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++-----\n t/t5327-multi-pack-bitmaps-rev.sh | 24 +-\n 18 files changed, 1378 insertions(+), 710 deletions(-)\n create mode 100755 t/perf/p5312-pack-bitmaps-revs.sh\n\n\nbase-commit: afa70145a25e81faa685dc0b465e52b45d2444bd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1266%2FAbhra303%2Fbitmap-commit-table-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1266/Abhra303/bitmap-commit-table-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1266\n\nRange-diff vs v5:\n\n 1: 33aca8f3dc8 ! 1: 67b71be8c85 Documentation/technical: describe bitmap lookup table extension\n @@ Commit message\n \n ## Documentation/technical/bitmap-format.txt ##\n @@ Documentation/technical/bitmap-format.txt: MIDXs, both the bit-cache and rev-cache extensions are required.\n - \t\t\tpack/MIDX. The format and meaning of the name-hash is\n - \t\t\tdescribed below.\n + \t pack/MIDX. The format and meaning of the name-hash is\n + \t described below.\n \n -+\t\t\t** {empty}\n -+\t\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n -+\t\t\tIf present, the end of the bitmap file contains a table\n -+\t\t\tcontaining a list of `N` \n -+\t\t\ttriplets. The format and meaning of the table is described\n -+\t\t\tbelow.\n ++\t\t** {empty}\n ++\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n ++\t\tIf present, the end of the bitmap file contains a table\n ++\t\tcontaining a list of `N` \n ++\t\ttriplets. The format and meaning of the table is described\n ++\t\tbelow.\n ++\n +NOTE: Unlike the xor_offset used to compress an individual bitmap,\n +`xor_row` stores an *absolute* index into the lookup table, not a location\n +relative to the current entry.\n +\n - \t\t4-byte entry count (network byte order)\n + \t4-byte entry count (network byte order): ::\n + \t The total count of entries (bitmapped commits) in this bitmap index.\n \n - \t\t\tThe total count of entries (bitmapped commits) in this bitmap index.\n @@ Documentation/technical/bitmap-format.txt: Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n -: ----------- > 2: 92ca58fbeeb bitmap: move `get commit positions` code to `bitmap_writer_finish`\n 2: a913e6a2cb3 ! 3: 090becaabe0 pack-bitmap-write.c: write lookup table extension\n @@ Commit message\n \n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: static const struct object_id *oid_access(size_t pos, const void *table)\n - \n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n --\t\t\t\t uint32_t index_nr)\n -+\t\t\t\t uint32_t index_nr,\n -+\t\t\t\t off_t *offsets,\n -+\t\t\t\t uint32_t *commit_positions)\n + \t\t\t\t uint32_t index_nr,\n +-\t\t\t\t uint32_t *commit_positions)\n ++\t\t\t\t uint32_t *commit_positions,\n ++\t\t\t\t off_t *offsets)\n {\n \tint i;\n \n \tfor (i = 0; i < writer.selected_nr; ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n --\t\tint commit_pos =\n --\t\t\toid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n +\t\tif (offsets)\n +\t\t\toffsets[i] = hashfile_total(f);\n - \n --\t\tif (commit_pos < 0)\n --\t\t\tBUG(\"trying to write commit not in index\");\n --\n --\t\thashwrite_be32(f, commit_pos);\n -+\t\thashwrite_be32(f, commit_positions[i]);\n ++\n + \t\thashwrite_be32(f, commit_positions[i]);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n - \n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +static void write_lookup_table(struct hashfile *f,\n +\t\t\t struct pack_idx_entry **index,\n +\t\t\t uint32_t index_nr,\n -+\t\t\t off_t *offsets,\n -+\t\t\t uint32_t *commit_positions)\n ++\t\t\t uint32_t *commit_positions,\n ++\t\t\t off_t *offsets)\n +{\n +\tuint32_t i;\n +\tuint32_t *table, *table_inv;\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n - {\n - \tstatic uint16_t default_version = 1;\n - \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n -+\toff_t *offsets = NULL;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n -+\tuint32_t *commit_positions = NULL;\n + \tuint32_t *commit_positions = NULL;\n ++\toff_t *offsets = NULL;\n + \tuint32_t i;\n \n \tstruct bitmap_disk_header header;\n - \n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n - \tdump_bitmap(f, writer.trees);\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n --\twrite_selected_commits_v1(f, index, index_nr);\n -+\n -+\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n -+\tfor (uint32_t i = 0; i < writer.selected_nr; ++i) {\n -+\t\tstruct bitmapped_commit *stored = &writer.selected[i];\n -+\t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n -+\n -+\t\tif (commit_pos < 0)\n -+\t\t\tBUG(_(\"trying to write commit not in index\"));\n -+\n -+\t\tcommit_positions[i] = commit_pos;\n -+\t}\n -+\n + \n +\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n +\t\tCALLOC_ARRAY(offsets, index_nr);\n +\n -+\twrite_selected_commits_v1(f, index, index_nr, offsets, commit_positions);\n + \tALLOC_ARRAY(commit_positions, writer.selected_nr);\n + \n + \tfor (i = 0; i < writer.selected_nr; i++) {\n +@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n + \t\tcommit_positions[i] = commit_pos;\n + \t}\n + \n +-\twrite_selected_commits_v1(f, index, index_nr, commit_positions);\n ++\twrite_selected_commits_v1(f, index, index_nr, commit_positions, offsets);\n +\n +\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n -+\t\twrite_lookup_table(f, index, index_nr, offsets, commit_positions);\n ++\t\twrite_lookup_table(f, index, index_nr, commit_positions, offsets);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n - \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n \n \tstrbuf_release(&tmp_file);\n + \tfree(commit_positions);\n +\tfree(offsets);\n -+\tfree(commit_positions);\n }\n \n ## pack-bitmap.h ##\n 3: 59b465e5a78 ! 4: b2b7c5c1703 pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests\n @@ builtin/pack-objects.c: static int git_pack_config(const char *k, const char *v,\n \t\treturn 0;\n \n ## midx.c ##\n -@@ midx.c: static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n +@@ midx.c: static int write_midx_bitmap(const char *midx_name,\n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n \n +\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n +\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n +\n - \tprepare_midx_packing_data(&pdata, ctx);\n - \n - \tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\n + \t/*\n + \t * Build the MIDX-order index based on pdata.objects (which is already\n + \t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n \n ## midx.h ##\n @@ midx.h: struct multi_pack_index {\n @@ t/t5327-multi-pack-bitmaps-rev.sh: GIT_TEST_MIDX_READ_RIDX=0\n -midx_bitmap_core rev\n -midx_bitmap_partial_tests rev\n +test_midx_bitmap_rev () {\n -+ writeLookupTable=false\n -+\n -+ \tfor i in \"$@\"\n -+ \tdo\n -+ \t\tcase $i in\n -+ \t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n -+ \t\tesac\n -+ \tdone\n -+\n -+ test_expect_success 'setup bitmap config' '\n -+ rm -rf * .git &&\n -+ git init &&\n -+ git config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n -+ '\n -+\n -+ midx_bitmap_core rev\n -+ midx_bitmap_partial_tests rev\n -+ }\n -+\n -+ test_midx_bitmap_rev\n -+ test_midx_bitmap_rev \"pack.writeBitmapLookupTable\"\n ++\twriteLookupTable=false\n ++\n ++\tfor i in \"$@\"\n ++\tdo\n ++\t\tcase $i in\n ++\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n ++\t\tesac\n ++\tdone\n ++\n ++\ttest_expect_success 'setup bitmap config' '\n ++\t\trm -rf * .git &&\n ++\t\tgit init &&\n ++\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n ++\t'\n ++\n ++\tmidx_bitmap_core rev\n ++\tmidx_bitmap_partial_tests rev\n ++}\n ++\n ++test_midx_bitmap_rev\n ++test_midx_bitmap_rev \"pack.writeBitmapLookupTable\"\n \n test_done\n 4: 6918f0860ad ! 5: 79842ca590c pack-bitmap: prepare to read lookup table extension\n @@ pack-bitmap.c: static struct stored_bitmap *store_bitmap(struct bitmap_index *in\n +\t * shouldn't be duplicated commits in the index.\n +\t */\n \tif (ret == 0) {\n --\t\terror(\"Duplicate entry in bitmap index: %s\", oid_to_hex(oid));\n -+\t\terror(_(\"duplicate entry in bitmap index: %s\"), oid_to_hex(oid));\n + \t\terror(_(\"duplicate entry in bitmap index: '%s'\"), oid_to_hex(oid));\n \t\treturn NULL;\n - \t}\n - \n @@ pack-bitmap.c: static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n @@ pack-bitmap.c: struct include_data {\n + * Note that this function assumes that there is enough memory\n + * left for filling the `triplet` struct from `p`.\n + */\n -+static int lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n -+\t\t\t\t\t const unsigned char *p)\n ++static int bitmap_lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n ++\t\t\t\t\t\t const unsigned char *p)\n +{\n +\tif (!triplet)\n +\t\treturn -1;\n @@ pack-bitmap.c: struct include_data {\n + * This function gets the raw triplet from `row`'th row in the\n + * lookup table and fills that data to the `triplet`.\n + */\n -+static int lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n -+\t\t\t\t uint32_t pos,\n -+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n ++static int bitmap_lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n ++\t\t\t\t\t uint32_t pos,\n ++\t\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n +{\n +\tunsigned char *p = NULL;\n +\tif (pos >= bitmap_git->entry_count)\n @@ pack-bitmap.c: struct include_data {\n +\n +\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n +\n -+\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n ++\treturn bitmap_lookup_table_get_triplet_by_pointer(triplet, p);\n +}\n +\n +/*\n @@ pack-bitmap.c: struct include_data {\n +\treturn 0;\n +}\n +\n -+static uint32_t bsearch_pos(struct bitmap_index *bitmap_git,\n ++static uint32_t bitmap_bsearch_pos(struct bitmap_index *bitmap_git,\n +\t\t\t struct object_id *oid,\n +\t\t\t uint32_t *result)\n +{\n @@ pack-bitmap.c: struct include_data {\n + * object from the raw triplet. Returns 1 on success and 0 on\n + * failure.\n + */\n -+static int bsearch_triplet_by_pos(uint32_t commit_pos,\n ++static int bitmap_bsearch_triplet_by_pos(uint32_t commit_pos,\n +\t\t\t\t struct bitmap_index *bitmap_git,\n +\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n +{\n @@ pack-bitmap.c: struct include_data {\n +\tif (!p)\n +\t\treturn -1;\n +\n -+\treturn lookup_table_get_triplet_by_pointer(triplet, p);\n ++\treturn bitmap_lookup_table_get_triplet_by_pointer(triplet, p);\n +}\n +\n +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n -+\t\t\t\t\t struct commit *commit)\n ++\t\t\t\t\t\t struct commit *commit)\n +{\n +\tuint32_t commit_pos, xor_row;\n +\tuint64_t offset;\n -+\tint flags, found;\n ++\tint flags;\n +\tstruct bitmap_lookup_table_triplet triplet;\n +\tstruct object_id *oid = &commit->object.oid;\n +\tstruct ewah_bitmap *bitmap;\n @@ pack-bitmap.c: struct include_data {\n +\tstatic struct bitmap_lookup_table_xor_item *xor_items = NULL;\n +\tstatic size_t xor_items_nr = 0, xor_items_alloc = 0;\n +\tstatic int is_corrupt = 0;\n ++\tint xor_flags;\n ++\tkhiter_t hash_pos;\n ++\tstruct bitmap_lookup_table_xor_item *xor_item;\n +\n +\tif (is_corrupt)\n +\t\treturn NULL;\n +\n -+\tfound = bsearch_pos(bitmap_git, oid, &commit_pos);\n -+\n -+\tif (!found)\n ++\tif (!bitmap_bsearch_pos(bitmap_git, oid, &commit_pos))\n +\t\treturn NULL;\n +\n -+\tif (bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n ++\tif (bitmap_bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n +\t\treturn NULL;\n +\n +\txor_items_nr = 0;\n +\toffset = triplet.offset;\n +\txor_row = triplet.xor_row;\n +\n -+\tif (xor_row != 0xffffffff) {\n -+\t\tint xor_flags;\n -+\t\tkhiter_t hash_pos;\n -+\t\tstruct bitmap_lookup_table_xor_item *xor_item;\n -+\n -+\t\twhile (xor_row != 0xffffffff) {\n -+\t\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n -+\n -+\t\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n -+\t\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n -+\t\t\t\tgoto corrupt;\n -+\t\t\t}\n -+\n -+\t\t\tif (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n -+\t\t\t\tgoto corrupt;\n -+\n -+\t\t\txor_item = &xor_items[xor_items_nr];\n -+\t\t\txor_item->offset = triplet.offset;\n -+\n -+\t\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n -+\t\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n -+\t\t\t\t\ttriplet.commit_pos);\n -+\t\t\t\tgoto corrupt;\n -+\t\t\t}\n -+\n -+\t\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n -+\n -+\t\t\t/*\n -+\t\t\t * If desired bitmap is already stored, we don't need\n -+\t\t\t * to iterate further. Because we know that bitmaps\n -+\t\t\t * that are needed to be parsed to parse this bitmap\n -+\t\t\t * has already been stored. So, assign this stored bitmap\n -+\t\t\t * to the xor_bitmap.\n -+\t\t\t */\n -+\t\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n -+\t\t\t (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n -+\t\t\t\tbreak;\n -+\t\t\txor_items_nr++;\n -+\t\t\txor_row = triplet.xor_row;\n ++\twhile (xor_row != 0xffffffff) {\n ++\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n ++\n ++\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n ++\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n ++\t\t\tgoto corrupt;\n +\t\t}\n +\n -+\t\twhile (xor_items_nr) {\n -+\t\t\txor_item = &xor_items[xor_items_nr - 1];\n -+\t\t\tbitmap_git->map_pos = xor_item->offset;\n -+\t\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n -+\t\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n -+\t\t\t\t\toid_to_hex(&xor_item->oid));\n -+\t\t\t\tgoto corrupt;\n -+\t\t\t}\n ++\t\tif (bitmap_lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n ++\t\t\tgoto corrupt;\n ++\n ++\t\txor_item = &xor_items[xor_items_nr];\n ++\t\txor_item->offset = triplet.offset;\n +\n -+\t\t\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n -+\t\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n -+\t\t\tbitmap = read_bitmap_1(bitmap_git);\n ++\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n ++\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n ++\t\t\t\ttriplet.commit_pos);\n ++\t\t\tgoto corrupt;\n ++\t\t}\n +\n -+\t\t\tif (!bitmap)\n -+\t\t\t\tgoto corrupt;\n ++\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n ++\n ++\t\t/*\n ++\t\t * If desired bitmap is already stored, we don't need\n ++\t\t * to iterate further. Because we know that bitmaps\n ++\t\t * that are needed to be parsed to parse this bitmap\n ++\t\t * has already been stored. So, assign this stored bitmap\n ++\t\t * to the xor_bitmap.\n ++\t\t */\n ++\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n ++\t\t\t(xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n ++\t\t\tbreak;\n ++\t\txor_items_nr++;\n ++\t\txor_row = triplet.xor_row;\n ++\t}\n +\n -+\t\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n -+\t\t\txor_items_nr--;\n ++\twhile (xor_items_nr) {\n ++\t\txor_item = &xor_items[xor_items_nr - 1];\n ++\t\tbitmap_git->map_pos = xor_item->offset;\n ++\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n ++\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n ++\t\t\t\toid_to_hex(&xor_item->oid));\n ++\t\t\tgoto corrupt;\n +\t\t}\n ++\n ++\t\tbitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t);\n ++\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n ++\t\tbitmap = read_bitmap_1(bitmap_git);\n ++\n ++\t\tif (!bitmap)\n ++\t\t\tgoto corrupt;\n ++\n ++\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n ++\t\txor_items_nr--;\n +\t}\n +\n +\tbitmap_git->map_pos = offset;\n @@ pack-bitmap.c: struct include_data {\n +\t\tgoto corrupt;\n +\t}\n +\n -+\tbitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t);\n ++\t/*\n ++\t * Don't bother reading the commit's index position or its xor\n ++\t * offset:\n ++\t *\n ++\t * - The commit's index position is irrelevant to us, since\n ++\t * load_bitmap_entries_v1 only uses it to learn the object\n ++\t * id which is used to compute the hashmap's key. We already\n ++\t * have an object id, so no need to look it up again.\n ++\t *\n ++\t * - The xor_offset is unusable for us, since it specifies how\n ++\t * many entries previous to ours we should look at. This\n ++\t * makes sense when reading the bitmaps sequentially (as in\n ++\t * load_bitmap_entries_v1()), since we can keep track of\n ++\t * each bitmap as we read them.\n ++\t *\n ++\t * But it can't work for us, since the bitmap's don't have a\n ++\t * fixed size. So we learn the position of the xor'd bitmap\n ++\t * from the commit table (and resolve it to a bitmap in the\n ++\t * above if-statement).\n ++\t *\n ++\t * Instead, we can skip ahead and immediately read the flags and\n ++\t * ewah bitmap.\n ++\t */\n ++\tbitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t);\n +\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n +\tbitmap = read_bitmap_1(bitmap_git);\n +\n @@ pack-bitmap.c: struct include_data {\n \n @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n - \t\tdie(\"you must specify exactly one commit to test\");\n + \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n --\tfprintf(stderr, \"Bitmap v%d test (%d entries loaded)\\n\",\n +-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries loaded)\",\n -\t\tbitmap_git->version, bitmap_git->entry_count);\n -+\tfprintf(stderr, \"Bitmap v%d test (%d entries%s)\",\n ++\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n +\t\tbitmap_git->version,\n +\t\tbitmap_git->entry_count,\n +\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n +\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n -+\n + \n + \tif (!bitmap_git)\n + \t\tdie(_(\"failed to load bitmap indexes\"));\n + \n +\t/*\n +\t * As this function is only used to print bitmap selected\n +\t * commits, we don't have to read the commit table.\n +\t */\n - \n - \tif (!bitmap_git)\n - \t\tdie(\"failed to load bitmap indexes\");\n - \n +\tif (bitmap_git->table_lookup) {\n +\t\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n +\t\t\tdie(_(\"failed to load bitmap indexes\"));\n +\t}\n +\n \tkh_foreach(bitmap_git->bitmaps, oid, value, {\n - \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n + \t\tprintf_ln(\"%s\", oid_to_hex(&oid));\n \t});\n \n ## pack-bitmap.h ##\n 5: e7ef420f321 < -: ----------- p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex`\n 6: 6628001241d ! 6: b460516b306 bitmap-lookup-table: add performance tests for lookup table\n @@ Metadata\n ## Commit message ##\n bitmap-lookup-table: add performance tests for lookup table\n \n - Add performance tests to verify the performance of lookup table with\n - `pack.writeReverseIndex` enabled. This is to check the performance\n - when the above configuration is set.\n + Add performance tests to verify the performance of lookup table.\n + `p5310-pack-bitmaps.sh` contain tests with and without lookup table.\n + `p5312-pack-bitmaps-revs.sh` contain same tests with and without\n + lookup table but with `pack.writeReverseIndex` enabled.\n \n Lookup table makes Git run faster in most of the cases. Below is the\n result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\n gives similar result. The repository used in the test is linux kernel.\n \n - Test this tree\n - ---------------------------------------------------------------------------\n - 5310.4: repack to disk (lookup=false) 296.55(256.53+14.52)\n - 5310.5: simulated clone 15.64(8.88+1.39)\n - 5310.6: simulated fetch 1.65(2.75+0.20)\n - 5310.7: pack to file (bitmap) 48.71(30.20+7.58)\n - 5310.8: rev-list (commits) 0.61(0.41+0.08)\n - 5310.9: rev-list (objects) 4.38(4.26+0.09)\n - 5310.10: rev-list with tag negated via --not 0.07(0.02+0.04)\n + Test this tree\n + -----------------------------------------------------------------------\n + 5310.4: enable lookup table: false 0.01(0.00+0.00)\n + 5310.5: repack to disk 320.89(230.20+23.45)\n + 5310.6: simulated clone 14.04(5.78+1.79)\n + 5310.7: simulated fetch 1.95(3.05+0.20)\n + 5310.8: pack to file (bitmap) 44.73(20.55+7.45)\n + 5310.9: rev-list (commits) 0.78(0.46+0.10)\n + 5310.10: rev-list (objects) 4.07(3.97+0.08)\n + 5310.11: rev-list with tag negated via --not 0.06(0.02+0.03)\n --all (objects)\n - 5310.11: rev-list with negative tag (objects) 0.05(0.01+0.03)\n - 5310.12: rev-list count with blob:none 0.08(0.03+0.04)\n - 5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30)\n - 5310.14: rev-list count with tree:0 0.08(0.03+0.04)\n - 5310.15: simulated partial clone 9.45(8.12+0.41)\n - 5310.17: clone (partial bitmap) 21.00(15.04+2.39)\n - 5310.18: pack to file (partial bitmap) 47.98(38.13+5.23)\n - 5310.19: rev-list with tree filter (partial bitmap) 0.70(0.07+0.20)\n - 5310.22: repack to disk (lookup=true) 255.92(188.13+20.47)\n - 5310.23: simulated clone 13.78(8.84+1.09)\n - 5310.24: simulated fetch 0.52(0.63+0.14)\n - 5310.25: pack to file (bitmap) 44.34(28.94+6.84)\n - 5310.26: rev-list (commits) 0.48(0.31+0.06)\n - 5310.27: rev-list (objects) 4.02(3.93+0.07)\n - 5310.28: rev-list with tag negated via --not 0.04(0.00+0.03)\n + 5310.12: rev-list with negative tag (objects) 0.21(0.15+0.05)\n + 5310.13: rev-list count with blob:none 0.24(0.17+0.06)\n + 5310.14: rev-list count with blob:limit=1k 7.07(5.92+0.48)\n + 5310.15: rev-list count with tree:0 0.25(0.17+0.07)\n + 5310.16: simulated partial clone 5.67(3.28+0.64)\n + 5310.18: clone (partial bitmap) 16.05(8.34+1.86)\n + 5310.19: pack to file (partial bitmap) 59.76(27.22+7.43)\n + 5310.20: rev-list with tree filter (partial bitmap) 0.90(0.18+0.16)\n + 5310.24: enable lookup table: true 0.01(0.00+0.00)\n + 5310.25: repack to disk 319.73(229.30+23.01)\n + 5310.26: simulated clone 13.69(5.72+1.78)\n + 5310.27: simulated fetch 1.84(3.02+0.16)\n + 5310.28: pack to file (bitmap) 45.63(20.67+7.50)\n + 5310.29: rev-list (commits) 0.56(0.39+0.8)\n + 5310.30: rev-list (objects) 3.77(3.74+0.08)\n + 5310.31: rev-list with tag negated via --not 0.05(0.02+0.03)\n --all (objects)\n - 5310.29: rev-list with negative tag (objects) 0.04(0.00+0.03)\n - 5310.30: rev-list count with blob:none 0.04(0.01+0.03)\n - 5310.31: rev-list count with blob:limit=1k 6.48(6.23+0.22)\n - 5310.32: rev-list count with tree:0 0.04(0.01+0.03)\n - 5310.33: simulated partial clone 8.30(7.21+0.36)\n - 5310.35: clone (partial bitmap) 20.34(15.00+2.41)\n - 5310.36: pack to file (partial bitmap) 46.45(38.05+5.20)\n - 5310.37: rev-list with tree filter (partial bitmap) 0.61(0.06+0.20)\n + 5310.32: rev-list with negative tag (objects) 0.21(0.15+0.05)\n + 5310.33: rev-list count with blob:none 0.23(0.17+0.05)\n + 5310.34: rev-list count with blob:limit=1k 6.65(5.72+0.40)\n + 5310.35: rev-list count with tree:0 0.23(0.16+0.06)\n + 5310.36: simulated partial clone 5.57(3.26+0.59)\n + 5310.38: clone (partial bitmap) 15.89(8.39+1.84)\n + 5310.39: pack to file (partial bitmap) 58.32(27.55+7.47)\n + 5310.40: rev-list with tree filter (partial bitmap) 0.73(0.18+0.15)\n \n Test 4-15 are tested without using lookup table. Same tests are\n repeated in 16-30 (using lookup table).\n @@ Commit message\n Co-Mentored-by: Kaartic Sivaraam \n Signed-off-by: Abhradeep Chakraborty \n \n + ## t/perf/lib-bitmap.sh ##\n +@@ t/perf/lib-bitmap.sh: test_partial_bitmap () {\n + \t\t\t--filter=tree:0 >/dev/null\n + \t'\n + }\n ++\n ++test_pack_bitmap () {\n ++\ttest_perf \"repack to disk\" '\n ++\t\tgit repack -ad\n ++\t'\n ++\n ++\ttest_full_bitmap\n ++\n ++\ttest_expect_success \"create partial bitmap state\" '\n ++\t\t# pick a commit to represent the repo tip in the past\n ++\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n ++\t\torig_tip=$(git rev-parse HEAD) &&\n ++\n ++\t\t# now kill off all of the refs and pretend we had\n ++\t\t# just the one tip\n ++\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n ++\t\tgit update-ref HEAD $cutoff &&\n ++\n ++\t\t# and then repack, which will leave us with a nice\n ++\t\t# big bitmap pack of the \"old\" history, and all of\n ++\t\t# the new history will be loose, as if it had been pushed\n ++\t\t# up incrementally and exploded via unpack-objects\n ++\t\tgit repack -Ad &&\n ++\n ++\t\t# and now restore our original tip, as if the pushes\n ++\t\t# had happened\n ++\t\tgit update-ref HEAD $orig_tip\n ++\t'\n ++\n ++\ttest_partial_bitmap\n ++}\n +\n ## t/perf/p5310-pack-bitmaps.sh ##\n -@@ t/perf/p5310-pack-bitmaps.sh: test_expect_success 'setup bitmap config' '\n - \tgit config pack.writeReverseIndex true\n - '\n +@@ t/perf/p5310-pack-bitmaps.sh: test_description='Tests pack performance using bitmaps'\n + . ./perf-lib.sh\n + . \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n +-test_perf_large_repo\n +-\n +-# note that we do everything through config,\n +-# since we want to be able to compare bitmap-aware\n +-# git versus non-bitmap git\n +-#\n +-# We intentionally use the deprecated pack.writebitmaps\n +-# config so that we can test against older versions of git.\n +-test_expect_success 'setup bitmap config' '\n +-\tgit config pack.writebitmaps true\n +-'\n +-\n -# we need to create the tag up front such that it is covered by the repack and\n -# thus by generated bitmaps.\n -test_expect_success 'create tags' '\n -\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n -'\n -+test_bitmap () {\n -+\tlocal enabled=\"$1\"\n - \n +-\n -test_perf 'repack to disk' '\n -\tgit repack -ad\n -'\n -+\t# we need to create the tag up front such that it is covered by the repack and\n -+\t# thus by generated bitmaps.\n -+\ttest_expect_success 'create tags' '\n -+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n -+\t'\n - \n +-\n -test_full_bitmap\n -+\ttest_expect_success \"use lookup table: $enabled\" '\n -+\t\tgit config pack.writeBitmapLookupTable '\"$enabled\"'\n -+\t'\n - \n +-\n -test_expect_success 'create partial bitmap state' '\n -\t# pick a commit to represent the repo tip in the past\n -\tcutoff=$(git rev-list HEAD~100 -1) &&\n -\torig_tip=$(git rev-parse HEAD) &&\n -+\ttest_perf \"repack to disk (lookup=$enabled)\" '\n -+\t\tgit repack -ad\n -+\t'\n - \n +-\n -\t# now kill off all of the refs and pretend we had\n -\t# just the one tip\n -\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n -\tgit update-ref HEAD $cutoff &&\n -+\ttest_full_bitmap\n - \n +-\n -\t# and then repack, which will leave us with a nice\n -\t# big bitmap pack of the \"old\" history, and all of\n -\t# the new history will be loose, as if it had been pushed\n -\t# up incrementally and exploded via unpack-objects\n -\tgit repack -Ad &&\n -+\ttest_expect_success \"create partial bitmap state (lookup=$enabled)\" '\n -+\t\t# pick a commit to represent the repo tip in the past\n -+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n -+\t\torig_tip=$(git rev-parse HEAD) &&\n - \n +-\n -\t# and now restore our original tip, as if the pushes\n -\t# had happened\n -\tgit update-ref HEAD $orig_tip\n -'\n -+\t\t# now kill off all of the refs and pretend we had\n -+\t\t# just the one tip\n -+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n -+\t\tgit update-ref HEAD $cutoff &&\n +-\n +-test_partial_bitmap\n ++test_lookup_pack_bitmap () {\n ++\ttest_expect_success 'start the test from scratch' '\n ++\t\trm -rf * .git\n ++\t'\n +\n -+\t\t# and then repack, which will leave us with a nice\n -+\t\t# big bitmap pack of the \"old\" history, and all of\n -+\t\t# the new history will be loose, as if it had been pushed\n -+\t\t# up incrementally and exploded via unpack-objects\n -+\t\tgit repack -Ad &&\n ++\ttest_perf_large_repo\n +\n -+\t\t# and now restore our original tip, as if the pushes\n -+\t\t# had happened\n -+\t\tgit update-ref HEAD $orig_tip\n ++\t# note that we do everything through config,\n ++\t# since we want to be able to compare bitmap-aware\n ++\t# git versus non-bitmap git\n ++\t#\n ++\t# We intentionally use the deprecated pack.writebitmaps\n ++\t# config so that we can test against older versions of git.\n ++\ttest_expect_success 'setup bitmap config' '\n ++\t\tgit config pack.writebitmaps true\n +\t'\n +\n -+\ttest_partial_bitmap\n ++\t# we need to create the tag up front such that it is covered by the repack and\n ++\t# thus by generated bitmaps.\n ++\ttest_expect_success 'create tags' '\n ++\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n ++\t'\n ++\n ++\ttest_perf \"enable lookup table: $1\" '\n ++\t\tgit config pack.writeBitmapLookupTable '\"$1\"'\n ++\t'\n ++\n ++\ttest_pack_bitmap\n +}\n ++\n ++test_lookup_pack_bitmap false\n ++test_lookup_pack_bitmap true\n \n --test_partial_bitmap\n -+test_bitmap false\n -+test_bitmap true\n + test_done\n +\n + ## t/perf/p5311-pack-bitmaps-fetch.sh ##\n +@@\n + test_description='performance of fetches from bitmapped packs'\n + . ./perf-lib.sh\n + \n +-test_perf_default_repo\n +-\n +-test_expect_success 'create bitmapped server repo' '\n +-\tgit config pack.writebitmaps true &&\n +-\tgit repack -ad\n +-'\n +-\n +-# simulate a fetch from a repository that last fetched N days ago, for\n +-# various values of N. We do so by following the first-parent chain,\n +-# and assume the first entry in the chain that is N days older than the current\n +-# HEAD is where the HEAD would have been then.\n +-for days in 1 2 4 8 16 32 64 128; do\n +-\ttitle=$(printf '%10s' \"($days days)\")\n +-\ttest_expect_success \"setup revs from $days days ago\" '\n +-\t\tnow=$(git log -1 --format=%ct HEAD) &&\n +-\t\tthen=$(($now - ($days * 86400))) &&\n +-\t\ttip=$(git rev-list -1 --first-parent --until=$then HEAD) &&\n +-\t\t{\n +-\t\t\techo HEAD &&\n +-\t\t\techo ^$tip\n +-\t\t} >revs\n ++test_fetch_bitmaps () {\n ++\ttest_expect_success 'setup test directory' '\n ++\t\trm -fr * .git\n + \t'\n + \n +-\ttest_perf \"server $title\" '\n +-\t\tgit pack-objects --stdout --revs \\\n +-\t\t\t\t --thin --delta-base-offset \\\n +-\t\t\t\t tmp.pack\n +-\t'\n ++\ttest_perf_default_repo\n + \n +-\ttest_size \"size $title\" '\n +-\t\twc -c revs\n ++\t\t'\n ++\n ++\t\ttest_perf \"server $title (lookup=$1)\" '\n ++\t\t\tgit pack-objects --stdout --revs \\\n ++\t\t\t\t\t--thin --delta-base-offset \\\n ++\t\t\t\t\ttmp.pack\n ++\t\t'\n ++\n ++\t\ttest_size \"size $title\" '\n ++\t\t\twc -c \n\nWhen reading bitmap file, Git loads each and every bitmap one by one\neven if all the bitmaps are not required. A \"bitmap lookup table\"\nextension to the bitmap format can reduce the overhead of loading\nbitmaps which stores a list of bitmapped commit id pos (in the midx\nor pack, along with their offset and xor offset. This way Git can\nload only the necessary bitmaps without loading the previous bitmaps.\n\nOlder versions of Git ignore the lookup table extension and don't\nthrow any kind of warning or error while parsing the bitmap file.\n\nAdd some information for the new \"bitmap lookup table\" extension in the\nbitmap-format documentation.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/technical/bitmap-format.txt | 39 +++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex a85f58f5153..c2e652b71a7 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -72,6 +72,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required.\n \t pack/MIDX. The format and meaning of the name-hash is\n \t described below.\n \n+\t\t** {empty}\n+\t\tBITMAP_OPT_LOOKUP_TABLE (0x10): :::\n+\t\tIf present, the end of the bitmap file contains a table\n+\t\tcontaining a list of `N` \n+\t\ttriplets. The format and meaning of the table is described\n+\t\tbelow.\n++\n+NOTE: Unlike the xor_offset used to compress an individual bitmap,\n+`xor_row` stores an *absolute* index into the lookup table, not a location\n+relative to the current entry.\n+\n \t4-byte entry count (network byte order): ::\n \t The total count of entries (bitmapped commits) in this bitmap index.\n \n@@ -216,3 +227,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.\n If implementations want to choose a different hashing scheme, they are\n free to do so, but MUST allocate a new header flag (because comparing\n hashes made under two different schemes would be pointless).\n+\n+Commit lookup table\n+-------------------\n+\n+If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`\n+bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`\n+file contains a lookup table specifying the information needed to get\n+the desired bitmap from the entries without parsing previous unnecessary\n+bitmaps.\n+\n+For a `.bitmap` containing `nr_entries` reachability bitmaps, the table\n+contains a list of `nr_entries` triplets\n+(sorted in the ascending order of `commit_pos`). The content of i'th\n+triplet is -\n+\n+\t* {empty}\n+\tcommit_pos (4 byte integer, network byte order): ::\n+\tIt stores the object position of a commit (in the midx or pack\n+\tindex).\n+\n+\t* {empty}\n+\toffset (8 byte integer, network byte order): ::\n+\tThe offset from which that commit's bitmap can be read.\n+\n+\t* {empty}\n+\txor_row (4 byte integer, network byte order): ::\n+\tThe position of the triplet whose bitmap is used to compress\n+\tthis one, or `0xffffffff` if no such bitmap exists.\n-- \ngitgitgadget\n\n"},{"id":"461195","messageId":"090becaabe0c47fb5fed4ec5ce7628deeafeded1.1660496112.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"[PATCH v6 3/6] pack-bitmap-write.c: write lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-14T16:55:08Z","receivedAt":"2022-08-14T17:00:15Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nThe bitmap lookup table extension was documented by an earlier\nchange, but Git does not yet know how to write that extension.\n\nTeach Git to write bitmap lookup table extension. The table contains\nthe list of `N` ` triplets. These\ntriplets are sorted according to their commit pos (ascending order).\nThe meaning of each data in the i'th triplet is given below:\n\n - commit_pos stores commit position (in the pack-index or midx).\n It is a 4 byte network byte order unsigned integer.\n\n - offset is the position (in the bitmap file) from which that\n commit's bitmap can be read.\n\n - xor_row is the position of the triplet in the lookup table\n whose bitmap is used to compress this bitmap, or `0xffffffff`\n if no such bitmap exists.\n\nMentored-by: Taylor Blau \nCo-mentored-by: Kaartic Sivaraam \nCo-authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap-write.c | 91 ++++++++++++++++++++++++++++++++++++++++++++-\n pack-bitmap.h | 5 ++-\n 2 files changed, 92 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9b1be59f6d3..2cfc92f2871 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -651,13 +651,17 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n static void write_selected_commits_v1(struct hashfile *f,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr,\n-\t\t\t\t uint32_t *commit_positions)\n+\t\t\t\t uint32_t *commit_positions,\n+\t\t\t\t off_t *offsets)\n {\n \tint i;\n \n \tfor (i = 0; i < writer.selected_nr; ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \n+\t\tif (offsets)\n+\t\t\toffsets[i] = hashfile_total(f);\n+\n \t\thashwrite_be32(f, commit_positions[i]);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n@@ -666,6 +670,81 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static int table_cmp(const void *_va, const void *_vb, void *_data)\n+{\n+\tuint32_t *commit_positions = _data;\n+\tuint32_t a = commit_positions[*(uint32_t *)_va];\n+\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n+\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static void write_lookup_table(struct hashfile *f,\n+\t\t\t struct pack_idx_entry **index,\n+\t\t\t uint32_t index_nr,\n+\t\t\t uint32_t *commit_positions,\n+\t\t\t off_t *offsets)\n+{\n+\tuint32_t i;\n+\tuint32_t *table, *table_inv;\n+\n+\tALLOC_ARRAY(table, writer.selected_nr);\n+\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable[i] = i;\n+\n+\t/*\n+\t * At the end of this sort table[j] = i means that the i'th\n+\t * bitmap corresponds to j'th bitmapped commit (among the selected\n+\t * commits) in lex order of OIDs.\n+\t */\n+\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\n+\t/* table_inv helps us discover that relationship (i'th bitmap\n+\t * to j'th commit by j = table_inv[i])\n+\t */\n+\tfor (i = 0; i < writer.selected_nr; i++)\n+\t\ttable_inv[table[i]] = i;\n+\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\tfor (i = 0; i < writer.selected_nr; i++) {\n+\t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n+\t\tuint32_t xor_offset = selected->xor_offset;\n+\t\tuint32_t xor_row;\n+\n+\t\tif (xor_offset) {\n+\t\t\t/*\n+\t\t\t * xor_index stores the index (in the bitmap entries)\n+\t\t\t * of the corresponding xor bitmap. But we need to convert\n+\t\t\t * this index into lookup table's index. So, table_inv[xor_index]\n+\t\t\t * gives us the index position w.r.t. the lookup table.\n+\t\t\t *\n+\t\t\t * If \"k = table[i] - xor_offset\" then the xor base is the k'th\n+\t\t\t * bitmap. `table_inv[k]` gives us the position of that bitmap\n+\t\t\t * in the lookup table.\n+\t\t\t */\n+\t\t\tuint32_t xor_index = table[i] - xor_offset;\n+\t\t\txor_row = table_inv[xor_index];\n+\t\t} else {\n+\t\t\txor_row = 0xffffffff;\n+\t\t}\n+\n+\t\thashwrite_be32(f, commit_positions[table[i]]);\n+\t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n+\t\thashwrite_be32(f, xor_row);\n+\t}\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n+\n+\tfree(table);\n+\tfree(table_inv);\n+}\n+\n static void write_hash_cache(struct hashfile *f,\n \t\t\t struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr)\n@@ -693,6 +772,7 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n \tuint32_t *commit_positions = NULL;\n+\toff_t *offsets = NULL;\n \tuint32_t i;\n \n \tstruct bitmap_disk_header header;\n@@ -713,6 +793,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tdump_bitmap(f, writer.blobs);\n \tdump_bitmap(f, writer.tags);\n \n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\tCALLOC_ARRAY(offsets, index_nr);\n+\n \tALLOC_ARRAY(commit_positions, writer.selected_nr);\n \n \tfor (i = 0; i < writer.selected_nr; i++) {\n@@ -725,7 +808,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tcommit_positions[i] = commit_pos;\n \t}\n \n-\twrite_selected_commits_v1(f, index, index_nr, commit_positions);\n+\twrite_selected_commits_v1(f, index, index_nr, commit_positions, offsets);\n+\n+\tif (options & BITMAP_OPT_LOOKUP_TABLE)\n+\t\twrite_lookup_table(f, index, index_nr, commit_positions, offsets);\n \n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n@@ -741,4 +827,5 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \tstrbuf_release(&tmp_file);\n \tfree(commit_positions);\n+\tfree(offsets);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex f3a57ca065f..cb065a263cb 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -24,8 +24,9 @@ struct bitmap_disk_header {\n #define NEEDS_BITMAP (1u<<22)\n \n enum pack_bitmap_opts {\n-\tBITMAP_OPT_FULL_DAG = 1,\n-\tBITMAP_OPT_HASH_CACHE = 4,\n+\tBITMAP_OPT_FULL_DAG = 0x1,\n+\tBITMAP_OPT_HASH_CACHE = 0x4,\n+\tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n };\n \n enum pack_bitmap_flags {\n-- \ngitgitgadget\n\n"},{"id":"461196","messageId":"79842ca590c8f3af41d416bfb5d2b0109b1b918e.1660496112.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"[PATCH v6 5/6] pack-bitmap: prepare to read lookup table extension","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-14T16:55:10Z","receivedAt":"2022-08-14T17:00:18Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nEarlier change teaches Git to write bitmap lookup table. But Git\ndoes not know how to parse them.\n\nTeach Git to parse the existing bitmap lookup table. The older\nversions of Git are not affected by it. Those versions ignore the\nlookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n pack-bitmap.c | 290 ++++++++++++++++++++++++++++++++++++++--\n pack-bitmap.h | 9 ++\n t/t5310-pack-bitmaps.sh | 22 +++\n 3 files changed, 312 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex ef580be9e3f..9a208abc1fd 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -83,6 +83,12 @@ struct bitmap_index {\n \t/* The checksum of the packfile or MIDX; points into map. */\n \tconst unsigned char *checksum;\n \n+\t/*\n+\t * If not NULL, this point into the commit table extension\n+\t * (within the memory mapped region `map`).\n+\t */\n+\tunsigned char *table_lookup;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -186,6 +192,16 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\tindex->hashes = (void *)(index_end - cache_size);\n \t\t\tindex_end -= cache_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_LOOKUP_TABLE) {\n+\t\t\tsize_t table_size = st_mult(ntohl(header->entry_count),\n+\t\t\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit lookup table)\"));\n+\t\t\tif (git_env_bool(\"GIT_TEST_READ_COMMIT_TABLE\", 1))\n+\t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n@@ -212,9 +228,11 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,\n \n \thash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret);\n \n-\t/* a 0 return code means the insertion succeeded with no changes,\n-\t * because the SHA1 already existed on the map. this is bad, there\n-\t * shouldn't be duplicated commits in the index */\n+\t/*\n+\t * A 0 return code means the insertion succeeded with no changes,\n+\t * because the SHA1 already existed on the map. This is bad, there\n+\t * shouldn't be duplicated commits in the index.\n+\t */\n \tif (ret == 0) {\n \t\terror(_(\"duplicate entry in bitmap index: '%s'\"), oid_to_hex(oid));\n \t\treturn NULL;\n@@ -482,7 +500,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git)\n \t\t!(bitmap_git->tags = read_bitmap_1(bitmap_git)))\n \t\tgoto failed;\n \n-\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\tif (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)\n \t\tgoto failed;\n \n \treturn 0;\n@@ -570,13 +588,256 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct bitmap_lookup_table_triplet {\n+\tuint32_t commit_pos;\n+\tuint64_t offset;\n+\tuint32_t xor_row;\n+};\n+\n+struct bitmap_lookup_table_xor_item {\n+\tstruct object_id oid;\n+\tuint64_t offset;\n+};\n+\n+/*\n+ * Given a `triplet` struct pointer and pointer `p`, this\n+ * function reads the triplet beginning at `p` into the struct.\n+ * Note that this function assumes that there is enough memory\n+ * left for filling the `triplet` struct from `p`.\n+ */\n+static int bitmap_lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet,\n+\t\t\t\t\t\t const unsigned char *p)\n+{\n+\tif (!triplet)\n+\t\treturn -1;\n+\n+\ttriplet->commit_pos = get_be32(p);\n+\tp += sizeof(uint32_t);\n+\ttriplet->offset = get_be64(p);\n+\tp += sizeof(uint64_t);\n+\ttriplet->xor_row = get_be32(p);\n+\treturn 0;\n+}\n+\n+/*\n+ * This function gets the raw triplet from `row`'th row in the\n+ * lookup table and fills that data to the `triplet`.\n+ */\n+static int bitmap_lookup_table_get_triplet(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t uint32_t pos,\n+\t\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = NULL;\n+\tif (pos >= bitmap_git->entry_count)\n+\t\treturn error(_(\"corrupt bitmap lookup table: triplet position out of index\"));\n+\n+\tp = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH);\n+\n+\treturn bitmap_lookup_table_get_triplet_by_pointer(triplet, p);\n+}\n+\n+/*\n+ * Searches for a matching triplet. `commit_pos` is a pointer\n+ * to the wanted commit position value. `table_entry` points to\n+ * a triplet in lookup table. The first 4 bytes of each\n+ * triplet (pointed by `table_entry`) are compared with `*commit_pos`.\n+ */\n+static int triplet_cmp(const void *commit_pos, const void *table_entry)\n+{\n+\n+\tuint32_t a = *(uint32_t *)commit_pos;\n+\tuint32_t b = get_be32(table_entry);\n+\tif (a > b)\n+\t\treturn 1;\n+\telse if (a < b)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t bitmap_bsearch_pos(struct bitmap_index *bitmap_git,\n+\t\t\t struct object_id *oid,\n+\t\t\t uint32_t *result)\n+{\n+\tint found;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tfound = bsearch_midx(oid, bitmap_git->midx, result);\n+\telse\n+\t\tfound = bsearch_pack(oid, bitmap_git->pack, result);\n+\n+\treturn found;\n+}\n+\n+/*\n+ * `bsearch_triplet_by_pos` function searches for the raw triplet\n+ * having commit position same as `commit_pos` and fills `triplet`\n+ * object from the raw triplet. Returns 1 on success and 0 on\n+ * failure.\n+ */\n+static int bitmap_bsearch_triplet_by_pos(uint32_t commit_pos,\n+\t\t\t\t struct bitmap_index *bitmap_git,\n+\t\t\t\t struct bitmap_lookup_table_triplet *triplet)\n+{\n+\tunsigned char *p = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count,\n+\t\t\t\t BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp);\n+\n+\tif (!p)\n+\t\treturn -1;\n+\n+\treturn bitmap_lookup_table_get_triplet_by_pointer(triplet, p);\n+}\n+\n+static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit)\n+{\n+\tuint32_t commit_pos, xor_row;\n+\tuint64_t offset;\n+\tint flags;\n+\tstruct bitmap_lookup_table_triplet triplet;\n+\tstruct object_id *oid = &commit->object.oid;\n+\tstruct ewah_bitmap *bitmap;\n+\tstruct stored_bitmap *xor_bitmap = NULL;\n+\tconst int bitmap_header_size = 6;\n+\tstatic struct bitmap_lookup_table_xor_item *xor_items = NULL;\n+\tstatic size_t xor_items_nr = 0, xor_items_alloc = 0;\n+\tstatic int is_corrupt = 0;\n+\tint xor_flags;\n+\tkhiter_t hash_pos;\n+\tstruct bitmap_lookup_table_xor_item *xor_item;\n+\n+\tif (is_corrupt)\n+\t\treturn NULL;\n+\n+\tif (!bitmap_bsearch_pos(bitmap_git, oid, &commit_pos))\n+\t\treturn NULL;\n+\n+\tif (bitmap_bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0)\n+\t\treturn NULL;\n+\n+\txor_items_nr = 0;\n+\toffset = triplet.offset;\n+\txor_row = triplet.xor_row;\n+\n+\twhile (xor_row != 0xffffffff) {\n+\t\tALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc);\n+\n+\t\tif (xor_items_nr + 1 >= bitmap_git->entry_count) {\n+\t\t\terror(_(\"corrupt bitmap lookup table: xor chain exceed entry count\"));\n+\t\t\tgoto corrupt;\n+\t\t}\n+\n+\t\tif (bitmap_lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0)\n+\t\t\tgoto corrupt;\n+\n+\t\txor_item = &xor_items[xor_items_nr];\n+\t\txor_item->offset = triplet.offset;\n+\n+\t\tif (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) {\n+\t\t\terror(_(\"corrupt bitmap lookup table: commit index %u out of range\"),\n+\t\t\t\ttriplet.commit_pos);\n+\t\t\tgoto corrupt;\n+\t\t}\n+\n+\t\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid);\n+\n+\t\t/*\n+\t\t * If desired bitmap is already stored, we don't need\n+\t\t * to iterate further. Because we know that bitmaps\n+\t\t * that are needed to be parsed to parse this bitmap\n+\t\t * has already been stored. So, assign this stored bitmap\n+\t\t * to the xor_bitmap.\n+\t\t */\n+\t\tif (hash_pos < kh_end(bitmap_git->bitmaps) &&\n+\t\t\t(xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos)))\n+\t\t\tbreak;\n+\t\txor_items_nr++;\n+\t\txor_row = triplet.xor_row;\n+\t}\n+\n+\twhile (xor_items_nr) {\n+\t\txor_item = &xor_items[xor_items_nr - 1];\n+\t\tbitmap_git->map_pos = xor_item->offset;\n+\t\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n+\t\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\t\toid_to_hex(&xor_item->oid));\n+\t\t\tgoto corrupt;\n+\t\t}\n+\n+\t\tbitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t);\n+\t\txor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\t\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\t\tif (!bitmap)\n+\t\t\tgoto corrupt;\n+\n+\t\txor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags);\n+\t\txor_items_nr--;\n+\t}\n+\n+\tbitmap_git->map_pos = offset;\n+\tif (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) {\n+\t\terror(_(\"corrupt ewah bitmap: truncated header for bitmap of commit \\\"%s\\\"\"),\n+\t\t\toid_to_hex(oid));\n+\t\tgoto corrupt;\n+\t}\n+\n+\t/*\n+\t * Don't bother reading the commit's index position or its xor\n+\t * offset:\n+\t *\n+\t * - The commit's index position is irrelevant to us, since\n+\t * load_bitmap_entries_v1 only uses it to learn the object\n+\t * id which is used to compute the hashmap's key. We already\n+\t * have an object id, so no need to look it up again.\n+\t *\n+\t * - The xor_offset is unusable for us, since it specifies how\n+\t * many entries previous to ours we should look at. This\n+\t * makes sense when reading the bitmaps sequentially (as in\n+\t * load_bitmap_entries_v1()), since we can keep track of\n+\t * each bitmap as we read them.\n+\t *\n+\t * But it can't work for us, since the bitmap's don't have a\n+\t * fixed size. So we learn the position of the xor'd bitmap\n+\t * from the commit table (and resolve it to a bitmap in the\n+\t * above if-statement).\n+\t *\n+\t * Instead, we can skip ahead and immediately read the flags and\n+\t * ewah bitmap.\n+\t */\n+\tbitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t);\n+\tflags = read_u8(bitmap_git->map, &bitmap_git->map_pos);\n+\tbitmap = read_bitmap_1(bitmap_git);\n+\n+\tif (!bitmap)\n+\t\tgoto corrupt;\n+\n+\treturn store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags);\n+\n+corrupt:\n+\tfree(xor_items);\n+\tis_corrupt = 1;\n+\treturn NULL;\n+}\n+\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit)\n {\n \tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n \t\t\t\t\t commit->object.oid);\n-\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n-\t\treturn NULL;\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps)) {\n+\t\tstruct stored_bitmap *bitmap = NULL;\n+\t\tif (!bitmap_git->table_lookup)\n+\t\t\treturn NULL;\n+\n+\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\t/* NEEDSWORK: cache misses aren't recorded */\n+\t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n+\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\tif (!bitmap)\n+\t\t\treturn NULL;\n+\t\treturn lookup_stored_bitmap(bitmap);\n+\t}\n \treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n }\n \n@@ -1712,8 +1973,10 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (revs->pending.nr != 1)\n \t\tdie(_(\"you must specify exactly one commit to test\"));\n \n-\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries loaded)\",\n-\t\tbitmap_git->version, bitmap_git->entry_count);\n+\tfprintf_ln(stderr, \"Bitmap v%d test (%d entries%s)\",\n+\t\tbitmap_git->version,\n+\t\tbitmap_git->entry_count,\n+\t\tbitmap_git->table_lookup ? \"\" : \" loaded\");\n \n \troot = revs->pending.objects[0].item;\n \tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n@@ -1766,13 +2029,22 @@ void test_bitmap_walk(struct rev_info *revs)\n \n int test_bitmap_commits(struct repository *r)\n {\n-\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n \tstruct object_id oid;\n \tMAYBE_UNUSED void *value;\n+\tstruct bitmap_index *bitmap_git = prepare_bitmap_git(r);\n \n \tif (!bitmap_git)\n \t\tdie(_(\"failed to load bitmap indexes\"));\n \n+\t/*\n+\t * As this function is only used to print bitmap selected\n+\t * commits, we don't have to read the commit table.\n+\t */\n+\tif (bitmap_git->table_lookup) {\n+\t\tif (load_bitmap_entries_v1(bitmap_git) < 0)\n+\t\t\tdie(_(\"failed to load bitmap indexes\"));\n+\t}\n+\n \tkh_foreach(bitmap_git->bitmaps, oid, value, {\n \t\tprintf_ln(\"%s\", oid_to_hex(&oid));\n \t});\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex cb065a263cb..f0180b5276b 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -23,6 +23,15 @@ struct bitmap_disk_header {\n \n #define NEEDS_BITMAP (1u<<22)\n \n+/*\n+ * The width in bytes of a single triplet in the lookup table\n+ * extension:\n+ * (commit_pos, offset, xor_row)\n+ *\n+ * whose fields ar 32-, 64-, 32- bits wide, respectively.\n+ */\n+#define BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH (16)\n+\n enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c0607172827..7e50f8e7653 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -258,6 +258,7 @@ test_bitmap_cases () {\n \n \ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \t\ttest_config pack.writebitmaphashcache false &&\n+\t\ttest_config pack.writebitmaplookuptable false &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -270,6 +271,7 @@ test_bitmap_cases () {\n \t'\n \n \ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\tgit repack -ad &&\n \t\tgit rev-list --use-bitmap-index --count --all >expect &&\n \t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -453,4 +455,24 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n+test_expect_success 'lookup table is actually used to traverse objects' '\n+\tgit repack -adb &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n+\t\tgit rev-list --use-bitmap-index --count --all &&\n+\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n+\ttest_config pack.writebitmaphashcache false &&\n+\tgit repack -adb &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"461197","messageId":"b2b7c5c1703b0e0b64599cef26ead46a4ff46afb.1660496112.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"[PATCH v6 4/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-14T16:55:09Z","receivedAt":"2022-08-14T17:00:22Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nTeach Git to provide a way for users to enable/disable bitmap lookup\ntable extension by providing a config option named 'writeBitmapLookupTable'.\nDefault is false.\n\nAlso add test to verify writting of lookup table.\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nCo-Authored-by: Taylor Blau \nSigned-off-by: Abhradeep Chakraborty \n---\n Documentation/config/pack.txt | 7 +\n builtin/multi-pack-index.c | 7 +\n builtin/pack-objects.c | 8 +\n midx.c | 3 +\n midx.h | 1 +\n t/t5310-pack-bitmaps.sh | 792 ++++++++++++++++--------------\n t/t5311-pack-bitmaps-shallow.sh | 53 +-\n t/t5326-multi-pack-bitmaps.sh | 421 +++++++++-------\n t/t5327-multi-pack-bitmaps-rev.sh | 24 +-\n 9 files changed, 733 insertions(+), 583 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex ad7f73a1ead..b955ca572ec 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are\n computed; instead, any namehashes stored in an existing bitmap are\n permuted into their appropriate location when writing a new bitmap.\n \n+pack.writeBitmapLookupTable::\n+\tWhen true, Git will include a \"lookup table\" section in the\n+\tbitmap index (if one is written). This table is used to defer\n+\tloading individual bitmaps as late as possible. This can be\n+\tbeneficial in repositories that have relatively large bitmap\n+\tindexes. Defaults to false.\n+\n pack.writeReverseIndex::\n \tWhen true, git will write a corresponding .rev file (see:\n \tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 8f24d59a753..e7cce1d26ee 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n \t\t\topts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;\n \t}\n \n+\tif (!strcmp(var, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(var, value))\n+\t\t\topts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t\telse\n+\t\t\topts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE;\n+\t}\n+\n \t/*\n \t * We should never make a fall-back call to 'git_default_config', since\n \t * this was already called in 'cmd_multi_pack_index()'.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39e28cfcafc..46e26774963 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n+\n+\tif (!strcmp(k, \"pack.writebitmaplookuptable\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\twrite_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE;\n+\t\telse\n+\t\t\twrite_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE;\n+\t}\n+\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\ndiff --git a/midx.c b/midx.c\nindex 4e956cacb71..3ff6e91e6ee 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1070,6 +1070,9 @@ static int write_midx_bitmap(const char *midx_name,\n \tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n \t\toptions |= BITMAP_OPT_HASH_CACHE;\n \n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n \t/*\n \t * Build the MIDX-order index based on pdata.objects (which is already\n \t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\ndiff --git a/midx.h b/midx.h\nindex 22e8e53288e..5578cd7b835 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,6 +47,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_REV_INDEX (1 << 1)\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n+#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex f775fc1ce69..c0607172827 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -26,22 +26,413 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n-setup_bitmap_history\n-\n-test_expect_success 'setup writing bitmaps during repack' '\n-\tgit config repack.writeBitmaps true\n-'\n-\n-test_expect_success 'full repack creates bitmaps' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+test_bitmap_cases () {\n+\twriteLookupTable=false\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase \"$i\" in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test repository' '\n+\t\trm -fr * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'setup writing bitmaps during repack' '\n+\t\tgit config repack.writeBitmaps true\n+\t'\n+\n+\ttest_expect_success 'full repack creates bitmaps' '\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit repack -ad &&\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\t\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n+\t'\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local loose)' '\n+\t\tgit init --bare alt.git &&\n+\t\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\t\techo content1 >file1 &&\n+\t\t# non-local loose object which is not present in bitmapped pack\n+\t\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t\t# non-local loose object which is also present in bitmapped pack\n+\t\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\t\tgit add file1 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file1 &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\t\tgit index-pack 1.pack &&\n+\t\tlist_packed_objects 1.idx >1.objects &&\n+\t\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t\t! has_any nonlocal-loose 1.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\t\techo content2 >file2 &&\n+\t\tblob2=$(git hash-object -w file2) &&\n+\t\tgit add file2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -m commit_file2 &&\n+\t\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\t\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\t\trm $(objpath $blob2) &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\t\tgit index-pack 2a.pack &&\n+\t\tlist_packed_objects 2a.idx >2a.objects &&\n+\t\t! has_any keepobjects 2a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local pack)' '\n+\t\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\t\tgit index-pack 2b.pack &&\n+\t\tlist_packed_objects 2b.idx >2b.objects &&\n+\t\t! has_any keepobjects 2b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\t\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\t\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t\t>.git/objects/pack/$packbitmap.keep &&\n+\t\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\t\tgit index-pack 3a.pack &&\n+\t\tlist_packed_objects 3a.idx >3a.objects &&\n+\t\t! has_any packbitmap.objects 3a.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\t\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\t\trm -f .git/objects/pack/multi-pack-index &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\t\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\t\tgit index-pack 3b.pack &&\n+\t\tlist_packed_objects 3b.idx >3b.objects &&\n+\t\t! has_any packbitmap.objects 3b.objects\n+\t'\n+\n+\ttest_expect_success 'pack-objects to file can use bitmap' '\n+\t\t# make sure we still have 1 bitmap index from previous tests\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output &&\n+\t\t# verify equivalent packs are generated with/without using bitmap index\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n+\t\ttest_cmp packa.objects packb.objects\n+\t'\n+\n+\ttest_expect_success 'full repack, reusing previous bitmaps' '\n \t\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n-'\n+\t\tls .git/objects/pack/ | grep bitmap >output &&\n+\t\ttest_line_count = 1 output\n+\t'\n+\n+\ttest_expect_success 'fetch (full bitmap)' '\n+\t\tgit --git-dir=clone.git fetch origin second:second &&\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'create objects for missing-HAVE tests' '\n+\t\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n+\t\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n+\t\tparent=$(echo parent | git commit-tree $tree) &&\n+\t\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n+\t\tcat >revs <<-EOF\n+\t\tHEAD\n+\t\t^HEAD^\n+\t\t^$commit\n+\t\tEOF\n+\t'\n+\n+\ttest_expect_success 'pack-objects respects --incremental' '\n+\t\tcat >revs2 <<-EOF &&\n+\t\tHEAD\n+\t\t$commit\n+\t\tEOF\n+\t\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\t\tgit index-pack 4.pack &&\n+\t\tlist_packed_objects 4.idx >4.objects &&\n+\t\ttest_line_count = 4 4.objects &&\n+\t\tgit rev-list --objects $commit >revlist &&\n+\t\tcut -d\" \" -f1 revlist |sort >objects &&\n+\t\ttest_cmp 4.objects objects\n+\t'\n+\n+\ttest_expect_success 'pack with missing blob' '\n+\t\trm $(objpath $blob) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing tree' '\n+\t\trm $(objpath $tree) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success 'pack with missing parent' '\n+\t\trm $(objpath $parent) &&\n+\t\tgit pack-objects --stdout --revs /dev/null\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n+\t\tgit clone --bare . compat-jgit.git &&\n+\t\t(\n+\t\t\tcd compat-jgit.git &&\n+\t\t\trm -f objects/pack/*.bitmap &&\n+\t\t\tjgit gc &&\n+\t\t\tgit rev-list --test-bitmap HEAD\n+\t\t)\n+\t'\n+\n+\ttest_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n+\t\tgit clone --bare . compat-us.git &&\n+\t\t(\n+\t\t\tcd compat-us.git &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\tgit repack -adb &&\n+\t\t\t# jgit gc will barf if it does not like our bitmaps\n+\t\t\tjgit gc\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'splitting packs does not generate bogus bitmaps' '\n+\t\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n+\t\tgit add rand &&\n+\t\tgit commit -m \"commit with big file\" &&\n+\t\tgit -c pack.packSizeLimit=500k repack -adb &&\n+\t\tgit init --bare no-bitmaps.git &&\n+\t\tgit -C no-bitmaps.git fetch .. HEAD\n+\t'\n+\n+\ttest_expect_success 'set up reusable pack' '\n+\t\trm -f .git/objects/pack/*.keep &&\n+\t\tgit repack -adb &&\n+\t\treusable_pack () {\n+\t\t\tgit for-each-ref --format=\"%(objectname)\" |\n+\t\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n+\t\t}\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --honor-pack-keep' '\n+\t\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n+\t\tfor i in .git/objects/pack/*.pack\n+\t\tdo\n+\t\t\t>${i%.pack}.keep || return 1\n+\t\tdone &&\n+\t\treusable_pack --honor-pack-keep >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --local' '\n+\t\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n+\t\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n+\t\treusable_pack --local >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'pack reuse respects --incremental' '\n+\t\treusable_pack --incremental >empty.pack &&\n+\t\tgit index-pack empty.pack &&\n+\t\tgit show-index actual &&\n+\t\ttest_must_be_empty actual\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\t\ttest_config pack.writebitmaphashcache false &&\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+\t'\n+\n+\ttest_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\t\tgit repack -ad &&\n+\t\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\t\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\t\ttest_when_finished \"rm -f $bitmap\" &&\n+\t\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\t\tmv -f $bitmap.tmp $bitmap &&\n+\t\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest_i18ngrep corrupted.bitmap.index stderr\n+\t'\n+\n+\t# Create a state of history with these properties:\n+\t#\n+\t# - refs that allow a client to fetch some new history, while sharing some old\n+\t# history with the server; we use branches delta-reuse-old and\n+\t# delta-reuse-new here\n+\t#\n+\t# - the new history contains an object that is stored on the server as a delta\n+\t# against a base that is in the old history\n+\t#\n+\t# - the base object is not immediately reachable from the tip of the old\n+\t# history; finding it would involve digging down through history we know the\n+\t# other side has\n+\t#\n+\t# This should result in a state where fetching from old->new would not\n+\t# traditionally reuse the on-disk delta (because we'd have to dig to realize\n+\t# that the client has it), but we will do so if bitmaps can tell us cheaply\n+\t# that the other side has it.\n+\ttest_expect_success 'set up thin delta-reuse parent' '\n+\t\t# This first commit contains the buried base object.\n+\t\ttest-tool genrandom delta 16384 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m \"delta base\" &&\n+\t\tbase=$(git rev-parse --verify HEAD:file) &&\n+\n+\t\t# These intermediate commits bury the base back in history.\n+\t\t# This becomes the \"old\" state.\n+\t\tfor i in 1 2 3 4 5\n+\t\tdo\n+\t\t\techo $i >file &&\n+\t\t\tgit commit -am \"intermediate $i\" || return 1\n+\t\tdone &&\n+\t\tgit branch delta-reuse-old &&\n+\n+\t\t# And now our new history has a delta against the buried base. Note\n+\t\t# that this must be smaller than the original file, since pack-objects\n+\t\t# prefers to create deltas from smaller objects to larger.\n+\t\ttest-tool genrandom delta 16300 >file &&\n+\t\tgit commit -am \"delta result\" &&\n+\t\tdelta=$(git rev-parse --verify HEAD:file) &&\n+\t\tgit branch delta-reuse-new &&\n+\n+\t\t# Repack with bitmaps and double check that we have the expected delta\n+\t\t# relationship.\n+\t\tgit repack -adb &&\n+\t\thave_delta $delta $base\n+\t'\n+\n+\t# Now we can sanity-check the non-bitmap behavior (that the server is not able\n+\t# to reuse the delta). This isn't strictly something we care about, so this\n+\t# test could be scrapped in the future. But it makes sure that the next test is\n+\t# actually triggering the feature we want.\n+\t#\n+\t# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n+\t# we actually perform the fetch, retain the resulting pack, and inspect the\n+\t# result.\n+\ttest_expect_success 'fetch without bitmaps ignores delta against old base' '\n+\t\ttest_config pack.usebitmaps false &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $ZERO_OID\n+\t\t)\n+\t'\n+\n+\t# And do the same for the bitmap case, where we do expect to find the delta.\n+\ttest_expect_success 'fetch with bitmaps can reuse old base' '\n+\t\ttest_config pack.usebitmaps true &&\n+\t\ttest_when_finished \"rm -rf client.git\" &&\n+\t\tgit init --bare client.git &&\n+\t\t(\n+\t\t\tcd client.git &&\n+\t\t\tgit config transfer.unpackLimit 1 &&\n+\t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n+\t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n+\t\t\thave_delta $delta $base\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\t# create enough commits that not all are receive bitmap\n+\t\t\t# coverage even if they are all at the tip of some reference.\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\t\tgit rev-list HEAD >commits.raw &&\n+\t\t\tsort commits &&\n+\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin bitmaps &&\n+\n+\t\t\t# remember which commits did not receive bitmaps\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_file_not_empty before &&\n+\n+\t\t\t# mark the commits which did not receive bitmaps as preferred,\n+\t\t\t# and generate the bitmap again\n+\t\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'complains about multiple pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\n+\t\t\ttest_commit base &&\n+\n+\t\t\tgit repack -adb &&\n+\t\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n+\t\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n+\n+\t\t\ttest_commit other &&\n+\t\t\tgit repack -ab &&\n+\n+\t\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n+\n+\t\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n+\t\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n+\t\t\ttest_line_count = 2 packs &&\n+\t\t\ttest_line_count = 2 bitmaps &&\n+\n+\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\tgrep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-basic_bitmap_tests\n+test_bitmap_cases\n \n test_expect_success 'incremental repack fails when bitmaps are requested' '\n \ttest_commit more-1 &&\n@@ -54,375 +445,12 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n-test_expect_success 'pack-objects respects --local (non-local loose)' '\n-\tgit init --bare alt.git &&\n-\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n-\techo content1 >file1 &&\n-\t# non-local loose object which is not present in bitmapped pack\n-\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n-\t# non-local loose object which is also present in bitmapped pack\n-\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n-\tgit add file1 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file1 &&\n-\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n-\tgit index-pack 1.pack &&\n-\tlist_packed_objects 1.idx >1.objects &&\n-\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n-\t! has_any nonlocal-loose 1.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n-\techo content2 >file2 &&\n-\tblob2=$(git hash-object -w file2) &&\n-\tgit add file2 &&\n-\ttest_tick &&\n-\tgit commit -m commit_file2 &&\n-\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n-\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n-\trm $(objpath $blob2) &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n-\tgit index-pack 2a.pack &&\n-\tlist_packed_objects 2a.idx >2a.objects &&\n-\t! has_any keepobjects 2a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local pack)' '\n-\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n-\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n-\tgit index-pack 2b.pack &&\n-\tlist_packed_objects 2b.idx >2b.objects &&\n-\t! has_any keepobjects 2b.objects\n-'\n-\n-test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\tpackbitmap=$(basename $(cat output) .bitmap) &&\n-\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n-\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n-\t>.git/objects/pack/$packbitmap.keep &&\n-\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n-\tgit index-pack 3a.pack &&\n-\tlist_packed_objects 3a.idx >3a.objects &&\n-\t! has_any packbitmap.objects 3a.objects\n-'\n-\n-test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n-\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n-\trm -f .git/objects/pack/multi-pack-index &&\n-\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n-\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n-\tgit index-pack 3b.pack &&\n-\tlist_packed_objects 3b.idx >3b.objects &&\n-\t! has_any packbitmap.objects 3b.objects\n-'\n-\n-test_expect_success 'pack-objects to file can use bitmap' '\n-\t# make sure we still have 1 bitmap index from previous tests\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output &&\n-\t# verify equivalent packs are generated with/without using bitmap index\n-\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n-\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n-\ttest_cmp packa.objects packb.objects\n-'\n-\n-test_expect_success 'full repack, reusing previous bitmaps' '\n-\tgit repack -ad &&\n-\tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n-'\n-\n-test_expect_success 'fetch (full bitmap)' '\n-\tgit --git-dir=clone.git fetch origin second:second &&\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'create objects for missing-HAVE tests' '\n-\tblob=$(echo \"missing have\" | git hash-object -w --stdin) &&\n-\ttree=$(printf \"100644 blob $blob\\tfile\\n\" | git mktree) &&\n-\tparent=$(echo parent | git commit-tree $tree) &&\n-\tcommit=$(echo commit | git commit-tree $tree -p $parent) &&\n-\tcat >revs <<-EOF\n-\tHEAD\n-\t^HEAD^\n-\t^$commit\n-\tEOF\n-'\n-\n-test_expect_success 'pack-objects respects --incremental' '\n-\tcat >revs2 <<-EOF &&\n-\tHEAD\n-\t$commit\n-\tEOF\n-\tgit pack-objects --incremental --stdout --revs 4.pack &&\n-\tgit index-pack 4.pack &&\n-\tlist_packed_objects 4.idx >4.objects &&\n-\ttest_line_count = 4 4.objects &&\n-\tgit rev-list --objects $commit >revlist &&\n-\tcut -d\" \" -f1 revlist |sort >objects &&\n-\ttest_cmp 4.objects objects\n-'\n-\n-test_expect_success 'pack with missing blob' '\n-\trm $(objpath $blob) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n+test_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'pack with missing tree' '\n-\trm $(objpath $tree) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success 'pack with missing parent' '\n-\trm $(objpath $parent) &&\n-\tgit pack-objects --stdout --revs /dev/null\n-'\n-\n-test_expect_success JGIT,SHA1 'we can read jgit bitmaps' '\n-\tgit clone --bare . compat-jgit.git &&\n-\t(\n-\t\tcd compat-jgit.git &&\n-\t\trm -f objects/pack/*.bitmap &&\n-\t\tjgit gc &&\n-\t\tgit rev-list --test-bitmap HEAD\n-\t)\n-'\n-\n-test_expect_success JGIT,SHA1 'jgit can read our bitmaps' '\n-\tgit clone --bare . compat-us.git &&\n-\t(\n-\t\tcd compat-us.git &&\n-\t\tgit repack -adb &&\n-\t\t# jgit gc will barf if it does not like our bitmaps\n-\t\tjgit gc\n-\t)\n-'\n-\n-test_expect_success 'splitting packs does not generate bogus bitmaps' '\n-\ttest-tool genrandom foo $((1024 * 1024)) >rand &&\n-\tgit add rand &&\n-\tgit commit -m \"commit with big file\" &&\n-\tgit -c pack.packSizeLimit=500k repack -adb &&\n-\tgit init --bare no-bitmaps.git &&\n-\tgit -C no-bitmaps.git fetch .. HEAD\n-'\n-\n-test_expect_success 'set up reusable pack' '\n-\trm -f .git/objects/pack/*.keep &&\n-\tgit repack -adb &&\n-\treusable_pack () {\n-\t\tgit for-each-ref --format=\"%(objectname)\" |\n-\t\tgit pack-objects --delta-base-offset --revs --stdout \"$@\"\n-\t}\n-'\n-\n-test_expect_success 'pack reuse respects --honor-pack-keep' '\n-\ttest_when_finished \"rm -f .git/objects/pack/*.keep\" &&\n-\tfor i in .git/objects/pack/*.pack\n-\tdo\n-\t\t>${i%.pack}.keep || return 1\n-\tdone &&\n-\treusable_pack --honor-pack-keep >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --local' '\n-\tmv .git/objects/pack/* alt.git/objects/pack/ &&\n-\ttest_when_finished \"mv alt.git/objects/pack/* .git/objects/pack/\" &&\n-\treusable_pack --local >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'pack reuse respects --incremental' '\n-\treusable_pack --incremental >empty.pack &&\n-\tgit index-pack empty.pack &&\n-\tgit show-index actual &&\n-\ttest_must_be_empty actual\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n-\ttest_config pack.writebitmaphashcache false &&\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt.ewah.bitmap stderr\n-'\n-\n-test_expect_success 'truncated bitmap fails gracefully (cache)' '\n-\tgit repack -ad &&\n-\tgit rev-list --use-bitmap-index --count --all >expect &&\n-\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n-\ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n-\tmv -f $bitmap.tmp $bitmap &&\n-\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n-\ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupted.bitmap.index stderr\n-'\n-\n-# Create a state of history with these properties:\n-#\n-# - refs that allow a client to fetch some new history, while sharing some old\n-# history with the server; we use branches delta-reuse-old and\n-# delta-reuse-new here\n-#\n-# - the new history contains an object that is stored on the server as a delta\n-# against a base that is in the old history\n-#\n-# - the base object is not immediately reachable from the tip of the old\n-# history; finding it would involve digging down through history we know the\n-# other side has\n-#\n-# This should result in a state where fetching from old->new would not\n-# traditionally reuse the on-disk delta (because we'd have to dig to realize\n-# that the client has it), but we will do so if bitmaps can tell us cheaply\n-# that the other side has it.\n-test_expect_success 'set up thin delta-reuse parent' '\n-\t# This first commit contains the buried base object.\n-\ttest-tool genrandom delta 16384 >file &&\n-\tgit add file &&\n-\tgit commit -m \"delta base\" &&\n-\tbase=$(git rev-parse --verify HEAD:file) &&\n-\n-\t# These intermediate commits bury the base back in history.\n-\t# This becomes the \"old\" state.\n-\tfor i in 1 2 3 4 5\n-\tdo\n-\t\techo $i >file &&\n-\t\tgit commit -am \"intermediate $i\" || return 1\n-\tdone &&\n-\tgit branch delta-reuse-old &&\n-\n-\t# And now our new history has a delta against the buried base. Note\n-\t# that this must be smaller than the original file, since pack-objects\n-\t# prefers to create deltas from smaller objects to larger.\n-\ttest-tool genrandom delta 16300 >file &&\n-\tgit commit -am \"delta result\" &&\n-\tdelta=$(git rev-parse --verify HEAD:file) &&\n-\tgit branch delta-reuse-new &&\n-\n-\t# Repack with bitmaps and double check that we have the expected delta\n-\t# relationship.\n-\tgit repack -adb &&\n-\thave_delta $delta $base\n-'\n-\n-# Now we can sanity-check the non-bitmap behavior (that the server is not able\n-# to reuse the delta). This isn't strictly something we care about, so this\n-# test could be scrapped in the future. But it makes sure that the next test is\n-# actually triggering the feature we want.\n-#\n-# Note that our tools for working with on-the-wire \"thin\" packs are limited. So\n-# we actually perform the fetch, retain the resulting pack, and inspect the\n-# result.\n-test_expect_success 'fetch without bitmaps ignores delta against old base' '\n-\ttest_config pack.usebitmaps false &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $ZERO_OID\n-\t)\n-'\n-\n-# And do the same for the bitmap case, where we do expect to find the delta.\n-test_expect_success 'fetch with bitmaps can reuse old base' '\n-\ttest_config pack.usebitmaps true &&\n-\ttest_when_finished \"rm -rf client.git\" &&\n-\tgit init --bare client.git &&\n-\t(\n-\t\tcd client.git &&\n-\t\tgit config transfer.unpackLimit 1 &&\n-\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n-\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\n-\t\thave_delta $delta $base\n-\t)\n-'\n-\n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\t# create enough commits that not all are receive bitmap\n-\t\t# coverage even if they are all at the tip of some reference.\n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n-\n-\t\tgit rev-list HEAD >commits.raw &&\n-\t\tsort commits &&\n-\n-\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n-\n-\t\t# remember which commits did not receive bitmaps\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_file_not_empty before &&\n-\n-\t\t# mark the commits which did not receive bitmaps as preferred,\n-\t\t# and generate the bitmap again\n-\t\tperl -pe \"s{^}{create refs/tags/include/$. }\" bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n-\n-\t\t! test_cmp before after\n-\t)\n-'\n-\n-test_expect_success 'complains about multiple pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit base &&\n-\n-\t\tgit repack -adb &&\n-\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n-\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n-\n-\t\ttest_commit other &&\n-\t\tgit repack -ab &&\n-\n-\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n-\n-\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n-\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n-\t\ttest_line_count = 2 packs &&\n-\t\ttest_line_count = 2 bitmaps &&\n-\n-\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\tgrep \"ignoring extra bitmap file\" err\n-\t)\n+test_expect_success 'verify writing bitmap lookup table when enabled' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2\" \\\n+\t\tgit repack -ad &&\n+\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n test_done\ndiff --git a/t/t5311-pack-bitmaps-shallow.sh b/t/t5311-pack-bitmaps-shallow.sh\nindex 872a95df338..9dae60f73e3 100755\n--- a/t/t5311-pack-bitmaps-shallow.sh\n+++ b/t/t5311-pack-bitmaps-shallow.sh\n@@ -17,23 +17,40 @@ test_description='check bitmap operation with shallow repositories'\n # the tree for A. But in a shallow one, we've grafted away\n # A, and fetching A to B requires that the other side send\n # us the tree for file=1.\n-test_expect_success 'setup shallow repo' '\n-\techo 1 >file &&\n-\tgit add file &&\n-\tgit commit -m orig &&\n-\techo 2 >file &&\n-\tgit commit -a -m update &&\n-\tgit clone --no-local --bare --depth=1 . shallow.git &&\n-\techo 1 >file &&\n-\tgit commit -a -m repeat\n-'\n-\n-test_expect_success 'turn on bitmaps in the parent' '\n-\tgit repack -adb\n-'\n-\n-test_expect_success 'shallow fetch from bitmapped repo' '\n-\t(cd shallow.git && git fetch)\n-'\n+test_shallow_bitmaps () {\n+\twriteLookupTable=false\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup shallow repo' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\techo 1 >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m orig &&\n+\t\techo 2 >file &&\n+\t\tgit commit -a -m update &&\n+\t\tgit clone --no-local --bare --depth=1 . shallow.git &&\n+\t\techo 1 >file &&\n+\t\tgit commit -a -m repeat\n+\t'\n+\n+\ttest_expect_success 'turn on bitmaps in the parent' '\n+\t\tgit repack -adb\n+\t'\n+\n+\ttest_expect_success 'shallow fetch from bitmapped repo' '\n+\t\t(cd shallow.git && git fetch)\n+\t'\n+}\n+\n+test_shallow_bitmaps\n+test_shallow_bitmaps \"pack.writeBitmapLookupTable\"\n \n test_done\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4fe57414c13..3b206adcee6 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -15,17 +15,24 @@ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n sane_unset GIT_TEST_MIDX_WRITE_REV\n sane_unset GIT_TEST_MIDX_READ_RIDX\n \n-midx_bitmap_core\n-\n bitmap_reuse_tests() {\n \tfrom=$1\n \tto=$2\n+\twriteLookupTable=false\n+\n+\tfor i in $3-${$#}\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n \n \ttest_expect_success \"setup pack reuse tests ($from -> $to)\" '\n \t\trm -fr repo &&\n \t\tgit init repo &&\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk 16 &&\n \t\t\tgit tag old-tip &&\n \n@@ -43,6 +50,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"build bitmap from existing ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\ttest_commit_bulk --id=further 16 &&\n \t\t\tgit tag new-tip &&\n \n@@ -59,6 +67,7 @@ bitmap_reuse_tests() {\n \ttest_expect_success \"verify resulting bitmaps ($from -> $to)\" '\n \t\t(\n \t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \t\t\tgit for-each-ref &&\n \t\t\tgit rev-list --test-bitmap refs/tags/old-tip &&\n \t\t\tgit rev-list --test-bitmap refs/tags/new-tip\n@@ -66,244 +75,294 @@ bitmap_reuse_tests() {\n \t'\n }\n \n-bitmap_reuse_tests 'pack' 'MIDX'\n-bitmap_reuse_tests 'MIDX' 'pack'\n-bitmap_reuse_tests 'MIDX' 'MIDX'\n+test_midx_bitmap_cases () {\n+\twriteLookupTable=false\n+\twriteBitmapLookupTable=\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\")\n+\t\t\twriteLookupTable=true\n+\t\t\twriteBitmapLookupTable=\"$i\"\n+\t\t\t;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup test_repository' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n \n-test_expect_success 'missing object closure fails gracefully' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\tmidx_bitmap_core\n \n-\t\ttest_commit loose &&\n-\t\ttest_commit packed &&\n+\tbitmap_reuse_tests 'pack' 'MIDX' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'pack' \"$writeBitmapLookupTable\"\n+\tbitmap_reuse_tests 'MIDX' 'MIDX' \"$writeBitmapLookupTable\"\n \n-\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n-\t\t# commit.\n-\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n-\t\t$(git rev-parse packed)\n-\t\tEOF\n+\ttest_expect_success 'missing object closure fails gracefully' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n-\t\tgrep \"doesn.t have full closure\" err &&\n-\t\ttest_path_is_missing $midx\n-\t)\n-'\n+\t\t\ttest_commit loose &&\n+\t\t\ttest_commit packed &&\n \n-midx_bitmap_partial_tests\n+\t\t\t# Do not pass \"--revs\"; we want a pack without the \"loose\"\n+\t\t\t# commit.\n+\t\t\tgit pack-objects $objdir/pack/pack <<-EOF &&\n+\t\t\t$(git rev-parse packed)\n+\t\t\tEOF\n \n-test_expect_success 'removing a MIDX clears stale bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\t\ttest_commit base &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_must_fail git multi-pack-index write --bitmap 2>err &&\n+\t\t\tgrep \"doesn.t have full closure\" err &&\n+\t\t\ttest_path_is_missing $midx\n+\t\t)\n+\t'\n \n-\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n-\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm $midx &&\n+\tmidx_bitmap_partial_tests\n \n-\t\t# Then write a new MIDX.\n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\ttest_expect_success 'removing a MIDX clears stale bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n+\t\t\ttest_commit base &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n+\t\t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm $midx &&\n+\n+\t\t\t# Then write a new MIDX.\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest_path_is_missing $stale_bitmap\n+\t\t)\n+\t'\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_missing $stale_bitmap\n-\t)\n-'\n+\ttest_expect_success 'pack.preferBitmapTips' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'pack.preferBitmapTips' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\tperl -ne \"printf(\\\"create refs/tags/include/%d \\\", $.); print\" \\\n+\t\t\t\tbitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n \n-\t\tgit -c pack.preferBitmapTips=refs/tags/include \\\n-\t\t\tmulti-pack-index write --bitmap &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\ttest_expect_success 'writing a bitmap with --refs-snapshot' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'writing a bitmap with --refs-snapshot' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\ttest_commit two &&\n \n-\t\ttest_commit one &&\n-\t\ttest_commit two &&\n+\t\t\tgit rev-parse one >snapshot &&\n \n-\t\tgit rev-parse one >snapshot &&\n+\t\t\tgit repack -ad &&\n \n-\t\tgit repack -ad &&\n+\t\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t\t# bitmaps).\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\t# First, write a MIDX which see both refs/tags/one and\n-\t\t# refs/tags/two (causing both of those commits to receive\n-\t\t# bitmaps).\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\t# Then again, but with a refs snapshot which only sees\n+\t\t\t# refs/tags/one.\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n \n-\t\t# Then again, but with a refs snapshot which only sees\n-\t\t# refs/tags/one.\n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n-\t\t! grep \"$(git rev-parse two)\" bitmaps\n-\t)\n-'\n+\ttest_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_commit_bulk --message=\"%s\" 103 &&\n \n-\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\t\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\t\tsort commits &&\n \n-\t\tgit log --format=\"%H\" >commits.raw &&\n-\t\tsort commits &&\n+\t\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\t\tgit update-ref --stdin refs &&\n-\t\tgit update-ref --stdin bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >before &&\n+\t\t\ttest_line_count = 1 before &&\n \n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >before &&\n-\t\ttest_line_count = 1 before &&\n+\t\t\t(\n+\t\t\t\tgrep -vf before commits.raw &&\n+\t\t\t\t# mark missing commits as preferred\n+\t\t\t\tsed \"s/^/+/\" before\n+\t\t\t) >snapshot &&\n \n+\t\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\t\trm -fr $midx &&\n+\n+\t\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t\t! test_cmp before after\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'hash-cache values are propagated from pack bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n \t\t(\n-\t\t\tgrep -vf before commits.raw &&\n-\t\t\t# mark missing commits as preferred\n-\t\t\tsed \"s/^/+/\" before\n-\t\t) >snapshot &&\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx &&\n+\t\t\ttest_commit base &&\n+\t\t\ttest_commit base2 &&\n+\t\t\tgit repack -adb &&\n \n-\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n-\t\tcomm -13 bitmaps commits >after &&\n+\t\t\ttest-tool bitmap dump-hashes >pack.raw &&\n+\t\t\ttest_file_not_empty pack.raw &&\n+\t\t\tsort pack.raw >pack.hashes &&\n \n-\t\t! test_cmp before after\n-\t)\n-'\n+\t\t\ttest_commit new &&\n+\t\t\tgit repack &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest-tool bitmap dump-hashes >midx.raw &&\n+\t\t\tsort midx.raw >midx.hashes &&\n \n-\t\ttest_commit base &&\n-\t\ttest_commit base2 &&\n-\t\tgit repack -adb &&\n+\t\t\t# ensure that every namehash in the pack bitmap can be found in\n+\t\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n+\t\t\t# unique to the pack bitmap).\n+\t\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n+\t\t\ttest_must_be_empty dropped.hashes\n+\t\t)\n+\t'\n \n-\t\ttest-tool bitmap dump-hashes >pack.raw &&\n-\t\ttest_file_not_empty pack.raw &&\n-\t\tsort pack.raw >pack.hashes &&\n+\ttest_expect_success 'no .bitmap is written without any objects' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\ttest_commit new &&\n-\t\tgit repack &&\n-\t\tgit multi-pack-index write --bitmap &&\n+\t\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n+\t\t\tpack-$empty.idx\n+\t\t\tEOF\n \n-\t\ttest-tool bitmap dump-hashes >midx.raw &&\n-\t\tsort midx.raw >midx.hashes &&\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\terr &&\n \n-\t\t# ensure that every namehash in the pack bitmap can be found in\n-\t\t# the midx bitmap (i.e., that there are no oid-namehash pairs\n-\t\t# unique to the pack bitmap).\n-\t\tcomm -23 pack.hashes midx.hashes >dropped.hashes &&\n-\t\ttest_must_be_empty dropped.hashes\n-\t)\n-'\n+\t\t\tgrep \"bitmap without any objects\" err &&\n \n-test_expect_success 'no .bitmap is written without any objects' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n+\t\t\ttest_path_is_file $midx &&\n+\t\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n+\t\t)\n+\t'\n+\n+\ttest_expect_success 'graceful fallback when missing reverse index' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"' &&\n \n-\t\tempty=\"$(git pack-objects $objdir/pack/pack packs <<-EOF &&\n-\t\tpack-$empty.idx\n-\t\tEOF\n+\t\t\ttest_commit base &&\n \n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\terr &&\n+\t\t\t# write a pack and MIDX bitmap containing base\n+\t\t\tgit repack -adb &&\n+\t\t\tgit multi-pack-index write --bitmap &&\n \n-\t\tgrep \"bitmap without any objects\" err &&\n+\t\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n+\t\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t\t! grep \"ignoring extra bitmap file\" err\n+\t\t)\n+\t'\n+}\n \n-\t\ttest_path_is_file $midx &&\n-\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap\n-\t)\n-'\n+test_midx_bitmap_cases\n+\n+test_midx_bitmap_cases \"pack.writeBitmapLookupTable\"\n \n-test_expect_success 'graceful fallback when missing reverse index' '\n+test_expect_success 'multi-pack-index write writes lookup table if enabled' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n \t(\n \t\tcd repo &&\n-\n \t\ttest_commit base &&\n-\n-\t\t# write a pack and MIDX bitmap containing base\n-\t\tgit repack -adb &&\n-\t\tgit multi-pack-index write --bitmap &&\n-\n-\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n-\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n-\t\t! grep \"ignoring extra bitmap file\" err\n+\t\tgit config pack.writeBitmapLookupTable true &&\n+\t\tgit repack -ad &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit multi-pack-index write --bitmap &&\n+\t\tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace\n \t)\n '\n \ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nindex d30ba632c87..e65e311cd73 100755\n--- a/t/t5327-multi-pack-bitmaps-rev.sh\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -17,7 +17,27 @@ GIT_TEST_MIDX_READ_RIDX=0\n export GIT_TEST_MIDX_WRITE_REV\n export GIT_TEST_MIDX_READ_RIDX\n \n-midx_bitmap_core rev\n-midx_bitmap_partial_tests rev\n+test_midx_bitmap_rev () {\n+\twriteLookupTable=false\n+\n+\tfor i in \"$@\"\n+\tdo\n+\t\tcase $i in\n+\t\t\"pack.writeBitmapLookupTable\") writeLookupTable=true;;\n+\t\tesac\n+\tdone\n+\n+\ttest_expect_success 'setup bitmap config' '\n+\t\trm -rf * .git &&\n+\t\tgit init &&\n+\t\tgit config pack.writeBitmapLookupTable '\"$writeLookupTable\"'\n+\t'\n+\n+\tmidx_bitmap_core rev\n+\tmidx_bitmap_partial_tests rev\n+}\n+\n+test_midx_bitmap_rev\n+test_midx_bitmap_rev \"pack.writeBitmapLookupTable\"\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"461198","messageId":"b460516b306e6885cd1c0af1c3379fb953952de2.1660496112.git.gitgitgadget@gmail.com","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"[PATCH v6 6/6] bitmap-lookup-table: add performance tests for lookup table","fromName":"Abhradeep Chakraborty via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-14T16:55:11Z","receivedAt":"2022-08-14T17:00:25Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty \n\nAdd performance tests to verify the performance of lookup table.\n`p5310-pack-bitmaps.sh` contain tests with and without lookup table.\n`p5312-pack-bitmaps-revs.sh` contain same tests with and without\nlookup table but with `pack.writeReverseIndex` enabled.\n\nLookup table makes Git run faster in most of the cases. Below is the\nresult of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh`\ngives similar result. The repository used in the test is linux kernel.\n\nTest this tree\n-----------------------------------------------------------------------\n5310.4: enable lookup table: false 0.01(0.00+0.00)\n5310.5: repack to disk 320.89(230.20+23.45)\n5310.6: simulated clone 14.04(5.78+1.79)\n5310.7: simulated fetch 1.95(3.05+0.20)\n5310.8: pack to file (bitmap) 44.73(20.55+7.45)\n5310.9: rev-list (commits) 0.78(0.46+0.10)\n5310.10: rev-list (objects) 4.07(3.97+0.08)\n5310.11: rev-list with tag negated via --not 0.06(0.02+0.03)\n --all (objects)\n5310.12: rev-list with negative tag (objects) 0.21(0.15+0.05)\n5310.13: rev-list count with blob:none 0.24(0.17+0.06)\n5310.14: rev-list count with blob:limit=1k 7.07(5.92+0.48)\n5310.15: rev-list count with tree:0 0.25(0.17+0.07)\n5310.16: simulated partial clone 5.67(3.28+0.64)\n5310.18: clone (partial bitmap) 16.05(8.34+1.86)\n5310.19: pack to file (partial bitmap) 59.76(27.22+7.43)\n5310.20: rev-list with tree filter (partial bitmap) 0.90(0.18+0.16)\n5310.24: enable lookup table: true 0.01(0.00+0.00)\n5310.25: repack to disk 319.73(229.30+23.01)\n5310.26: simulated clone 13.69(5.72+1.78)\n5310.27: simulated fetch 1.84(3.02+0.16)\n5310.28: pack to file (bitmap) 45.63(20.67+7.50)\n5310.29: rev-list (commits) 0.56(0.39+0.8)\n5310.30: rev-list (objects) 3.77(3.74+0.08)\n5310.31: rev-list with tag negated via --not 0.05(0.02+0.03)\n --all (objects)\n5310.32: rev-list with negative tag (objects) 0.21(0.15+0.05)\n5310.33: rev-list count with blob:none 0.23(0.17+0.05)\n5310.34: rev-list count with blob:limit=1k 6.65(5.72+0.40)\n5310.35: rev-list count with tree:0 0.23(0.16+0.06)\n5310.36: simulated partial clone 5.57(3.26+0.59)\n5310.38: clone (partial bitmap) 15.89(8.39+1.84)\n5310.39: pack to file (partial bitmap) 58.32(27.55+7.47)\n5310.40: rev-list with tree filter (partial bitmap) 0.73(0.18+0.15)\n\nTest 4-15 are tested without using lookup table. Same tests are\nrepeated in 16-30 (using lookup table).\n\nMentored-by: Taylor Blau \nCo-Mentored-by: Kaartic Sivaraam \nSigned-off-by: Abhradeep Chakraborty \n---\n t/perf/lib-bitmap.sh | 31 +++++++++\n t/perf/p5310-pack-bitmaps.sh | 78 +++++++++-------------\n t/perf/p5311-pack-bitmaps-fetch.sh | 74 ++++++++++++---------\n t/perf/p5312-pack-bitmaps-revs.sh | 35 ++++++++++\n t/perf/p5326-multi-pack-bitmaps.sh | 103 +++++++++++++++++------------\n 5 files changed, 199 insertions(+), 122 deletions(-)\n create mode 100755 t/perf/p5312-pack-bitmaps-revs.sh\n\ndiff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh\nindex 63d3bc7cece..55a8feb1dc4 100644\n--- a/t/perf/lib-bitmap.sh\n+++ b/t/perf/lib-bitmap.sh\n@@ -67,3 +67,34 @@ test_partial_bitmap () {\n \t\t\t--filter=tree:0 >/dev/null\n \t'\n }\n+\n+test_pack_bitmap () {\n+\ttest_perf \"repack to disk\" '\n+\t\tgit repack -ad\n+\t'\n+\n+\ttest_full_bitmap\n+\n+\ttest_expect_success \"create partial bitmap state\" '\n+\t\t# pick a commit to represent the repo tip in the past\n+\t\tcutoff=$(git rev-list HEAD~100 -1) &&\n+\t\torig_tip=$(git rev-parse HEAD) &&\n+\n+\t\t# now kill off all of the refs and pretend we had\n+\t\t# just the one tip\n+\t\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n+\t\tgit update-ref HEAD $cutoff &&\n+\n+\t\t# and then repack, which will leave us with a nice\n+\t\t# big bitmap pack of the \"old\" history, and all of\n+\t\t# the new history will be loose, as if it had been pushed\n+\t\t# up incrementally and exploded via unpack-objects\n+\t\tgit repack -Ad &&\n+\n+\t\t# and now restore our original tip, as if the pushes\n+\t\t# had happened\n+\t\tgit update-ref HEAD $orig_tip\n+\t'\n+\n+\ttest_partial_bitmap\n+}\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex 7ad4f237bc3..b1399f1007e 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -4,51 +4,37 @@ test_description='Tests pack performance using bitmaps'\n . ./perf-lib.sh\n . \"${TEST_DIRECTORY}/perf/lib-bitmap.sh\"\n \n-test_perf_large_repo\n-\n-# note that we do everything through config,\n-# since we want to be able to compare bitmap-aware\n-# git versus non-bitmap git\n-#\n-# We intentionally use the deprecated pack.writebitmaps\n-# config so that we can test against older versions of git.\n-test_expect_success 'setup bitmap config' '\n-\tgit config pack.writebitmaps true\n-'\n-\n-# we need to create the tag up front such that it is covered by the repack and\n-# thus by generated bitmaps.\n-test_expect_success 'create tags' '\n-\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n-'\n-\n-test_perf 'repack to disk' '\n-\tgit repack -ad\n-'\n-\n-test_full_bitmap\n-\n-test_expect_success 'create partial bitmap state' '\n-\t# pick a commit to represent the repo tip in the past\n-\tcutoff=$(git rev-list HEAD~100 -1) &&\n-\torig_tip=$(git rev-parse HEAD) &&\n-\n-\t# now kill off all of the refs and pretend we had\n-\t# just the one tip\n-\trm -rf .git/logs .git/refs/* .git/packed-refs &&\n-\tgit update-ref HEAD $cutoff &&\n-\n-\t# and then repack, which will leave us with a nice\n-\t# big bitmap pack of the \"old\" history, and all of\n-\t# the new history will be loose, as if it had been pushed\n-\t# up incrementally and exploded via unpack-objects\n-\tgit repack -Ad &&\n-\n-\t# and now restore our original tip, as if the pushes\n-\t# had happened\n-\tgit update-ref HEAD $orig_tip\n-'\n-\n-test_partial_bitmap\n+test_lookup_pack_bitmap () {\n+\ttest_expect_success 'start the test from scratch' '\n+\t\trm -rf * .git\n+\t'\n+\n+\ttest_perf_large_repo\n+\n+\t# note that we do everything through config,\n+\t# since we want to be able to compare bitmap-aware\n+\t# git versus non-bitmap git\n+\t#\n+\t# We intentionally use the deprecated pack.writebitmaps\n+\t# config so that we can test against older versions of git.\n+\ttest_expect_success 'setup bitmap config' '\n+\t\tgit config pack.writebitmaps true\n+\t'\n+\n+\t# we need to create the tag up front such that it is covered by the repack and\n+\t# thus by generated bitmaps.\n+\ttest_expect_success 'create tags' '\n+\t\tgit tag --message=\"tag pointing to HEAD\" perf-tag HEAD\n+\t'\n+\n+\ttest_perf \"enable lookup table: $1\" '\n+\t\tgit config pack.writeBitmapLookupTable '\"$1\"'\n+\t'\n+\n+\ttest_pack_bitmap\n+}\n+\n+test_lookup_pack_bitmap false\n+test_lookup_pack_bitmap true\n \n test_done\ndiff --git a/t/perf/p5311-pack-bitmaps-fetch.sh b/t/perf/p5311-pack-bitmaps-fetch.sh\nindex 47c3fd7581c..426fab87e32 100755\n--- a/t/perf/p5311-pack-bitmaps-fetch.sh\n+++ b/t/perf/p5311-pack-bitmaps-fetch.sh\n@@ -3,42 +3,52 @@\n test_description='performance of fetches from bitmapped packs'\n . ./perf-lib.sh\n \n-test_perf_default_repo\n-\n-test_expect_success 'create bitmapped server repo' '\n-\tgit config pack.writebitmaps true &&\n-\tgit repack -ad\n-'\n-\n-# simulate a fetch from a repository that last fetched N days ago, for\n-# various values of N. We do so by following the first-parent chain,\n-# and assume the first entry in the chain that is N days older than the current\n-# HEAD is where the HEAD would have been then.\n-for days in 1 2 4 8 16 32 64 128; do\n-\ttitle=$(printf '%10s' \"($days days)\")\n-\ttest_expect_success \"setup revs from $days days ago\" '\n-\t\tnow=$(git log -1 --format=%ct HEAD) &&\n-\t\tthen=$(($now - ($days * 86400))) &&\n-\t\ttip=$(git rev-list -1 --first-parent --until=$then HEAD) &&\n-\t\t{\n-\t\t\techo HEAD &&\n-\t\t\techo ^$tip\n-\t\t} >revs\n+test_fetch_bitmaps () {\n+\ttest_expect_success 'setup test directory' '\n+\t\trm -fr * .git\n \t'\n \n-\ttest_perf \"server $title\" '\n-\t\tgit pack-objects --stdout --revs \\\n-\t\t\t\t --thin --delta-base-offset \\\n-\t\t\t\t tmp.pack\n-\t'\n+\ttest_perf_default_repo\n \n-\ttest_size \"size $title\" '\n-\t\twc -c revs\n+\t\t'\n+\n+\t\ttest_perf \"server $title (lookup=$1)\" '\n+\t\t\tgit pack-objects --stdout --revs \\\n+\t\t\t\t\t--thin --delta-base-offset \\\n+\t\t\t\t\ttmp.pack\n+\t\t'\n+\n+\t\ttest_size \"size $title\" '\n+\t\t\twc -c Hi Abhradeep,\n>\n> On Tue, 9 Aug 2022, Abhradeep Chakraborty wrote:\n>\n> > I noticed in the 'setup partial bitmaps' test case that if we comment\n> > out the line `git repack &&` , it runs successfully.\n> >\n> > test_expect_success 'setup partial bitmaps' '\n> > test_commit packed &&\n> > # git repack &&\n> > test_commit loose &&\n> > git multi-pack-index write --bitmap 2>err &&\n> > ...\n> > '\n>\n> That's interesting. Are the `.bitmap` and `.midx` files updated as part of\n> that `repack`?\n\nThey aren't. You can cause a MIDX / bitmap to be updated during `git\nrepack` provided that the flags `--write-midx` and\n`--write-bitmap-index` are given to `repack`.\n\nBut the point of that `git repack` in this test case specifically is to\nensure that the commit generated on the previous line is included in a\nnew pack, and that that pack makes its way into the MIDX.\n\nSo removing that invocation of `git repack` means that the set of packs\nwould be unchanged, and the `git multi-pack-index write --bitmap` would\nbe a noop. That should rule out the theory that the existing MIDX is\nbroken, since without the `git repack`, we'd be using that MIDX in\nsubsequent tests (which pass).\n\nThanks,\nTaylor\n"},{"id":"461314","messageId":"YvwS5RlEMvgDm93m@nand.local","threadId":"58038","inReplyTo":"CAPOJW5z99b0_NGBYDbZUvmzbWECJKxGvB4RffoPJYszfFB0cEg@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-08-16T21:57:57Z","receivedAt":"2022-08-16T21:58:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Abhradeep,\n\nOn Sat, Aug 13, 2022 at 04:29:32PM +0530, Abhradeep Chakraborty wrote:\n> One thing that really worries me is what if the failure is not related\n> to calling `oe_map_new_pack()? I did all my work assuming that this\n> function is the culprit. But I don't know if it is.\n\nAfter much consternation, I was able to rule out `oe_map_new_pack()` as\nthe culprit.\n\n(Your find that we call `add_packed_git()` with arguments corresponding\nto pack(s) that we've already loaded is good, and I think that is\ndefinitely something we can and should consider cleaning up. But it\nultimately doesn't affect correctness, just the memory efficiency of the\nprocess).\n\nWhen I took a close look at the process to generate MIDX bitmaps, I found a\ncouple of interesting things. The first more trivial fix is that we\nincorrectly propagate the \"preferred\"-ness bit from packs in an existing\nMIDX when generating a new one. If the identity of the preferred pack\nchanges, we should not drag forward those bits on objects already known\n(and preferred) by the existing MIDX:\n\n--- >8 ---\n\ndiff --git a/midx.c b/midx.c\nindex 3ff6e91e6e..40e520534c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -619,6 +619,9 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n\n \t\tif (m) {\n \t\t\tuint32_t start = 0, end;\n+\t\t\tint orig_preferred_pack = -1;\n+\t\t\tif (0 <= preferred_pack && preferred_pack < m->num_packs)\n+\t\t\t\torig_preferred_pack = info[preferred_pack].orig_pack_int_id;\n\n \t\t\tif (cur_fanout)\n \t\t\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n@@ -629,7 +632,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n-\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n+\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == orig_preferred_pack)\n \t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 1;\n \t\t\t\telse\n \t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 0;\n\n--- 8< ---\n\nBut a more interesting problem arose when I took a closer look at the\npsuedo-pack order of objects generated according to\n`prepare_midx_packing_data()`. With Johannes' fixed $test_tick value, I\nwas able to see the following in runs that succeeded:\n\n 27bb4ecd3e96cd0b3bc37d92a78cb5cbf34c418afa67f74cc52517ff7df418e1 (12 in pack-63c460f99a5c08f631396b1828c64006170a9d543b064506fd11b504a62acf52.idx)\n c68154d69c19f010afce786c6debe926ae6e7decfb946a4549085a792cf9de7e (202 in pack-63c460f99a5c08f631396b1828c64006170a9d543b064506fd11b504a62acf52.idx)\n a0b85b314ede46aa9f9b5796a284a4cf0b86ebb8fa32f87ae246e21b5378b11c (392 in pack-63c460f99a5c08f631396b1828c64006170a9d543b064506fd11b504a62acf52.idx)\n [...]\n\nand the following in runs that failed:\n\n 46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b (221 in pack-3fc052de674e3d48096af7cc5125675c0ae1082aa798eb9358de357b2655f9ad.idx)\n 67df8a01ac84cf5f028855c48384eac3336bb02a52603bac285c4b31d66b3ab5 (12 in pack-2021cdedb33b542b244eacf3d009d1384471a53286b0c1235c91d124355dc818.idx)\n 1556b5f0ad7cb0c25a1fc47355fcffc00775e90d94ae8c511e5776b204796ce6 (200 in pack-2021cdedb33b542b244eacf3d009d1384471a53286b0c1235c91d124355dc818.idx)\n\nIn the successful case, pack 63c460f99a... is preferred, and its objects\nappear in ascending order of their pack offsets. But in the other case,\npack 3fc052de67... is preferred, but its first object starts at offset\n221. Huh? That's not right:\n\n $ git show-index <.git/objects/pack/pack-3fc052de674e3d48096af7cc5125675c0ae1082aa798eb9358de357b2655f9ad.idx\n 221 46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b (1f4bd28e)\n 12 4d332072f161629ffe4652ecd3ce377ef88447bec73f05ab0f3515f98bd061cf (fadf885b)\n\nIndeed, there is another object there at offset 12. Missing that object\n(since it comes from a preferred pack) is an invariant violation (since\nall objects from the preferred pack should be selected when multiple\ncopies are available).\n\nIt's missing because the existing MIDX selects that object from a\ndifferent pack, and when we get to fanout 0x4d (the one which should\ninclude that object), we skip over seeing its copy in the preferred pack\nbecause that pack already appears in the existing MIDX, though it wasn't\npreferred.\n\nI think there are a couple of ways to fix this. The easiest thing to do\nwould be to force the identity of the preferred pack to be the same when\ngenerating a MIDX bitmap *while reusing an existing MIDX*, since that is\nthe only time this bug can happen.\n\nBut that's a little magical for my taste. I think a more reasonable fix\nwould be to include copies of all objects from the preferred pack\nincluding in the case where that pack was non-preferred in an existing\nMIDX and at least one object in that pack was selected from a different\npack in the existing MIDX.\n\nAbhradeep -- let me know if this is something you want to look into. I\nthink it's a very worthwhile bug to fix, since it is definitely\ntrigger-able in the wild (notably, only with `git multi-pack-index write\n--bitmap` without `--stdin-packs` and only under certain circumstances),\nand not just limited to SHA-256 mode.\n\nIf you are busy experimenting with CRoaring, that's no problem and I can\nfix this up, too. Either way, it would be worth you and others weighing\nin on which fix you think is worth pursuing.\n\nPhew.\n\nThanks,\nTaylor\n"},{"id":"461358","messageId":"CAPOJW5znYngr4n4tBBCgqZY4Hr38NArHC7Go=ujDkmsFXY57mQ@mail.gmail.com","threadId":"58038","inReplyTo":"YvwS5RlEMvgDm93m@nand.local","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-17T10:02:31Z","receivedAt":"2022-08-17T10:02:48Z","isPatch":true,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Hello Taylor, extremely thanks for finding the reason for this failure.\n\nOn Wed, Aug 17, 2022 at 3:28 AM Taylor Blau wrote:\n>\n> Hi Abhradeep,\n>\n> When I took a close look at the process to generate MIDX bitmaps, I found a\n> couple of interesting things. The first more trivial fix is that we\n> incorrectly propagate the \"preferred\"-ness bit from packs in an existing\n> MIDX when generating a new one. If the identity of the preferred pack\n> changes, we should not drag forward those bits on objects already known\n> (and preferred) by the existing MIDX:\n>\n> --- >8 ---\n>\n> diff --git a/midx.c b/midx.c\n> index 3ff6e91e6e..40e520534c 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -619,6 +619,9 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n>\n> if (m) {\n> uint32_t start = 0, end;\n> + int orig_preferred_pack = -1;\n> + if (0 <= preferred_pack && preferred_pack < m->num_packs)\n> + orig_preferred_pack = info[preferred_pack].orig_pack_int_id;\n>\n> if (cur_fanout)\n> start = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n> @@ -629,7 +632,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> nth_midxed_pack_midx_entry(m,\n> &entries_by_fanout[nr_fanout],\n> cur_object);\n> - if (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n> + if (nth_midxed_pack_int_id(m, cur_object) == orig_preferred_pack)\n> entries_by_fanout[nr_fanout].preferred = 1;\n> else\n> entries_by_fanout[nr_fanout].preferred = 0;\n>\n> --- 8< ---\n\nI am not able to understand this modification.\n`info[preferred_pack].orig_pack_int_id` and `preferred_pack` have the\nsame value, right? I see `ctx.info` getting sorted only after calling\n`get_sorted_entries()` function.\n\n> But a more interesting problem arose when I took a closer look at the\n> psuedo-pack order of objects generated according to\n> `prepare_midx_packing_data()`. With Johannes' fixed $test_tick value, I\n> was able to see the following in runs that succeeded:\n>\n> 27bb4ecd3e96cd0b3bc37d92a78cb5cbf34c418afa67f74cc52517ff7df418e1 (12 in pack-63c460f99a5c08f631396b1828c64006170a9d543b064506fd11b504a62acf52.idx)\n> c68154d69c19f010afce786c6debe926ae6e7decfb946a4549085a792cf9de7e (202 in pack-63c460f99a5c08f631396b1828c64006170a9d543b064506fd11b504a62acf52.idx)\n> a0b85b314ede46aa9f9b5796a284a4cf0b86ebb8fa32f87ae246e21b5378b11c (392 in pack-63c460f99a5c08f631396b1828c64006170a9d543b064506fd11b504a62acf52.idx)\n> [...]\n>\n> and the following in runs that failed:\n>\n> 46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b (221 in pack-3fc052de674e3d48096af7cc5125675c0ae1082aa798eb9358de357b2655f9ad.idx)\n> 67df8a01ac84cf5f028855c48384eac3336bb02a52603bac285c4b31d66b3ab5 (12 in pack-2021cdedb33b542b244eacf3d009d1384471a53286b0c1235c91d124355dc818.idx)\n> 1556b5f0ad7cb0c25a1fc47355fcffc00775e90d94ae8c511e5776b204796ce6 (200 in pack-2021cdedb33b542b244eacf3d009d1384471a53286b0c1235c91d124355dc818.idx)\n>\n> In the successful case, pack 63c460f99a... is preferred, and its objects\n> appear in ascending order of their pack offsets. But in the other case,\n> pack 3fc052de67... is preferred, but its first object starts at offset\n> 221. Huh? That's not right:\n>\n> $ git show-index <.git/objects/pack/pack-3fc052de674e3d48096af7cc5125675c0ae1082aa798eb9358de357b2655f9ad.idx\n> 221 46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b (1f4bd28e)\n> 12 4d332072f161629ffe4652ecd3ce377ef88447bec73f05ab0f3515f98bd061cf (fadf885b)\n>\n> Indeed, there is another object there at offset 12. Missing that object\n> (since it comes from a preferred pack) is an invariant violation (since\n> all objects from the preferred pack should be selected when multiple\n> copies are available).\n>\n> It's missing because the existing MIDX selects that object from a\n> different pack, and when we get to fanout 0x4d (the one which should\n> include that object), we skip over seeing its copy in the preferred pack\n> because that pack already appears in the existing MIDX, though it wasn't\n> preferred.\n\nahh, now I understand what the problem was actually. Thanks :)\n\n> I think there are a couple of ways to fix this. The easiest thing to do\n> would be to force the identity of the preferred pack to be the same when\n> generating a MIDX bitmap *while reusing an existing MIDX*, since that is\n> the only time this bug can happen.\n>\n> But that's a little magical for my taste. I think a more reasonable fix\n> would be to include copies of all objects from the preferred pack\n> including in the case where that pack was non-preferred in an existing\n> MIDX and at least one object in that pack was selected from a different\n> pack in the existing MIDX.\n\nI think the later approach makes the most sense to me. It might not be\na good idea to keep the same pack as `preferred` as a better candidate\nwould be ignored in that case.\n\n> Abhradeep -- let me know if this is something you want to look into. I\n> think it's a very worthwhile bug to fix, since it is definitely\n> trigger-able in the wild (notably, only with `git multi-pack-index write\n> --bitmap` without `--stdin-packs` and only under certain circumstances),\n> and not just limited to SHA-256 mode.\n>\n> If you are busy experimenting with CRoaring, that's no problem and I can\n> fix this up, too. Either way, it would be worth you and others weighing\n> in on which fix you think is worth pursuing.\n\nI will be happy to fix it but I can't work on it right now (neither on\nCRoaring) because I am currently preparing for my exam. I can continue\nmy work after that (i.e. from 19 aug). If you feel it is getting too\nlate then you can do this too. I am also thinking of writing a patch\nfor bitmap specific test dump tool (as Johannes proposed previously).\n\nMy exam dates are 18 Aug, 31 Aug, 1 Sep, 2 Sep and 3 Sep (I know the\ndates are weird) The dates are adjusted on request for Smart India\nHackathon ( 24 Aug - 27 Aug).\n\nThanks :)\n"},{"id":"461388","messageId":"Yv1RtjTkByAdOvjE@nand.local","threadId":"58038","inReplyTo":"CAPOJW5znYngr4n4tBBCgqZY4Hr38NArHC7Go=ujDkmsFXY57mQ@mail.gmail.com","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-08-17T20:38:14Z","receivedAt":"2022-08-17T20:38:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 17, 2022 at 03:32:31PM +0530, Abhradeep Chakraborty wrote:\n> Hello Taylor, extremely thanks for finding the reason for this failure.\n\nNo problem. I appreciate all of the time and effort that you, Dscho, and\nStolee all put into looking into this (especially while I was out).\n\nMy experience with the bitmap code is that it can be somewhat difficult\nto work in, because there are both (a) many ways to introduce bugs, and\n(b) the effect of a bug can occur far away from the source of the bug.\nThose two together make debugging difficult, at least for me.\n\nI think that having a test-tool (like Dscho suggested) to dump some\nbasic information about a bitmap's structure would be quite helpful in\nthe future.\n\n> On Wed, Aug 17, 2022 at 3:28 AM Taylor Blau wrote:\n> I am not able to understand this modification.\n> `info[preferred_pack].orig_pack_int_id` and `preferred_pack` have the\n> same value, right? I see `ctx.info` getting sorted only after calling\n> `get_sorted_entries()` function.\n\nYeah, I realized that this is bogus. For one, (as you note) those have\nthe same value before setting up the pack_perm array. But it also goes\nagainst the grain of what we're trying to do: the point is that the\nprefered-ness of objects in an existing MIDX should be discarded when\ngenerating a new pseudo-pack order.\n\n> > I think there are a couple of ways to fix this. The easiest thing to do\n> > would be to force the identity of the preferred pack to be the same when\n> > generating a MIDX bitmap *while reusing an existing MIDX*, since that is\n> > the only time this bug can happen.\n> >\n> > But that's a little magical for my taste. I think a more reasonable fix\n> > would be to include copies of all objects from the preferred pack\n> > including in the case where that pack was non-preferred in an existing\n> > MIDX and at least one object in that pack was selected from a different\n> > pack in the existing MIDX.\n>\n> I think the later approach makes the most sense to me. It might not be\n> a good idea to keep the same pack as `preferred` as a better candidate\n> would be ignored in that case.\n\nYep, I agree. Users should feel free to change the identity of the\npreferred pack when rewriting a MIDX regardless of whether or not they\nare using `--stdin-packs`.\n\n> > Abhradeep -- let me know if this is something you want to look into. I\n> > think it's a very worthwhile bug to fix, since it is definitely\n> > trigger-able in the wild (notably, only with `git multi-pack-index write\n> > --bitmap` without `--stdin-packs` and only under certain circumstances),\n> > and not just limited to SHA-256 mode.\n> >\n> > If you are busy experimenting with CRoaring, that's no problem and I can\n> > fix this up, too. Either way, it would be worth you and others weighing\n> > in on which fix you think is worth pursuing.\n>\n> I will be happy to fix it but I can't work on it right now (neither on\n> CRoaring) because I am currently preparing for my exam. I can continue\n> my work after that (i.e. from 19 aug). If you feel it is getting too\n> late then you can do this too. I am also thinking of writing a patch\n> for bitmap specific test dump tool (as Johannes proposed previously).\n\nNo problem. I wrote up some patches today myself that implement the\nabove fix. I haven't polished them up yet, but they are available here:\n\n https://github.com/ttaylorr/git/compare/master...ttaylorr:git:tb/bitmap-use-existing-preferred\n\nI want to add a more direct reproduction that works in both SHA-1 and\nSHA-256 to demonstrate that these patches fix the issue. But in the\nmeantime, you can use Dscho's reproduction with these patches (based on\nthe tip of `master`) applied on top and observe that it passes\nconsistently.\n\nThanks,\nTaylor\n"},{"id":"461622","messageId":"xmqqlerkj5f9.fsf@gitster.g","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-19T21:21:14Z","receivedAt":"2022-08-19T21:21:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Abhradeep Chakraborty via GitGitGadget\" \nwrites:\n\n> When parsing the .bitmap file, git loads all the bitmaps one by one even if\n> some of the bitmaps are not necessary. We can remove this overhead by\n> loading only the necessary bitmaps. A look up table extension can solve this\n> issue.\n>\n> Changes since v5:\n>\n> As the failure in the test case is not due to this code, I think it makes no\n> sense to delay the patch further.\n>\n> * The performance test changes were not accurate as the second\n> test_bitmap_cases call using the repo built for the previous call. This\n> version fixes that.\n> * Taylor suggested some minor changes. Those are addressed in this version.\n\nThe discussion on v5 was quite active, but we haven't seen any\ntraffic on this round. Is everybody happy with what we see here?\n\n"},{"id":"461632","messageId":"YwAFcax1Lmei6NMS@nand.local","threadId":"58038","inReplyTo":"Yv1RtjTkByAdOvjE@nand.local","subject":"Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-08-19T21:49:37Z","receivedAt":"2022-08-19T21:49:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Abhradeep,\n\nOn Wed, Aug 17, 2022 at 04:38:14PM -0400, Taylor Blau wrote:\n> > > Abhradeep -- let me know if this is something you want to look into. I\n> > > think it's a very worthwhile bug to fix, since it is definitely\n> > > trigger-able in the wild (notably, only with `git multi-pack-index write\n> > > --bitmap` without `--stdin-packs` and only under certain circumstances),\n> > > and not just limited to SHA-256 mode.\n> > >\n> > > If you are busy experimenting with CRoaring, that's no problem and I can\n> > > fix this up, too. Either way, it would be worth you and others weighing\n> > > in on which fix you think is worth pursuing.\n> >\n> > I will be happy to fix it but I can't work on it right now (neither on\n> > CRoaring) because I am currently preparing for my exam. I can continue\n> > my work after that (i.e. from 19 aug). If you feel it is getting too\n> > late then you can do this too. I am also thinking of writing a patch\n> > for bitmap specific test dump tool (as Johannes proposed previously).\n>\n> No problem. I wrote up some patches today myself that implement the\n> above fix. I haven't polished them up yet, but they are available here:\n>\n> https://github.com/ttaylorr/git/compare/master...ttaylorr:git:tb/bitmap-use-existing-preferred\n>\n> I want to add a more direct reproduction that works in both SHA-1 and\n> SHA-256 to demonstrate that these patches fix the issue. But in the\n> meantime, you can use Dscho's reproduction with these patches (based on\n> the tip of `master`) applied on top and observe that it passes\n> consistently.\n\nThat is now done and I sent the resulting patch series to the list,\nwhich I'd encourage you to review here:\n\n https://lore.kernel.org/git/cover.1660944574.git.me@ttaylorr.com/T/#t\n\nPhew!\n\nThanks,\nTaylor\n"},{"id":"461757","messageId":"80852679-rsso-rp45-q328-99n36q0639sq@tzk.qr","threadId":"58038","inReplyTo":"xmqqlerkj5f9.fsf@gitster.g","subject":"Re: [PATCH v6 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-08-22T14:42:49Z","receivedAt":"2022-08-22T14:43:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 19 Aug 2022, Junio C Hamano wrote:\n\n> \"Abhradeep Chakraborty via GitGitGadget\" \n> writes:\n>\n> > When parsing the .bitmap file, git loads all the bitmaps one by one even if\n> > some of the bitmaps are not necessary. We can remove this overhead by\n> > loading only the necessary bitmaps. A look up table extension can solve this\n> > issue.\n> >\n> > Changes since v5:\n> >\n> > As the failure in the test case is not due to this code, I think it makes no\n> > sense to delay the patch further.\n> >\n> > * The performance test changes were not accurate as the second\n> > test_bitmap_cases call using the repo built for the previous call. This\n> > version fixes that.\n> > * Taylor suggested some minor changes. Those are addressed in this version.\n>\n> The discussion on v5 was quite active, but we haven't seen any\n> traffic on this round. Is everybody happy with what we see here?\n\nThe part of the lively discussion in which I participated exclusively\nfocused on the failed CI runs and trying to get to the bottom of this bug.\n\nTaylor contributed to address the\nbug. While he seems grateful for my help, I am honestly puzzled because I\nlack too much knowledge about the code to have been of assistance in any\nmeaningful way.\n\nMy participation in this thread should not be mistaken for a review: I am\nwoefully unfamiliar with the bitmap design (let alone code) and would\ntherefore not _dare_ to offer anything that I would claim is a code\nreview.\n\nCiao,\nDscho\n"},{"id":"461758","messageId":"YwOXIE8K0GJRLuDT@nand.local","threadId":"58038","inReplyTo":"80852679-rsso-rp45-q328-99n36q0639sq@tzk.qr","subject":"Re: [PATCH v6 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-08-22T14:48:00Z","receivedAt":"2022-08-22T14:48:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Aug 22, 2022 at 04:42:49PM +0200, Johannes Schindelin wrote:\n> Taylor contributed to address the\n> bug. While he seems grateful for my help, I am honestly puzzled because I\n> lack too much knowledge about the code to have been of assistance in any\n> meaningful way.\n\nI was grateful ;-). Your contributions were quite helpful, especially\nmaking the bug more easily reproducible (doubly so since that test\n*could* have failed on master since its introduction, but didn't).\n\nPinning down some of the effects of the bug and documenting those were\nhelpful, too.\n\n> My participation in this thread should not be mistaken for a review: I am\n> woefully unfamiliar with the bitmap design (let alone code) and would\n> therefore not _dare_ to offer anything that I would claim is a code\n> review.\n\nReviewing Abhradeep's patches are on my list of things to get to,\nhopefully today. I had hoped to get to it last week after getting back\nfrom vacation, but was stymied by the aforementioned bug.\n\nThanks,\nTaylor\n"},{"id":"461969","messageId":"Ywf01YqJKNsGfffx@nand.local","threadId":"58038","inReplyTo":"pull.1266.v6.git.1660496112.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-08-25T22:16:53Z","receivedAt":"2022-08-25T22:17:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Abhradeep,\n\nOn Sun, Aug 14, 2022 at 04:55:05PM +0000, Abhradeep Chakraborty via GitGitGadget wrote:\n> Changes since v5:\n>\n> As the failure in the test case is not due to this code, I think it makes no\n> sense to delay the patch further.\n>\n> * The performance test changes were not accurate as the second\n> test_bitmap_cases call using the repo built for the previous call. This\n> version fixes that.\n> * Taylor suggested some minor changes. Those are addressed in this version.\n\nApologies for my slow reaction time reviewing this series. Between\nlooking at that preferred pack bug you and Dscho spotted to catching up\nafter my vacation, it has taken me longer than I wanted to to take a\nlook at this.\n\nI read through v6 carefully and am happy with the current state of\nthings. I think there are some small incremental clean-ups that we could\ndo on top, but they need not block this series, especially since the new\ncode is made opt-in behind a configuration knob.\n\nThis series all looks great to me, and the performance numbers that you\nachieved at the end are a nice payoff for all of your hard work. Well\ndone!\n\n Reviewed-by: Taylor Blau \n\nThanks,\nTaylor\n"},{"id":"461998","messageId":"xmqqlerb0z8l.fsf@gitster.g","threadId":"58038","inReplyTo":"Ywf01YqJKNsGfffx@nand.local","subject":"Re: [PATCH v6 0/6] [GSoC] bitmap: integrate a lookup table extension to the bitmap format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-26T16:02:34Z","receivedAt":"2022-08-26T16:02:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> This series all looks great to me, and the performance numbers that you\n> achieved at the end are a nice payoff for all of your hard work. Well\n> done!\n>\n> Reviewed-by: Taylor Blau \n\nThanks, both.\n"}]}