{"thread":{"id":"54622","subject":"[PATCH 00/23] pack-bitmap: bitmap generation improvements","startedAt":"2020-11-11T19:41:45Z","lastAt":"2020-12-08T22:06:48Z","messageCount":173,"participants":["Taylor Blau","Derrick Stolee","Junio C Hamano","Martin Ågren","Jeff King","SZEDER Gábor","Johannes Schindelin","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":23},"messages":[{"id":"409670","messageId":"cover.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":null,"subject":"[PATCH 00/23] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:41:34Z","receivedAt":"2020-11-11T19:41:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series contains some patches that GitHub has been using in its fork\nfor the past few months to improve generating reachability bitmaps,\nparticularly in pathological cases, such as the repo containing all\nforks of chromium/chromium.\n\nThe patches that follow are organized into five parts:\n\n - The first nine patches do some basic clean-up and fix a bug that we\n were able to exercise in tests while writing these patches.\n\n - The next two patches reimplements bitmap writing in order to avoid making\n multiple passes over the object graph. This approach ends up\n regressing both the time and memory used to generate bitmaps on the\n kernel's fork-network, but ends up being a useful stepping stone for\n further improvements.\n\n - The six patches that follow that culminates in a patch to build\n fewer intermediate bitmaps during the walk in order to reduce both\n memory and time for reasonably-sized repositories. (Which\n intermediate bitmaps are considered \"important\" is discussed in\n detail in the seventeenth patch).\n\n - The next several patches make reusing previously generated reachability\n bitmaps purely an optimization for generating new bitmaps. Importantly, that\n allows the bitmap selection process to pick better commits to bitmap going\n forward, rather than blindly reusing previously selected ones. They also\n include some light refactoring, and a patch to avoid tree walks when\n existing bitmaps suffice.\n\n - The final two patches address a trade-off in the prior patches between\n walking a wide history only once with high memory cost, and walking the same\n history multiple times with lower memory cost. Here, the walk is reduced to\n only cover the first-parent history. The final patch treats existing bitmaps\n as maximal in order to make it more difficult for a different set of\n selected commits to \"walk around\" the previously selected commits and force\n a large number of new bitmaps to be computed.\n\nIn the end, no block-buster performance improvements are attained on\nnormal-to-large sized repositories, but the new bitmap generation routine helps\nsubstantially on enormous repositories, like the chromium/chromium fork-network.\n\nIndividual performance numbers are available in the patches throughout.\n\nThis series is a prerequisite to a list of other bitmap-related patches in\nGitHub's fork, including multi-pack bitmaps.\n\nDerrick Stolee (9):\n pack-bitmap-write: fill bitmap with commit history\n bitmap: add bitmap_diff_nonzero()\n commit: implement commit_list_contains()\n t5310: add branch-based checks\n pack-bitmap-write: rename children to reverse_edges\n pack-bitmap-write: build fewer intermediate bitmaps\n pack-bitmap-write: use existing bitmaps\n pack-bitmap-write: relax unique rewalk condition\n pack-bitmap-write: better reuse bitmaps\n\nJeff King (11):\n pack-bitmap: fix header size check\n pack-bitmap: bounds-check size of cache extension\n t5310: drop size of truncated ewah bitmap\n rev-list: die when --test-bitmap detects a mismatch\n ewah: factor out bitmap growth\n ewah: make bitmap growth less aggressive\n ewah: implement bitmap_or()\n ewah: add bitmap_dup() function\n pack-bitmap-write: reimplement bitmap writing\n pack-bitmap-write: pass ownership of intermediate bitmaps\n pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n\nTaylor Blau (3):\n ewah/ewah_bitmap.c: grow buffer past 1\n pack-bitmap: factor out 'bitmap_for_commit()'\n pack-bitmap: factor out 'add_commit_to_bitmap()'\n\n builtin/pack-objects.c | 1 -\n commit.c | 11 +\n commit.h | 2 +\n ewah/bitmap.c | 54 ++++-\n ewah/ewah_bitmap.c | 2 +-\n ewah/ewok.h | 3 +-\n pack-bitmap-write.c | 452 +++++++++++++++++++++++++---------------\n pack-bitmap.c | 130 +++++-------\n pack-bitmap.h | 8 +-\n t/t5310-pack-bitmaps.sh | 164 ++++++++++++---\n 10 files changed, 548 insertions(+), 279 deletions(-)\n\n--\n2.29.2.156.gc03786897f\n"},{"id":"409671","messageId":"36deaad366d66d10b96755dd6969bfe51123a2d4.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:41:44Z","receivedAt":"2020-11-11T19:41:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When the buffer size is exactly 1, we fail to grow it properly, since\nthe integer truncation means that 1 * 3 / 2 = 1. This can cause a bad\nwrite on the line below.\n\nBandaid this by first padding the buffer by 16, and then growing it.\nThis still allows old blocks to fit into new ones, but fixes the case\nwhere the block size equals 1.\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/ewah_bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex d59b1afe3d..3fae04ad00 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -45,7 +45,7 @@ static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n static inline void buffer_push(struct ewah_bitmap *self, eword_t value)\n {\n \tif (self->buffer_size + 1 >= self->alloc_size)\n-\t\tbuffer_grow(self, self->buffer_size * 3 / 2);\n+\t\tbuffer_grow(self, (self->buffer_size + 16) * 3 / 2);\n \n \tself->buffer[self->buffer_size++] = value;\n }\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409672","messageId":"c59fcbcc67556c5c9c5a22a2ee745a2f58234efd.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 02/23] pack-bitmap: fix header size check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:41:54Z","receivedAt":"2020-11-11T19:42:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWhen we parse a .bitmap header, we first check that we have enough bytes\nto make a valid header. We do that based on sizeof(struct\nbitmap_disk_header). However, as of 0f4d6cada8 (pack-bitmap: make bitmap\nheader handling hash agnostic, 2019-02-19), that struct oversizes its\nchecksum member to GIT_MAX_RAWSZ. That means we need to adjust for the\ndifference between that constant and the size of the actual hash we're\nusing. That commit adjusted the code which moves our pointer forward,\nbut forgot to update the size check.\n\nThis meant we were overly strict about the header size (requiring room\nfor a 32-byte worst-case hash, when sha1 is only 20 bytes). But in\npractice it didn't matter because bitmap files tend to have at least 12\nbytes of actual data anyway, so it was unlikely for a valid file to be\ncaught by this.\n\nLet's fix it by pulling the header size into a separate variable and\nusing it in both spots. That fixes the bug and simplifies the code to make\nit harder to have a mismatch like this in the future. It will also come\nin handy in the next patch for more bounds checking.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4077e731e8..cea3bb88bf 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -138,8 +138,9 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n+\tsize_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n \n-\tif (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n+\tif (index->map_size < header_size)\n \t\treturn error(\"Corrupted bitmap index (missing header data)\");\n \n \tif (memcmp(header->magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)) != 0)\n@@ -164,7 +165,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-\tindex->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n+\tindex->map_pos += header_size;\n \treturn 0;\n }\n \n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409673","messageId":"1573902df00e8a14a9cb68c37f55474388b1dc2e.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:42:00Z","receivedAt":"2020-11-11T19:42:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nA .bitmap file may have a \"name hash cache\" extension, which puts a\nsequence of uint32_t bytes (one per object) at the end of the file. When\nwe see a flag indicating this extension, we blindly subtract the\nappropriate number of bytes from our available length. However, if the\n.bitmap file is too short, we'll underflow our length variable and wrap\naround, thinking we have a very large length. This can lead to reading\nout-of-bounds bytes while loading individual ewah bitmaps.\n\nWe can fix this by checking the number of available bytes when we parse\nthe header. The existing \"truncated bitmap\" test is now split into two\ntests: one where we don't have this extension at all (and hence actually\ndo try to read a truncated ewah bitmap) and one where we realize\nup-front that we can't even fit in the cache structure. We'll check\nstderr in each case to make sure we hit the error we're expecting.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 8 ++++++--\n t/t5310-pack-bitmaps.sh | 17 +++++++++++++++--\n 2 files changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex cea3bb88bf..42d4824c76 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -153,14 +153,18 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n+\t\tuint32_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n \t\t\treturn error(\"Unsupported options for bitmap index file \"\n \t\t\t\t\"(Git requires BITMAP_OPT_FULL_DAG)\");\n \n \t\tif (flags & BITMAP_OPT_HASH_CACHE) {\n-\t\t\tunsigned char *end = index->map + index->map_size - the_hash_algo->rawsz;\n-\t\t\tindex->hashes = ((uint32_t *)end) - index->pack->num_objects;\n+\t\t\tif (index->map + header_size + cache_size > index_end)\n+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit hash cache)\");\n+\t\t\tindex->hashes = (void *)(index_end - cache_size);\n+\t\t\tindex_end -= cache_size;\n \t\t}\n \t}\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 8318781d2b..e2c3907a68 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -343,7 +343,8 @@ test_expect_success 'pack reuse respects --incremental' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_success 'truncated bitmap fails gracefully' '\n+test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\ttest_config pack.writebitmaphashcache false &&\n \tgit repack -ad &&\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -352,7 +353,19 @@ test_expect_success 'truncated bitmap fails gracefully' '\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt stderr\n+\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\tgit repack -ad &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n # have_delta \n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409674","messageId":"9db61a254af0f8369872e0ef91cbc38acfb079d0.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 04/23] t5310: drop size of truncated ewah bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:42:08Z","receivedAt":"2020-11-11T19:42:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe truncate the .bitmap file to 512 bytes and expect to run into\nproblems reading an individual ewah file. But this length is somewhat\narbitrary, and just happened to work when the test was added in\n9d2e330b17 (ewah_read_mmap: bounds-check mmap reads, 2018-06-14).\n\nAn upcoming commit will change the size of the history we create in the\ntest repo, which will cause this test to fail. We can future-proof it a\nbit more by reducing the size of the truncated bitmap file.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex e2c3907a68..70a4fc4843 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -349,7 +349,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409675","messageId":"37af96311503c3515ebec8d6215af70c48337d40.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 05/23] rev-list: die when --test-bitmap detects a mismatch","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:42:44Z","receivedAt":"2020-11-11T19:42:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nYou can use \"git rev-list --test-bitmap HEAD\" to check that bitmaps\nproduce the same answer we'd get from a regular traversal. But if we\ndetect an error, we only print \"mismatch\", and still exit with a\nsuccessful error code.\n\nThat makes the uses of --test-bitmap in the test suite (e.g., in t5310)\nmostly pointless: even if we saw an error, the tests wouldn't notice.\nLet's instead call die(), which will let these tests work as designed,\nand alert us if the bitmaps are bogus.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 42d4824c76..82c6bf2843 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1328,7 +1328,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (bitmap_equals(result, tdata.base))\n \t\tfprintf(stderr, \"OK!\\n\");\n \telse\n-\t\tfprintf(stderr, \"Mismatch!\\n\");\n+\t\tdie(\"mismatch in bitmap results\");\n \n \tfree_bitmap_index(bitmap_git);\n }\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409676","messageId":"f6eaf05482d46dce58bcf8af884688afb102c386.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 06/23] ewah: factor out bitmap growth","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:42:51Z","receivedAt":"2020-11-11T19:43:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe auto-grow bitmaps when somebody asks to set a bit whose position is\noutside of our currently allocated range. Other operations besides\nsingle bit-setting might need to do this, too, so let's pull it into its\nown function.\n\nNote that we change the semantics a little: you now ask for the number\nof words you'd like to have, not the id of the block you'd like to write\nto.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 +++++++++-----\n 1 file changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d8cec585af..7c1ecfa6fd 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,18 +35,22 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n-void bitmap_set(struct bitmap *self, size_t pos)\n+static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n-\tsize_t block = EWAH_BLOCK(pos);\n-\n-\tif (block >= self->word_alloc) {\n+\tif (word_alloc > self->word_alloc) {\n \t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = block ? block * 2 : 1;\n+\t\tself->word_alloc = word_alloc * 2;\n \t\tREALLOC_ARRAY(self->words, self->word_alloc);\n \t\tmemset(self->words + old_size, 0x0,\n \t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n \t}\n+}\n \n+void bitmap_set(struct bitmap *self, size_t pos)\n+{\n+\tsize_t block = EWAH_BLOCK(pos);\n+\n+\tbitmap_grow(self, block + 1);\n \tself->words[block] |= EWAH_MASK(pos);\n }\n \n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409677","messageId":"c7db594fae4d0447a55a92e830475d9bc418ae7f.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 07/23] ewah: make bitmap growth less aggressive","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:42:58Z","receivedAt":"2020-11-11T19:43:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nIf you ask to set a bit in the Nth word and we haven't yet allocated\nthat many slots in our array, we'll increase the bitmap size to 2*N.\nThis means we might frequently end up with bitmaps that are twice the\nnecessary size (as soon as you ask for the biggest bit, we'll size up to\ntwice that).\n\nBut if we just allocate as many words as were asked for, we may not grow\nfast enough. The worst case there is setting bit 0, then 1, etc. Each\ntime we grow we'd just extend by one more word, giving us linear\nreallocations (and quadratic memory copies).\n\nLet's combine those by allocating the maximum of:\n\n - what the caller asked for\n\n - a geometric increase in existing size; we'll switch to 3/2 instead of\n 2 here. That's less aggressive and may help avoid fragmenting memory\n (N + 3N/2 > 9N/4, so old chunks can be reused as we scale up).\n\nOur worst case is still 3/2N wasted bits (you set bit N-1, then setting\nbit N causes us to grow by 3/2), but our average should be much better.\n\nThis isn't usually that big a deal, but it will matter as we shift the\nreachability bitmap generation code to store more bitmaps in memory.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 7c1ecfa6fd..43a59d7fed 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -39,7 +39,9 @@ static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n \tif (word_alloc > self->word_alloc) {\n \t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = word_alloc * 2;\n+\t\tself->word_alloc = old_size * 3 / 2;\n+\t\tif (word_alloc > self->word_alloc)\n+\t\t\tself->word_alloc = word_alloc;\n \t\tREALLOC_ARRAY(self->words, self->word_alloc);\n \t\tmemset(self->words + old_size, 0x0,\n \t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409678","messageId":"dce9b6da0ad38da0a92b39d780d7b56f83d52950.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 08/23] ewah: implement bitmap_or()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:03Z","receivedAt":"2020-11-11T19:43:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe have a function to bitwise-OR an ewah into an uncompressed bitmap,\nbut not to OR two uncompressed bitmaps. Let's add it.\n\nInterestingly, we have a public header declaration going back to\ne1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\nfunction was never implemented.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 43a59d7fed..c3f8e7242b 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -127,6 +127,15 @@ void bitmap_and_not(struct bitmap *self, struct bitmap *other)\n \t\tself->words[i] &= ~other->words[i];\n }\n \n+void bitmap_or(struct bitmap *self, const struct bitmap *other)\n+{\n+\tsize_t i;\n+\n+\tbitmap_grow(self, other->word_alloc);\n+\tfor (i = 0; i < other->word_alloc; i++)\n+\t\tself->words[i] |= other->words[i];\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409679","messageId":"242bd3f110fe43c127208a7351215f1b238c714b.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 09/23] ewah: add bitmap_dup() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:08Z","receivedAt":"2020-11-11T19:43:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThere's no easy way to make a copy of a bitmap. Obviously a caller can\niterate over the bits and set them one by one in a new bitmap, but we\ncan go much faster by copying whole words with memcpy().\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 7 +++++++\n ewah/ewok.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex c3f8e7242b..eb7e2539be 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,6 +35,13 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n+struct bitmap *bitmap_dup(const struct bitmap *src)\n+{\n+\tstruct bitmap *dst = bitmap_word_alloc(src->word_alloc);\n+\tCOPY_ARRAY(dst->words, src->words, src->word_alloc);\n+\treturn dst;\n+}\n+\n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n \tif (word_alloc > self->word_alloc) {\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 011852bef1..1fc555e672 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -173,6 +173,7 @@ struct bitmap {\n \n struct bitmap *bitmap_new(void);\n struct bitmap *bitmap_word_alloc(size_t word_alloc);\n+struct bitmap *bitmap_dup(const struct bitmap *src);\n void bitmap_set(struct bitmap *self, size_t pos);\n void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409680","messageId":"2f18d76cfd2c6ab3df7f4391aa824de1ad5f2d5f.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 10/23] pack-bitmap-write: reimplement bitmap writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:12Z","receivedAt":"2020-11-11T19:43:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe bitmap generation code works by iterating over the set of commits\nfor which we plan to write bitmaps, and then for each one performing a\ntraditional traversal over the reachable commits and trees, filling in\nthe bitmap. Between two traversals, we can often reuse the previous\nbitmap result as long as the first commit is an ancestor of the second.\nHowever, our worst case is that we may end up doing \"n\" complete\ncomplete traversals to the root in order to create \"n\" bitmaps.\n\nIn a real-world case (the shared-storage repo consisting of all GitHub\nforks of chromium/chromium), we perform very poorly: generating bitmaps\ntakes ~3 hours, whereas we can walk the whole object graph in ~3\nminutes.\n\nThis commit completely rewrites the algorithm, with the goal of\naccessing each object only once. It works roughly like this:\n\n - generate a list of commits in topo-order using a single traversal\n\n - invert the edges of the graph (so have parents point at their\n children)\n\n - make one pass in reverse topo-order, generating a bitmap for each\n commit and passing the result along to child nodes\n\nWe generate correct results because each node we visit has already had\nall of its ancestors added to the bitmap. And we make only two linear\npasses over the commits.\n\nWe also visit each tree usually only once. When filling in a bitmap, we\ndon't bother to recurse into trees whose bit is already set in the\nbitmap (since we know we've already done so when setting their bit).\nThat means that if commit A references tree T, none of its descendants\nwill need to open T again. I say \"usually\", though, because it is\npossible for a given tree to be mentioned in unrelated parts of history\n(e.g., cherry-picking to a parallel branch).\n\nSo we've accomplished our goal, and the resulting algorithm is pretty\nsimple to understand. But there are some downsides, at least with this\ninitial implementation:\n\n - we no longer reuse the results of any on-disk bitmaps when\n generating. So we'd expect to sometimes be slower than the original\n when bitmaps already exist. However, this is something we'll be able\n to add back in later.\n\n - we use much more memory. Instead of keeping one bitmap in memory at\n a time, we're passing them up through the graph. So our memory use\n should scale with the graph width (times the size of a bitmap).\n\nSo how does it perform?\n\nFor a clone of linux.git, generating bitmaps from scratch with the old\nalgorithm took 63s. Using this algorithm it takes 205s. Which is much\nworse, but _might_ be acceptable if it behaved linearly as the size\ngrew. It also increases peak heap usage by ~1G. That's not impossibly\nlarge, but not encouraging.\n\nOn the complete fork-network of torvalds/linux, it increases the peak\nRAM usage by 40GB. Yikes. (I forgot to record the time it took, but the\nmemory usage was too much to consider this reasonable anyway).\n\nOn the complete fork-network of chromium/chromium, I ran out of memory\nbefore succeeding. Some back-of-the-envelope calculations indicate it\nwould need 80+GB to complete.\n\nSo at this stage, we've managed to make things much worse. But because\nof the way this new algorithm is structured, there are a lot of\nopportunities for optimization on top. We'll start implementing those in\nthe follow-on patches.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 303 ++++++++++++++++++++++++--------------------\n 1 file changed, 169 insertions(+), 134 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 5e998bdaa7..f2f0b6b2c2 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -110,8 +110,6 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n /**\n * Compute the actual bitmaps\n */\n-static struct object **seen_objects;\n-static unsigned int seen_objects_nr, seen_objects_alloc;\n \n static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n {\n@@ -127,21 +125,6 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \twriter.selected_nr++;\n }\n \n-static inline void mark_as_seen(struct object *object)\n-{\n-\tALLOC_GROW(seen_objects, seen_objects_nr + 1, seen_objects_alloc);\n-\tseen_objects[seen_objects_nr++] = object;\n-}\n-\n-static inline void reset_all_seen(void)\n-{\n-\tunsigned int i;\n-\tfor (i = 0; i < seen_objects_nr; ++i) {\n-\t\tseen_objects[i]->flags &= ~(SEEN | ADDED | SHOWN);\n-\t}\n-\tseen_objects_nr = 0;\n-}\n-\n static uint32_t find_object_pos(const struct object_id *oid)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n@@ -154,60 +137,6 @@ static uint32_t find_object_pos(const struct object_id *oid)\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n-static void show_object(struct object *object, const char *name, void *data)\n-{\n-\tstruct bitmap *base = data;\n-\tbitmap_set(base, find_object_pos(&object->oid));\n-\tmark_as_seen(object);\n-}\n-\n-static void show_commit(struct commit *commit, void *data)\n-{\n-\tmark_as_seen((struct object *)commit);\n-}\n-\n-static int\n-add_to_include_set(struct bitmap *base, struct commit *commit)\n-{\n-\tkhiter_t hash_pos;\n-\tuint32_t bitmap_pos = find_object_pos(&commit->object.oid);\n-\n-\tif (bitmap_get(base, bitmap_pos))\n-\t\treturn 0;\n-\n-\thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n-\tif (hash_pos < kh_end(writer.bitmaps)) {\n-\t\tstruct bitmapped_commit *bc = kh_value(writer.bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(base, bc->bitmap);\n-\t\treturn 0;\n-\t}\n-\n-\tbitmap_set(base, bitmap_pos);\n-\treturn 1;\n-}\n-\n-static int\n-should_include(struct commit *commit, void *_data)\n-{\n-\tstruct bitmap *base = _data;\n-\n-\tif (!add_to_include_set(base, commit)) {\n-\t\tstruct commit_list *parent = commit->parents;\n-\n-\t\tmark_as_seen((struct object *)commit);\n-\n-\t\twhile (parent) {\n-\t\t\tparent->item->object.flags |= SEEN;\n-\t\t\tmark_as_seen((struct object *)parent->item);\n-\t\t\tparent = parent->next;\n-\t\t}\n-\n-\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n static void compute_xor_offsets(void)\n {\n \tstatic const int MAX_XOR_OFFSET_SEARCH = 10;\n@@ -248,79 +177,185 @@ static void compute_xor_offsets(void)\n \t}\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+struct bb_commit {\n+\tstruct commit_list *children;\n+\tstruct bitmap *bitmap;\n+\tunsigned selected:1;\n+\tunsigned idx; /* within selected array */\n+};\n+\n+define_commit_slab(bb_data, struct bb_commit);\n+\n+struct bitmap_builder {\n+\tstruct bb_data data;\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+};\n+\n+static void bitmap_builder_init(struct bitmap_builder *bb,\n+\t\t\t\tstruct bitmap_writer *writer)\n {\n-\tstatic const double REUSE_BITMAP_THRESHOLD = 0.2;\n-\n-\tint i, reuse_after, need_reset;\n-\tstruct bitmap *base = bitmap_new();\n \tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tunsigned int i;\n+\n+\tmemset(bb, 0, sizeof(*bb));\n+\tinit_bb_data(&bb->data);\n+\n+\treset_revision_walk();\n+\trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n+\trevs.topo_order = 1;\n+\n+\tfor (i = 0; i < writer->selected_nr; i++) {\n+\t\tstruct commit *c = writer->selected[i].commit;\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tent->selected = 1;\n+\t\tent->idx = i;\n+\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(\"revision walk setup failed\");\n+\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tstruct commit_list *p;\n+\n+\t\tparse_commit_or_die(commit);\n+\n+\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\tbb->commits[bb->commits_nr++] = commit;\n+\n+\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n+\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t}\n+\t}\n+}\n+\n+static void bitmap_builder_clear(struct bitmap_builder *bb)\n+{\n+\tclear_bb_data(&bb->data);\n+\tfree(bb->commits);\n+\tbb->commits_nr = bb->commits_alloc = 0;\n+}\n+\n+static void fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n+{\n+\tuint32_t pos;\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\t/*\n+\t * If our bit is already set, then there is nothing to do. Both this\n+\t * tree and all of its children will be set.\n+\t */\n+\tpos = find_object_pos(&tree->object.oid);\n+\tif (bitmap_get(bitmap, pos))\n+\t\treturn;\n+\tbitmap_set(bitmap, pos);\n+\n+\tif (parse_tree(tree) < 0)\n+\t\tdie(\"unable to load tree object %s\",\n+\t\t oid_to_hex(&tree->object.oid));\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tfill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Gitlink, etc; not reachable */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n+static void fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit)\n+{\n+\tif (!ent->bitmap)\n+\t\tent->bitmap = bitmap_new();\n+\n+\t/*\n+\t * mark ourselves, but do not bother with parents; their values\n+\t * will already have been propagated to us\n+\t */\n+\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n+\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+}\n+\n+static void store_selected(struct bb_commit *ent, struct commit *commit)\n+{\n+\tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n+\tkhiter_t hash_pos;\n+\tint hash_ret;\n+\n+\t/*\n+\t * the \"reuse bitmaps\" phase may have stored something here, but\n+\t * our new algorithm doesn't use it. Drop it.\n+\t */\n+\tif (stored->bitmap)\n+\t\tewah_free(stored->bitmap);\n+\n+\tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n+\n+\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n+\tif (hash_ret == 0)\n+\t\tdie(\"Duplicate entry when writing index: %s\",\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer.bitmaps, hash_pos) = stored;\n+}\n+\n+void bitmap_writer_build(struct packing_data *to_pack)\n+{\n+\tstruct bitmap_builder bb;\n+\tsize_t i;\n+\tint nr_stored = 0; /* for progress */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n \n \tif (writer.show_progress)\n \t\twriter.progress = start_progress(\"Building bitmaps\", writer.selected_nr);\n-\n-\trepo_init_revisions(to_pack->repo, &revs, NULL);\n-\trevs.tag_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.blob_objects = 1;\n-\trevs.no_walk = 0;\n-\n-\trevs.include_check = should_include;\n-\treset_revision_walk();\n-\n-\treuse_after = writer.selected_nr * REUSE_BITMAP_THRESHOLD;\n-\tneed_reset = 0;\n-\n-\tfor (i = writer.selected_nr - 1; i >= 0; --i) {\n-\t\tstruct bitmapped_commit *stored;\n-\t\tstruct object *object;\n-\n-\t\tkhiter_t hash_pos;\n-\t\tint hash_ret;\n-\n-\t\tstored = &writer.selected[i];\n-\t\tobject = (struct object *)stored->commit;\n-\n-\t\tif (stored->bitmap == NULL) {\n-\t\t\tif (i < writer.selected_nr - 1 &&\n-\t\t\t (need_reset ||\n-\t\t\t !in_merge_bases(writer.selected[i + 1].commit,\n-\t\t\t\t\t stored->commit))) {\n-\t\t\t bitmap_reset(base);\n-\t\t\t reset_all_seen();\n-\t\t\t}\n-\n-\t\t\tadd_pending_object(&revs, object, \"\");\n-\t\t\trevs.include_check_data = base;\n-\n-\t\t\tif (prepare_revision_walk(&revs))\n-\t\t\t\tdie(\"revision walk setup failed\");\n-\n-\t\t\ttraverse_commit_list(&revs, show_commit, show_object, base);\n-\n-\t\t\tobject_array_clear(&revs.pending);\n-\n-\t\t\tstored->bitmap = bitmap_to_ewah(base);\n-\t\t\tneed_reset = 0;\n-\t\t} else\n-\t\t\tneed_reset = 1;\n-\n-\t\tif (i >= reuse_after)\n-\t\t\tstored->flags |= BITMAP_FLAG_REUSE;\n-\n-\t\thash_pos = kh_put_oid_map(writer.bitmaps, object->oid, &hash_ret);\n-\t\tif (hash_ret == 0)\n-\t\t\tdie(\"Duplicate entry when writing index: %s\",\n-\t\t\t oid_to_hex(&object->oid));\n-\n-\t\tkh_value(writer.bitmaps, hash_pos) = stored;\n-\t\tdisplay_progress(writer.progress, writer.selected_nr - i);\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n+\t\tthe_repository);\n+\n+\tbitmap_builder_init(&bb, &writer);\n+\tfor (i = bb.commits_nr; i > 0; i--) {\n+\t\tstruct commit *commit = bb.commits[i-1];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n+\t\tstruct commit *child;\n+\n+\t\tfill_bitmap_commit(ent, commit);\n+\n+\t\tif (ent->selected) {\n+\t\t\tstore_selected(ent, commit);\n+\t\t\tnr_stored++;\n+\t\t\tdisplay_progress(writer.progress, nr_stored);\n+\t\t}\n+\n+\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\t\tstruct bb_commit *child_ent =\n+\t\t\t\tbb_data_at(&bb.data, child);\n+\n+\t\t\tif (child_ent->bitmap)\n+\t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n+\t\t\telse\n+\t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t}\n+\t\tbitmap_free(ent->bitmap);\n+\t\tent->bitmap = NULL;\n \t}\n+\tbitmap_builder_clear(&bb);\n \n-\tbitmap_free(base);\n \tstop_progress(&writer.progress);\n \n \tcompute_xor_offsets();\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409681","messageId":"8328d8cc9930ca10d939e4b35abe1c4987aba115.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 11/23] pack-bitmap-write: pass ownership of intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:18Z","receivedAt":"2020-11-11T19:43:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nOur algorithm to generate reachability bitmaps walks through the commit\ngraph from the bottom up, passing bitmap data from each commit to its\ndescendants. For a linear stretch of history like:\n\n A -- B -- C\n\nour sequence of steps is:\n\n - compute the bitmap for A by walking its trees, etc\n\n - duplicate A's bitmap as a starting point for B; we can now free A's\n bitmap, since we only needed it as an intermediate result\n\n - OR in any extra objects that B can reach into its bitmap\n\n - duplicate B's bitmap as a starting point for C; likewise, free B's\n bitmap\n\n - OR in objects for C, and so on...\n\nRather than duplicating bitmaps and immediately freeing the original, we\ncan just pass ownership from commit to commit. Note that this doesn't\nalways work:\n\n - the recipient may be a merge which already has an intermediate\n bitmap from its other ancestor. In that case we have to OR our\n result into it. Note that the first ancestor to reach the merge does\n get to pass ownership, though.\n\n - we may have multiple children; we can only pass ownership to one of\n them\n\nHowever, it happens often enough and copying bitmaps is expensive enough\nthat this provides a noticeable speedup. On a clone of linux.git, this\nreduces the time to generate bitmaps from 205s to 70s. This is about the\nsame amount of time it took to generate bitmaps using our old \"many\ntraversals\" algorithm (the previous commit measures the identical\nscenario as taking 63s). It unfortunately provides only a very modest\nreduction in the peak memory usage, though.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f2f0b6b2c2..d2d46ff5f4 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -333,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n \t\tstruct commit *child;\n+\t\tint reused = 0;\n \n \t\tfill_bitmap_commit(ent, commit);\n \n@@ -348,10 +349,15 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \t\t\tif (child_ent->bitmap)\n \t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n-\t\t\telse\n+\t\t\telse if (reused)\n \t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t\telse {\n+\t\t\t\tchild_ent->bitmap = ent->bitmap;\n+\t\t\t\treused = 1;\n+\t\t\t}\n \t\t}\n-\t\tbitmap_free(ent->bitmap);\n+\t\tif (!reused)\n+\t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n \tbitmap_builder_clear(&bb);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409682","messageId":"88e7988751fca329a8e453727c614fdfbbba426a.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 12/23] pack-bitmap-write: fill bitmap with commit history","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:24Z","receivedAt":"2020-11-11T19:43:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe fill_bitmap_commit() method assumes that every parent of the given\ncommit is already part of the current bitmap. Instead of making that\nassumption, let's walk parents until we reach commits already part of\nthe bitmap. Set the value for that parent immediately after querying to\nsave time doing double calls to find_object_pos() and to avoid inserting\nthe parent into the queue multiple times.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 30 +++++++++++++++++++++++-------\n 1 file changed, 23 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d2d46ff5f4..361f3305a2 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -12,6 +12,7 @@\n #include \"sha1-lookup.h\"\n #include \"pack-objects.h\"\n #include \"commit-reach.h\"\n+#include \"prio-queue.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -279,17 +280,30 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n }\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit)\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\t/*\n-\t * mark ourselves, but do not bother with parents; their values\n-\t * will already have been propagated to us\n-\t */\n \tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n-\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+\tprio_queue_put(queue, commit);\n+\n+\twhile (queue->nr) {\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = prio_queue_get(queue);\n+\n+\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n+\t\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\t\tprio_queue_put(queue, p->item);\n+\t\t\t}\n+\t\t}\n+\t}\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -319,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct bitmap_builder bb;\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -335,7 +350,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit);\n+\t\tfill_bitmap_commit(ent, commit, &queue);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -360,6 +375,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n+\tclear_prio_queue(&queue);\n \tbitmap_builder_clear(&bb);\n \n \tstop_progress(&writer.progress);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409683","messageId":"3f25315cf7960dcdb33e56ce8ea083b4b9688f99.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 13/23] bitmap: add bitmap_diff_nonzero()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:29Z","receivedAt":"2020-11-11T19:43:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_diff_nonzero() checks if the 'self' bitmap contains any bits\nthat are not on in the 'other' bitmap.\n\nAlso, delete the declaration of bitmap_is_subset() as it is not used or\nimplemented.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 24 ++++++++++++++++++++++++\n ewah/ewok.h | 2 +-\n 2 files changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex eb7e2539be..e2ebeac0e5 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -200,6 +200,30 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other)\n+{\n+\tstruct bitmap *small;\n+\tsize_t i;\n+\n+\tif (self->word_alloc < other->word_alloc) {\n+\t\tsmall = self;\n+\t} else {\n+\t\tsmall = other;\n+\n+\t\tfor (i = other->word_alloc; i < self->word_alloc; i++) {\n+\t\t\tif (self->words[i] != 0)\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < small->word_alloc; i++) {\n+\t\tif ((self->words[i] & ~other->words[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void bitmap_reset(struct bitmap *bitmap)\n {\n \tmemset(bitmap->words, 0x0, bitmap->word_alloc * sizeof(eword_t));\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 1fc555e672..156c71d06d 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -180,7 +180,7 @@ int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_reset(struct bitmap *self);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n-int bitmap_is_subset(struct bitmap *self, struct bitmap *super);\n+int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409684","messageId":"7f9b45b118cdd48372dd60d5e2f3e9a588175dab.1605123652.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 14/23] commit: implement commit_list_contains()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:33Z","receivedAt":"2020-11-11T19:43:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to check if a commit_list contains a commit. Use\npointer equality, assuming lookup_commit() was used.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit.c | 11 +++++++++++\n commit.h | 2 ++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/commit.c b/commit.c\nindex fe1fa3dc41..9a785bf906 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -544,6 +544,17 @@ struct commit_list *commit_list_insert(struct commit *item, struct commit_list *\n \treturn new_list;\n }\n \n+int commit_list_contains(struct commit *item, struct commit_list *list)\n+{\n+\twhile (list) {\n+\t\tif (list->item == item)\n+\t\t\treturn 1;\n+\t\tlist = list->next;\n+\t}\n+\n+\treturn 0;\n+}\n+\n unsigned commit_list_count(const struct commit_list *l)\n {\n \tunsigned c = 0;\ndiff --git a/commit.h b/commit.h\nindex 5467786c7b..742a6de460 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -167,6 +167,8 @@ int find_commit_subject(const char *commit_buffer, const char **subject);\n \n struct commit_list *commit_list_insert(struct commit *item,\n \t\t\t\t\tstruct commit_list **list);\n+int commit_list_contains(struct commit *item,\n+\t\t\t struct commit_list *list);\n struct commit_list **commit_list_append(struct commit *commit,\n \t\t\t\t\tstruct commit_list **next);\n unsigned commit_list_count(const struct commit_list *l);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409685","messageId":"9ab4b94b3573346b31e710486799ab3d95bade8e.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 15/23] t5310: add branch-based checks","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:41Z","receivedAt":"2020-11-11T19:43:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe current rev-list tests that check the bitmap data only work on HEAD\ninstead of multiple branches. Expand the test cases to handle both\n'master' and 'other' branches.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 61 +++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 70a4fc4843..6bf68fee85 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -41,63 +41,70 @@ test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n \tgit rev-list --test-bitmap HEAD\n '\n \n-rev_list_tests() {\n-\tstate=$1\n-\n-\ttest_expect_success \"counting commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD >actual &&\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting partial commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD~5..HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD~5..HEAD >actual &&\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limit ($state)\" '\n-\t\tgit rev-list --count -n 1 HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 HEAD >actual &&\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting non-linear history ($state)\" '\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n \t\tgit rev-list --count other...master >expect &&\n \t\tgit rev-list --use-bitmap-index --count other...master >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limiting ($state)\" '\n-\t\tgit rev-list --count HEAD -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD -- 1.t >actual &&\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting objects via bitmap ($state)\" '\n-\t\tgit rev-list --count --objects HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects HEAD >actual &&\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"enumerate commits ($state)\" '\n-\t\tgit rev-list --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list HEAD >expect &&\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n \t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n \t'\n \n-\ttest_expect_success \"enumerate --objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list --objects HEAD >expect &&\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n \t\ttest_bitmap_traversal expect actual\n \t'\n \n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD tagged-blob >actual &&\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n \t\tgrep $blob actual\n \t'\n }\n \n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"master\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n rev_list_tests 'full bitmap'\n \n test_expect_success 'clone from bitmapped repository' '\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409686","messageId":"ac92e76a98591e6131a8e7a1b2f2e2bbf6d92564.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 16/23] pack-bitmap-write: rename children to reverse_edges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:47Z","receivedAt":"2020-11-11T19:43:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_builder_init() method walks the reachable commits in\ntopological order and constructs a \"reverse graph\" along the way. At the\nmoment, this reverse graph contains an edge from commit A to commit B if\nand only if A is a parent of B. Thus, the name \"children\" is appropriate\nfor for this reverse graph.\n\nIn the next change, we will repurpose the reverse graph to not be\ndirectly-adjacent commits in the commit-graph, but instead a more\nabstract relationship. The previous changes have already incorporated\nthe necessary updates to fill_bitmap_commit() that allow these edges to\nnot be immediate children.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 361f3305a2..369c76a87c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -179,7 +179,7 @@ static void compute_xor_offsets(void)\n }\n \n struct bb_commit {\n-\tstruct commit_list *children;\n+\tstruct commit_list *reverse_edges;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1;\n \tunsigned idx; /* within selected array */\n@@ -228,7 +228,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n \t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n \t\t}\n \t}\n }\n@@ -358,7 +358,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tdisplay_progress(writer.progress, nr_stored);\n \t\t}\n \n-\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\twhile ((child = pop_commit(&ent->reverse_edges))) {\n \t\t\tstruct bb_commit *child_ent =\n \t\t\t\tbb_data_at(&bb.data, child);\n \n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409687","messageId":"ab64354851e2aa61e901e37814b2ae33d8f855d1.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 17/23] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:51Z","receivedAt":"2020-11-11T19:43:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_writer_build() method calls bitmap_builder_init() to\nconstruct a list of commits reachable from the selected commits along\nwith a \"reverse graph\". This reverse graph has edges pointing from a\ncommit to other commits that can reach that commit. After computing a\nreachability bitmap for a commit, the values in that bitmap are then\ncopied to the reachability bitmaps across the edges in the reverse\ngraph.\n\nWe can now relax the role of the reverse graph to greatly reduce the\nnumber of intermediate reachability bitmaps we compute during this\nreverse walk. The end result is that we walk objects the same number of\ntimes as before when constructing the reachability bitmaps, but we also\nspend much less time copying bits between bitmaps and have much lower\nmemory pressure in the process.\n\nThe core idea is to select a set of \"important\" commits based on\ninteractions among the sets of commits reachable from each selected commit.\n\nThe first technical concept is to create a new 'commit_mask' member in the\nbb_commit struct. Note that the selected commits are provided in an\nordered array. The first thing to do is to mark the ith bit in the\ncommit_mask for the ith selected commit. As we walk the commit-graph, we\ncopy the bits in a commit's commit_mask to its parents. At the end of\nthe walk, the ith bit in the commit_mask for a commit C stores a boolean\nrepresenting \"The ith selected commit can reach C.\"\n\nAs we walk, we will discover non-selected commits that are important. We\nwill get into this later, but those important commits must also receive\nbit positions, growing the width of the bitmasks as we walk. At the true\nend of the walk, the ith bit means \"the ith _important_ commit can reach\nC.\"\n\nMAXIMAL COMMITS\n---------------\n\nWe use a new 'maximal' bit in the bb_commit struct to represent whether\na commit is important or not. The term \"maximal\" comes from the\npartially-ordered set of commits in the commit-graph where C >= P if P\nis a parent of C, and then extending the relationship transitively.\nInstead of taking the maximal commits across the entire commit-graph, we\ninstead focus on selecting each commit that is maximal among commits\nwith the same bits on in their commit_mask. This definition is\nimportant, so let's consider an example.\n\nSuppose we have three selected commits A, B, and C. These are assigned\nbitmasks 100, 010, and 001 to start. Each of these can be marked as\nmaximal immediately because they each will be the uniquely maximal\ncommit that contains their own bit. Keep in mind that that these commits\nmay have different bitmasks after the walk; for example, if B can reach\nC but A cannot, then the final bitmask for C is 011. Even in these\ncases, C would still be a maximal commit among all commits with the\nthird bit on in their masks.\n\nNow define sets X, Y, and Z to be the sets of commits reachable from A,\nB, and C, respectively. The intersections of these sets correspond to\ndifferent bitmasks:\n\n * 100: X - (Y union Z)\n * 010: Y - (X union Z)\n * 001: Z - (X union Y)\n * 110: (X intersect Y) - Z\n * 101: (X intersect Z) - Y\n * 011: (Y intersect Z) - X\n * 111: X intersect Y intersect Z\n\nThis can be visualized with the following Hasse diagram:\n\n\t100 010 001\n | \\ / \\ / |\n | \\/ \\/ |\n | /\\ /\\ |\n | / \\ / \\ |\n 110 101 011\n \\___ | ___/\n \\ | /\n 111\n\nSome of these bitmasks may not be represented, depending on the topology\nof the commit-graph. In fact, we are counting on it, since the number of\npossible bitmasks is exponential in the number of selected commits, but\nis also limited by the total number of commits. In practice, very few\nbitmasks are possible because most commits converge on a common \"trunk\"\nin the commit history.\n\nWith this three-bit example, we wish to find commits that are maximal\nfor each bitmask. How can we identify this as we are walking?\n\nAs we walk, we visit a commit C. Since we are walking the commits in\ntopo-order, we know that C is visited after all of its children are\nvisited. Thus, when we get C from the revision walk we inspect the\n'maximal' property of its bb_data and use that to determine if C is truly\nimportant. Its commit_mask is also nearly final. If C is not one of the\noriginally-selected commits, then assign a bit position to C (by\nincrementing num_maximal) and set that bit on in commit_mask. See\n\"MULTIPLE MAXIMAL COMMITS\" below for more detail on this.\n\nNow that the commit C is known to be maximal or not, consider each\nparent P of C. Compute two new values:\n\n * c_not_p : true if and only if the commit_mask for C contains a bit\n that is not contained in the commit_mask for P.\n\n * p_not_c : true if and only if the commit_mask for P contains a bit\n that is not contained in the commit_mask for P.\n\nIf c_not_p is false, then P already has all of the bits that C would\nprovide to its commit_mask. In this case, move on to other parents as C\nhas nothing to contribute to P's state that was not already provided by\nother children of P.\n\nWe continue with the case that c_not_p is true. This means there are\nbits in C's commit_mask to copy to P's commit_mask, so use bitmap_or()\nto add those bits.\n\nIf p_not_c is also true, then set the maximal bit for P to one. This means\nthat if no other commit has P as a parent, then P is definitely maximal.\nThis is because no child had the same bitmask. It is important to think\nabout the maximal bit for P at this point as a temporary state: \"P is\nmaximal based on current information.\"\n\nIn contrast, if p_not_c is false, then set the maximal bit for P to\nzero. Further, clear all reverse_edges for P since any edges that were\npreviously assigned to P are no longer important. P will gain all\nreverse edges based on C.\n\nThe final thing we need to do is to update the reverse edges for P.\nThese reverse edges respresent \"which closest maximal commits\ncontributed bits to my commit_mask?\" Since C contributed bits to P's\ncommit_mask in this case, C must add to the reverse edges of P.\n\nIf C is maximal, then C is a 'closest' maximal commit that contributed\nbits to P. Add C to P's reverse_edges list.\n\nOtherwise, C has a list of maximal commits that contributed bits to its\nbitmask (and this list is exactly one element). Add all of these items\nto P's reverse_edges list. Be careful to ignore duplicates here.\n\nAfter inspecting all parents P for a commit C, we can clear the\ncommit_mask for C. This reduces the memory load to be limited to the\n\"width\" of the commit graph.\n\nConsider our ABC/XYZ example from earlier and let's inspect the state of\nthe commits for an interesting bitmask, say 011. Suppose that D is the\nonly maximal commit with this bitmask (in the first three bits). All\nother commits with bitmask 011 have D as the only entry in their\nreverse_edges list. D's reverse_edges list contains B and C.\n\nCOMPUTING REACHABILITY BITMAPS\n------------------------------\n\nNow that we have our definition, let's zoom out and consider what\nhappens with our new reverse graph when computing reachability bitmaps.\nWe walk the reverse graph in reverse-topo-order, so we visit commits\nwith largest commit_masks first. After we compute the reachability\nbitmap for a commit C, we push the bits in that bitmap to each commit D\nin the reverse edge list for C. Then, when we finally visit D we already\nhave the bits for everything reachable from maximal commits that D can\nreach and we only need to walk the objects in the set-difference.\n\nIn our ABC/XYZ example, when we finally walk for the commit A we only\nneed to walk commits with bitmask equal to A's bitmask. If that bitmask\nis 100, then we are only walking commits in X - (Y union Z) because the\nbitmap already contains the bits for objects reachable from (X intersect\nY) union (X intersect Z) (i.e. the bits from the reachability bitmaps\nfor the maximal commits with bitmasks 110 and 101).\n\nThe behavior is intended to walk each commit (and the trees that commit\nintroduces) at most once while allocating and copying fewer reachability\nbitmaps. There is one caveat: what happens when there are multiple\nmaximal commits with the same bitmask, with respect to the initial set\nof selected commits?\n\nMULTIPLE MAXIMAL COMMITS\n------------------------\n\nEarlier, we mentioned that when we discover a new maximal commit, we\nassign a new bit position to that commit and set that bit position to\none for that commit. This is absolutely important for interesting\ncommit-graphs such as git/git and torvalds/linux. The reason is due to\nthe existence of \"butterflies\" in the commit-graph partial order.\n\nHere is an example of four commits forming a butterfly:\n\n I J\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N\n \\ /\n |/\n Q\n\nHere, I and J both have parents M and N. In general, these do not need\nto be exact parent relationships, but reachability relationships. The\nmost important part is that M and N cannot reach each other, so they are\nindependent in the partial order. If I had commit_mask 10 and J had\ncommit_mask 01, then M and N would both be assigned commit_mask 11 and\nbe maximal commits with the bitmask 11. Then, what happens when M and N\ncan both reach a commit Q? If Q is also assigned the bitmask 11, then it\nis not maximal but is reachable from both M and N.\n\nWhile this is not necessarily a deal-breaker for our abstract definition\nof finding maximal commits according to a given bitmask, we have a few\nissues that can come up in our larger picture of constructing\nreachability bitmaps.\n\nIn particular, if we do not also consider Q to be a \"maximal\" commit,\nthen we will walk commits reachable from Q twice: once when computing\nthe reachability bitmap for M and another time when computing the\nreachability bitmap for N. This becomes much worse if the topology\ncontinues this pattern with multiple butterflies.\n\nThe solution has already been mentioned: each of M and N are assigned\ntheir own bits to the bitmask and hence they become uniquely maximal for\ntheir bitmasks. Finally, Q also becomes maximal and thus we do not need\nto walk its commits multiple times. The final bitmasks for these commits\nare as follows:\n\n I:10 J:01\n |\\ /|\n | \\ _____/ |\n | /\\____ |\n |/ \\ |\n M:111 N:1101\n \\ /\n Q:1111\n\nFurther, Q's reverse edge list is { M, N }, while M and N both have\nreverse edge list { I, J }.\n\nPERFORMANCE MEASUREMENTS\n------------------------\n\nNow that we've spent a LOT of time on the theory of this algorithm,\nlet's show that this is actually worth all that effort.\n\nTo test the performance, use GIT_TRACE2_PERF=1 when running\n'git repack -abd' in a repository with no existing reachability bitmaps.\nThis avoids any issues with keeping existing bitmaps to skew the\nnumbers.\n\nInspect the \"building_bitmaps_total\" region in the trace2 output to\nfocus on the portion of work that is affected by this change. Here are\nthe performance comparisons for a few repositories. The timings are for\nthe following versions of Git: \"multi\" is the timing from before any\nreverse graph is constructed, where we might perform multiple\ntraversals. \"reverse\" is for the previous change where the reverse graph\nhas every reachable commit. Finally \"maximal\" is the version introduced\nhere where the reverse graph only contains the maximal commits.\n\n Repository: git/git\n multi: 2.628 sec\n reverse: 2.344 sec\n maximal: 2.047 sec\n\n Repository: torvalds/linux\n multi: 64.7 sec\n reverse: 205.3 sec\n maximal: 44.7 sec\n\nSo in all cases we've not only recovered any time lost to switching to\nthe reverse-edge algorithm, but we come out ahead of \"multi\" in all\ncases. Likewise, peak heap has gone back to something reasonable:\n\n Repository: torvalds/linux\n multi: 2.087 GB\n reverse: 3.141 GB\n maximal: 2.288 GB\n\nWhile I do not have access to full fork networks on GitHub, Peff has run\nthis algorithm on the chromium/chromium fork network and reported a\nchange from 3 hours to ~233 seconds. That network is particularly\nbeneficial for this approach because it has a long, linear history along\nwith many tags. The \"multi\" approach was obviously quadratic and the new\napproach is linear.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++---\n t/t5310-pack-bitmaps.sh | 85 +++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 148 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 369c76a87c..7b4fc0f304 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -180,8 +180,10 @@ static void compute_xor_offsets(void)\n \n struct bb_commit {\n \tstruct commit_list *reverse_edges;\n+\tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n-\tunsigned selected:1;\n+\tunsigned selected:1,\n+\t\t maximal:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -198,7 +200,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i;\n+\tunsigned int i, num_maximal;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -210,27 +212,85 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n \t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\n \t\tent->selected = 1;\n+\t\tent->maximal = 1;\n \t\tent->idx = i;\n+\n+\t\tent->commit_mask = bitmap_new();\n+\t\tbitmap_set(ent->commit_mask, i);\n+\n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n+\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n \t\tstruct commit_list *p;\n+\t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n \n-\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n-\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\tc_ent = bb_data_at(&bb->data, commit);\n+\n+\t\tif (c_ent->maximal) {\n+\t\t\tif (!c_ent->selected) {\n+\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n+\t\t\t\tnum_maximal++;\n+\t\t\t}\n+\n+\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\t}\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n-\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n+\t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n+\t\t\tint c_not_p, p_not_c;\n+\n+\t\t\tif (!p_ent->commit_mask) {\n+\t\t\t\tp_ent->commit_mask = bitmap_new();\n+\t\t\t\tc_not_p = 1;\n+\t\t\t\tp_not_c = 0;\n+\t\t\t} else {\n+\t\t\t\tc_not_p = bitmap_diff_nonzero(c_ent->commit_mask, p_ent->commit_mask);\n+\t\t\t\tp_not_c = bitmap_diff_nonzero(p_ent->commit_mask, c_ent->commit_mask);\n+\t\t\t}\n+\n+\t\t\tif (!c_not_p)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_or(p_ent->commit_mask, c_ent->commit_mask);\n+\n+\t\t\tif (p_not_c)\n+\t\t\t\tp_ent->maximal = 1;\n+\t\t\telse {\n+\t\t\t\tp_ent->maximal = 0;\n+\t\t\t\tfree_commit_list(p_ent->reverse_edges);\n+\t\t\t\tp_ent->reverse_edges = NULL;\n+\t\t\t}\n+\n+\t\t\tif (c_ent->maximal) {\n+\t\t\t\tcommit_list_insert(commit, &p_ent->reverse_edges);\n+\t\t\t} else {\n+\t\t\t\tstruct commit_list *cc = c_ent->reverse_edges;\n+\n+\t\t\t\tfor (; cc; cc = cc->next) {\n+\t\t\t\t\tif (!commit_list_contains(cc->item, p_ent->reverse_edges))\n+\t\t\t\t\t\tcommit_list_insert(cc->item, &p_ent->reverse_edges);\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n+\n+\t\tbitmap_free(c_ent->commit_mask);\n+\t\tc_ent->commit_mask = NULL;\n \t}\n+\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_selected_commits\", writer->selected_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_maximal_commits\", num_maximal);\n }\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 6bf68fee85..33ef9a098d 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -20,11 +20,87 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other master\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-master * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\____________...____________ |\n+# \\|\n+# * (base)\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for master (bit 0) and other (bit 1), and some flexibility\n+# in the order that merge bases are visited, the bitmasks at\n+# the end should be:\n+#\n+# master: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# octo-master: 1\n+# octo-other: 01\n+# merge-right: 111 (maximal)\n+# (l1): 111\n+# (r1): 111\n+# merge-left: 1101 (maximal)\n+# (l2): 11111 (maximal)\n+# (r2): 111101 (maximal)\n+# (base): 1111111 (maximal)\n+\n test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 100 &&\n+\ttest_commit_bulk --id=file 10 &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n+\n+\t# add complicated history setup, including merges and\n+\t# ambiguous merge-bases\n+\n+\tgit checkout -b merge-left other~2 &&\n+\tgit merge master~2 -m \"merge-left\" &&\n+\n+\tgit checkout -b merge-right master~1 &&\n+\tgit merge other~1 -m \"merge-right\" &&\n+\n+\tgit checkout -b octo-master master &&\n+\tgit merge merge-left merge-right -m \"octopus-master\" &&\n+\n+\tgit checkout -b octo-other other &&\n+\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\tgit checkout other &&\n+\tgit merge octo-other -m \"pull octopus\" &&\n+\n \tgit checkout master &&\n+\tgit merge octo-master -m \"pull octopus\" &&\n+\n+\t# Remove these branches so they are not selected\n+\t# as bitmap tips\n+\tgit branch -D merge-left &&\n+\tgit branch -D merge-right &&\n+\tgit branch -D octo-other &&\n+\tgit branch -D octo-master &&\n+\n+\t# add padding to make these merges less interesting\n+\t# and avoid having them selected for bitmaps\n+\ttest_commit_bulk --id=file 100 &&\n+\tgit checkout other &&\n+\ttest_commit_bulk --id=side 100 &&\n+\tgit checkout master &&\n+\n \tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n@@ -32,9 +108,12 @@ test_expect_success 'setup repo with moderate-sized history' '\n '\n \n test_expect_success 'full repack creates bitmaps' '\n-\tgit repack -ad &&\n+\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n+\ttest_line_count = 1 output &&\n+\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409688","messageId":"99a295416b2714c183aa39dcf515fb625090c2f2.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 18/23] pack-bitmap-write: ignore BITMAP_FLAG_REUSE","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:43:59Z","receivedAt":"2020-11-11T19:44:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe on-disk bitmap format has a flag to mark a bitmap to be \"reused\".\nThis is a rather curious feature, and works like this:\n\n - a run of pack-objects would decide to mark the last 80% of the\n bitmaps it generates with the reuse flag\n\n - the next time we generate bitmaps, we'd see those reuse flags from\n the last run, and mark those commits as special:\n\n - we'd be more likely to select those commits to get bitmaps in\n the new output\n\n - when generating the bitmap for a selected commit, we'd reuse the\n old bitmap as-is (rearranging the bits to match the new pack, of\n course)\n\nHowever, neither of these behaviors particularly makes sense.\n\nJust because a commit happened to be bitmapped last time does not make\nit a good candidate for having a bitmap this time. In particular, we may\nchoose bitmaps based on how recent they are in history, or whether a ref\ntip points to them, and those things will change. We're better off\nre-considering fresh which commits are good candidates.\n\nReusing the existing bitmap _is_ a reasonable thing to do to save\ncomputation. But only reusing exact bitmaps is a weak form of this. If\nwe have an old bitmap for A and now want a new bitmap for its child, we\nshould be able to compute that only by looking at trees and that are new\nto the child. But this code would consider only exact reuse (which is\nperhaps why it was eager to select those commits in the first place).\n\nFurthermore, the recent switch to the reverse-edge algorithm for\ngenerating bitmaps dropped this optimization entirely (and yet still\nperforms better).\n\nSo let's do a few cleanups:\n\n - drop the whole \"reusing bitmaps\" phase of generating bitmaps. It's\n not helping anything, and is mostly unused code (or worse, code that\n is using CPU but not doing anything useful)\n\n - drop the use of the on-disk reuse flag to select commits to bitmap\n\n - stop setting the on-disk reuse flag in bitmaps we generate (since\n nothing respects it anymore)\n\nWe will keep a few innards of the reuse code, which will help us\nimplement a more capable version of the \"reuse\" optimization:\n\n - simplify rebuild_existing_bitmaps() into a function that only builds\n the mapping of bits between the old and new orders, but doesn't\n actually convert any bitmaps\n\n - make rebuild_bitmap() public; we'll call it lazily to convert bitmaps\n as we traverse (using the mapping created above)\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 -\n pack-bitmap-write.c | 50 +++++-------------------------------------\n pack-bitmap.c | 46 +++++---------------------------------\n pack-bitmap.h | 6 ++++-\n 4 files changed, 16 insertions(+), 87 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5617c01b5a..2a00358f34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1104,7 +1104,6 @@ static void write_pack_file(void)\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n-\t\t\t\tbitmap_writer_reuse_bitmaps(&to_pack);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 7b4fc0f304..1995f75818 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -30,7 +30,6 @@ struct bitmap_writer {\n \tstruct ewah_bitmap *tags;\n \n \tkh_oid_map_t *bitmaps;\n-\tkh_oid_map_t *reused;\n \tstruct packing_data *to_pack;\n \n \tstruct bitmapped_commit *selected;\n@@ -112,7 +111,7 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n+static inline void push_bitmapped_commit(struct commit *commit)\n {\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n@@ -120,7 +119,7 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \t}\n \n \twriter.selected[writer.selected_nr].commit = commit;\n-\twriter.selected[writer.selected_nr].bitmap = reused;\n+\twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n \n \twriter.selected_nr++;\n@@ -372,13 +371,6 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkhiter_t hash_pos;\n \tint hash_ret;\n \n-\t/*\n-\t * the \"reuse bitmaps\" phase may have stored something here, but\n-\t * our new algorithm doesn't use it. Drop it.\n-\t */\n-\tif (stored->bitmap)\n-\t\tewah_free(stored->bitmap);\n-\n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n \thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n@@ -477,35 +469,6 @@ static int date_compare(const void *_a, const void *_b)\n \treturn (long)b->date - (long)a->date;\n }\n \n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack)\n-{\n-\tstruct bitmap_index *bitmap_git;\n-\tif (!(bitmap_git = prepare_bitmap_git(to_pack->repo)))\n-\t\treturn;\n-\n-\twriter.reused = kh_init_oid_map();\n-\trebuild_existing_bitmaps(bitmap_git, to_pack, writer.reused,\n-\t\t\t\t writer.show_progress);\n-\t/*\n-\t * NEEDSWORK: rebuild_existing_bitmaps() makes writer.reused reference\n-\t * some bitmaps in bitmap_git, so we can't free the latter.\n-\t */\n-}\n-\n-static struct ewah_bitmap *find_reused_bitmap(const struct object_id *oid)\n-{\n-\tkhiter_t hash_pos;\n-\n-\tif (!writer.reused)\n-\t\treturn NULL;\n-\n-\thash_pos = kh_get_oid_map(writer.reused, *oid);\n-\tif (hash_pos >= kh_end(writer.reused))\n-\t\treturn NULL;\n-\n-\treturn kh_value(writer.reused, hash_pos);\n-}\n-\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr,\n \t\t\t\t int max_bitmaps)\n@@ -519,12 +482,11 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(indexed_commits[i], NULL);\n+\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n \tfor (;;) {\n-\t\tstruct ewah_bitmap *reused_bitmap = NULL;\n \t\tstruct commit *chosen = NULL;\n \n \t\tnext = next_commit_index(i);\n@@ -539,15 +501,13 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \t\tif (next == 0) {\n \t\t\tchosen = indexed_commits[i];\n-\t\t\treused_bitmap = find_reused_bitmap(&chosen->object.oid);\n \t\t} else {\n \t\t\tchosen = indexed_commits[i + next];\n \n \t\t\tfor (j = 0; j <= next; ++j) {\n \t\t\t\tstruct commit *cm = indexed_commits[i + j];\n \n-\t\t\t\treused_bitmap = find_reused_bitmap(&cm->object.oid);\n-\t\t\t\tif (reused_bitmap || (cm->object.flags & NEEDS_BITMAP) != 0) {\n+\t\t\t\tif ((cm->object.flags & NEEDS_BITMAP) != 0) {\n \t\t\t\t\tchosen = cm;\n \t\t\t\t\tbreak;\n \t\t\t\t}\n@@ -557,7 +517,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(chosen, reused_bitmap);\n+\t\tpush_bitmapped_commit(chosen);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer.progress, i);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 82c6bf2843..682f4d19dd 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1333,9 +1333,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \tfree_bitmap_index(bitmap_git);\n }\n \n-static int rebuild_bitmap(uint32_t *reposition,\n-\t\t\t struct ewah_bitmap *source,\n-\t\t\t struct bitmap *dest)\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest)\n {\n \tuint32_t pos = 0;\n \tstruct ewah_iterator it;\n@@ -1364,19 +1364,11 @@ static int rebuild_bitmap(uint32_t *reposition,\n \treturn 0;\n }\n \n-int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n-\t\t\t struct packing_data *mapping,\n-\t\t\t kh_oid_map_t *reused_bitmaps,\n-\t\t\t int show_progress)\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping)\n {\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n-\tstruct bitmap *rebuild;\n-\tstruct stored_bitmap *stored;\n-\tstruct progress *progress = NULL;\n-\n-\tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tnum_objects = bitmap_git->pack->num_objects;\n \treposition = xcalloc(num_objects, sizeof(uint32_t));\n@@ -1394,33 +1386,7 @@ int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n \t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n-\trebuild = bitmap_new();\n-\ti = 0;\n-\n-\tif (show_progress)\n-\t\tprogress = start_progress(\"Reusing bitmaps\", 0);\n-\n-\tkh_foreach_value(bitmap_git->bitmaps, stored, {\n-\t\tif (stored->flags & BITMAP_FLAG_REUSE) {\n-\t\t\tif (!rebuild_bitmap(reposition,\n-\t\t\t\t\t lookup_stored_bitmap(stored),\n-\t\t\t\t\t rebuild)) {\n-\t\t\t\thash_pos = kh_put_oid_map(reused_bitmaps,\n-\t\t\t\t\t\t\t stored->oid,\n-\t\t\t\t\t\t\t &hash_ret);\n-\t\t\t\tkh_value(reused_bitmaps, hash_pos) =\n-\t\t\t\t\tbitmap_to_ewah(rebuild);\n-\t\t\t}\n-\t\t\tbitmap_reset(rebuild);\n-\t\t\tdisplay_progress(progress, ++i);\n-\t\t}\n-\t});\n-\n-\tstop_progress(&progress);\n-\n-\tfree(reposition);\n-\tbitmap_free(rebuild);\n-\treturn 0;\n+\treturn reposition;\n }\n \n void free_bitmap_index(struct bitmap_index *b)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 1203120c43..afa4115136 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,7 +73,11 @@ void bitmap_writer_set_checksum(unsigned char *sha1);\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr);\n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping);\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409689","messageId":"3aecb42215c31db61397edc9620f1e038753709e.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 19/23] pack-bitmap: factor out 'bitmap_for_commit()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:44:06Z","receivedAt":"2020-11-11T19:44:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A couple of callers within pack-bitmap.c duplicate logic to lookup a\ngiven object id in the bitamps khash. Factor this out into a new\nfunction, 'bitmap_for_commit()' to reduce some code duplication.\n\nMake this new function non-static, since it will be used in later\ncommits from outside of pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 33 +++++++++++++++++++--------------\n pack-bitmap.h | 2 ++\n 2 files changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 682f4d19dd..99a0683f49 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -375,6 +375,16 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit)\n+{\n+\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n+\t\t\t\t\t commit->object.oid);\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n+\t\treturn NULL;\n+\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t const struct object_id *oid)\n {\n@@ -460,10 +470,10 @@ static void show_commit(struct commit *commit, void *data)\n \n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n-\t\t\t const struct object_id *oid,\n+\t\t\t struct commit *commit,\n \t\t\t int bitmap_pos)\n {\n-\tkhiter_t hash_pos;\n+\tstruct ewah_bitmap *partial;\n \n \tif (data->seen && bitmap_get(data->seen, bitmap_pos))\n \t\treturn 0;\n@@ -471,10 +481,9 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \tif (bitmap_get(data->base, bitmap_pos))\n \t\treturn 0;\n \n-\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, *oid);\n-\tif (hash_pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(data->base, lookup_stored_bitmap(st));\n+\tpartial = bitmap_for_commit(bitmap_git, commit);\n+\tif (partial) {\n+\t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n@@ -493,8 +502,7 @@ static int should_include(struct commit *commit, void *_data)\n \t\t\t\t\t\t (struct object *)commit,\n \t\t\t\t\t\t NULL);\n \n-\tif (!add_to_include_set(data->bitmap_git, data, &commit->object.oid,\n-\t\t\t\tbitmap_pos)) {\n+\tif (!add_to_include_set(data->bitmap_git, data, commit, bitmap_pos)) {\n \t\tstruct commit_list *parent = commit->parents;\n \n \t\twhile (parent) {\n@@ -1277,10 +1285,10 @@ void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n-\tkhiter_t pos;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n \tstruct bitmap_index *bitmap_git;\n+\tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n \t\tdie(\"failed to load bitmap indexes\");\n@@ -1292,12 +1300,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tbitmap_git->version, bitmap_git->entry_count);\n \n \troot = revs->pending.objects[0].item;\n-\tpos = kh_get_oid_map(bitmap_git->bitmaps, root->oid);\n-\n-\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\tstruct ewah_bitmap *bm = lookup_stored_bitmap(st);\n+\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n \n+\tif (bm) {\n \t\tfprintf(stderr, \"Found bitmap for %s. %d bits / %08x checksum\\n\",\n \t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex afa4115136..25dfcf5615 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,6 +78,8 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest);\n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409690","messageId":"2ee47f31ab7718e3b6b0d0079f70b0a9195677b3.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 20/23] pack-bitmap: factor out 'add_commit_to_bitmap()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:44:11Z","receivedAt":"2020-11-11T19:44:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'find_objects()' currently needs to interact with the bitmaps khash\npretty closely. To make 'find_objects()' read a little more\nstraightforwardly, remove some of the khash-level details into a new\nfunction that describes what it does: 'add_commit_to_bitmap()'.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 36 +++++++++++++++++++++---------------\n 1 file changed, 21 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 99a0683f49..dc811ebae8 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -516,6 +516,23 @@ static int should_include(struct commit *commit, void *_data)\n \treturn 1;\n }\n \n+static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct bitmap **base,\n+\t\t\t\tstruct commit *commit)\n+{\n+\tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n+\n+\tif (!or_with)\n+\t\treturn 0;\n+\n+\tif (*base == NULL)\n+\t\t*base = ewah_to_bitmap(or_with);\n+\telse\n+\t\tbitmap_or_ewah(*base, or_with);\n+\n+\treturn 1;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -539,21 +556,10 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (object->type == OBJ_COMMIT) {\n-\t\t\tkhiter_t pos = kh_get_oid_map(bitmap_git->bitmaps, object->oid);\n-\n-\t\t\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\t\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\t\t\tstruct ewah_bitmap *or_with = lookup_stored_bitmap(st);\n-\n-\t\t\t\tif (base == NULL)\n-\t\t\t\t\tbase = ewah_to_bitmap(or_with);\n-\t\t\t\telse\n-\t\t\t\t\tbitmap_or_ewah(base, or_with);\n-\n-\t\t\t\tobject->flags |= SEEN;\n-\t\t\t\tcontinue;\n-\t\t\t}\n+\t\tif (object->type == OBJ_COMMIT &&\n+\t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n+\t\t\tobject->flags |= SEEN;\n+\t\t\tcontinue;\n \t\t}\n \n \t\tobject_list_insert(object, ¬_mapped);\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409691","messageId":"3f8235180636ea6735302c6e415d406676282090.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 21/23] pack-bitmap-write: use existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:44:18Z","receivedAt":"2020-11-11T19:44:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nWhen constructing new bitmaps, we perform a commit and tree walk in\nfill_bitmap_commit() and fill_bitmap_tree(). This walk would benefit\nfrom using existing bitmaps when available. We must track the existing\nbitmaps and translate them into the new object order, but this is\ngenerally faster than parsing trees.\n\nIn fill_bitmap_commit(), we must reorder thing somewhat. The priority\nqueue walks commits from newest-to-oldest, which means we correctly stop\nwalking when reaching a commit with a bitmap. However, if we walk trees\nfrom top to bottom, then we might be parsing trees that are actually\npart of a re-used bitmap. To avoid over-walking trees, add them to a\nLIFO queue and walk them from bottom-to-top after exploring commits\ncompletely.\n\nOn git.git, this reduces a second immediate bitmap computation from 2.0s\nto 1.0s. On linux.git, we go from 32s to 22s. On chromium's fork\nnetwork, we go from 227s to 198s.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 42 ++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 38 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 1995f75818..37204b691c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -340,20 +340,39 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue)\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n \tprio_queue_put(queue, commit);\n \n \twhile (queue->nr) {\n \t\tstruct commit_list *p;\n \t\tstruct commit *c = prio_queue_get(queue);\n \n+\t\t/*\n+\t\t * If this commit has an old bitmap, then translate that\n+\t\t * bitmap and add its bits to this one. No need to walk\n+\t\t * parents or the tree for this commit.\n+\t\t */\n+\t\tif (old_bitmap && mapping) {\n+\t\t\tstruct ewah_bitmap *old;\n+\n+\t\t\told = bitmap_for_commit(old_bitmap, c);\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Mark ourselves and queue our tree. The commit\n+\t\t * walk ensures we cover all parents.\n+\t\t */\n \t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n-\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tint pos = find_object_pos(&p->item->object.oid);\n@@ -363,6 +382,9 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t}\n \t\t}\n \t}\n+\n+\twhile (tree_queue->nr)\n+\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -386,6 +408,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n+\tstruct prio_queue tree_queue = { NULL };\n+\tstruct bitmap_index *old_bitmap;\n+\tuint32_t *mapping;\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -395,6 +420,12 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n \t\tthe_repository);\n \n+\told_bitmap = prepare_bitmap_git(to_pack->repo);\n+\tif (old_bitmap)\n+\t\tmapping = create_bitmap_mapping(old_bitmap, to_pack);\n+\telse\n+\t\tmapping = NULL;\n+\n \tbitmap_builder_init(&bb, &writer);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n@@ -402,7 +433,8 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue);\n+\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -428,7 +460,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tent->bitmap = NULL;\n \t}\n \tclear_prio_queue(&queue);\n+\tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree(mapping);\n \n \tstop_progress(&writer.progress);\n \n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409692","messageId":"ba53182c0de70a0a6529e06ad306b44bcaef30fe.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 22/23] pack-bitmap-write: relax unique rewalk condition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:44:24Z","receivedAt":"2020-11-11T19:44:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe previous commits improved the bitmap computation process for very\nlong, linear histories with many refs by removing quadratic growth in\nhow many objects were walked. The strategy of computing \"intermediate\ncommits\" using bitmasks for which refs can reach those commits\npartitioned the poset of reachable objects so each part could be walked\nexactly once. This was effective for linear histories.\n\nHowever, there was a (significant) drawback: wide histories with many\nrefs had an explosion of memory costs to compute the commit bitmasks\nduring the exploration that discovers these intermediate commits. Since\nthese wide histories are unlikely to repeat walking objects, the benefit\nof walking objects multiple times was not expensive before. But now, the\ncommit walk *before computing bitmaps* is incredibly expensive.\n\nIn an effort to discover a happy medium, this change reduces the walk\nfor intermediate commits to only the first-parent history. This focuses\nthe walk on how the histories converge, which still has significant\nreduction in repeat object walks. It is still possible to create\nquadratic behavior in this version, but it is probably less likely in\nrealistic data shapes.\n\nHere is some data taken on a fresh clone of the kernel:\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n original | 64.044 | 83.241 | 2.088 | 2.194 |\n last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 14 +++++---------\n t/t5310-pack-bitmaps.sh | 27 ++++++++++++++-------------\n 2 files changed, 19 insertions(+), 22 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 37204b691c..b0493d971d 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -199,7 +199,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i, num_maximal;\n+\tunsigned int i, num_maximal = 0;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -207,6 +207,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \treset_revision_walk();\n \trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n \trevs.topo_order = 1;\n+\trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n@@ -221,13 +222,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n-\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n-\t\tstruct commit_list *p;\n+\t\tstruct commit_list *p = commit->parents;\n \t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n@@ -235,16 +235,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n \t\tif (c_ent->maximal) {\n-\t\t\tif (!c_ent->selected) {\n-\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n-\t\t\t\tnum_maximal++;\n-\t\t\t}\n-\n+\t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n \t\t\tbb->commits[bb->commits_nr++] = commit;\n \t\t}\n \n-\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\tif (p) {\n \t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n \t\t\tint c_not_p, p_not_c;\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 33ef9a098d..68badd63cb 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -43,23 +43,24 @@ has_any () {\n # \\|\n # * (base)\n #\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n # The important part for the maximal commit algorithm is how\n # the bitmasks are extended. Assuming starting bit positions\n-# for master (bit 0) and other (bit 1), and some flexibility\n-# in the order that merge bases are visited, the bitmasks at\n-# the end should be:\n+# for master (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n #\n # master: 1 (maximal, selected)\n # other: 01 (maximal, selected)\n-# octo-master: 1\n-# octo-other: 01\n-# merge-right: 111 (maximal)\n-# (l1): 111\n-# (r1): 111\n-# merge-left: 1101 (maximal)\n-# (l2): 11111 (maximal)\n-# (r2): 111101 (maximal)\n-# (base): 1111111 (maximal)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n \n test_expect_success 'setup repo with moderate-sized history' '\n \ttest_commit_bulk --id=file 10 &&\n@@ -113,7 +114,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.156.gc03786897f\n\n"},{"id":"409693","messageId":"35beb4f098991bb3886bbaf74f9d2fc5561bca42.1605123653.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH 23/23] pack-bitmap-write: better reuse bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-11T19:44:28Z","receivedAt":"2020-11-11T19:44:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIf the old bitmap file contains a bitmap for a given commit, then that\ncommit does not need help from intermediate commits in its history to\ncompute its final bitmap. Eject that commit from the walk and insert it\nas a maximal commit in the list of commits for computing bitmaps.\n\nThis helps the repeat bitmap computation task, even if the selected\ncommits shift drastically. This helps when a previously-bitmapped commit\nexists in the first-parent history of a newly-selected commit. Since we\nstop the walk at these commits and we use a first-parent walk, it is\nharder to walk \"around\" these bitmapped commits. It's not impossible,\nbut we can greatly reduce the computation time for many selected\ncommits.\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n last patch | 100.641 | 35.560 | 2.152 | 2.224 |\n this patch | 99.720 | 11.696 | 2.152 | 2.217 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex b0493d971d..3ac90ae410 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -195,7 +195,8 @@ struct bitmap_builder {\n };\n \n static void bitmap_builder_init(struct bitmap_builder *bb,\n-\t\t\t\tstruct bitmap_writer *writer)\n+\t\t\t\tstruct bitmap_writer *writer,\n+\t\t\t\tstruct bitmap_index *old_bitmap)\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n@@ -234,12 +235,26 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n+\t\tif (old_bitmap && bitmap_for_commit(old_bitmap, commit)) {\n+\t\t\t/*\n+\t\t\t * This commit has an existing bitmap, so we can\n+\t\t\t * get its bits immediately without an object\n+\t\t\t * walk. There is no need to continue walking\n+\t\t\t * beyond this commit.\n+\t\t\t */\n+\t\t\tc_ent->maximal = 1;\n+\t\t\tp = NULL;\n+\t\t}\n+\n \t\tif (c_ent->maximal) {\n \t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n \t\t\tbb->commits[bb->commits_nr++] = commit;\n \t\t}\n \n+\t\tif (!c_ent->commit_mask)\n+\t\t\tcontinue;\n+\n \t\tif (p) {\n \t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n \t\t\tint c_not_p, p_not_c;\n@@ -422,7 +437,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \telse\n \t\tmapping = NULL;\n \n-\tbitmap_builder_init(&bb, &writer);\n+\tbitmap_builder_init(&bb, &writer, old_bitmap);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n-- \n2.29.2.156.gc03786897f\n"},{"id":"409720","messageId":"abf9273f-4795-5a48-c28b-15e68d40b910@gmail.com","threadId":"54622","inReplyTo":"9ab4b94b3573346b31e710486799ab3d95bade8e.1605123653.git.me@ttaylorr.com","subject":"Re: [PATCH 15/23] t5310: add branch-based checks","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T20:58:52Z","receivedAt":"2020-11-11T20:58:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/11/2020 2:43 PM, Taylor Blau wrote:\n> From: Derrick Stolee \n> \n> The current rev-list tests that check the bitmap data only work on HEAD\n> instead of multiple branches. Expand the test cases to handle both\n> 'master' and 'other' branches.\n\nAdding Johannes to CC since this likely will start colliding with his\ndefault branch rename efforts.\n\n> +rev_list_tests () {\n> +\tstate=$1\n> +\n> +\tfor branch in \"master\" \"other\"\n> +\tdo\n> +\t\trev_list_tests_head\n> +\tdone\n> +}\n\nSpecifically, this is a _new_ instance of \"master\", but all the\nother instances of \"master\" are likely being converted to \"main\"\nin parallel. It would certainly be easier to convert this test\n_after_ these changes are applied, but that's unlikely to happen\nwith the current schedule of things.\n\nThanks,\n-Stolee\n\n\n"},{"id":"409722","messageId":"xmqqwnyr38rh.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"abf9273f-4795-5a48-c28b-15e68d40b910@gmail.com","subject":"Re: [PATCH 15/23] t5310: add branch-based checks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-11T21:04:34Z","receivedAt":"2020-11-11T21:04:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 11/11/2020 2:43 PM, Taylor Blau wrote:\n>> From: Derrick Stolee \n>> \n>> The current rev-list tests that check the bitmap data only work on HEAD\n>> instead of multiple branches. Expand the test cases to handle both\n>> 'master' and 'other' branches.\n>\n> Adding Johannes to CC since this likely will start colliding with his\n> default branch rename efforts.\n>\n>> +rev_list_tests () {\n>> +\tstate=$1\n>> +\n>> +\tfor branch in \"master\" \"other\"\n>> +\tdo\n>> +\t\trev_list_tests_head\n>> +\tdone\n>> +}\n>\n> Specifically, this is a _new_ instance of \"master\", but all the\n> other instances of \"master\" are likely being converted to \"main\"\n> in parallel. It would certainly be easier to convert this test\n> _after_ these changes are applied, but that's unlikely to happen\n> with the current schedule of things.\n\nIn some tests, it may make sense to configure init.defaultbranchname\nin $HOME/.gitconfig upfront and either (1) leave instances of\n'master' as they are (we may want to avoid 'slave', but 'master' is\nnot all that wrong), or (2) rewrite instances of 'master' to 'main'\n(or 'primary' or whatever init.defaultbranchname gets configured).\n\n\n\n"},{"id":"409775","messageId":"CAN0heSoVS0MhEN_91da5Uv=jimLRcEu+EX1mocm2+quB559vsQ@mail.gmail.com","threadId":"54622","inReplyTo":"c59fcbcc67556c5c9c5a22a2ee745a2f58234efd.1605123652.git.me@ttaylorr.com","subject":"Re: [PATCH 02/23] pack-bitmap: fix header size check","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-11-12T17:39:53Z","receivedAt":"2020-11-12T17:40:08Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Hi Taylor/Peff,\n\nOn Wed, 11 Nov 2020 at 20:43, Taylor Blau wrote:\n>\n> This meant we were overly strict about the header size (requiring room\n> for a 32-byte worst-case hash, when sha1 is only 20 bytes). But in\n> practice it didn't matter because bitmap files tend to have at least 12\n> bytes of actual data anyway, so it was unlikely for a valid file to be\n> caught by this.\n\nGood catch.\n\n> + size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n>\n> - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> + if (index->map_size < header_size)\n> return error(\"Corrupted bitmap index (missing header data)\");\n\nI wondered if the \"12\" in the commit message shouldn't be \"32\". We used\nto count the hash bytes twice: first 32 that are included in the\n`sizeof()` and then another 20 or 32 on top of that. So we'd always\ncount 32 too many.\n\nExcept, what the addition of `the_hash_algo->rawsz` tries to account for\nis the hash aaaaall the way at the end of the file -- not the one at the\nend of the header. That's my reading of the state before 0f4d6cada8\n(\"pack-bitmap: make bitmap header handling hash agnostic\", 2019-02-19),\nanyway. So with that in mind, \"12\" makes sense.\n\nI think we should actually check that we have room for the footer\nhash. I'll comment more on the next patch.\n\n> - index->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> + index->map_pos += header_size;\n\nMakes sense.\n\nMartin\n"},{"id":"409777","messageId":"CAN0heSqiiMZgT+rEgWVVR_cEmPK2bS3QNnJuHahrqVQet7_Qug@mail.gmail.com","threadId":"54622","inReplyTo":"1573902df00e8a14a9cb68c37f55474388b1dc2e.1605123652.git.me@ttaylorr.com","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-11-12T17:47:09Z","receivedAt":"2020-11-12T17:47:26Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 11 Nov 2020 at 20:43, Taylor Blau wrote:\n>\n> A .bitmap file may have a \"name hash cache\" extension, which puts a\n> sequence of uint32_t bytes (one per object) at the end of the file. When\n\ns/bytes/values/, perhaps?\n\n> we see a flag indicating this extension, we blindly subtract the\n> appropriate number of bytes from our available length. However, if the\n> .bitmap file is too short, we'll underflow our length variable and wrap\n> around, thinking we have a very large length. This can lead to reading\n> out-of-bounds bytes while loading individual ewah bitmaps.\n\n> + uint32_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n\nHmm. If `sizeof(size_t)` is 8, then this multiplication can't possibly\noverflow. A huge value of `num_objects` (say, 0xffffffff) would give a\nhuge return value (0xffffffff<<2) which would be truncated (0xfffffffc).\nI think?\n\nDo we want a `u32_mult()`?\n\n> + unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n\nThe addition should be ok or mmap has failed on us. Do we know that we\nhave room for the final hash there so that the subtraction is ok? Yes,\nfrom the previous commit, we know we have room for the header, which is\neven larger. But that's cheating a bit -- see below.\n\n> + if (index->map + header_size + cache_size > index_end)\n> + return error(\"corrupted bitmap index file (too short to fit hash cache)\");\n> + index->hashes = (void *)(index_end - cache_size);\n> + index_end -= cache_size;\n\nIf the header size we're adding is indeed too large, the addition in the\ncheck would be undefined behavior, if I'm not mistaken. In practical\nterms, with 32-bit pointers and a huge size, we'd wrap around, decide\nthat everything is ok and go on to do the same erroneous subtraction as\nbefore.\n\nMaybe shuffle a few things over from the left to the right to only make\nsubtractions that we know are ok:\n\n if (cache_size > index_end - index->map - header_size)\n\nOne could substitute for `index_end - index_map` and end up with\n\n if (cache_size > index->map_size - header_size - the_hash_algo->rawsz)\n\nMaybe that's clearer in a way, or maybe then it's not so obvious that\nthe subtraction that follows matches this check.\n\nBut I don't think we can fully trust those subtractions. We're\nsubtracting the size of two hashes (one in the header, one in the\nfooter), but after the previous patch, we only know that there's room\nfor one. So probably the previous patch could go\n\n + /*\n + * Verify that we have room for the header and the\n + * trailing checksum hash, so we can safely subtract\n + * their sizes from map_size. We can afford to be\n + * a bit imprecise with the error message.\n + */\n - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n + if (index->map_size < header_size + the_hash_algo->rawsz)\n\nI *think* I've got most of my comments here somewhat right, but I could\neasily have missed something.\n\nMartin\n"},{"id":"409843","messageId":"20201113045700.GA743619@coredump.intra.peff.net","threadId":"54622","inReplyTo":"CAN0heSqiiMZgT+rEgWVVR_cEmPK2bS3QNnJuHahrqVQet7_Qug@mail.gmail.com","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-13T04:57:00Z","receivedAt":"2020-11-13T04:58:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 12, 2020 at 06:47:09PM +0100, Martin Ågren wrote:\n\n> > A .bitmap file may have a \"name hash cache\" extension, which puts a\n> > sequence of uint32_t bytes (one per object) at the end of the file. When\n> \n> s/bytes/values/, perhaps?\n\nYeah, definitely.\n\n> > we see a flag indicating this extension, we blindly subtract the\n> > appropriate number of bytes from our available length. However, if the\n> > .bitmap file is too short, we'll underflow our length variable and wrap\n> > around, thinking we have a very large length. This can lead to reading\n> > out-of-bounds bytes while loading individual ewah bitmaps.\n> \n> > + uint32_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n> \n> Hmm. If `sizeof(size_t)` is 8, then this multiplication can't possibly\n> overflow. A huge value of `num_objects` (say, 0xffffffff) would give a\n> huge return value (0xffffffff<<2) which would be truncated (0xfffffffc).\n> I think?\n\nYeah, `cache_size` should absolutely be a `size_t`. If you have more\nthan a billion objects, obviously your cache is going to be bigger than\nthat. But most importantly, somebody can _claim_ to have a huge number\nof objects and foil the size checks by wrapping around.\n\n> Do we want a `u32_mult()`?\n\nNah, we should be doing this as a size_t in the first place. There are\nsimilar problems with the .idx format, btw. I have a series to deal with\nthat which I've been meaning to post.\n\n> > + unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n> \n> The addition should be ok or mmap has failed on us. Do we know that we\n> have room for the final hash there so that the subtraction is ok? Yes,\n> from the previous commit, we know we have room for the header, which is\n> even larger. But that's cheating a bit -- see below.\n\nYeah, I agree this ought to be checking the minimum size against the\nheader _plus_ the trailer.\n\nI think the previous patch is actually where it goes wrong. The original\nwas checking for a minimum of:\n\n if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n\nwhich is the header plus the trailer. We want to readjust for the\nMAX_RAWSZ part of the header, so it should be:\n\n size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n\n> > + if (index->map + header_size + cache_size > index_end)\n> > + return error(\"corrupted bitmap index file (too short to fit hash cache)\");\n> > + index->hashes = (void *)(index_end - cache_size);\n> > + index_end -= cache_size;\n> \n> If the header size we're adding is indeed too large, the addition in the\n> check would be undefined behavior, if I'm not mistaken. In practical\n> terms, with 32-bit pointers and a huge size, we'd wrap around, decide\n> that everything is ok and go on to do the same erroneous subtraction as\n> before.\n> \n> Maybe shuffle a few things over from the left to the right to only make\n> subtractions that we know are ok:\n> \n> if (cache_size > index_end - index->map - header_size)\n\nYes, I agree this should be done as a subtraction as you showed to avoid\ninteger overflow.\n\n> But I don't think we can fully trust those subtractions. We're\n> subtracting the size of two hashes (one in the header, one in the\n> footer), but after the previous patch, we only know that there's room\n> for one. So probably the previous patch could go\n> \n> + /*\n> + * Verify that we have room for the header and the\n> + * trailing checksum hash, so we can safely subtract\n> + * their sizes from map_size. We can afford to be\n> + * a bit imprecise with the error message.\n> + */\n> - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> + if (index->map_size < header_size + the_hash_algo->rawsz)\n> \n> I *think* I've got most of my comments here somewhat right, but I could\n> easily have missed something.\n\nRight. I think that's right, and the previous patch is just buggy.\n\n-Peff\n"},{"id":"409851","messageId":"CAN0heSpQURCY4xnLHzq8ok7as-YUq4VingPrZ-NJnetsv-RG1w@mail.gmail.com","threadId":"54622","inReplyTo":"20201113045700.GA743619@coredump.intra.peff.net","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-11-13T05:26:02Z","receivedAt":"2020-11-13T05:26:18Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 13 Nov 2020 at 05:57, Jeff King wrote:\n>\n> On Thu, Nov 12, 2020 at 06:47:09PM +0100, Martin Ågren wrote:\n>\n> > > + uint32_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n> >\n> > Hmm. If `sizeof(size_t)` is 8, then this multiplication can't possibly\n> > overflow. A huge value of `num_objects` (say, 0xffffffff) would give a\n> > huge return value (0xffffffff<<2) which would be truncated (0xfffffffc).\n> > I think?\n>\n> Yeah, `cache_size` should absolutely be a `size_t`. If you have more\n> than a billion objects, obviously your cache is going to be bigger than\n> that. But most importantly, somebody can _claim_ to have a huge number\n> of objects and foil the size checks by wrapping around.\n>\n> > Do we want a `u32_mult()`?\n>\n> Nah, we should be doing this as a size_t in the first place. There are\n> similar problems with the .idx format, btw. I have a series to deal with\n> that which I've been meaning to post.\n\nYes, that makes sense!\n\n> > if (cache_size > index_end - index->map - header_size)\n>\n> Yes, I agree this should be done as a subtraction as you showed to avoid\n> integer overflow.\n\n> > - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> > + if (index->map_size < header_size + the_hash_algo->rawsz)\n\n> Right. I think that's right, and the previous patch is just buggy.\n\nMartin\n"},{"id":"409922","messageId":"X6760infcF0hRYTG@nand.local","threadId":"54622","inReplyTo":"20201113045700.GA743619@coredump.intra.peff.net","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-13T21:29:54Z","receivedAt":"2020-11-13T21:30:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 12, 2020 at 11:57:00PM -0500, Jeff King wrote:\n> On Thu, Nov 12, 2020 at 06:47:09PM +0100, Martin Ågren wrote:\n>\n> > The addition should be ok or mmap has failed on us. Do we know that we\n> > have room for the final hash there so that the subtraction is ok? Yes,\n> > from the previous commit, we know we have room for the header, which is\n> > even larger. But that's cheating a bit -- see below.\n>\n> Yeah, I agree this ought to be checking the minimum size against the\n> header _plus_ the trailer.\n>\n> I think the previous patch is actually where it goes wrong. The original\n> was checking for a minimum of:\n>\n> if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n>\n> which is the header plus the trailer. We want to readjust for the\n> MAX_RAWSZ part of the header, so it should be:\n>\n> size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n\nI'm not sure that I follow. If you apply this to the second patch in\nthis series, the only thing that changes is that it factors out:\n\n index->map_pos += ...;\n\ninto\n\n size_t header_size = ...;\n // ...\n index->map_pos += header_size;\n\nWhat am I missing here?\n\nThanks,\nTaylor\n"},{"id":"409924","messageId":"20201113213942.GB780435@coredump.intra.peff.net","threadId":"54622","inReplyTo":"X6760infcF0hRYTG@nand.local","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-13T21:39:42Z","receivedAt":"2020-11-13T21:39:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 13, 2020 at 04:29:54PM -0500, Taylor Blau wrote:\n\n> > which is the header plus the trailer. We want to readjust for the\n> > MAX_RAWSZ part of the header, so it should be:\n> >\n> > size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> > if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> \n> I'm not sure that I follow. If you apply this to the second patch in\n> this series, the only thing that changes is that it factors out:\n> \n> index->map_pos += ...;\n> \n> into\n> \n> size_t header_size = ...;\n> // ...\n> index->map_pos += header_size;\n> \n> What am I missing here?\n\nThe problem is this hunk from patch 2:\n\n> + size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> \n> - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> + if (index->map_size < header_size)\n> return error(\"Corrupted bitmap index (missing header data)\");\n\nThe header struct contains a field for the hash of the pack. So the\noriginal code as taking that full header, and adding in another\ncurrent-algo rawsz to account for the trailer.\n\nAfterwards, we adjust header_size to swap out the MAX_RAWSZ for the\ncurrent-algo rawsz. So header_size is a correct substitution for\nsizeof(*header) now. But we still have to add back in\nthe_hash_algo->rawsz to account for the trailer. The second \"+\" line is\nwrong to have removed it.\n\nThe later line we adjust:\n\n> - index->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> + index->map_pos += header_size;\n\nis correct. It's just skipping past the header, and doesn't care about\nthe trailer at all (and confusing the two is probably what led me to\nwrite the bug in the first place).\n\n-Peff\n"},{"id":"409926","messageId":"X67/aExL78Fxyobl@nand.local","threadId":"54622","inReplyTo":"20201113213942.GB780435@coredump.intra.peff.net","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-13T21:49:28Z","receivedAt":"2020-11-13T21:49:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Nov 13, 2020 at 04:39:42PM -0500, Jeff King wrote:\n> The problem is this hunk from patch 2:\n>\n> > + size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> >\n> > - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> > + if (index->map_size < header_size)\n> > return error(\"Corrupted bitmap index (missing header data)\");\n>\n> The header struct contains a field for the hash of the pack. So the\n> original code as taking that full header, and adding in another\n> current-algo rawsz to account for the trailer.\n>\n> Afterwards, we adjust header_size to swap out the MAX_RAWSZ for the\n> current-algo rawsz. So header_size is a correct substitution for\n> sizeof(*header) now. But we still have to add back in\n> the_hash_algo->rawsz to account for the trailer. The second \"+\" line is\n> wrong to have removed it.\n\nThanks for your patient explanation. This hunk should instead read:\n\n+ size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n\n- if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n+ if (index->map_size < header_size + the_hash_algo->rawsz)\n return error(\"Corrupted bitmap index (missing header data)\");\n\nThat error might not necessarily be right (it could say \"missing header\nor trailer data\"), though. I'm open to if you think it should be\nchanged or not.\n\nSince we didn't realize this bug at the time, the rest of the patch\nmessage is worded correctly, I believe.\n\n> The later line we adjust:\n>\n> > - index->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> > + index->map_pos += header_size;\n>\n> is correct. It's just skipping past the header, and doesn't care about\n> the trailer at all (and confusing the two is probably what led me to\n> write the bug in the first place).\n\nRight, makes sense.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"409930","messageId":"20201113221111.GA783373@coredump.intra.peff.net","threadId":"54622","inReplyTo":"X67/aExL78Fxyobl@nand.local","subject":"Re: [PATCH 03/23] pack-bitmap: bounds-check size of cache extension","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-13T22:11:11Z","receivedAt":"2020-11-13T22:11:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 13, 2020 at 04:49:28PM -0500, Taylor Blau wrote:\n\n> Thanks for your patient explanation. This hunk should instead read:\n> \n> + size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n> \n> - if (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n> + if (index->map_size < header_size + the_hash_algo->rawsz)\n> return error(\"Corrupted bitmap index (missing header data)\");\n> \n> That error might not necessarily be right (it could say \"missing header\n> or trailer data\"), though. I'm open to if you think it should be\n> changed or not.\n\nYeah, I agree it's misleading. In the idx code path we just say \"%s is\ntoo small\", which is more accurate.\n\n-Peff\n"},{"id":"409931","messageId":"20201113222328.GA8033@szeder.dev","threadId":"54622","inReplyTo":"ab64354851e2aa61e901e37814b2ae33d8f855d1.1605123653.git.me@ttaylorr.com","subject":"Re: [PATCH 17/23] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-11-13T22:23:28Z","receivedAt":"2020-11-13T22:23:48Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Nov 11, 2020 at 02:43:51PM -0500, Taylor Blau wrote:\n> From: Derrick Stolee \n> \n> The bitmap_writer_build() method calls bitmap_builder_init() to\n> construct a list of commits reachable from the selected commits along\n> with a \"reverse graph\". This reverse graph has edges pointing from a\n> commit to other commits that can reach that commit. After computing a\n> reachability bitmap for a commit, the values in that bitmap are then\n> copied to the reachability bitmaps across the edges in the reverse\n> graph.\n> \n> We can now relax the role of the reverse graph to greatly reduce the\n> number of intermediate reachability bitmaps we compute during this\n> reverse walk. The end result is that we walk objects the same number of\n> times as before when constructing the reachability bitmaps, but we also\n> spend much less time copying bits between bitmaps and have much lower\n> memory pressure in the process.\n> \n> The core idea is to select a set of \"important\" commits based on\n> interactions among the sets of commits reachable from each selected commit.\n\nThis patch breaks the test 'truncated bitmap fails gracefully (ewah)'\nwhen run with GIT_TEST_DEFAULT_HASH=sha256:\n\n expecting success of 5310.66 'truncated bitmap fails gracefully (ewah)':\n test_config pack.writebitmaphashcache false &&\n git repack -ad &&\n git rev-list --use-bitmap-index --count --all >expect &&\n bitmap=$(ls .git/objects/pack/*.bitmap) &&\n test_when_finished \"rm -f $bitmap\" &&\n test_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n mv -f $bitmap.tmp $bitmap &&\n git rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n test_cmp expect actual &&\n test_i18ngrep corrupt.ewah.bitmap stderr\n \n + test_config pack.writebitmaphashcache false\n + git repack -ad\n + git rev-list --use-bitmap-index --count --all\n + ls .git/objects/pack/pack-23fe19963d67a1d4797a39622c15144bbf35ab76a2c0638ba9288cc688c24c16.bitmap\n + bitmap=.git/objects/pack/pack-23fe19963d67a1d4797a39622c15144bbf35ab76a2c0638ba9288cc688c24c16.bitmap\n + test_when_finished rm -f .git/objects/pack/pack-23fe19963d67a1d4797a39622c15144bbf35ab76a2c0638ba9288cc688c24c16.bitmap\n + test_copy_bytes 256\n + mv -f .git/objects/pack/pack-23fe19963d67a1d4797a39622c15144bbf35ab76a2c0638ba9288cc688c24c16.bitmap.tmp .git/objects/pack/pack-23fe19963d67a1d4797a39622c15144bbf35ab76a2c0638ba9288cc688c24c16.bitmap\n + git rev-list --use-bitmap-index --count --all\n + test_cmp expect actual\n --- expect 2020-11-13 22:20:39.246355100 +0000\n +++ actual 2020-11-13 22:20:39.254355294 +0000\n @@ -1 +1 @@\n -239\n +236\n error: last command exited with $?=1\n\n"},{"id":"409933","messageId":"20201113230324.GA784144@coredump.intra.peff.net","threadId":"54622","inReplyTo":"20201113222328.GA8033@szeder.dev","subject":"Re: [PATCH 17/23] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-13T23:03:24Z","receivedAt":"2020-11-13T23:03:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 13, 2020 at 11:23:28PM +0100, SZEDER Gábor wrote:\n\n> This patch breaks the test 'truncated bitmap fails gracefully (ewah)'\n> when run with GIT_TEST_DEFAULT_HASH=sha256:\n\nThanks for reporting. It's mostly unluckiness that is unrelated to this\ncommit.\n\nWe're corrupting the bitmap by truncating it:\n\n> test_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n> mv -f $bitmap.tmp $bitmap &&\n\nand then expecting to notice the problem. But it really depends on which\nbitmaps we try to look at, and exactly where the truncation is. And this\ncommit just happens to rearrange the exact bytes we write to the bitmap\nfile.\n\nIf I do this:\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 68badd63cb..a83e7a93fb 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -436,7 +436,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n\nthen it passes with both sha1 and sha256.\n\nBut what's slightly disturbing is this output:\n\n> --- expect 2020-11-13 22:20:39.246355100 +0000\n> +++ actual 2020-11-13 22:20:39.254355294 +0000\n> @@ -1 +1 @@\n> -239\n> +236\n> error: last command exited with $?=1\n\nWe're actually producing the wrong answer here, which implies that\newah_read_mmap() is not being careful enough. Or possibly we are feeding\nit extra bytes (e.g., letting it run over into the name-hash cache or\ninto the trailer checksum).\n\nI think we'll have to dig further into this, probably running the sha256\ncase in a debugger to see what offsets we actually end up reading.\n\n-Peff\n"},{"id":"409946","messageId":"20201114062310.GA828053@coredump.intra.peff.net","threadId":"54622","inReplyTo":"20201113230324.GA784144@coredump.intra.peff.net","subject":"Re: [PATCH 17/23] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-14T06:23:10Z","receivedAt":"2020-11-14T06:32:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 13, 2020 at 06:03:24PM -0500, Jeff King wrote:\n\n> But what's slightly disturbing is this output:\n> \n> > --- expect 2020-11-13 22:20:39.246355100 +0000\n> > +++ actual 2020-11-13 22:20:39.254355294 +0000\n> > @@ -1 +1 @@\n> > -239\n> > +236\n> > error: last command exited with $?=1\n> \n> We're actually producing the wrong answer here, which implies that\n> ewah_read_mmap() is not being careful enough. Or possibly we are feeding\n> it extra bytes (e.g., letting it run over into the name-hash cache or\n> into the trailer checksum).\n> \n> I think we'll have to dig further into this, probably running the sha256\n> case in a debugger to see what offsets we actually end up reading.\n\nYep, the problem is in the caller, which is not careful about size\nchecks before reading the header before the actual ewah.\n\nThe first hunk here fixes it (the second is just another possible\ncorruption I noticed, but not triggered by the test):\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex dc811ebae8..785009b04e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -229,11 +229,16 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\tuint32_t commit_idx_pos;\n \t\tstruct object_id oid;\n \n+\t\tif (index->map_size - index->map_pos < 6)\n+\t\t\treturn error(\"corrupt ewah bitmap: truncated header for entry %d\", i);\n+\n \t\tcommit_idx_pos = read_be32(index->map, &index->map_pos);\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tnth_packed_object_id(&oid, index->pack, commit_idx_pos);\n+\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n+\t\t\t\t (unsigned)commit_idx_pos);\n \n \t\tbitmap = read_bitmap_1(index);\n \t\tif (!bitmap)\n\nWe should definitely do something like this, but there are some possible\nfurther improvements:\n\n - I think that map_size includes the trailing hash, and almost\n certainly any post-index extensions. We could probably compute the\n correct boundary of the bitmaps themselves in the caller and make\n sure we don't read past it. I'm not sure if it's worth the effort,\n though. In a truncation situation, basically all bets are off (is\n the trailer still there and the bitmap entries malformed, or is the\n trailer truncated?). The best we can do is try to read what's there\n as if it's correct data (and protect ourselves when it's obviously\n bogus).\n\n - we could avoid the magic \"6\" if read_be32() and read_u8(), which are\n custom helpers for this function, checked sizes before advancing the\n pointers.\n\n - I'm hesitant to add more tests in this area. As you can see from the\n commit which \"broke\" the test, truncating at byte N is going to be\n sensitive to small variations in the bitmap generation. So unless\n we're actually parsing the bitmaps and doing targeted corruptions,\n the tests will be somewhat brittle.\n\n-Peff\n"},{"id":"409997","messageId":"nycvar.QRO.7.76.6.2011160024050.18437@tvgsbejvaqbjf.bet","threadId":"54622","inReplyTo":"xmqqwnyr38rh.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 15/23] t5310: add branch-based checks","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-11-15T23:26:36Z","receivedAt":"2020-11-16T12:38:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 11 Nov 2020, Junio C Hamano wrote:\n\n> Derrick Stolee writes:\n>\n> > On 11/11/2020 2:43 PM, Taylor Blau wrote:\n> >> From: Derrick Stolee \n> >>\n> >> The current rev-list tests that check the bitmap data only work on HEAD\n> >> instead of multiple branches. Expand the test cases to handle both\n> >> 'master' and 'other' branches.\n> >\n> > Adding Johannes to CC since this likely will start colliding with his\n> > default branch rename efforts.\n\nIt's okay. It's not like this is the only topic I have to navigate around.\n\n> >> +rev_list_tests () {\n> >> +\tstate=$1\n> >> +\n> >> +\tfor branch in \"master\" \"other\"\n> >> +\tdo\n> >> +\t\trev_list_tests_head\n> >> +\tdone\n> >> +}\n> >\n> > Specifically, this is a _new_ instance of \"master\", but all the\n> > other instances of \"master\" are likely being converted to \"main\"\n> > in parallel. It would certainly be easier to convert this test\n> > _after_ these changes are applied, but that's unlikely to happen\n> > with the current schedule of things.\n>\n> In some tests, it may make sense to configure init.defaultbranchname\n> in $HOME/.gitconfig upfront and either (1) leave instances of\n> 'master' as they are (we may want to avoid 'slave', but 'master' is\n> not all that wrong), or (2) rewrite instances of 'master' to 'main'\n> (or 'primary' or whatever init.defaultbranchname gets configured).\n\nI explored this option very early on (so long ago that I failed to mention\nit). The problem with that is that `$HOME` is set thusly in `test-lib.sh`:\n\n\tHOME=\"$TRASH_DIRECTORY\"\n\nIn other words, the test repository's top-level directory is the home\ndirectory. Which means that `git status`, when run directly after `.\ntest-lib.sh` would already show `.gitignore` as untracked, something that\nwould trip up a couple of test scripts.\n\nCiao,\nDscho\n"},{"id":"410142","messageId":"cover.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:16Z","receivedAt":"2020-11-17T21:46:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is an updated version of this series, which improves the\nperformance of generating reachability bitmaps in large repositories.\n\nNot very much has changed since last time, but a range-diff is below\nnonetheless. The major changes are:\n\n - Avoid an overflow when bounds checking in the second and third\n patches (thanks, Martin, for noticing).\n - Incorporate a fix to avoid reading beyond an EWAH bitmap by double\n checking our read before actually doing it (thanks, Peff).\n - Harden the tests so that they pass under sha256-mode (thanks SZEDER,\n and Peff).\n\nDerrick Stolee (9):\n pack-bitmap-write: fill bitmap with commit history\n bitmap: add bitmap_diff_nonzero()\n commit: implement commit_list_contains()\n t5310: add branch-based checks\n pack-bitmap-write: rename children to reverse_edges\n pack-bitmap-write: build fewer intermediate bitmaps\n pack-bitmap-write: use existing bitmaps\n pack-bitmap-write: relax unique rewalk condition\n pack-bitmap-write: better reuse bitmaps\n\nJeff King (11):\n pack-bitmap: fix header size check\n pack-bitmap: bounds-check size of cache extension\n t5310: drop size of truncated ewah bitmap\n rev-list: die when --test-bitmap detects a mismatch\n ewah: factor out bitmap growth\n ewah: make bitmap growth less aggressive\n ewah: implement bitmap_or()\n ewah: add bitmap_dup() function\n pack-bitmap-write: reimplement bitmap writing\n pack-bitmap-write: pass ownership of intermediate bitmaps\n pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n\nTaylor Blau (4):\n ewah/ewah_bitmap.c: grow buffer past 1\n pack-bitmap.c: check reads more aggressively when loading\n pack-bitmap: factor out 'bitmap_for_commit()'\n pack-bitmap: factor out 'add_commit_to_bitmap()'\n\n builtin/pack-objects.c | 1 -\n commit.c | 11 +\n commit.h | 2 +\n ewah/bitmap.c | 54 ++++-\n ewah/ewah_bitmap.c | 2 +-\n ewah/ewok.h | 3 +-\n pack-bitmap-write.c | 452 +++++++++++++++++++++++++---------------\n pack-bitmap.c | 139 ++++++------\n pack-bitmap.h | 8 +-\n t/t5310-pack-bitmaps.sh | 164 ++++++++++++---\n 10 files changed, 555 insertions(+), 281 deletions(-)\n\nRange-diff against v1:\n -: ---------- > 1: 07054ff8ee ewah/ewah_bitmap.c: grow buffer past 1\n 1: 1970a70207 ! 2: 74a13b4a6e pack-bitmap: fix header size check\n @@ pack-bitmap.c: static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *ind\n +\tsize_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n\n -\tif (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n -+\tif (index->map_size < header_size)\n - \t\treturn error(\"Corrupted bitmap index (missing header data)\");\n +-\t\treturn error(\"Corrupted bitmap index (missing header data)\");\n ++\tif (index->map_size < header_size + the_hash_algo->rawsz)\n ++\t\treturn error(\"Corrupted bitmap index (too small)\");\n\n \tif (memcmp(header->magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)) != 0)\n + \t\treturn error(\"Corrupted bitmap index file (wrong header)\");\n @@ pack-bitmap.c: static int load_bitmap_header(struct bitmap_index *index)\n \t}\n\n 2: 36b1815d03 ! 3: db11116dac pack-bitmap: bounds-check size of cache extension\n @@ Commit message\n pack-bitmap: bounds-check size of cache extension\n\n A .bitmap file may have a \"name hash cache\" extension, which puts a\n - sequence of uint32_t bytes (one per object) at the end of the file. When\n - we see a flag indicating this extension, we blindly subtract the\n + sequence of uint32_t values (one per object) at the end of the file.\n + When we see a flag indicating this extension, we blindly subtract the\n appropriate number of bytes from our available length. However, if the\n .bitmap file is too short, we'll underflow our length variable and wrap\n around, thinking we have a very large length. This can lead to reading\n @@ pack-bitmap.c: static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n -+\t\tuint32_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n ++\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n +\t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n\n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n @@ pack-bitmap.c: static int load_bitmap_header(struct bitmap_index *index)\n \t\tif (flags & BITMAP_OPT_HASH_CACHE) {\n -\t\t\tunsigned char *end = index->map + index->map_size - the_hash_algo->rawsz;\n -\t\t\tindex->hashes = ((uint32_t *)end) - index->pack->num_objects;\n -+\t\t\tif (index->map + header_size + cache_size > index_end)\n ++\t\t\tif (cache_size > index_end - index->map - header_size)\n +\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit hash cache)\");\n +\t\t\tindex->hashes = (void *)(index_end - cache_size);\n +\t\t\tindex_end -= cache_size;\n 3: edfec2ea62 = 4: f779e76f82 t5310: drop size of truncated ewah bitmap\n 4: f3fec466f7 = 5: 1a9ac1c4ae rev-list: die when --test-bitmap detects a mismatch\n 5: b35012f44d = 6: 9bb1ea3b19 ewah: factor out bitmap growth\n 6: 53b8bea98c = 7: f8426c7e8b ewah: make bitmap growth less aggressive\n 7: 98e3bfc1b2 = 8: 674e31f98e ewah: implement bitmap_or()\n 8: 1bd115fc51 = 9: a903c949d8 ewah: add bitmap_dup() function\n 9: adf16557c2 = 10: c951206729 pack-bitmap-write: reimplement bitmap writing\n10: 27992687c9 = 11: 466dd3036a pack-bitmap-write: pass ownership of intermediate bitmaps\n11: d92fb0e1e1 = 12: 8e5607929d pack-bitmap-write: fill bitmap with commit history\n12: bf86cb6196 = 13: 4840c64c51 bitmap: add bitmap_diff_nonzero()\n13: 78cdf847aa = 14: 63e846f4e8 commit: implement commit_list_contains()\n14: 778e9e9c44 = 15: 8b5d239333 t5310: add branch-based checks\n15: 526d3509ef = 16: 60a46091bb pack-bitmap-write: rename children to reverse_edges\n -: ---------- > 17: 8f7bb2dd2e pack-bitmap.c: check reads more aggressively when loading\n16: 86d77fd085 ! 18: 5262daa330 pack-bitmap-write: build fewer intermediate bitmaps\n @@ t/t5310-pack-bitmaps.sh: test_expect_success 'setup repo with moderate-sized his\n '\n\n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n + \tgit rev-list --use-bitmap-index --count --all >expect &&\n + \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n + \ttest_when_finished \"rm -f $bitmap\" &&\n +-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n ++\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n + \tmv -f $bitmap.tmp $bitmap &&\n + \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n + \ttest_cmp expect actual &&\n17: e4f296100c = 19: a206f48614 pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n18: 6e856bcf75 = 20: 9928b3c7da pack-bitmap: factor out 'bitmap_for_commit()'\n19: 9b5f595f50 = 21: f40a39a48a pack-bitmap: factor out 'add_commit_to_bitmap()'\n20: c458f98e11 = 22: 4bf5e78a54 pack-bitmap-write: use existing bitmaps\n21: 3026876e7a = 23: 1da4fa0fb8 pack-bitmap-write: relax unique rewalk condition\n22: ce2716e291 = 24: 42399a1c2e pack-bitmap-write: better reuse bitmaps\n--\n2.29.2.312.gabc4d358d8\n"},{"id":"410143","messageId":"07054ff8ee43c4361c472e40b72f767107f66ed8.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 01/24] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:24Z","receivedAt":"2020-11-17T21:46:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When the buffer size is exactly 1, we fail to grow it properly, since\nthe integer truncation means that 1 * 3 / 2 = 1. This can cause a bad\nwrite on the line below.\n\nBandaid this by first padding the buffer by 16, and then growing it.\nThis still allows old blocks to fit into new ones, but fixes the case\nwhere the block size equals 1.\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/ewah_bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex d59b1afe3d..3fae04ad00 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -45,7 +45,7 @@ static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n static inline void buffer_push(struct ewah_bitmap *self, eword_t value)\n {\n \tif (self->buffer_size + 1 >= self->alloc_size)\n-\t\tbuffer_grow(self, self->buffer_size * 3 / 2);\n+\t\tbuffer_grow(self, (self->buffer_size + 16) * 3 / 2);\n \n \tself->buffer[self->buffer_size++] = value;\n }\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410144","messageId":"74a13b4a6edaa8f9c679019017fafe2fd8512357.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 02/24] pack-bitmap: fix header size check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:33Z","receivedAt":"2020-11-17T21:46:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWhen we parse a .bitmap header, we first check that we have enough bytes\nto make a valid header. We do that based on sizeof(struct\nbitmap_disk_header). However, as of 0f4d6cada8 (pack-bitmap: make bitmap\nheader handling hash agnostic, 2019-02-19), that struct oversizes its\nchecksum member to GIT_MAX_RAWSZ. That means we need to adjust for the\ndifference between that constant and the size of the actual hash we're\nusing. That commit adjusted the code which moves our pointer forward,\nbut forgot to update the size check.\n\nThis meant we were overly strict about the header size (requiring room\nfor a 32-byte worst-case hash, when sha1 is only 20 bytes). But in\npractice it didn't matter because bitmap files tend to have at least 12\nbytes of actual data anyway, so it was unlikely for a valid file to be\ncaught by this.\n\nLet's fix it by pulling the header size into a separate variable and\nusing it in both spots. That fixes the bug and simplifies the code to make\nit harder to have a mismatch like this in the future. It will also come\nin handy in the next patch for more bounds checking.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4077e731e8..fe5647e72e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -138,9 +138,10 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n+\tsize_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n \n-\tif (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n-\t\treturn error(\"Corrupted bitmap index (missing header data)\");\n+\tif (index->map_size < header_size + the_hash_algo->rawsz)\n+\t\treturn error(\"Corrupted bitmap index (too small)\");\n \n \tif (memcmp(header->magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)) != 0)\n \t\treturn error(\"Corrupted bitmap index file (wrong header)\");\n@@ -164,7 +165,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-\tindex->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n+\tindex->map_pos += header_size;\n \treturn 0;\n }\n \n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410145","messageId":"db11116dac1bda88d8b80fbb8197c92d9ff4369c.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 03/24] pack-bitmap: bounds-check size of cache extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:38Z","receivedAt":"2020-11-17T21:46:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nA .bitmap file may have a \"name hash cache\" extension, which puts a\nsequence of uint32_t values (one per object) at the end of the file.\nWhen we see a flag indicating this extension, we blindly subtract the\nappropriate number of bytes from our available length. However, if the\n.bitmap file is too short, we'll underflow our length variable and wrap\naround, thinking we have a very large length. This can lead to reading\nout-of-bounds bytes while loading individual ewah bitmaps.\n\nWe can fix this by checking the number of available bytes when we parse\nthe header. The existing \"truncated bitmap\" test is now split into two\ntests: one where we don't have this extension at all (and hence actually\ndo try to read a truncated ewah bitmap) and one where we realize\nup-front that we can't even fit in the cache structure. We'll check\nstderr in each case to make sure we hit the error we're expecting.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 8 ++++++--\n t/t5310-pack-bitmaps.sh | 17 +++++++++++++++--\n 2 files changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fe5647e72e..074d9ac8f2 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -153,14 +153,18 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n+\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n \t\t\treturn error(\"Unsupported options for bitmap index file \"\n \t\t\t\t\"(Git requires BITMAP_OPT_FULL_DAG)\");\n \n \t\tif (flags & BITMAP_OPT_HASH_CACHE) {\n-\t\t\tunsigned char *end = index->map + index->map_size - the_hash_algo->rawsz;\n-\t\t\tindex->hashes = ((uint32_t *)end) - index->pack->num_objects;\n+\t\t\tif (cache_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit hash cache)\");\n+\t\t\tindex->hashes = (void *)(index_end - cache_size);\n+\t\t\tindex_end -= cache_size;\n \t\t}\n \t}\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 8318781d2b..e2c3907a68 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -343,7 +343,8 @@ test_expect_success 'pack reuse respects --incremental' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_success 'truncated bitmap fails gracefully' '\n+test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\ttest_config pack.writebitmaphashcache false &&\n \tgit repack -ad &&\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -352,7 +353,19 @@ test_expect_success 'truncated bitmap fails gracefully' '\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt stderr\n+\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\tgit repack -ad &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n # have_delta \n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410146","messageId":"f779e76f82bd2684e835affa45583063772d1c1b.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 04/24] t5310: drop size of truncated ewah bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:44Z","receivedAt":"2020-11-17T21:46:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe truncate the .bitmap file to 512 bytes and expect to run into\nproblems reading an individual ewah file. But this length is somewhat\narbitrary, and just happened to work when the test was added in\n9d2e330b17 (ewah_read_mmap: bounds-check mmap reads, 2018-06-14).\n\nAn upcoming commit will change the size of the history we create in the\ntest repo, which will cause this test to fail. We can future-proof it a\nbit more by reducing the size of the truncated bitmap file.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex e2c3907a68..70a4fc4843 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -349,7 +349,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410147","messageId":"1a9ac1c4aefcc8e4665106a39ce6c6c0b2a4a52c.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 05/24] rev-list: die when --test-bitmap detects a mismatch","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:48Z","receivedAt":"2020-11-17T21:46:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nYou can use \"git rev-list --test-bitmap HEAD\" to check that bitmaps\nproduce the same answer we'd get from a regular traversal. But if we\ndetect an error, we only print \"mismatch\", and still exit with a\nsuccessful error code.\n\nThat makes the uses of --test-bitmap in the test suite (e.g., in t5310)\nmostly pointless: even if we saw an error, the tests wouldn't notice.\nLet's instead call die(), which will let these tests work as designed,\nand alert us if the bitmaps are bogus.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 074d9ac8f2..4431f9f120 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1328,7 +1328,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (bitmap_equals(result, tdata.base))\n \t\tfprintf(stderr, \"OK!\\n\");\n \telse\n-\t\tfprintf(stderr, \"Mismatch!\\n\");\n+\t\tdie(\"mismatch in bitmap results\");\n \n \tfree_bitmap_index(bitmap_git);\n }\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410148","messageId":"9bb1ea3b19c86507b7485ae872a8ef350cd0aedc.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 06/24] ewah: factor out bitmap growth","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:46:54Z","receivedAt":"2020-11-17T21:47:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe auto-grow bitmaps when somebody asks to set a bit whose position is\noutside of our currently allocated range. Other operations besides\nsingle bit-setting might need to do this, too, so let's pull it into its\nown function.\n\nNote that we change the semantics a little: you now ask for the number\nof words you'd like to have, not the id of the block you'd like to write\nto.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 +++++++++-----\n 1 file changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d8cec585af..7c1ecfa6fd 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,18 +35,22 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n-void bitmap_set(struct bitmap *self, size_t pos)\n+static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n-\tsize_t block = EWAH_BLOCK(pos);\n-\n-\tif (block >= self->word_alloc) {\n+\tif (word_alloc > self->word_alloc) {\n \t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = block ? block * 2 : 1;\n+\t\tself->word_alloc = word_alloc * 2;\n \t\tREALLOC_ARRAY(self->words, self->word_alloc);\n \t\tmemset(self->words + old_size, 0x0,\n \t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n \t}\n+}\n \n+void bitmap_set(struct bitmap *self, size_t pos)\n+{\n+\tsize_t block = EWAH_BLOCK(pos);\n+\n+\tbitmap_grow(self, block + 1);\n \tself->words[block] |= EWAH_MASK(pos);\n }\n \n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410149","messageId":"f8426c7e8b2523711f8d6f917dff4d89881f4340.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 07/24] ewah: make bitmap growth less aggressive","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:00Z","receivedAt":"2020-11-17T21:47:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nIf you ask to set a bit in the Nth word and we haven't yet allocated\nthat many slots in our array, we'll increase the bitmap size to 2*N.\nThis means we might frequently end up with bitmaps that are twice the\nnecessary size (as soon as you ask for the biggest bit, we'll size up to\ntwice that).\n\nBut if we just allocate as many words as were asked for, we may not grow\nfast enough. The worst case there is setting bit 0, then 1, etc. Each\ntime we grow we'd just extend by one more word, giving us linear\nreallocations (and quadratic memory copies).\n\nLet's combine those by allocating the maximum of:\n\n - what the caller asked for\n\n - a geometric increase in existing size; we'll switch to 3/2 instead of\n 2 here. That's less aggressive and may help avoid fragmenting memory\n (N + 3N/2 > 9N/4, so old chunks can be reused as we scale up).\n\nOur worst case is still 3/2N wasted bits (you set bit N-1, then setting\nbit N causes us to grow by 3/2), but our average should be much better.\n\nThis isn't usually that big a deal, but it will matter as we shift the\nreachability bitmap generation code to store more bitmaps in memory.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 7c1ecfa6fd..43a59d7fed 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -39,7 +39,9 @@ static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n \tif (word_alloc > self->word_alloc) {\n \t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = word_alloc * 2;\n+\t\tself->word_alloc = old_size * 3 / 2;\n+\t\tif (word_alloc > self->word_alloc)\n+\t\t\tself->word_alloc = word_alloc;\n \t\tREALLOC_ARRAY(self->words, self->word_alloc);\n \t\tmemset(self->words + old_size, 0x0,\n \t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410150","messageId":"674e31f98e9ad3a69504125a9aaa6e914383858b.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 08/24] ewah: implement bitmap_or()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:05Z","receivedAt":"2020-11-17T21:47:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe have a function to bitwise-OR an ewah into an uncompressed bitmap,\nbut not to OR two uncompressed bitmaps. Let's add it.\n\nInterestingly, we have a public header declaration going back to\ne1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\nfunction was never implemented.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 43a59d7fed..c3f8e7242b 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -127,6 +127,15 @@ void bitmap_and_not(struct bitmap *self, struct bitmap *other)\n \t\tself->words[i] &= ~other->words[i];\n }\n \n+void bitmap_or(struct bitmap *self, const struct bitmap *other)\n+{\n+\tsize_t i;\n+\n+\tbitmap_grow(self, other->word_alloc);\n+\tfor (i = 0; i < other->word_alloc; i++)\n+\t\tself->words[i] |= other->words[i];\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410151","messageId":"a903c949d8b597bb48094c100974369e259d4b40.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 09/24] ewah: add bitmap_dup() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:10Z","receivedAt":"2020-11-17T21:47:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThere's no easy way to make a copy of a bitmap. Obviously a caller can\niterate over the bits and set them one by one in a new bitmap, but we\ncan go much faster by copying whole words with memcpy().\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 7 +++++++\n ewah/ewok.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex c3f8e7242b..eb7e2539be 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,6 +35,13 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n+struct bitmap *bitmap_dup(const struct bitmap *src)\n+{\n+\tstruct bitmap *dst = bitmap_word_alloc(src->word_alloc);\n+\tCOPY_ARRAY(dst->words, src->words, src->word_alloc);\n+\treturn dst;\n+}\n+\n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n \tif (word_alloc > self->word_alloc) {\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 011852bef1..1fc555e672 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -173,6 +173,7 @@ struct bitmap {\n \n struct bitmap *bitmap_new(void);\n struct bitmap *bitmap_word_alloc(size_t word_alloc);\n+struct bitmap *bitmap_dup(const struct bitmap *src);\n void bitmap_set(struct bitmap *self, size_t pos);\n void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410152","messageId":"c9512067293c082ad3082262e50dfd04f1bc1648.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 10/24] pack-bitmap-write: reimplement bitmap writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:14Z","receivedAt":"2020-11-17T21:47:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe bitmap generation code works by iterating over the set of commits\nfor which we plan to write bitmaps, and then for each one performing a\ntraditional traversal over the reachable commits and trees, filling in\nthe bitmap. Between two traversals, we can often reuse the previous\nbitmap result as long as the first commit is an ancestor of the second.\nHowever, our worst case is that we may end up doing \"n\" complete\ncomplete traversals to the root in order to create \"n\" bitmaps.\n\nIn a real-world case (the shared-storage repo consisting of all GitHub\nforks of chromium/chromium), we perform very poorly: generating bitmaps\ntakes ~3 hours, whereas we can walk the whole object graph in ~3\nminutes.\n\nThis commit completely rewrites the algorithm, with the goal of\naccessing each object only once. It works roughly like this:\n\n - generate a list of commits in topo-order using a single traversal\n\n - invert the edges of the graph (so have parents point at their\n children)\n\n - make one pass in reverse topo-order, generating a bitmap for each\n commit and passing the result along to child nodes\n\nWe generate correct results because each node we visit has already had\nall of its ancestors added to the bitmap. And we make only two linear\npasses over the commits.\n\nWe also visit each tree usually only once. When filling in a bitmap, we\ndon't bother to recurse into trees whose bit is already set in the\nbitmap (since we know we've already done so when setting their bit).\nThat means that if commit A references tree T, none of its descendants\nwill need to open T again. I say \"usually\", though, because it is\npossible for a given tree to be mentioned in unrelated parts of history\n(e.g., cherry-picking to a parallel branch).\n\nSo we've accomplished our goal, and the resulting algorithm is pretty\nsimple to understand. But there are some downsides, at least with this\ninitial implementation:\n\n - we no longer reuse the results of any on-disk bitmaps when\n generating. So we'd expect to sometimes be slower than the original\n when bitmaps already exist. However, this is something we'll be able\n to add back in later.\n\n - we use much more memory. Instead of keeping one bitmap in memory at\n a time, we're passing them up through the graph. So our memory use\n should scale with the graph width (times the size of a bitmap).\n\nSo how does it perform?\n\nFor a clone of linux.git, generating bitmaps from scratch with the old\nalgorithm took 63s. Using this algorithm it takes 205s. Which is much\nworse, but _might_ be acceptable if it behaved linearly as the size\ngrew. It also increases peak heap usage by ~1G. That's not impossibly\nlarge, but not encouraging.\n\nOn the complete fork-network of torvalds/linux, it increases the peak\nRAM usage by 40GB. Yikes. (I forgot to record the time it took, but the\nmemory usage was too much to consider this reasonable anyway).\n\nOn the complete fork-network of chromium/chromium, I ran out of memory\nbefore succeeding. Some back-of-the-envelope calculations indicate it\nwould need 80+GB to complete.\n\nSo at this stage, we've managed to make things much worse. But because\nof the way this new algorithm is structured, there are a lot of\nopportunities for optimization on top. We'll start implementing those in\nthe follow-on patches.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 303 ++++++++++++++++++++++++--------------------\n 1 file changed, 169 insertions(+), 134 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 5e998bdaa7..f2f0b6b2c2 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -110,8 +110,6 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n /**\n * Compute the actual bitmaps\n */\n-static struct object **seen_objects;\n-static unsigned int seen_objects_nr, seen_objects_alloc;\n \n static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n {\n@@ -127,21 +125,6 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \twriter.selected_nr++;\n }\n \n-static inline void mark_as_seen(struct object *object)\n-{\n-\tALLOC_GROW(seen_objects, seen_objects_nr + 1, seen_objects_alloc);\n-\tseen_objects[seen_objects_nr++] = object;\n-}\n-\n-static inline void reset_all_seen(void)\n-{\n-\tunsigned int i;\n-\tfor (i = 0; i < seen_objects_nr; ++i) {\n-\t\tseen_objects[i]->flags &= ~(SEEN | ADDED | SHOWN);\n-\t}\n-\tseen_objects_nr = 0;\n-}\n-\n static uint32_t find_object_pos(const struct object_id *oid)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n@@ -154,60 +137,6 @@ static uint32_t find_object_pos(const struct object_id *oid)\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n-static void show_object(struct object *object, const char *name, void *data)\n-{\n-\tstruct bitmap *base = data;\n-\tbitmap_set(base, find_object_pos(&object->oid));\n-\tmark_as_seen(object);\n-}\n-\n-static void show_commit(struct commit *commit, void *data)\n-{\n-\tmark_as_seen((struct object *)commit);\n-}\n-\n-static int\n-add_to_include_set(struct bitmap *base, struct commit *commit)\n-{\n-\tkhiter_t hash_pos;\n-\tuint32_t bitmap_pos = find_object_pos(&commit->object.oid);\n-\n-\tif (bitmap_get(base, bitmap_pos))\n-\t\treturn 0;\n-\n-\thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n-\tif (hash_pos < kh_end(writer.bitmaps)) {\n-\t\tstruct bitmapped_commit *bc = kh_value(writer.bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(base, bc->bitmap);\n-\t\treturn 0;\n-\t}\n-\n-\tbitmap_set(base, bitmap_pos);\n-\treturn 1;\n-}\n-\n-static int\n-should_include(struct commit *commit, void *_data)\n-{\n-\tstruct bitmap *base = _data;\n-\n-\tif (!add_to_include_set(base, commit)) {\n-\t\tstruct commit_list *parent = commit->parents;\n-\n-\t\tmark_as_seen((struct object *)commit);\n-\n-\t\twhile (parent) {\n-\t\t\tparent->item->object.flags |= SEEN;\n-\t\t\tmark_as_seen((struct object *)parent->item);\n-\t\t\tparent = parent->next;\n-\t\t}\n-\n-\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n static void compute_xor_offsets(void)\n {\n \tstatic const int MAX_XOR_OFFSET_SEARCH = 10;\n@@ -248,79 +177,185 @@ static void compute_xor_offsets(void)\n \t}\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+struct bb_commit {\n+\tstruct commit_list *children;\n+\tstruct bitmap *bitmap;\n+\tunsigned selected:1;\n+\tunsigned idx; /* within selected array */\n+};\n+\n+define_commit_slab(bb_data, struct bb_commit);\n+\n+struct bitmap_builder {\n+\tstruct bb_data data;\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+};\n+\n+static void bitmap_builder_init(struct bitmap_builder *bb,\n+\t\t\t\tstruct bitmap_writer *writer)\n {\n-\tstatic const double REUSE_BITMAP_THRESHOLD = 0.2;\n-\n-\tint i, reuse_after, need_reset;\n-\tstruct bitmap *base = bitmap_new();\n \tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tunsigned int i;\n+\n+\tmemset(bb, 0, sizeof(*bb));\n+\tinit_bb_data(&bb->data);\n+\n+\treset_revision_walk();\n+\trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n+\trevs.topo_order = 1;\n+\n+\tfor (i = 0; i < writer->selected_nr; i++) {\n+\t\tstruct commit *c = writer->selected[i].commit;\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tent->selected = 1;\n+\t\tent->idx = i;\n+\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(\"revision walk setup failed\");\n+\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tstruct commit_list *p;\n+\n+\t\tparse_commit_or_die(commit);\n+\n+\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\tbb->commits[bb->commits_nr++] = commit;\n+\n+\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n+\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t}\n+\t}\n+}\n+\n+static void bitmap_builder_clear(struct bitmap_builder *bb)\n+{\n+\tclear_bb_data(&bb->data);\n+\tfree(bb->commits);\n+\tbb->commits_nr = bb->commits_alloc = 0;\n+}\n+\n+static void fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n+{\n+\tuint32_t pos;\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\t/*\n+\t * If our bit is already set, then there is nothing to do. Both this\n+\t * tree and all of its children will be set.\n+\t */\n+\tpos = find_object_pos(&tree->object.oid);\n+\tif (bitmap_get(bitmap, pos))\n+\t\treturn;\n+\tbitmap_set(bitmap, pos);\n+\n+\tif (parse_tree(tree) < 0)\n+\t\tdie(\"unable to load tree object %s\",\n+\t\t oid_to_hex(&tree->object.oid));\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tfill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Gitlink, etc; not reachable */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n+static void fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit)\n+{\n+\tif (!ent->bitmap)\n+\t\tent->bitmap = bitmap_new();\n+\n+\t/*\n+\t * mark ourselves, but do not bother with parents; their values\n+\t * will already have been propagated to us\n+\t */\n+\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n+\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+}\n+\n+static void store_selected(struct bb_commit *ent, struct commit *commit)\n+{\n+\tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n+\tkhiter_t hash_pos;\n+\tint hash_ret;\n+\n+\t/*\n+\t * the \"reuse bitmaps\" phase may have stored something here, but\n+\t * our new algorithm doesn't use it. Drop it.\n+\t */\n+\tif (stored->bitmap)\n+\t\tewah_free(stored->bitmap);\n+\n+\tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n+\n+\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n+\tif (hash_ret == 0)\n+\t\tdie(\"Duplicate entry when writing index: %s\",\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer.bitmaps, hash_pos) = stored;\n+}\n+\n+void bitmap_writer_build(struct packing_data *to_pack)\n+{\n+\tstruct bitmap_builder bb;\n+\tsize_t i;\n+\tint nr_stored = 0; /* for progress */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n \n \tif (writer.show_progress)\n \t\twriter.progress = start_progress(\"Building bitmaps\", writer.selected_nr);\n-\n-\trepo_init_revisions(to_pack->repo, &revs, NULL);\n-\trevs.tag_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.blob_objects = 1;\n-\trevs.no_walk = 0;\n-\n-\trevs.include_check = should_include;\n-\treset_revision_walk();\n-\n-\treuse_after = writer.selected_nr * REUSE_BITMAP_THRESHOLD;\n-\tneed_reset = 0;\n-\n-\tfor (i = writer.selected_nr - 1; i >= 0; --i) {\n-\t\tstruct bitmapped_commit *stored;\n-\t\tstruct object *object;\n-\n-\t\tkhiter_t hash_pos;\n-\t\tint hash_ret;\n-\n-\t\tstored = &writer.selected[i];\n-\t\tobject = (struct object *)stored->commit;\n-\n-\t\tif (stored->bitmap == NULL) {\n-\t\t\tif (i < writer.selected_nr - 1 &&\n-\t\t\t (need_reset ||\n-\t\t\t !in_merge_bases(writer.selected[i + 1].commit,\n-\t\t\t\t\t stored->commit))) {\n-\t\t\t bitmap_reset(base);\n-\t\t\t reset_all_seen();\n-\t\t\t}\n-\n-\t\t\tadd_pending_object(&revs, object, \"\");\n-\t\t\trevs.include_check_data = base;\n-\n-\t\t\tif (prepare_revision_walk(&revs))\n-\t\t\t\tdie(\"revision walk setup failed\");\n-\n-\t\t\ttraverse_commit_list(&revs, show_commit, show_object, base);\n-\n-\t\t\tobject_array_clear(&revs.pending);\n-\n-\t\t\tstored->bitmap = bitmap_to_ewah(base);\n-\t\t\tneed_reset = 0;\n-\t\t} else\n-\t\t\tneed_reset = 1;\n-\n-\t\tif (i >= reuse_after)\n-\t\t\tstored->flags |= BITMAP_FLAG_REUSE;\n-\n-\t\thash_pos = kh_put_oid_map(writer.bitmaps, object->oid, &hash_ret);\n-\t\tif (hash_ret == 0)\n-\t\t\tdie(\"Duplicate entry when writing index: %s\",\n-\t\t\t oid_to_hex(&object->oid));\n-\n-\t\tkh_value(writer.bitmaps, hash_pos) = stored;\n-\t\tdisplay_progress(writer.progress, writer.selected_nr - i);\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n+\t\tthe_repository);\n+\n+\tbitmap_builder_init(&bb, &writer);\n+\tfor (i = bb.commits_nr; i > 0; i--) {\n+\t\tstruct commit *commit = bb.commits[i-1];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n+\t\tstruct commit *child;\n+\n+\t\tfill_bitmap_commit(ent, commit);\n+\n+\t\tif (ent->selected) {\n+\t\t\tstore_selected(ent, commit);\n+\t\t\tnr_stored++;\n+\t\t\tdisplay_progress(writer.progress, nr_stored);\n+\t\t}\n+\n+\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\t\tstruct bb_commit *child_ent =\n+\t\t\t\tbb_data_at(&bb.data, child);\n+\n+\t\t\tif (child_ent->bitmap)\n+\t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n+\t\t\telse\n+\t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t}\n+\t\tbitmap_free(ent->bitmap);\n+\t\tent->bitmap = NULL;\n \t}\n+\tbitmap_builder_clear(&bb);\n \n-\tbitmap_free(base);\n \tstop_progress(&writer.progress);\n \n \tcompute_xor_offsets();\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410153","messageId":"466dd3036a8ca7dc9718a53f17cf87e0eb22c066.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:20Z","receivedAt":"2020-11-17T21:47:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nOur algorithm to generate reachability bitmaps walks through the commit\ngraph from the bottom up, passing bitmap data from each commit to its\ndescendants. For a linear stretch of history like:\n\n A -- B -- C\n\nour sequence of steps is:\n\n - compute the bitmap for A by walking its trees, etc\n\n - duplicate A's bitmap as a starting point for B; we can now free A's\n bitmap, since we only needed it as an intermediate result\n\n - OR in any extra objects that B can reach into its bitmap\n\n - duplicate B's bitmap as a starting point for C; likewise, free B's\n bitmap\n\n - OR in objects for C, and so on...\n\nRather than duplicating bitmaps and immediately freeing the original, we\ncan just pass ownership from commit to commit. Note that this doesn't\nalways work:\n\n - the recipient may be a merge which already has an intermediate\n bitmap from its other ancestor. In that case we have to OR our\n result into it. Note that the first ancestor to reach the merge does\n get to pass ownership, though.\n\n - we may have multiple children; we can only pass ownership to one of\n them\n\nHowever, it happens often enough and copying bitmaps is expensive enough\nthat this provides a noticeable speedup. On a clone of linux.git, this\nreduces the time to generate bitmaps from 205s to 70s. This is about the\nsame amount of time it took to generate bitmaps using our old \"many\ntraversals\" algorithm (the previous commit measures the identical\nscenario as taking 63s). It unfortunately provides only a very modest\nreduction in the peak memory usage, though.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f2f0b6b2c2..d2d46ff5f4 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -333,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n \t\tstruct commit *child;\n+\t\tint reused = 0;\n \n \t\tfill_bitmap_commit(ent, commit);\n \n@@ -348,10 +349,15 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \t\t\tif (child_ent->bitmap)\n \t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n-\t\t\telse\n+\t\t\telse if (reused)\n \t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t\telse {\n+\t\t\t\tchild_ent->bitmap = ent->bitmap;\n+\t\t\t\treused = 1;\n+\t\t\t}\n \t\t}\n-\t\tbitmap_free(ent->bitmap);\n+\t\tif (!reused)\n+\t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n \tbitmap_builder_clear(&bb);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410154","messageId":"8e5607929d66a3c808dbe3a06c312d0cda1ef568.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:24Z","receivedAt":"2020-11-17T21:47:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe fill_bitmap_commit() method assumes that every parent of the given\ncommit is already part of the current bitmap. Instead of making that\nassumption, let's walk parents until we reach commits already part of\nthe bitmap. Set the value for that parent immediately after querying to\nsave time doing double calls to find_object_pos() and to avoid inserting\nthe parent into the queue multiple times.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 30 +++++++++++++++++++++++-------\n 1 file changed, 23 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d2d46ff5f4..361f3305a2 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -12,6 +12,7 @@\n #include \"sha1-lookup.h\"\n #include \"pack-objects.h\"\n #include \"commit-reach.h\"\n+#include \"prio-queue.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -279,17 +280,30 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n }\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit)\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\t/*\n-\t * mark ourselves, but do not bother with parents; their values\n-\t * will already have been propagated to us\n-\t */\n \tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n-\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+\tprio_queue_put(queue, commit);\n+\n+\twhile (queue->nr) {\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = prio_queue_get(queue);\n+\n+\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n+\t\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\t\tprio_queue_put(queue, p->item);\n+\t\t\t}\n+\t\t}\n+\t}\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -319,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct bitmap_builder bb;\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -335,7 +350,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit);\n+\t\tfill_bitmap_commit(ent, commit, &queue);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -360,6 +375,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n+\tclear_prio_queue(&queue);\n \tbitmap_builder_clear(&bb);\n \n \tstop_progress(&writer.progress);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410155","messageId":"4840c64c51d65ea7bf1ebe03cad4588267db0207.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 13/24] bitmap: add bitmap_diff_nonzero()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:29Z","receivedAt":"2020-11-17T21:47:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_diff_nonzero() checks if the 'self' bitmap contains any bits\nthat are not on in the 'other' bitmap.\n\nAlso, delete the declaration of bitmap_is_subset() as it is not used or\nimplemented.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 24 ++++++++++++++++++++++++\n ewah/ewok.h | 2 +-\n 2 files changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex eb7e2539be..e2ebeac0e5 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -200,6 +200,30 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other)\n+{\n+\tstruct bitmap *small;\n+\tsize_t i;\n+\n+\tif (self->word_alloc < other->word_alloc) {\n+\t\tsmall = self;\n+\t} else {\n+\t\tsmall = other;\n+\n+\t\tfor (i = other->word_alloc; i < self->word_alloc; i++) {\n+\t\t\tif (self->words[i] != 0)\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < small->word_alloc; i++) {\n+\t\tif ((self->words[i] & ~other->words[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void bitmap_reset(struct bitmap *bitmap)\n {\n \tmemset(bitmap->words, 0x0, bitmap->word_alloc * sizeof(eword_t));\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 1fc555e672..156c71d06d 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -180,7 +180,7 @@ int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_reset(struct bitmap *self);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n-int bitmap_is_subset(struct bitmap *self, struct bitmap *super);\n+int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410156","messageId":"63e846f4e8841edeecccb46efa1645293f7452a8.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 14/24] commit: implement commit_list_contains()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:34Z","receivedAt":"2020-11-17T21:47:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to check if a commit_list contains a commit. Use\npointer equality, assuming lookup_commit() was used.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit.c | 11 +++++++++++\n commit.h | 2 ++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/commit.c b/commit.c\nindex fe1fa3dc41..9a785bf906 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -544,6 +544,17 @@ struct commit_list *commit_list_insert(struct commit *item, struct commit_list *\n \treturn new_list;\n }\n \n+int commit_list_contains(struct commit *item, struct commit_list *list)\n+{\n+\twhile (list) {\n+\t\tif (list->item == item)\n+\t\t\treturn 1;\n+\t\tlist = list->next;\n+\t}\n+\n+\treturn 0;\n+}\n+\n unsigned commit_list_count(const struct commit_list *l)\n {\n \tunsigned c = 0;\ndiff --git a/commit.h b/commit.h\nindex 5467786c7b..742a6de460 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -167,6 +167,8 @@ int find_commit_subject(const char *commit_buffer, const char **subject);\n \n struct commit_list *commit_list_insert(struct commit *item,\n \t\t\t\t\tstruct commit_list **list);\n+int commit_list_contains(struct commit *item,\n+\t\t\t struct commit_list *list);\n struct commit_list **commit_list_append(struct commit *commit,\n \t\t\t\t\tstruct commit_list **next);\n unsigned commit_list_count(const struct commit_list *l);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410157","messageId":"8b5d23933301988e42ddd57687e0535f8749367f.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 15/24] t5310: add branch-based checks","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:41Z","receivedAt":"2020-11-17T21:47:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe current rev-list tests that check the bitmap data only work on HEAD\ninstead of multiple branches. Expand the test cases to handle both\n'master' and 'other' branches.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 61 +++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 70a4fc4843..6bf68fee85 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -41,63 +41,70 @@ test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n \tgit rev-list --test-bitmap HEAD\n '\n \n-rev_list_tests() {\n-\tstate=$1\n-\n-\ttest_expect_success \"counting commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD >actual &&\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting partial commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD~5..HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD~5..HEAD >actual &&\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limit ($state)\" '\n-\t\tgit rev-list --count -n 1 HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 HEAD >actual &&\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting non-linear history ($state)\" '\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n \t\tgit rev-list --count other...master >expect &&\n \t\tgit rev-list --use-bitmap-index --count other...master >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limiting ($state)\" '\n-\t\tgit rev-list --count HEAD -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD -- 1.t >actual &&\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting objects via bitmap ($state)\" '\n-\t\tgit rev-list --count --objects HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects HEAD >actual &&\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"enumerate commits ($state)\" '\n-\t\tgit rev-list --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list HEAD >expect &&\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n \t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n \t'\n \n-\ttest_expect_success \"enumerate --objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list --objects HEAD >expect &&\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n \t\ttest_bitmap_traversal expect actual\n \t'\n \n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD tagged-blob >actual &&\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n \t\tgrep $blob actual\n \t'\n }\n \n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"master\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n rev_list_tests 'full bitmap'\n \n test_expect_success 'clone from bitmapped repository' '\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410158","messageId":"60a46091bb4661946985f66ec10062739770696a.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 16/24] pack-bitmap-write: rename children to reverse_edges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:47Z","receivedAt":"2020-11-17T21:47:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_builder_init() method walks the reachable commits in\ntopological order and constructs a \"reverse graph\" along the way. At the\nmoment, this reverse graph contains an edge from commit A to commit B if\nand only if A is a parent of B. Thus, the name \"children\" is appropriate\nfor for this reverse graph.\n\nIn the next change, we will repurpose the reverse graph to not be\ndirectly-adjacent commits in the commit-graph, but instead a more\nabstract relationship. The previous changes have already incorporated\nthe necessary updates to fill_bitmap_commit() that allow these edges to\nnot be immediate children.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 361f3305a2..369c76a87c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -179,7 +179,7 @@ static void compute_xor_offsets(void)\n }\n \n struct bb_commit {\n-\tstruct commit_list *children;\n+\tstruct commit_list *reverse_edges;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1;\n \tunsigned idx; /* within selected array */\n@@ -228,7 +228,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n \t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n \t\t}\n \t}\n }\n@@ -358,7 +358,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tdisplay_progress(writer.progress, nr_stored);\n \t\t}\n \n-\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\twhile ((child = pop_commit(&ent->reverse_edges))) {\n \t\t\tstruct bb_commit *child_ent =\n \t\t\t\tbb_data_at(&bb.data, child);\n \n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410159","messageId":"8f7bb2dd2e192562395bb815d891ec2ad28e6644.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 17/24] pack-bitmap.c: check reads more aggressively when loading","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:47:54Z","receivedAt":"2020-11-17T21:47:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Before 'load_bitmap_entries_v1()' reads an actual EWAH bitmap, it should\ncheck that it can safely do so by ensuring that there are at least 6\nbytes available to be read (four for the commit's index position, and\nthen two more for the xor offset and flags, respectively).\n\nLikewise, it should check that the commit index it read refers to a\nlegitimate object in the pack.\n\nThe first fix catches a truncation bug that was exposed when testing,\nand the second is purely precautionary.\n\nThere are some possible future improvements, not pursued here. They are:\n\n - Computing the correct boundary of the bitmap itself in the caller\n and ensuring that we don't read past it. This may or may not be\n worth it, since in a truncation situation, all bets are off: (is the\n trailer still there and the bitmap entries malformed, or is the\n trailer truncated?). The best we can do is try to read what's there\n as if it's correct data (and protect ourselves when it's obviously\n bogus).\n\n - Avoid the magic \"6\" by teaching read_be32() and read_u8() (both of\n which are custom helpers for this function) to check sizes before\n advancing the pointers.\n\n - Adding more tests in this area. Testing these truncation situations\n are remarkably fragile to even subtle changes in the bitmap\n generation. So, the resulting tests are likely to be quite brittle.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4431f9f120..60c781d100 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -229,11 +229,16 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\tuint32_t commit_idx_pos;\n \t\tstruct object_id oid;\n \n+\t\tif (index->map_size - index->map_pos < 6)\n+\t\t\treturn error(\"corrupt ewah bitmap: truncated header for entry %d\", i);\n+\n \t\tcommit_idx_pos = read_be32(index->map, &index->map_pos);\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tnth_packed_object_id(&oid, index->pack, commit_idx_pos);\n+\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n+\t\t\t\t (unsigned)commit_idx_pos);\n \n \t\tbitmap = read_bitmap_1(index);\n \t\tif (!bitmap)\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410160","messageId":"5262daa3300114fbaccdbc7393882c5435f95f4f.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 18/24] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:01Z","receivedAt":"2020-11-17T21:48:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_writer_build() method calls bitmap_builder_init() to\nconstruct a list of commits reachable from the selected commits along\nwith a \"reverse graph\". This reverse graph has edges pointing from a\ncommit to other commits that can reach that commit. After computing a\nreachability bitmap for a commit, the values in that bitmap are then\ncopied to the reachability bitmaps across the edges in the reverse\ngraph.\n\nWe can now relax the role of the reverse graph to greatly reduce the\nnumber of intermediate reachability bitmaps we compute during this\nreverse walk. The end result is that we walk objects the same number of\ntimes as before when constructing the reachability bitmaps, but we also\nspend much less time copying bits between bitmaps and have much lower\nmemory pressure in the process.\n\nThe core idea is to select a set of \"important\" commits based on\ninteractions among the sets of commits reachable from each selected commit.\n\nThe first technical concept is to create a new 'commit_mask' member in the\nbb_commit struct. Note that the selected commits are provided in an\nordered array. The first thing to do is to mark the ith bit in the\ncommit_mask for the ith selected commit. As we walk the commit-graph, we\ncopy the bits in a commit's commit_mask to its parents. At the end of\nthe walk, the ith bit in the commit_mask for a commit C stores a boolean\nrepresenting \"The ith selected commit can reach C.\"\n\nAs we walk, we will discover non-selected commits that are important. We\nwill get into this later, but those important commits must also receive\nbit positions, growing the width of the bitmasks as we walk. At the true\nend of the walk, the ith bit means \"the ith _important_ commit can reach\nC.\"\n\nMAXIMAL COMMITS\n---------------\n\nWe use a new 'maximal' bit in the bb_commit struct to represent whether\na commit is important or not. The term \"maximal\" comes from the\npartially-ordered set of commits in the commit-graph where C >= P if P\nis a parent of C, and then extending the relationship transitively.\nInstead of taking the maximal commits across the entire commit-graph, we\ninstead focus on selecting each commit that is maximal among commits\nwith the same bits on in their commit_mask. This definition is\nimportant, so let's consider an example.\n\nSuppose we have three selected commits A, B, and C. These are assigned\nbitmasks 100, 010, and 001 to start. Each of these can be marked as\nmaximal immediately because they each will be the uniquely maximal\ncommit that contains their own bit. Keep in mind that that these commits\nmay have different bitmasks after the walk; for example, if B can reach\nC but A cannot, then the final bitmask for C is 011. Even in these\ncases, C would still be a maximal commit among all commits with the\nthird bit on in their masks.\n\nNow define sets X, Y, and Z to be the sets of commits reachable from A,\nB, and C, respectively. The intersections of these sets correspond to\ndifferent bitmasks:\n\n * 100: X - (Y union Z)\n * 010: Y - (X union Z)\n * 001: Z - (X union Y)\n * 110: (X intersect Y) - Z\n * 101: (X intersect Z) - Y\n * 011: (Y intersect Z) - X\n * 111: X intersect Y intersect Z\n\nThis can be visualized with the following Hasse diagram:\n\n\t100 010 001\n | \\ / \\ / |\n | \\/ \\/ |\n | /\\ /\\ |\n | / \\ / \\ |\n 110 101 011\n \\___ | ___/\n \\ | /\n 111\n\nSome of these bitmasks may not be represented, depending on the topology\nof the commit-graph. In fact, we are counting on it, since the number of\npossible bitmasks is exponential in the number of selected commits, but\nis also limited by the total number of commits. In practice, very few\nbitmasks are possible because most commits converge on a common \"trunk\"\nin the commit history.\n\nWith this three-bit example, we wish to find commits that are maximal\nfor each bitmask. How can we identify this as we are walking?\n\nAs we walk, we visit a commit C. Since we are walking the commits in\ntopo-order, we know that C is visited after all of its children are\nvisited. Thus, when we get C from the revision walk we inspect the\n'maximal' property of its bb_data and use that to determine if C is truly\nimportant. Its commit_mask is also nearly final. If C is not one of the\noriginally-selected commits, then assign a bit position to C (by\nincrementing num_maximal) and set that bit on in commit_mask. See\n\"MULTIPLE MAXIMAL COMMITS\" below for more detail on this.\n\nNow that the commit C is known to be maximal or not, consider each\nparent P of C. Compute two new values:\n\n * c_not_p : true if and only if the commit_mask for C contains a bit\n that is not contained in the commit_mask for P.\n\n * p_not_c : true if and only if the commit_mask for P contains a bit\n that is not contained in the commit_mask for P.\n\nIf c_not_p is false, then P already has all of the bits that C would\nprovide to its commit_mask. In this case, move on to other parents as C\nhas nothing to contribute to P's state that was not already provided by\nother children of P.\n\nWe continue with the case that c_not_p is true. This means there are\nbits in C's commit_mask to copy to P's commit_mask, so use bitmap_or()\nto add those bits.\n\nIf p_not_c is also true, then set the maximal bit for P to one. This means\nthat if no other commit has P as a parent, then P is definitely maximal.\nThis is because no child had the same bitmask. It is important to think\nabout the maximal bit for P at this point as a temporary state: \"P is\nmaximal based on current information.\"\n\nIn contrast, if p_not_c is false, then set the maximal bit for P to\nzero. Further, clear all reverse_edges for P since any edges that were\npreviously assigned to P are no longer important. P will gain all\nreverse edges based on C.\n\nThe final thing we need to do is to update the reverse edges for P.\nThese reverse edges respresent \"which closest maximal commits\ncontributed bits to my commit_mask?\" Since C contributed bits to P's\ncommit_mask in this case, C must add to the reverse edges of P.\n\nIf C is maximal, then C is a 'closest' maximal commit that contributed\nbits to P. Add C to P's reverse_edges list.\n\nOtherwise, C has a list of maximal commits that contributed bits to its\nbitmask (and this list is exactly one element). Add all of these items\nto P's reverse_edges list. Be careful to ignore duplicates here.\n\nAfter inspecting all parents P for a commit C, we can clear the\ncommit_mask for C. This reduces the memory load to be limited to the\n\"width\" of the commit graph.\n\nConsider our ABC/XYZ example from earlier and let's inspect the state of\nthe commits for an interesting bitmask, say 011. Suppose that D is the\nonly maximal commit with this bitmask (in the first three bits). All\nother commits with bitmask 011 have D as the only entry in their\nreverse_edges list. D's reverse_edges list contains B and C.\n\nCOMPUTING REACHABILITY BITMAPS\n------------------------------\n\nNow that we have our definition, let's zoom out and consider what\nhappens with our new reverse graph when computing reachability bitmaps.\nWe walk the reverse graph in reverse-topo-order, so we visit commits\nwith largest commit_masks first. After we compute the reachability\nbitmap for a commit C, we push the bits in that bitmap to each commit D\nin the reverse edge list for C. Then, when we finally visit D we already\nhave the bits for everything reachable from maximal commits that D can\nreach and we only need to walk the objects in the set-difference.\n\nIn our ABC/XYZ example, when we finally walk for the commit A we only\nneed to walk commits with bitmask equal to A's bitmask. If that bitmask\nis 100, then we are only walking commits in X - (Y union Z) because the\nbitmap already contains the bits for objects reachable from (X intersect\nY) union (X intersect Z) (i.e. the bits from the reachability bitmaps\nfor the maximal commits with bitmasks 110 and 101).\n\nThe behavior is intended to walk each commit (and the trees that commit\nintroduces) at most once while allocating and copying fewer reachability\nbitmaps. There is one caveat: what happens when there are multiple\nmaximal commits with the same bitmask, with respect to the initial set\nof selected commits?\n\nMULTIPLE MAXIMAL COMMITS\n------------------------\n\nEarlier, we mentioned that when we discover a new maximal commit, we\nassign a new bit position to that commit and set that bit position to\none for that commit. This is absolutely important for interesting\ncommit-graphs such as git/git and torvalds/linux. The reason is due to\nthe existence of \"butterflies\" in the commit-graph partial order.\n\nHere is an example of four commits forming a butterfly:\n\n I J\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N\n \\ /\n |/\n Q\n\nHere, I and J both have parents M and N. In general, these do not need\nto be exact parent relationships, but reachability relationships. The\nmost important part is that M and N cannot reach each other, so they are\nindependent in the partial order. If I had commit_mask 10 and J had\ncommit_mask 01, then M and N would both be assigned commit_mask 11 and\nbe maximal commits with the bitmask 11. Then, what happens when M and N\ncan both reach a commit Q? If Q is also assigned the bitmask 11, then it\nis not maximal but is reachable from both M and N.\n\nWhile this is not necessarily a deal-breaker for our abstract definition\nof finding maximal commits according to a given bitmask, we have a few\nissues that can come up in our larger picture of constructing\nreachability bitmaps.\n\nIn particular, if we do not also consider Q to be a \"maximal\" commit,\nthen we will walk commits reachable from Q twice: once when computing\nthe reachability bitmap for M and another time when computing the\nreachability bitmap for N. This becomes much worse if the topology\ncontinues this pattern with multiple butterflies.\n\nThe solution has already been mentioned: each of M and N are assigned\ntheir own bits to the bitmask and hence they become uniquely maximal for\ntheir bitmasks. Finally, Q also becomes maximal and thus we do not need\nto walk its commits multiple times. The final bitmasks for these commits\nare as follows:\n\n I:10 J:01\n |\\ /|\n | \\ _____/ |\n | /\\____ |\n |/ \\ |\n M:111 N:1101\n \\ /\n Q:1111\n\nFurther, Q's reverse edge list is { M, N }, while M and N both have\nreverse edge list { I, J }.\n\nPERFORMANCE MEASUREMENTS\n------------------------\n\nNow that we've spent a LOT of time on the theory of this algorithm,\nlet's show that this is actually worth all that effort.\n\nTo test the performance, use GIT_TRACE2_PERF=1 when running\n'git repack -abd' in a repository with no existing reachability bitmaps.\nThis avoids any issues with keeping existing bitmaps to skew the\nnumbers.\n\nInspect the \"building_bitmaps_total\" region in the trace2 output to\nfocus on the portion of work that is affected by this change. Here are\nthe performance comparisons for a few repositories. The timings are for\nthe following versions of Git: \"multi\" is the timing from before any\nreverse graph is constructed, where we might perform multiple\ntraversals. \"reverse\" is for the previous change where the reverse graph\nhas every reachable commit. Finally \"maximal\" is the version introduced\nhere where the reverse graph only contains the maximal commits.\n\n Repository: git/git\n multi: 2.628 sec\n reverse: 2.344 sec\n maximal: 2.047 sec\n\n Repository: torvalds/linux\n multi: 64.7 sec\n reverse: 205.3 sec\n maximal: 44.7 sec\n\nSo in all cases we've not only recovered any time lost to switching to\nthe reverse-edge algorithm, but we come out ahead of \"multi\" in all\ncases. Likewise, peak heap has gone back to something reasonable:\n\n Repository: torvalds/linux\n multi: 2.087 GB\n reverse: 3.141 GB\n maximal: 2.288 GB\n\nWhile I do not have access to full fork networks on GitHub, Peff has run\nthis algorithm on the chromium/chromium fork network and reported a\nchange from 3 hours to ~233 seconds. That network is particularly\nbeneficial for this approach because it has a long, linear history along\nwith many tags. The \"multi\" approach was obviously quadratic and the new\napproach is linear.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++---\n t/t5310-pack-bitmaps.sh | 87 +++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 149 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 369c76a87c..7b4fc0f304 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -180,8 +180,10 @@ static void compute_xor_offsets(void)\n \n struct bb_commit {\n \tstruct commit_list *reverse_edges;\n+\tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n-\tunsigned selected:1;\n+\tunsigned selected:1,\n+\t\t maximal:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -198,7 +200,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i;\n+\tunsigned int i, num_maximal;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -210,27 +212,85 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n \t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\n \t\tent->selected = 1;\n+\t\tent->maximal = 1;\n \t\tent->idx = i;\n+\n+\t\tent->commit_mask = bitmap_new();\n+\t\tbitmap_set(ent->commit_mask, i);\n+\n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n+\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n \t\tstruct commit_list *p;\n+\t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n \n-\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n-\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\tc_ent = bb_data_at(&bb->data, commit);\n+\n+\t\tif (c_ent->maximal) {\n+\t\t\tif (!c_ent->selected) {\n+\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n+\t\t\t\tnum_maximal++;\n+\t\t\t}\n+\n+\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\t}\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n-\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n+\t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n+\t\t\tint c_not_p, p_not_c;\n+\n+\t\t\tif (!p_ent->commit_mask) {\n+\t\t\t\tp_ent->commit_mask = bitmap_new();\n+\t\t\t\tc_not_p = 1;\n+\t\t\t\tp_not_c = 0;\n+\t\t\t} else {\n+\t\t\t\tc_not_p = bitmap_diff_nonzero(c_ent->commit_mask, p_ent->commit_mask);\n+\t\t\t\tp_not_c = bitmap_diff_nonzero(p_ent->commit_mask, c_ent->commit_mask);\n+\t\t\t}\n+\n+\t\t\tif (!c_not_p)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_or(p_ent->commit_mask, c_ent->commit_mask);\n+\n+\t\t\tif (p_not_c)\n+\t\t\t\tp_ent->maximal = 1;\n+\t\t\telse {\n+\t\t\t\tp_ent->maximal = 0;\n+\t\t\t\tfree_commit_list(p_ent->reverse_edges);\n+\t\t\t\tp_ent->reverse_edges = NULL;\n+\t\t\t}\n+\n+\t\t\tif (c_ent->maximal) {\n+\t\t\t\tcommit_list_insert(commit, &p_ent->reverse_edges);\n+\t\t\t} else {\n+\t\t\t\tstruct commit_list *cc = c_ent->reverse_edges;\n+\n+\t\t\t\tfor (; cc; cc = cc->next) {\n+\t\t\t\t\tif (!commit_list_contains(cc->item, p_ent->reverse_edges))\n+\t\t\t\t\t\tcommit_list_insert(cc->item, &p_ent->reverse_edges);\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n+\n+\t\tbitmap_free(c_ent->commit_mask);\n+\t\tc_ent->commit_mask = NULL;\n \t}\n+\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_selected_commits\", writer->selected_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_maximal_commits\", num_maximal);\n }\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 6bf68fee85..1691710ec1 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -20,11 +20,87 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other master\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-master * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\____________...____________ |\n+# \\|\n+# * (base)\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for master (bit 0) and other (bit 1), and some flexibility\n+# in the order that merge bases are visited, the bitmasks at\n+# the end should be:\n+#\n+# master: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# octo-master: 1\n+# octo-other: 01\n+# merge-right: 111 (maximal)\n+# (l1): 111\n+# (r1): 111\n+# merge-left: 1101 (maximal)\n+# (l2): 11111 (maximal)\n+# (r2): 111101 (maximal)\n+# (base): 1111111 (maximal)\n+\n test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 100 &&\n+\ttest_commit_bulk --id=file 10 &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n+\n+\t# add complicated history setup, including merges and\n+\t# ambiguous merge-bases\n+\n+\tgit checkout -b merge-left other~2 &&\n+\tgit merge master~2 -m \"merge-left\" &&\n+\n+\tgit checkout -b merge-right master~1 &&\n+\tgit merge other~1 -m \"merge-right\" &&\n+\n+\tgit checkout -b octo-master master &&\n+\tgit merge merge-left merge-right -m \"octopus-master\" &&\n+\n+\tgit checkout -b octo-other other &&\n+\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\tgit checkout other &&\n+\tgit merge octo-other -m \"pull octopus\" &&\n+\n \tgit checkout master &&\n+\tgit merge octo-master -m \"pull octopus\" &&\n+\n+\t# Remove these branches so they are not selected\n+\t# as bitmap tips\n+\tgit branch -D merge-left &&\n+\tgit branch -D merge-right &&\n+\tgit branch -D octo-other &&\n+\tgit branch -D octo-master &&\n+\n+\t# add padding to make these merges less interesting\n+\t# and avoid having them selected for bitmaps\n+\ttest_commit_bulk --id=file 100 &&\n+\tgit checkout other &&\n+\ttest_commit_bulk --id=side 100 &&\n+\tgit checkout master &&\n+\n \tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n@@ -32,9 +108,12 @@ test_expect_success 'setup repo with moderate-sized history' '\n '\n \n test_expect_success 'full repack creates bitmaps' '\n-\tgit repack -ad &&\n+\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n+\ttest_line_count = 1 output &&\n+\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n@@ -356,7 +435,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410161","messageId":"a206f486140097c1554ce92ebcfa7190554dd19f.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:07Z","receivedAt":"2020-11-17T21:48:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe on-disk bitmap format has a flag to mark a bitmap to be \"reused\".\nThis is a rather curious feature, and works like this:\n\n - a run of pack-objects would decide to mark the last 80% of the\n bitmaps it generates with the reuse flag\n\n - the next time we generate bitmaps, we'd see those reuse flags from\n the last run, and mark those commits as special:\n\n - we'd be more likely to select those commits to get bitmaps in\n the new output\n\n - when generating the bitmap for a selected commit, we'd reuse the\n old bitmap as-is (rearranging the bits to match the new pack, of\n course)\n\nHowever, neither of these behaviors particularly makes sense.\n\nJust because a commit happened to be bitmapped last time does not make\nit a good candidate for having a bitmap this time. In particular, we may\nchoose bitmaps based on how recent they are in history, or whether a ref\ntip points to them, and those things will change. We're better off\nre-considering fresh which commits are good candidates.\n\nReusing the existing bitmap _is_ a reasonable thing to do to save\ncomputation. But only reusing exact bitmaps is a weak form of this. If\nwe have an old bitmap for A and now want a new bitmap for its child, we\nshould be able to compute that only by looking at trees and that are new\nto the child. But this code would consider only exact reuse (which is\nperhaps why it was eager to select those commits in the first place).\n\nFurthermore, the recent switch to the reverse-edge algorithm for\ngenerating bitmaps dropped this optimization entirely (and yet still\nperforms better).\n\nSo let's do a few cleanups:\n\n - drop the whole \"reusing bitmaps\" phase of generating bitmaps. It's\n not helping anything, and is mostly unused code (or worse, code that\n is using CPU but not doing anything useful)\n\n - drop the use of the on-disk reuse flag to select commits to bitmap\n\n - stop setting the on-disk reuse flag in bitmaps we generate (since\n nothing respects it anymore)\n\nWe will keep a few innards of the reuse code, which will help us\nimplement a more capable version of the \"reuse\" optimization:\n\n - simplify rebuild_existing_bitmaps() into a function that only builds\n the mapping of bits between the old and new orders, but doesn't\n actually convert any bitmaps\n\n - make rebuild_bitmap() public; we'll call it lazily to convert bitmaps\n as we traverse (using the mapping created above)\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 -\n pack-bitmap-write.c | 50 +++++-------------------------------------\n pack-bitmap.c | 46 +++++---------------------------------\n pack-bitmap.h | 6 ++++-\n 4 files changed, 16 insertions(+), 87 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5617c01b5a..2a00358f34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1104,7 +1104,6 @@ static void write_pack_file(void)\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n-\t\t\t\tbitmap_writer_reuse_bitmaps(&to_pack);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 7b4fc0f304..1995f75818 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -30,7 +30,6 @@ struct bitmap_writer {\n \tstruct ewah_bitmap *tags;\n \n \tkh_oid_map_t *bitmaps;\n-\tkh_oid_map_t *reused;\n \tstruct packing_data *to_pack;\n \n \tstruct bitmapped_commit *selected;\n@@ -112,7 +111,7 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n+static inline void push_bitmapped_commit(struct commit *commit)\n {\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n@@ -120,7 +119,7 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \t}\n \n \twriter.selected[writer.selected_nr].commit = commit;\n-\twriter.selected[writer.selected_nr].bitmap = reused;\n+\twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n \n \twriter.selected_nr++;\n@@ -372,13 +371,6 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkhiter_t hash_pos;\n \tint hash_ret;\n \n-\t/*\n-\t * the \"reuse bitmaps\" phase may have stored something here, but\n-\t * our new algorithm doesn't use it. Drop it.\n-\t */\n-\tif (stored->bitmap)\n-\t\tewah_free(stored->bitmap);\n-\n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n \thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n@@ -477,35 +469,6 @@ static int date_compare(const void *_a, const void *_b)\n \treturn (long)b->date - (long)a->date;\n }\n \n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack)\n-{\n-\tstruct bitmap_index *bitmap_git;\n-\tif (!(bitmap_git = prepare_bitmap_git(to_pack->repo)))\n-\t\treturn;\n-\n-\twriter.reused = kh_init_oid_map();\n-\trebuild_existing_bitmaps(bitmap_git, to_pack, writer.reused,\n-\t\t\t\t writer.show_progress);\n-\t/*\n-\t * NEEDSWORK: rebuild_existing_bitmaps() makes writer.reused reference\n-\t * some bitmaps in bitmap_git, so we can't free the latter.\n-\t */\n-}\n-\n-static struct ewah_bitmap *find_reused_bitmap(const struct object_id *oid)\n-{\n-\tkhiter_t hash_pos;\n-\n-\tif (!writer.reused)\n-\t\treturn NULL;\n-\n-\thash_pos = kh_get_oid_map(writer.reused, *oid);\n-\tif (hash_pos >= kh_end(writer.reused))\n-\t\treturn NULL;\n-\n-\treturn kh_value(writer.reused, hash_pos);\n-}\n-\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr,\n \t\t\t\t int max_bitmaps)\n@@ -519,12 +482,11 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(indexed_commits[i], NULL);\n+\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n \tfor (;;) {\n-\t\tstruct ewah_bitmap *reused_bitmap = NULL;\n \t\tstruct commit *chosen = NULL;\n \n \t\tnext = next_commit_index(i);\n@@ -539,15 +501,13 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \t\tif (next == 0) {\n \t\t\tchosen = indexed_commits[i];\n-\t\t\treused_bitmap = find_reused_bitmap(&chosen->object.oid);\n \t\t} else {\n \t\t\tchosen = indexed_commits[i + next];\n \n \t\t\tfor (j = 0; j <= next; ++j) {\n \t\t\t\tstruct commit *cm = indexed_commits[i + j];\n \n-\t\t\t\treused_bitmap = find_reused_bitmap(&cm->object.oid);\n-\t\t\t\tif (reused_bitmap || (cm->object.flags & NEEDS_BITMAP) != 0) {\n+\t\t\t\tif ((cm->object.flags & NEEDS_BITMAP) != 0) {\n \t\t\t\t\tchosen = cm;\n \t\t\t\t\tbreak;\n \t\t\t\t}\n@@ -557,7 +517,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(chosen, reused_bitmap);\n+\t\tpush_bitmapped_commit(chosen);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer.progress, i);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 60c781d100..d1368b69bb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1338,9 +1338,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \tfree_bitmap_index(bitmap_git);\n }\n \n-static int rebuild_bitmap(uint32_t *reposition,\n-\t\t\t struct ewah_bitmap *source,\n-\t\t\t struct bitmap *dest)\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest)\n {\n \tuint32_t pos = 0;\n \tstruct ewah_iterator it;\n@@ -1369,19 +1369,11 @@ static int rebuild_bitmap(uint32_t *reposition,\n \treturn 0;\n }\n \n-int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n-\t\t\t struct packing_data *mapping,\n-\t\t\t kh_oid_map_t *reused_bitmaps,\n-\t\t\t int show_progress)\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping)\n {\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n-\tstruct bitmap *rebuild;\n-\tstruct stored_bitmap *stored;\n-\tstruct progress *progress = NULL;\n-\n-\tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tnum_objects = bitmap_git->pack->num_objects;\n \treposition = xcalloc(num_objects, sizeof(uint32_t));\n@@ -1399,33 +1391,7 @@ int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n \t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n-\trebuild = bitmap_new();\n-\ti = 0;\n-\n-\tif (show_progress)\n-\t\tprogress = start_progress(\"Reusing bitmaps\", 0);\n-\n-\tkh_foreach_value(bitmap_git->bitmaps, stored, {\n-\t\tif (stored->flags & BITMAP_FLAG_REUSE) {\n-\t\t\tif (!rebuild_bitmap(reposition,\n-\t\t\t\t\t lookup_stored_bitmap(stored),\n-\t\t\t\t\t rebuild)) {\n-\t\t\t\thash_pos = kh_put_oid_map(reused_bitmaps,\n-\t\t\t\t\t\t\t stored->oid,\n-\t\t\t\t\t\t\t &hash_ret);\n-\t\t\t\tkh_value(reused_bitmaps, hash_pos) =\n-\t\t\t\t\tbitmap_to_ewah(rebuild);\n-\t\t\t}\n-\t\t\tbitmap_reset(rebuild);\n-\t\t\tdisplay_progress(progress, ++i);\n-\t\t}\n-\t});\n-\n-\tstop_progress(&progress);\n-\n-\tfree(reposition);\n-\tbitmap_free(rebuild);\n-\treturn 0;\n+\treturn reposition;\n }\n \n void free_bitmap_index(struct bitmap_index *b)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 1203120c43..afa4115136 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,7 +73,11 @@ void bitmap_writer_set_checksum(unsigned char *sha1);\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr);\n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping);\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410162","messageId":"9928b3c7da33edd8d4beae006a74dd682daf5fa5.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 20/24] pack-bitmap: factor out 'bitmap_for_commit()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:15Z","receivedAt":"2020-11-17T21:48:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A couple of callers within pack-bitmap.c duplicate logic to lookup a\ngiven object id in the bitamps khash. Factor this out into a new\nfunction, 'bitmap_for_commit()' to reduce some code duplication.\n\nMake this new function non-static, since it will be used in later\ncommits from outside of pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 33 +++++++++++++++++++--------------\n pack-bitmap.h | 2 ++\n 2 files changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d1368b69bb..5efb8af121 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -380,6 +380,16 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit)\n+{\n+\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n+\t\t\t\t\t commit->object.oid);\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n+\t\treturn NULL;\n+\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t const struct object_id *oid)\n {\n@@ -465,10 +475,10 @@ static void show_commit(struct commit *commit, void *data)\n \n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n-\t\t\t const struct object_id *oid,\n+\t\t\t struct commit *commit,\n \t\t\t int bitmap_pos)\n {\n-\tkhiter_t hash_pos;\n+\tstruct ewah_bitmap *partial;\n \n \tif (data->seen && bitmap_get(data->seen, bitmap_pos))\n \t\treturn 0;\n@@ -476,10 +486,9 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \tif (bitmap_get(data->base, bitmap_pos))\n \t\treturn 0;\n \n-\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, *oid);\n-\tif (hash_pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(data->base, lookup_stored_bitmap(st));\n+\tpartial = bitmap_for_commit(bitmap_git, commit);\n+\tif (partial) {\n+\t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n@@ -498,8 +507,7 @@ static int should_include(struct commit *commit, void *_data)\n \t\t\t\t\t\t (struct object *)commit,\n \t\t\t\t\t\t NULL);\n \n-\tif (!add_to_include_set(data->bitmap_git, data, &commit->object.oid,\n-\t\t\t\tbitmap_pos)) {\n+\tif (!add_to_include_set(data->bitmap_git, data, commit, bitmap_pos)) {\n \t\tstruct commit_list *parent = commit->parents;\n \n \t\twhile (parent) {\n@@ -1282,10 +1290,10 @@ void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n-\tkhiter_t pos;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n \tstruct bitmap_index *bitmap_git;\n+\tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n \t\tdie(\"failed to load bitmap indexes\");\n@@ -1297,12 +1305,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tbitmap_git->version, bitmap_git->entry_count);\n \n \troot = revs->pending.objects[0].item;\n-\tpos = kh_get_oid_map(bitmap_git->bitmaps, root->oid);\n-\n-\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\tstruct ewah_bitmap *bm = lookup_stored_bitmap(st);\n+\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n \n+\tif (bm) {\n \t\tfprintf(stderr, \"Found bitmap for %s. %d bits / %08x checksum\\n\",\n \t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex afa4115136..25dfcf5615 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,6 +78,8 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest);\n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410163","messageId":"f40a39a48a834443f76015821c0e56021b58fc9a.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:19Z","receivedAt":"2020-11-17T21:48:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'find_objects()' currently needs to interact with the bitmaps khash\npretty closely. To make 'find_objects()' read a little more\nstraightforwardly, remove some of the khash-level details into a new\nfunction that describes what it does: 'add_commit_to_bitmap()'.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 36 +++++++++++++++++++++---------------\n 1 file changed, 21 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5efb8af121..d88745fb02 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -521,6 +521,23 @@ static int should_include(struct commit *commit, void *_data)\n \treturn 1;\n }\n \n+static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct bitmap **base,\n+\t\t\t\tstruct commit *commit)\n+{\n+\tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n+\n+\tif (!or_with)\n+\t\treturn 0;\n+\n+\tif (*base == NULL)\n+\t\t*base = ewah_to_bitmap(or_with);\n+\telse\n+\t\tbitmap_or_ewah(*base, or_with);\n+\n+\treturn 1;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -544,21 +561,10 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (object->type == OBJ_COMMIT) {\n-\t\t\tkhiter_t pos = kh_get_oid_map(bitmap_git->bitmaps, object->oid);\n-\n-\t\t\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\t\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\t\t\tstruct ewah_bitmap *or_with = lookup_stored_bitmap(st);\n-\n-\t\t\t\tif (base == NULL)\n-\t\t\t\t\tbase = ewah_to_bitmap(or_with);\n-\t\t\t\telse\n-\t\t\t\t\tbitmap_or_ewah(base, or_with);\n-\n-\t\t\t\tobject->flags |= SEEN;\n-\t\t\t\tcontinue;\n-\t\t\t}\n+\t\tif (object->type == OBJ_COMMIT &&\n+\t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n+\t\t\tobject->flags |= SEEN;\n+\t\t\tcontinue;\n \t\t}\n \n \t\tobject_list_insert(object, ¬_mapped);\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410164","messageId":"4bf5e78a54dfdcbe13dd66ba4c5955a159ea181d.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 22/24] pack-bitmap-write: use existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:26Z","receivedAt":"2020-11-17T21:48:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nWhen constructing new bitmaps, we perform a commit and tree walk in\nfill_bitmap_commit() and fill_bitmap_tree(). This walk would benefit\nfrom using existing bitmaps when available. We must track the existing\nbitmaps and translate them into the new object order, but this is\ngenerally faster than parsing trees.\n\nIn fill_bitmap_commit(), we must reorder thing somewhat. The priority\nqueue walks commits from newest-to-oldest, which means we correctly stop\nwalking when reaching a commit with a bitmap. However, if we walk trees\nfrom top to bottom, then we might be parsing trees that are actually\npart of a re-used bitmap. To avoid over-walking trees, add them to a\nLIFO queue and walk them from bottom-to-top after exploring commits\ncompletely.\n\nOn git.git, this reduces a second immediate bitmap computation from 2.0s\nto 1.0s. On linux.git, we go from 32s to 22s. On chromium's fork\nnetwork, we go from 227s to 198s.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 42 ++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 38 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 1995f75818..37204b691c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -340,20 +340,39 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue)\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n \tprio_queue_put(queue, commit);\n \n \twhile (queue->nr) {\n \t\tstruct commit_list *p;\n \t\tstruct commit *c = prio_queue_get(queue);\n \n+\t\t/*\n+\t\t * If this commit has an old bitmap, then translate that\n+\t\t * bitmap and add its bits to this one. No need to walk\n+\t\t * parents or the tree for this commit.\n+\t\t */\n+\t\tif (old_bitmap && mapping) {\n+\t\t\tstruct ewah_bitmap *old;\n+\n+\t\t\told = bitmap_for_commit(old_bitmap, c);\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Mark ourselves and queue our tree. The commit\n+\t\t * walk ensures we cover all parents.\n+\t\t */\n \t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n-\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tint pos = find_object_pos(&p->item->object.oid);\n@@ -363,6 +382,9 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t}\n \t\t}\n \t}\n+\n+\twhile (tree_queue->nr)\n+\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -386,6 +408,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n+\tstruct prio_queue tree_queue = { NULL };\n+\tstruct bitmap_index *old_bitmap;\n+\tuint32_t *mapping;\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -395,6 +420,12 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n \t\tthe_repository);\n \n+\told_bitmap = prepare_bitmap_git(to_pack->repo);\n+\tif (old_bitmap)\n+\t\tmapping = create_bitmap_mapping(old_bitmap, to_pack);\n+\telse\n+\t\tmapping = NULL;\n+\n \tbitmap_builder_init(&bb, &writer);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n@@ -402,7 +433,8 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue);\n+\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -428,7 +460,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tent->bitmap = NULL;\n \t}\n \tclear_prio_queue(&queue);\n+\tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree(mapping);\n \n \tstop_progress(&writer.progress);\n \n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410165","messageId":"1da4fa0fb85fe848aa86987e767b33d296f8f878.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:32Z","receivedAt":"2020-11-17T21:48:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe previous commits improved the bitmap computation process for very\nlong, linear histories with many refs by removing quadratic growth in\nhow many objects were walked. The strategy of computing \"intermediate\ncommits\" using bitmasks for which refs can reach those commits\npartitioned the poset of reachable objects so each part could be walked\nexactly once. This was effective for linear histories.\n\nHowever, there was a (significant) drawback: wide histories with many\nrefs had an explosion of memory costs to compute the commit bitmasks\nduring the exploration that discovers these intermediate commits. Since\nthese wide histories are unlikely to repeat walking objects, the benefit\nof walking objects multiple times was not expensive before. But now, the\ncommit walk *before computing bitmaps* is incredibly expensive.\n\nIn an effort to discover a happy medium, this change reduces the walk\nfor intermediate commits to only the first-parent history. This focuses\nthe walk on how the histories converge, which still has significant\nreduction in repeat object walks. It is still possible to create\nquadratic behavior in this version, but it is probably less likely in\nrealistic data shapes.\n\nHere is some data taken on a fresh clone of the kernel:\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n original | 64.044 | 83.241 | 2.088 | 2.194 |\n last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 14 +++++---------\n t/t5310-pack-bitmaps.sh | 27 ++++++++++++++-------------\n 2 files changed, 19 insertions(+), 22 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 37204b691c..b0493d971d 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -199,7 +199,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i, num_maximal;\n+\tunsigned int i, num_maximal = 0;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -207,6 +207,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \treset_revision_walk();\n \trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n \trevs.topo_order = 1;\n+\trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n@@ -221,13 +222,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n-\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n-\t\tstruct commit_list *p;\n+\t\tstruct commit_list *p = commit->parents;\n \t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n@@ -235,16 +235,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n \t\tif (c_ent->maximal) {\n-\t\t\tif (!c_ent->selected) {\n-\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n-\t\t\t\tnum_maximal++;\n-\t\t\t}\n-\n+\t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n \t\t\tbb->commits[bb->commits_nr++] = commit;\n \t\t}\n \n-\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\tif (p) {\n \t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n \t\t\tint c_not_p, p_not_c;\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 1691710ec1..a83e7a93fb 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -43,23 +43,24 @@ has_any () {\n # \\|\n # * (base)\n #\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n # The important part for the maximal commit algorithm is how\n # the bitmasks are extended. Assuming starting bit positions\n-# for master (bit 0) and other (bit 1), and some flexibility\n-# in the order that merge bases are visited, the bitmasks at\n-# the end should be:\n+# for master (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n #\n # master: 1 (maximal, selected)\n # other: 01 (maximal, selected)\n-# octo-master: 1\n-# octo-other: 01\n-# merge-right: 111 (maximal)\n-# (l1): 111\n-# (r1): 111\n-# merge-left: 1101 (maximal)\n-# (l2): 11111 (maximal)\n-# (r2): 111101 (maximal)\n-# (base): 1111111 (maximal)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n \n test_expect_success 'setup repo with moderate-sized history' '\n \ttest_commit_bulk --id=file 10 &&\n@@ -113,7 +114,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.312.gabc4d358d8\n\n"},{"id":"410166","messageId":"42399a1c2e52e1d055a2d0ad96af2ca4dce6b1a0.1605649533.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"[PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-17T21:48:36Z","receivedAt":"2020-11-17T21:48:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIf the old bitmap file contains a bitmap for a given commit, then that\ncommit does not need help from intermediate commits in its history to\ncompute its final bitmap. Eject that commit from the walk and insert it\nas a maximal commit in the list of commits for computing bitmaps.\n\nThis helps the repeat bitmap computation task, even if the selected\ncommits shift drastically. This helps when a previously-bitmapped commit\nexists in the first-parent history of a newly-selected commit. Since we\nstop the walk at these commits and we use a first-parent walk, it is\nharder to walk \"around\" these bitmapped commits. It's not impossible,\nbut we can greatly reduce the computation time for many selected\ncommits.\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n last patch | 100.641 | 35.560 | 2.152 | 2.224 |\n this patch | 99.720 | 11.696 | 2.152 | 2.217 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex b0493d971d..3ac90ae410 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -195,7 +195,8 @@ struct bitmap_builder {\n };\n \n static void bitmap_builder_init(struct bitmap_builder *bb,\n-\t\t\t\tstruct bitmap_writer *writer)\n+\t\t\t\tstruct bitmap_writer *writer,\n+\t\t\t\tstruct bitmap_index *old_bitmap)\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n@@ -234,12 +235,26 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n+\t\tif (old_bitmap && bitmap_for_commit(old_bitmap, commit)) {\n+\t\t\t/*\n+\t\t\t * This commit has an existing bitmap, so we can\n+\t\t\t * get its bits immediately without an object\n+\t\t\t * walk. There is no need to continue walking\n+\t\t\t * beyond this commit.\n+\t\t\t */\n+\t\t\tc_ent->maximal = 1;\n+\t\t\tp = NULL;\n+\t\t}\n+\n \t\tif (c_ent->maximal) {\n \t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n \t\t\tbb->commits[bb->commits_nr++] = commit;\n \t\t}\n \n+\t\tif (!c_ent->commit_mask)\n+\t\t\tcontinue;\n+\n \t\tif (p) {\n \t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n \t\t\tint c_not_p, p_not_c;\n@@ -422,7 +437,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \telse\n \t\tmapping = NULL;\n \n-\tbitmap_builder_init(&bb, &writer);\n+\tbitmap_builder_init(&bb, &writer, old_bitmap);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n-- \n2.29.2.312.gabc4d358d8\n"},{"id":"410258","messageId":"20201118183225.GB8396@szeder.dev","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-11-18T18:32:25Z","receivedAt":"2020-11-18T18:33:29Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 17, 2020 at 04:46:16PM -0500, Taylor Blau wrote:\n> - Harden the tests so that they pass under sha256-mode (thanks SZEDER,\n> and Peff).\n\nFixing this is good, of course, but...\n\n> 16: 86d77fd085 ! 18: 5262daa330 pack-bitmap-write: build fewer intermediate bitmaps\n> @@ t/t5310-pack-bitmaps.sh: test_expect_success 'setup repo with moderate-sized his\n> '\n> \n> test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n> +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n> + \tgit rev-list --use-bitmap-index --count --all >expect &&\n> + \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n> + \ttest_when_finished \"rm -f $bitmap\" &&\n> +-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n> ++\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n> + \tmv -f $bitmap.tmp $bitmap &&\n> + \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n> + \ttest_cmp expect actual &&\n\nPlease don't simply sneak in such a change without explaining it in\nthe commit message.\n\n"},{"id":"410268","messageId":"X7V7X/qDfN0udr2u@nand.local","threadId":"54622","inReplyTo":"20201118183225.GB8396@szeder.dev","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-18T19:51:59Z","receivedAt":"2020-11-18T19:52:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Nov 18, 2020 at 07:32:25PM +0100, SZEDER Gábor wrote:\n> On Tue, Nov 17, 2020 at 04:46:16PM -0500, Taylor Blau wrote:\n> > - Harden the tests so that they pass under sha256-mode (thanks SZEDER,\n> > and Peff).\n>\n> Fixing this is good, of course, but...\n>\n> > 16: 86d77fd085 ! 18: 5262daa330 pack-bitmap-write: build fewer intermediate bitmaps\n> > @@ t/t5310-pack-bitmaps.sh: test_expect_success 'setup repo with moderate-sized his\n> > '\n> >\n> > test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n> > +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n> > + \tgit rev-list --use-bitmap-index --count --all >expect &&\n> > + \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n> > + \ttest_when_finished \"rm -f $bitmap\" &&\n> > +-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n> > ++\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n> > + \tmv -f $bitmap.tmp $bitmap &&\n> > + \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n> > + \ttest_cmp expect actual &&\n>\n> Please don't simply sneak in such a change without explaining it in\n> the commit message.\n\nAh, I certainly didn't mean to go under the radar, so to speak ;-). From\nmy perspective, the final patch looks like it picked a magic number in\nthe same way was the original version of this patch did, so I didn't\nthink to add any more detail there.\n\nI did try and highlight this a little bit in the patch just before the\none you're commenting on, though:\n\n - Adding more tests in this area. Testing these truncation situations\n are remarkably fragile to even subtle changes in the bitmap\n generation. So, the resulting tests are likely to be quite brittle.\n\nThanks,\nTaylor\n"},{"id":"410406","messageId":"CAN0heSq59uX=4pqkhc904oLfeiwF5ctiEb_9cQXYY7T1t=Mt1g@mail.gmail.com","threadId":"54622","inReplyTo":"cover.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-11-20T06:34:35Z","receivedAt":"2020-11-20T06:35:10Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Tue, 17 Nov 2020 at 22:46, Taylor Blau wrote:\n> Not very much has changed since last time, but a range-diff is below\n> nonetheless. The major changes are:\n>\n> - Avoid an overflow when bounds checking in the second and third\n> patches (thanks, Martin, for noticing).\n\nFWIW, the updates to patches 2 and 3 look exactly like what I was\nexpecting after the discussion on v1. I have nothing to add.\n\n\nMartin\n"},{"id":"410467","messageId":"xmqqy2iusdpy.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"CAN0heSq59uX=4pqkhc904oLfeiwF5ctiEb_9cQXYY7T1t=Mt1g@mail.gmail.com","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-21T19:37:45Z","receivedAt":"2020-11-21T19:38:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n> On Tue, 17 Nov 2020 at 22:46, Taylor Blau wrote:\n>> Not very much has changed since last time, but a range-diff is below\n>> nonetheless. The major changes are:\n>>\n>> - Avoid an overflow when bounds checking in the second and third\n>> patches (thanks, Martin, for noticing).\n>\n> FWIW, the updates to patches 2 and 3 look exactly like what I was\n> expecting after the discussion on v1. I have nothing to add.\n\nThanks, both. Shall we move the topic down to 'next'?\n\n"},{"id":"410469","messageId":"CAN0heSpVnzyE5S5ReKQ0Q_UU48jQ77NVF1x1NTGx29+5KZsyRA@mail.gmail.com","threadId":"54622","inReplyTo":"xmqqy2iusdpy.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-11-21T20:11:21Z","receivedAt":"2020-11-21T20:11:53Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Sat, 21 Nov 2020 at 20:37, Junio C Hamano wrote:\n>\n> Martin Ågren writes:\n>\n> > On Tue, 17 Nov 2020 at 22:46, Taylor Blau wrote:\n> >> Not very much has changed since last time, but a range-diff is below\n> >> nonetheless. The major changes are:\n> >>\n> >> - Avoid an overflow when bounds checking in the second and third\n> >> patches (thanks, Martin, for noticing).\n> >\n> > FWIW, the updates to patches 2 and 3 look exactly like what I was\n> > expecting after the discussion on v1. I have nothing to add.\n>\n> Thanks, both. Shall we move the topic down to 'next'?\n\nI really only dug into those patches 2 and 3. I read the rest of the\npatches of v1 and went \"that makes sense\", but that's about it. I\nstarted looking at \"pack-bitmap-write: build fewer intermediate bitmaps\"\nand went \"this looks really cool -- I should try to understand this\". :-)\n\nThere was SZEDER's comment on that last patch in v2, where future\nreaders of that patch will have to wonder why it does s/256/270/ in a\ntest. I agree with SZEDER that the change should be mentioned in the\ncommit message, even if it's just \"unfortunately, we have some magicness\nhere, plus we want to pass both with SHA-1 and SHA-256; turns out 270\nhits the problem we want to test for\".\n\nMartin\n"},{"id":"410479","messageId":"X7nKRxNy47nTre3b@xnor.local","threadId":"54622","inReplyTo":"X7V7X/qDfN0udr2u@nand.local","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-22T02:17:43Z","receivedAt":"2020-11-22T02:20:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Nov 18, 2020 at 02:51:59PM -0500, Taylor Blau wrote:\n> On Wed, Nov 18, 2020 at 07:32:25PM +0100, SZEDER Gábor wrote:\n> > Please don't simply sneak in such a change without explaining it in\n> > the commit message.\n>\n> Ah, I certainly didn't mean to go under the radar, so to speak ;-). From\n> my perspective, the final patch looks like it picked a magic number in\n> the same way was the original version of this patch did, so I didn't\n> think to add any more detail there.\n\nOops; when I wrote that to you, I had in my mind that this patch already\nchanged that line in the tests, so the rerolled patch was simply\nchanging it to something different.\n\nBut, this isn't a new test from this patch's perspective, so I'll make a\nnote of why this changed in a replacement.\n\nThanks.\n\nTaylor\n"},{"id":"410480","messageId":"X7nMzzMfjm/p9qfj@xnor.local","threadId":"54622","inReplyTo":"X7nKRxNy47nTre3b@xnor.local","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-22T02:28:31Z","receivedAt":"2020-11-22T02:28:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Nov 21, 2020 at 09:17:43PM -0500, Taylor Blau wrote:\n> But, this isn't a new test from this patch's perspective, so I'll make a\n> note of why this changed in a replacement.\n\nHere's that patch: let's use it as a replacement when queueing. Thanks\nagain for noticing.\n\n--- 8< ---\n\nFrom: Derrick Stolee \nSubject: [PATCH] pack-bitmap-write: build fewer intermediate bitmaps\n\nThe bitmap_writer_build() method calls bitmap_builder_init() to\nconstruct a list of commits reachable from the selected commits along\nwith a \"reverse graph\". This reverse graph has edges pointing from a\ncommit to other commits that can reach that commit. After computing a\nreachability bitmap for a commit, the values in that bitmap are then\ncopied to the reachability bitmaps across the edges in the reverse\ngraph.\n\nWe can now relax the role of the reverse graph to greatly reduce the\nnumber of intermediate reachability bitmaps we compute during this\nreverse walk. The end result is that we walk objects the same number of\ntimes as before when constructing the reachability bitmaps, but we also\nspend much less time copying bits between bitmaps and have much lower\nmemory pressure in the process.\n\nThe core idea is to select a set of \"important\" commits based on\ninteractions among the sets of commits reachable from each selected commit.\n\nThe first technical concept is to create a new 'commit_mask' member in the\nbb_commit struct. Note that the selected commits are provided in an\nordered array. The first thing to do is to mark the ith bit in the\ncommit_mask for the ith selected commit. As we walk the commit-graph, we\ncopy the bits in a commit's commit_mask to its parents. At the end of\nthe walk, the ith bit in the commit_mask for a commit C stores a boolean\nrepresenting \"The ith selected commit can reach C.\"\n\nAs we walk, we will discover non-selected commits that are important. We\nwill get into this later, but those important commits must also receive\nbit positions, growing the width of the bitmasks as we walk. At the true\nend of the walk, the ith bit means \"the ith _important_ commit can reach\nC.\"\n\nMAXIMAL COMMITS\n---------------\n\nWe use a new 'maximal' bit in the bb_commit struct to represent whether\na commit is important or not. The term \"maximal\" comes from the\npartially-ordered set of commits in the commit-graph where C >= P if P\nis a parent of C, and then extending the relationship transitively.\nInstead of taking the maximal commits across the entire commit-graph, we\ninstead focus on selecting each commit that is maximal among commits\nwith the same bits on in their commit_mask. This definition is\nimportant, so let's consider an example.\n\nSuppose we have three selected commits A, B, and C. These are assigned\nbitmasks 100, 010, and 001 to start. Each of these can be marked as\nmaximal immediately because they each will be the uniquely maximal\ncommit that contains their own bit. Keep in mind that that these commits\nmay have different bitmasks after the walk; for example, if B can reach\nC but A cannot, then the final bitmask for C is 011. Even in these\ncases, C would still be a maximal commit among all commits with the\nthird bit on in their masks.\n\nNow define sets X, Y, and Z to be the sets of commits reachable from A,\nB, and C, respectively. The intersections of these sets correspond to\ndifferent bitmasks:\n\n * 100: X - (Y union Z)\n * 010: Y - (X union Z)\n * 001: Z - (X union Y)\n * 110: (X intersect Y) - Z\n * 101: (X intersect Z) - Y\n * 011: (Y intersect Z) - X\n * 111: X intersect Y intersect Z\n\nThis can be visualized with the following Hasse diagram:\n\n\t100 010 001\n | \\ / \\ / |\n | \\/ \\/ |\n | /\\ /\\ |\n | / \\ / \\ |\n 110 101 011\n \\___ | ___/\n \\ | /\n 111\n\nSome of these bitmasks may not be represented, depending on the topology\nof the commit-graph. In fact, we are counting on it, since the number of\npossible bitmasks is exponential in the number of selected commits, but\nis also limited by the total number of commits. In practice, very few\nbitmasks are possible because most commits converge on a common \"trunk\"\nin the commit history.\n\nWith this three-bit example, we wish to find commits that are maximal\nfor each bitmask. How can we identify this as we are walking?\n\nAs we walk, we visit a commit C. Since we are walking the commits in\ntopo-order, we know that C is visited after all of its children are\nvisited. Thus, when we get C from the revision walk we inspect the\n'maximal' property of its bb_data and use that to determine if C is truly\nimportant. Its commit_mask is also nearly final. If C is not one of the\noriginally-selected commits, then assign a bit position to C (by\nincrementing num_maximal) and set that bit on in commit_mask. See\n\"MULTIPLE MAXIMAL COMMITS\" below for more detail on this.\n\nNow that the commit C is known to be maximal or not, consider each\nparent P of C. Compute two new values:\n\n * c_not_p : true if and only if the commit_mask for C contains a bit\n that is not contained in the commit_mask for P.\n\n * p_not_c : true if and only if the commit_mask for P contains a bit\n that is not contained in the commit_mask for P.\n\nIf c_not_p is false, then P already has all of the bits that C would\nprovide to its commit_mask. In this case, move on to other parents as C\nhas nothing to contribute to P's state that was not already provided by\nother children of P.\n\nWe continue with the case that c_not_p is true. This means there are\nbits in C's commit_mask to copy to P's commit_mask, so use bitmap_or()\nto add those bits.\n\nIf p_not_c is also true, then set the maximal bit for P to one. This means\nthat if no other commit has P as a parent, then P is definitely maximal.\nThis is because no child had the same bitmask. It is important to think\nabout the maximal bit for P at this point as a temporary state: \"P is\nmaximal based on current information.\"\n\nIn contrast, if p_not_c is false, then set the maximal bit for P to\nzero. Further, clear all reverse_edges for P since any edges that were\npreviously assigned to P are no longer important. P will gain all\nreverse edges based on C.\n\nThe final thing we need to do is to update the reverse edges for P.\nThese reverse edges respresent \"which closest maximal commits\ncontributed bits to my commit_mask?\" Since C contributed bits to P's\ncommit_mask in this case, C must add to the reverse edges of P.\n\nIf C is maximal, then C is a 'closest' maximal commit that contributed\nbits to P. Add C to P's reverse_edges list.\n\nOtherwise, C has a list of maximal commits that contributed bits to its\nbitmask (and this list is exactly one element). Add all of these items\nto P's reverse_edges list. Be careful to ignore duplicates here.\n\nAfter inspecting all parents P for a commit C, we can clear the\ncommit_mask for C. This reduces the memory load to be limited to the\n\"width\" of the commit graph.\n\nConsider our ABC/XYZ example from earlier and let's inspect the state of\nthe commits for an interesting bitmask, say 011. Suppose that D is the\nonly maximal commit with this bitmask (in the first three bits). All\nother commits with bitmask 011 have D as the only entry in their\nreverse_edges list. D's reverse_edges list contains B and C.\n\nCOMPUTING REACHABILITY BITMAPS\n------------------------------\n\nNow that we have our definition, let's zoom out and consider what\nhappens with our new reverse graph when computing reachability bitmaps.\nWe walk the reverse graph in reverse-topo-order, so we visit commits\nwith largest commit_masks first. After we compute the reachability\nbitmap for a commit C, we push the bits in that bitmap to each commit D\nin the reverse edge list for C. Then, when we finally visit D we already\nhave the bits for everything reachable from maximal commits that D can\nreach and we only need to walk the objects in the set-difference.\n\nIn our ABC/XYZ example, when we finally walk for the commit A we only\nneed to walk commits with bitmask equal to A's bitmask. If that bitmask\nis 100, then we are only walking commits in X - (Y union Z) because the\nbitmap already contains the bits for objects reachable from (X intersect\nY) union (X intersect Z) (i.e. the bits from the reachability bitmaps\nfor the maximal commits with bitmasks 110 and 101).\n\nThe behavior is intended to walk each commit (and the trees that commit\nintroduces) at most once while allocating and copying fewer reachability\nbitmaps. There is one caveat: what happens when there are multiple\nmaximal commits with the same bitmask, with respect to the initial set\nof selected commits?\n\nMULTIPLE MAXIMAL COMMITS\n------------------------\n\nEarlier, we mentioned that when we discover a new maximal commit, we\nassign a new bit position to that commit and set that bit position to\none for that commit. This is absolutely important for interesting\ncommit-graphs such as git/git and torvalds/linux. The reason is due to\nthe existence of \"butterflies\" in the commit-graph partial order.\n\nHere is an example of four commits forming a butterfly:\n\n I J\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N\n \\ /\n |/\n Q\n\nHere, I and J both have parents M and N. In general, these do not need\nto be exact parent relationships, but reachability relationships. The\nmost important part is that M and N cannot reach each other, so they are\nindependent in the partial order. If I had commit_mask 10 and J had\ncommit_mask 01, then M and N would both be assigned commit_mask 11 and\nbe maximal commits with the bitmask 11. Then, what happens when M and N\ncan both reach a commit Q? If Q is also assigned the bitmask 11, then it\nis not maximal but is reachable from both M and N.\n\nWhile this is not necessarily a deal-breaker for our abstract definition\nof finding maximal commits according to a given bitmask, we have a few\nissues that can come up in our larger picture of constructing\nreachability bitmaps.\n\nIn particular, if we do not also consider Q to be a \"maximal\" commit,\nthen we will walk commits reachable from Q twice: once when computing\nthe reachability bitmap for M and another time when computing the\nreachability bitmap for N. This becomes much worse if the topology\ncontinues this pattern with multiple butterflies.\n\nThe solution has already been mentioned: each of M and N are assigned\ntheir own bits to the bitmask and hence they become uniquely maximal for\ntheir bitmasks. Finally, Q also becomes maximal and thus we do not need\nto walk its commits multiple times. The final bitmasks for these commits\nare as follows:\n\n I:10 J:01\n |\\ /|\n | \\ _____/ |\n | /\\____ |\n |/ \\ |\n M:111 N:1101\n \\ /\n Q:1111\n\nFurther, Q's reverse edge list is { M, N }, while M and N both have\nreverse edge list { I, J }.\n\nPERFORMANCE MEASUREMENTS\n------------------------\n\nNow that we've spent a LOT of time on the theory of this algorithm,\nlet's show that this is actually worth all that effort.\n\nTo test the performance, use GIT_TRACE2_PERF=1 when running\n'git repack -abd' in a repository with no existing reachability bitmaps.\nThis avoids any issues with keeping existing bitmaps to skew the\nnumbers.\n\nInspect the \"building_bitmaps_total\" region in the trace2 output to\nfocus on the portion of work that is affected by this change. Here are\nthe performance comparisons for a few repositories. The timings are for\nthe following versions of Git: \"multi\" is the timing from before any\nreverse graph is constructed, where we might perform multiple\ntraversals. \"reverse\" is for the previous change where the reverse graph\nhas every reachable commit. Finally \"maximal\" is the version introduced\nhere where the reverse graph only contains the maximal commits.\n\n Repository: git/git\n multi: 2.628 sec\n reverse: 2.344 sec\n maximal: 2.047 sec\n\n Repository: torvalds/linux\n multi: 64.7 sec\n reverse: 205.3 sec\n maximal: 44.7 sec\n\nSo in all cases we've not only recovered any time lost to switching to\nthe reverse-edge algorithm, but we come out ahead of \"multi\" in all\ncases. Likewise, peak heap has gone back to something reasonable:\n\n Repository: torvalds/linux\n multi: 2.087 GB\n reverse: 3.141 GB\n maximal: 2.288 GB\n\nWhile I do not have access to full fork networks on GitHub, Peff has run\nthis algorithm on the chromium/chromium fork network and reported a\nchange from 3 hours to ~233 seconds. That network is particularly\nbeneficial for this approach because it has a long, linear history along\nwith many tags. The \"multi\" approach was obviously quadratic and the new\napproach is linear.\n\nMISCELLANEOUS\n-------------\n\nUnluckily, this patch causes bitmaps to change in such a way that\nt5310.66 no longer passes as-is in SHA-256 mode. That test is asserting\nthat truncated .bitmap files fail gracefully. But noticing that\ntruncation depends on which bitmaps we try to look at, and exactly where\nthe truncation is.\n\nSince this commit happens to rearrange the bytes in that exact region by\nchanging the way commits are selected (and thus shifting around other\nbitmaps), move the truncation to a spot where it will be noticed in both\nSHA-1 and SHA-256 mode.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++---\n t/t5310-pack-bitmaps.sh | 87 +++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 149 insertions(+), 10 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 369c76a87c..7b4fc0f304 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -180,8 +180,10 @@ static void compute_xor_offsets(void)\n\n struct bb_commit {\n \tstruct commit_list *reverse_edges;\n+\tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n-\tunsigned selected:1;\n+\tunsigned selected:1,\n+\t\t maximal:1;\n \tunsigned idx; /* within selected array */\n };\n\n@@ -198,7 +200,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i;\n+\tunsigned int i, num_maximal;\n\n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -210,27 +212,85 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n \t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\n \t\tent->selected = 1;\n+\t\tent->maximal = 1;\n \t\tent->idx = i;\n+\n+\t\tent->commit_mask = bitmap_new();\n+\t\tbitmap_set(ent->commit_mask, i);\n+\n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n+\tnum_maximal = writer->selected_nr;\n\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n\n \twhile ((commit = get_revision(&revs))) {\n \t\tstruct commit_list *p;\n+\t\tstruct bb_commit *c_ent;\n\n \t\tparse_commit_or_die(commit);\n\n-\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n-\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\tc_ent = bb_data_at(&bb->data, commit);\n+\n+\t\tif (c_ent->maximal) {\n+\t\t\tif (!c_ent->selected) {\n+\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n+\t\t\t\tnum_maximal++;\n+\t\t\t}\n+\n+\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\t}\n\n \t\tfor (p = commit->parents; p; p = p->next) {\n-\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n+\t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n+\t\t\tint c_not_p, p_not_c;\n+\n+\t\t\tif (!p_ent->commit_mask) {\n+\t\t\t\tp_ent->commit_mask = bitmap_new();\n+\t\t\t\tc_not_p = 1;\n+\t\t\t\tp_not_c = 0;\n+\t\t\t} else {\n+\t\t\t\tc_not_p = bitmap_diff_nonzero(c_ent->commit_mask, p_ent->commit_mask);\n+\t\t\t\tp_not_c = bitmap_diff_nonzero(p_ent->commit_mask, c_ent->commit_mask);\n+\t\t\t}\n+\n+\t\t\tif (!c_not_p)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_or(p_ent->commit_mask, c_ent->commit_mask);\n+\n+\t\t\tif (p_not_c)\n+\t\t\t\tp_ent->maximal = 1;\n+\t\t\telse {\n+\t\t\t\tp_ent->maximal = 0;\n+\t\t\t\tfree_commit_list(p_ent->reverse_edges);\n+\t\t\t\tp_ent->reverse_edges = NULL;\n+\t\t\t}\n+\n+\t\t\tif (c_ent->maximal) {\n+\t\t\t\tcommit_list_insert(commit, &p_ent->reverse_edges);\n+\t\t\t} else {\n+\t\t\t\tstruct commit_list *cc = c_ent->reverse_edges;\n+\n+\t\t\t\tfor (; cc; cc = cc->next) {\n+\t\t\t\t\tif (!commit_list_contains(cc->item, p_ent->reverse_edges))\n+\t\t\t\t\t\tcommit_list_insert(cc->item, &p_ent->reverse_edges);\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n+\n+\t\tbitmap_free(c_ent->commit_mask);\n+\t\tc_ent->commit_mask = NULL;\n \t}\n+\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_selected_commits\", writer->selected_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_maximal_commits\", num_maximal);\n }\n\n static void bitmap_builder_clear(struct bitmap_builder *bb)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 6bf68fee85..1691710ec1 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -20,11 +20,87 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n\n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other master\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-master * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\____________...____________ |\n+# \\|\n+# * (base)\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for master (bit 0) and other (bit 1), and some flexibility\n+# in the order that merge bases are visited, the bitmasks at\n+# the end should be:\n+#\n+# master: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# octo-master: 1\n+# octo-other: 01\n+# merge-right: 111 (maximal)\n+# (l1): 111\n+# (r1): 111\n+# merge-left: 1101 (maximal)\n+# (l2): 11111 (maximal)\n+# (r2): 111101 (maximal)\n+# (base): 1111111 (maximal)\n+\n test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 100 &&\n+\ttest_commit_bulk --id=file 10 &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n+\n+\t# add complicated history setup, including merges and\n+\t# ambiguous merge-bases\n+\n+\tgit checkout -b merge-left other~2 &&\n+\tgit merge master~2 -m \"merge-left\" &&\n+\n+\tgit checkout -b merge-right master~1 &&\n+\tgit merge other~1 -m \"merge-right\" &&\n+\n+\tgit checkout -b octo-master master &&\n+\tgit merge merge-left merge-right -m \"octopus-master\" &&\n+\n+\tgit checkout -b octo-other other &&\n+\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\tgit checkout other &&\n+\tgit merge octo-other -m \"pull octopus\" &&\n+\n \tgit checkout master &&\n+\tgit merge octo-master -m \"pull octopus\" &&\n+\n+\t# Remove these branches so they are not selected\n+\t# as bitmap tips\n+\tgit branch -D merge-left &&\n+\tgit branch -D merge-right &&\n+\tgit branch -D octo-other &&\n+\tgit branch -D octo-master &&\n+\n+\t# add padding to make these merges less interesting\n+\t# and avoid having them selected for bitmaps\n+\ttest_commit_bulk --id=file 100 &&\n+\tgit checkout other &&\n+\ttest_commit_bulk --id=side 100 &&\n+\tgit checkout master &&\n+\n \tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n@@ -32,9 +108,12 @@ test_expect_success 'setup repo with moderate-sized history' '\n '\n\n test_expect_success 'full repack creates bitmaps' '\n-\tgit repack -ad &&\n+\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n+\ttest_line_count = 1 output &&\n+\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n '\n\n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n@@ -356,7 +435,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n--\n2.29.2.312.gabc4d358d8\n\n"},{"id":"410481","messageId":"X7nNlu8wBZw3xFjX@xnor.local","threadId":"54622","inReplyTo":"CAN0heSpVnzyE5S5ReKQ0Q_UU48jQ77NVF1x1NTGx29+5KZsyRA@mail.gmail.com","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-22T02:31:50Z","receivedAt":"2020-11-22T02:32:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Nov 21, 2020 at 09:11:21PM +0100, Martin Ågren wrote:\n> On Sat, 21 Nov 2020 at 20:37, Junio C Hamano wrote:\n> >\n> > Martin Ågren writes:\n> >\n> > > On Tue, 17 Nov 2020 at 22:46, Taylor Blau wrote:\n> > >> Not very much has changed since last time, but a range-diff is below\n> > >> nonetheless. The major changes are:\n> > >>\n> > >> - Avoid an overflow when bounds checking in the second and third\n> > >> patches (thanks, Martin, for noticing).\n> > >\n> > > FWIW, the updates to patches 2 and 3 look exactly like what I was\n> > > expecting after the discussion on v1. I have nothing to add.\n> >\n> > Thanks, both. Shall we move the topic down to 'next'?\n>\n> I really only dug into those patches 2 and 3. I read the rest of the\n> patches of v1 and went \"that makes sense\", but that's about it. I\n> started looking at \"pack-bitmap-write: build fewer intermediate bitmaps\"\n> and went \"this looks really cool -- I should try to understand this\". :-)\n>\n> There was SZEDER's comment on that last patch in v2, where future\n> readers of that patch will have to wonder why it does s/256/270/ in a\n> test. I agree with SZEDER that the change should be mentioned in the\n> commit message, even if it's just \"unfortunately, we have some magicness\n> here, plus we want to pass both with SHA-1 and SHA-256; turns out 270\n> hits the problem we want to test for\".\n\nThanks for reviewing it, and noticing a couple of problems in the\nearlier patches, too. If folks are happy with the replacement that I\nsent [1], then I am too :-).\n\nI don't think that the \"big\" patch generated a ton of review on the\nlist, but maybe that's OK. Peff, Stolee, and I all reviewed that patch\nextensively when deploying it at GitHub (where it has been running since\nlate Summer).\n\n> Martin\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/X7nMzzMfjm%2Fp9qfj@xnor.local/\n"},{"id":"410495","messageId":"xmqqblfpqj4e.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"36deaad366d66d10b96755dd6969bfe51123a2d4.1605123652.git.me@ttaylorr.com","subject":"Re: [PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-22T19:36:17Z","receivedAt":"2020-11-22T19:36:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> When the buffer size is exactly 1, we fail to grow it properly, since\n> the integer truncation means that 1 * 3 / 2 = 1. This can cause a bad\n> write on the line below.\n\nWhen the buffer_size is exactly (alloc_size - 1), we can fit the new\nelement at the last word in the buffer array, but we still grow. Is\nthis because we anticipate that we would need to add more soon?\n\n> Bandaid this by first padding the buffer by 16, and then growing it.\n> This still allows old blocks to fit into new ones, but fixes the case\n> where the block size equals 1.\n\nAdding 16 unconditionally is not \"to pad\". If somebody really wants\n\"to pad\", a likely implementation would be that the size resulting\nfrom some computation (e.g. multiplying by 1.5) is round up to a\nmultiple of some number, than rounding up the original number before\nmultiplying it by 1.5, so the use of that verb in the explanation\ndid not help me understand what is going on.\n\nHaving said that, I see you used the word \"bandaid\" to signal that\nwe shouldn't worry about this being optimal or even correct and we\nshould be happy as long as it is not wrong ;-), but is there any\nreason behind this 16 (as opposed to picking, say, 8 or 31), or is\nthat pulled out of thin air?\n\nI think this probably mimics what alloc_nr() computes for ALLOC_GROW().\nI wonder why buffer_grow() cannot be built around ALLOC_GROW() instead?\n\nNothing in the code is wrong per-se, but just what I noticed while\nre-reading the patch.\n\nThanks.\n\n> Co-authored-by: Jeff King \n> Signed-off-by: Jeff King \n> Signed-off-by: Taylor Blau \n> ---\n> ewah/ewah_bitmap.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\n> index d59b1afe3d..3fae04ad00 100644\n> --- a/ewah/ewah_bitmap.c\n> +++ b/ewah/ewah_bitmap.c\n> @@ -45,7 +45,7 @@ static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n> static inline void buffer_push(struct ewah_bitmap *self, eword_t value)\n> {\n> \tif (self->buffer_size + 1 >= self->alloc_size)\n> -\t\tbuffer_grow(self, self->buffer_size * 3 / 2);\n> +\t\tbuffer_grow(self, (self->buffer_size + 16) * 3 / 2);\n> \n> \tself->buffer[self->buffer_size++] = value;\n> }\n"},{"id":"410499","messageId":"xmqq7dqdqgji.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"c7db594fae4d0447a55a92e830475d9bc418ae7f.1605123652.git.me@ttaylorr.com","subject":"Re: [PATCH 07/23] ewah: make bitmap growth less aggressive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-22T20:32:01Z","receivedAt":"2020-11-22T20:32:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> - a geometric increase in existing size; we'll switch to 3/2 instead of\n> 2 here. That's less aggressive and may help avoid fragmenting memory\n> (N + 3N/2 > 9N/4, so old chunks can be reused as we scale up).\n\nI am sure this is something obvious to bitmap folks, but where does\n9N/4 come from (I get that the left-hand-side of the comparison is\nthe memory necessary to hold both the old and the new copy while\nreallocating the words[] array)?\n\nThanks.\n"},{"id":"410500","messageId":"xmqq3611qgg4.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"dce9b6da0ad38da0a92b39d780d7b56f83d52950.1605123652.git.me@ttaylorr.com","subject":"Re: [PATCH 08/23] ewah: implement bitmap_or()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-22T20:34:03Z","receivedAt":"2020-11-22T20:34:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> From: Jeff King \n>\n> We have a function to bitwise-OR an ewah into an uncompressed bitmap,\n> but not to OR two uncompressed bitmaps. Let's add it.\n>\n> Interestingly, we have a public header declaration going back to\n> e1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\n> function was never implemented.\n\nSo we have had decl, no impl, but it did not matter because there\nwas no user? Presumably we will see a real user soon in the series\n;-)\n\n"},{"id":"410501","messageId":"xmqqy2itoyco.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"8e5607929d66a3c808dbe3a06c312d0cda1ef568.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-22T21:50:15Z","receivedAt":"2020-11-22T21:50:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> From: Derrick Stolee \n>\n> The fill_bitmap_commit() method assumes that every parent of the given\n> commit is already part of the current bitmap. Instead of making that\n> assumption, let's walk parents until we reach commits already part of\n> the bitmap. Set the value for that parent immediately after querying to\n> save time doing double calls to find_object_pos() and to avoid inserting\n> the parent into the queue multiple times.\n\nIs it because somebody found a case where the assumption does not\nhold and the code with the assumption produces a wrong result? Is\nit because we can get a better result without making the assumption\nthe current code does?\n\nIn other words, can we explain why we are making the change in the\nproposed log message?\n\n> Signed-off-by: Derrick Stolee \n> Signed-off-by: Taylor Blau \n> ---\n> pack-bitmap-write.c | 30 +++++++++++++++++++++++-------\n> 1 file changed, 23 insertions(+), 7 deletions(-)\n>\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index d2d46ff5f4..361f3305a2 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -12,6 +12,7 @@\n> #include \"sha1-lookup.h\"\n> #include \"pack-objects.h\"\n> #include \"commit-reach.h\"\n> +#include \"prio-queue.h\"\n> \n> struct bitmapped_commit {\n> \tstruct commit *commit;\n> @@ -279,17 +280,30 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n> }\n> \n> static void fill_bitmap_commit(struct bb_commit *ent,\n> -\t\t\t struct commit *commit)\n> +\t\t\t struct commit *commit,\n> +\t\t\t struct prio_queue *queue)\n> {\n> \tif (!ent->bitmap)\n> \t\tent->bitmap = bitmap_new();\n> \n> -\t/*\n> -\t * mark ourselves, but do not bother with parents; their values\n> -\t * will already have been propagated to us\n> -\t */\n> \tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n> -\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n> +\tprio_queue_put(queue, commit);\n> +\n> +\twhile (queue->nr) {\n> +\t\tstruct commit_list *p;\n> +\t\tstruct commit *c = prio_queue_get(queue);\n> +\n> +\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n> +\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n> +\n> +\t\tfor (p = c->parents; p; p = p->next) {\n> +\t\t\tint pos = find_object_pos(&p->item->object.oid);\n> +\t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n> +\t\t\t\tbitmap_set(ent->bitmap, pos);\n> +\t\t\t\tprio_queue_put(queue, p->item);\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> }\n> \n> static void store_selected(struct bb_commit *ent, struct commit *commit)\n> @@ -319,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \tstruct bitmap_builder bb;\n> \tsize_t i;\n> \tint nr_stored = 0; /* for progress */\n> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> \n> \twriter.bitmaps = kh_init_oid_map();\n> \twriter.to_pack = to_pack;\n> @@ -335,7 +350,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \t\tstruct commit *child;\n> \t\tint reused = 0;\n> \n> -\t\tfill_bitmap_commit(ent, commit);\n> +\t\tfill_bitmap_commit(ent, commit, &queue);\n> \n> \t\tif (ent->selected) {\n> \t\t\tstore_selected(ent, commit);\n> @@ -360,6 +375,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \t\t\tbitmap_free(ent->bitmap);\n> \t\tent->bitmap = NULL;\n> \t}\n> +\tclear_prio_queue(&queue);\n> \tbitmap_builder_clear(&bb);\n> \n> \tstop_progress(&writer.progress);\n"},{"id":"410503","messageId":"xmqqtuthoxu6.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"4840c64c51d65ea7bf1ebe03cad4588267db0207.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 13/24] bitmap: add bitmap_diff_nonzero()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-22T22:01:21Z","receivedAt":"2020-11-22T22:01:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> From: Derrick Stolee \n>\n> The bitmap_diff_nonzero() checks if the 'self' bitmap contains any bits\n> that are not on in the 'other' bitmap.\n\nIn other words, it yields false if and only if self is a subset of\nother? I have to say that \"diff_nonzero\" is much less helpful than\nwords like \"subset\" or \"superset\" when I try to imagine what the\nfunction would compute.\n\nIf this were widely used helper function, I may insist on flipping\nthe polarity and call it bitmap_is_subset(), but I dunno...\n\n> Also, delete the declaration of bitmap_is_subset() as it is not used or\n> implemented.\n\n;-)\n\n> Signed-off-by: Derrick Stolee \n> Signed-off-by: Taylor Blau \n> ---\n> ewah/bitmap.c | 24 ++++++++++++++++++++++++\n> ewah/ewok.h | 2 +-\n> 2 files changed, 25 insertions(+), 1 deletion(-)\n>\n> diff --git a/ewah/bitmap.c b/ewah/bitmap.c\n> index eb7e2539be..e2ebeac0e5 100644\n> --- a/ewah/bitmap.c\n> +++ b/ewah/bitmap.c\n> @@ -200,6 +200,30 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n> \treturn 1;\n> }\n> \n> +int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other)\n> +{\n> +\tstruct bitmap *small;\n\nIt is not wrong per-se, but s/small/smaller/ would be more natural?\n\nI actually think it would be easier to follow the logic to replace\nthis pointer with\n\n\tsize_t common_size;\n\nThen the code becomes\n\n\tif (self->word_alloc < other->word_alloc)\n\t\tcommon_size = self->word_alloc;\n\telse {\n\t\tcommon_size = other->word_alloc;\n\t\tfor (i = common_size; i < self->word_alloc; i++)\n\t\t\tif (self->words[i])\n\t\t\t\t... self is *not* subset ...\n\t}\n\n\tfor (i = 0; i < common_size; i++)\n\t\tif (self->words[i] & ~other->words[i]))\n\t\t\t... self is *not* subset ...\n\n\n> +\tsize_t i;\n> +\n> +\tif (self->word_alloc < other->word_alloc) {\n> +\t\tsmall = self;\n> +\t} else {\n> +\t\tsmall = other;\n> +\n> +\t\tfor (i = other->word_alloc; i < self->word_alloc; i++) {\n> +\t\t\tif (self->words[i] != 0)\n> +\t\t\t\treturn 1;\n> +\t\t}\n> +\t}\n> +\n> +\tfor (i = 0; i < small->word_alloc; i++) {\n> +\t\tif ((self->words[i] & ~other->words[i]))\n> +\t\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> void bitmap_reset(struct bitmap *bitmap)\n> {\n> \tmemset(bitmap->words, 0x0, bitmap->word_alloc * sizeof(eword_t));\n> diff --git a/ewah/ewok.h b/ewah/ewok.h\n> index 1fc555e672..156c71d06d 100644\n> --- a/ewah/ewok.h\n> +++ b/ewah/ewok.h\n> @@ -180,7 +180,7 @@ int bitmap_get(struct bitmap *self, size_t pos);\n> void bitmap_reset(struct bitmap *self);\n> void bitmap_free(struct bitmap *self);\n> int bitmap_equals(struct bitmap *self, struct bitmap *other);\n> -int bitmap_is_subset(struct bitmap *self, struct bitmap *super);\n> +int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other);\n> \n> struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n> struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n"},{"id":"410528","messageId":"f6d89171-f343-0d4c-7cf4-725c49615b99@gmail.com","threadId":"54622","inReplyTo":"xmqqy2itoyco.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-23T14:54:25Z","receivedAt":"2020-11-23T14:54:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/22/2020 4:50 PM, Junio C Hamano wrote:\n> Taylor Blau writes:\n> \n>> From: Derrick Stolee \n>>\n>> The fill_bitmap_commit() method assumes that every parent of the given\n>> commit is already part of the current bitmap. Instead of making that\n>> assumption, let's walk parents until we reach commits already part of\n>> the bitmap. Set the value for that parent immediately after querying to\n>> save time doing double calls to find_object_pos() and to avoid inserting\n>> the parent into the queue multiple times.\n> \n> Is it because somebody found a case where the assumption does not\n> hold and the code with the assumption produces a wrong result? Is\n> it because we can get a better result without making the assumption\n> the current code does?\n\nThe algorithm from \"pack-bitmap-write: reimplement bitmap writing\"\nthat calls fill_bitmap_commit() satisfies this assumption, since it\ncomputes a reachability bitmap for every commit during the reverse\nwalk. We will soon change that algorithm to \"skip\" commits, so we\nneed this step in fill_bitmap_commit() to walk forward to fill the\ngaps.\n\n> In other words, can we explain why we are making the change in the\n> proposed log message?\nI'm sure Taylor and I can work out a better wording to make this\nmore clear.\n\nThanks,\n-Stolee\n\n\n"},{"id":"410540","messageId":"X7vhrCbeFzxaEVvv@nand.local","threadId":"54622","inReplyTo":"xmqqblfpqj4e.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-23T16:22:04Z","receivedAt":"2020-11-23T16:22:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Nov 22, 2020 at 11:36:17AM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > When the buffer size is exactly 1, we fail to grow it properly, since\n> > the integer truncation means that 1 * 3 / 2 = 1. This can cause a bad\n> > write on the line below.\n>\n> When the buffer_size is exactly (alloc_size - 1), we can fit the new\n> element at the last word in the buffer array, but we still grow. Is\n> this because we anticipate that we would need to add more soon?\n\nRight; the check 'if (self->buffer_size + 1 >= self->alloc_size)' could\nprobably be written as a strict inequality, but that check dates back to\nthe original ewah implementation that Vicent added.\n\nBut, that is not quite the point of this patch: instead we want to stop\nthe integer math on that line from preventing us from growing the\nbuffer.\n\nI think that this paragraph would be clarified by adding \"and we need to\ngrow\" to the end of \"when the buffer size is exactly 1\".\n\n> > Bandaid this by first padding the buffer by 16, and then growing it.\n> > This still allows old blocks to fit into new ones, but fixes the case\n> > where the block size equals 1.\n>\n> Adding 16 unconditionally is not \"to pad\". If somebody really wants\n> \"to pad\", a likely implementation would be that the size resulting\n> from some computation (e.g. multiplying by 1.5) is round up to a\n> multiple of some number, than rounding up the original number before\n> multiplying it by 1.5, so the use of that verb in the explanation\n> did not help me understand what is going on.\n>\n> Having said that, I see you used the word \"bandaid\" to signal that\n> we shouldn't worry about this being optimal or even correct and we\n> should be happy as long as it is not wrong ;-), but is there any\n> reason behind this 16 (as opposed to picking, say, 8 or 31), or is\n> that pulled out of thin air?\n\nAny phrase that more accurately states what's going on is fine by me,\nbut...\n\n> I think this probably mimics what alloc_nr() computes for ALLOC_GROW().\n> I wonder why buffer_grow() cannot be built around ALLOC_GROW() instead?\n\nI think that we probably could just use ALLOC_GROW() as you suggest.\nFunny enough, reading through GitHub's chat logs, apparently this is\nsomething that Peff and I talked about. So, 16 probably came from\nalloc_nr(), but we probably stopped short of realizing that we could\njust use ALLOC_GROW as-is.\n\nSo, maybe something along the lines of:\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex 3fae04ad00..9effcc0877 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -19,6 +19,7 @@\n #include \"git-compat-util.h\"\n #include \"ewok.h\"\n #include \"ewok_rlw.h\"\n+#include \"cache.h\"\n\n static inline size_t min_size(size_t a, size_t b)\n {\n@@ -33,12 +34,7 @@ static inline size_t max_size(size_t a, size_t b)\n static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n {\n \tsize_t rlw_offset = (uint8_t *)self->rlw - (uint8_t *)self->buffer;\n-\n-\tif (self->alloc_size >= new_size)\n-\t\treturn;\n-\n-\tself->alloc_size = new_size;\n-\tREALLOC_ARRAY(self->buffer, self->alloc_size);\n+\tALLOC_GROW(self->buffer, new_size, self->alloc_size);\n \tself->rlw = self->buffer + (rlw_offset / sizeof(eword_t));\n }\n\n> Nothing in the code is wrong per-se, but just what I noticed while\n> re-reading the patch.\n>\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"410541","messageId":"X7voLUlevHygqFg/@nand.local","threadId":"54622","inReplyTo":"xmqq7dqdqgji.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 07/23] ewah: make bitmap growth less aggressive","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-23T16:49:49Z","receivedAt":"2020-11-23T16:49:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Nov 22, 2020 at 12:32:01PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > - a geometric increase in existing size; we'll switch to 3/2 instead of\n> > 2 here. That's less aggressive and may help avoid fragmenting memory\n> > (N + 3N/2 > 9N/4, so old chunks can be reused as we scale up).\n>\n> I am sure this is something obvious to bitmap folks, but where does\n> 9N/4 come from (I get that the left-hand-side of the comparison is\n> the memory necessary to hold both the old and the new copy while\n> reallocating the words[] array)?\n\nI thought that I was in the group of \"bitmap folks\", but since it's not\nobvious to me either, I guess I'll have to hand in my bitmap folks\nmembership card ;).\n\nPeff: where does 9N/4 come from? On a similar note: we could certainly\nuse ALLOC_GROW here, too, but it would change the behavior slightly (by\nusing alloc_nr()'s \"add-16-first\" behavior). Maybe we should be using\nit, but I'll defer to your judgement.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"410542","messageId":"X7vouTlBlT62H7A4@nand.local","threadId":"54622","inReplyTo":"xmqq3611qgg4.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 08/23] ewah: implement bitmap_or()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-23T16:52:09Z","receivedAt":"2020-11-23T16:52:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Nov 22, 2020 at 12:34:03PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > From: Jeff King \n> >\n> > We have a function to bitwise-OR an ewah into an uncompressed bitmap,\n> > but not to OR two uncompressed bitmaps. Let's add it.\n> >\n> > Interestingly, we have a public header declaration going back to\n> > e1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\n> > function was never implemented.\n>\n> So we have had decl, no impl, but it did not matter because there\n> was no user? Presumably we will see a real user soon in the series\n> ;-)\n\nIndeed :-). I added a note to this patch's log message to indicate that\na new/first caller would be appearing in a couple of patches after this\none.\n\nThanks,\nTaylor\n"},{"id":"410563","messageId":"X7wZP9HO1gx850aO@nand.local","threadId":"54622","inReplyTo":"xmqqtuthoxu6.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 13/24] bitmap: add bitmap_diff_nonzero()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-23T20:19:11Z","receivedAt":"2020-11-23T20:19:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Nov 22, 2020 at 02:01:21PM -0800, Junio C Hamano wrote:\n> I actually think it would be easier to follow the logic to replace\n> this pointer with\n>\n> \tsize_t common_size;\n> [ ... ]\n\nYep, much clearer indeed. Thanks.\n\nTaylor\n"},{"id":"410622","messageId":"X7xzWClGr3bM3wcg@coredump.intra.peff.net","threadId":"54622","inReplyTo":"X7nNlu8wBZw3xFjX@xnor.local","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-24T02:43:36Z","receivedAt":"2020-11-24T02:44:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Nov 21, 2020 at 09:31:50PM -0500, Taylor Blau wrote:\n\n> > There was SZEDER's comment on that last patch in v2, where future\n> > readers of that patch will have to wonder why it does s/256/270/ in a\n> > test. I agree with SZEDER that the change should be mentioned in the\n> > commit message, even if it's just \"unfortunately, we have some magicness\n> > here, plus we want to pass both with SHA-1 and SHA-256; turns out 270\n> > hits the problem we want to test for\".\n> \n> Thanks for reviewing it, and noticing a couple of problems in the\n> earlier patches, too. If folks are happy with the replacement that I\n> sent [1], then I am too :-).\n> \n> I don't think that the \"big\" patch generated a ton of review on the\n> list, but maybe that's OK. Peff, Stolee, and I all reviewed that patch\n> extensively when deploying it at GitHub (where it has been running since\n> late Summer).\n\nHrm. I thought you were going to integrate the extra checks I suggested\nfor load_bitmap_entries_v1(). Which is looks like you did in patch 17.\nAfter that, the s/256/270/ hack should not be necessary anymore (if it\nis, then we should keep fixing more spots).\n\n-Peff\n"},{"id":"410623","messageId":"X7x0du3qoC4vuGtS@coredump.intra.peff.net","threadId":"54622","inReplyTo":"X7vhrCbeFzxaEVvv@nand.local","subject":"Re: [PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-24T02:48:22Z","receivedAt":"2020-11-24T02:48:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 23, 2020 at 11:22:04AM -0500, Taylor Blau wrote:\n\n> > I think this probably mimics what alloc_nr() computes for ALLOC_GROW().\n> > I wonder why buffer_grow() cannot be built around ALLOC_GROW() instead?\n> \n> I think that we probably could just use ALLOC_GROW() as you suggest.\n> Funny enough, reading through GitHub's chat logs, apparently this is\n> something that Peff and I talked about. So, 16 probably came from\n> alloc_nr(), but we probably stopped short of realizing that we could\n> just use ALLOC_GROW as-is.\n\nThat would probably be OK. It's a bit more aggressive, which could\nmatter if you have a large number of very small bitmaps. My original\ngoal of the \"grow less aggressively\" patch was to keep memory usage\ndown, knowing that I was going to be holding a lot of bitmaps in memory\nat once. But even with micro-optimizations like this, it turned out to\nbe far too big in practice (and hence Stolee's work on top to reduce the\ntotal number we hold at once).\n\n> @@ -33,12 +34,7 @@ static inline size_t max_size(size_t a, size_t b)\n> static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n> {\n> \tsize_t rlw_offset = (uint8_t *)self->rlw - (uint8_t *)self->buffer;\n> -\n> -\tif (self->alloc_size >= new_size)\n> -\t\treturn;\n> -\n> -\tself->alloc_size = new_size;\n> -\tREALLOC_ARRAY(self->buffer, self->alloc_size);\n> +\tALLOC_GROW(self->buffer, new_size, self->alloc_size);\n> \tself->rlw = self->buffer + (rlw_offset / sizeof(eword_t));\n> }\n\nI think the real test would be measuring the peak heap of the series as\nyou posted it in v2, and this version replacing this patch (and the\n\"grow less aggressively\" one) with ALLOC_GROW(). On something big, like\nrepacking all of the torvalds/linux or git/git fork networks.\n\nIf there's no appreciable difference, then definitely I think it's worth\nthe simplicity of reusing ALLOC_GROW().\n\n-Peff\n"},{"id":"410625","messageId":"X7x1PciN+bVDkJhr@coredump.intra.peff.net","threadId":"54622","inReplyTo":"X7x0du3qoC4vuGtS@coredump.intra.peff.net","subject":"Re: [PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-24T02:51:41Z","receivedAt":"2020-11-24T02:52:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 23, 2020 at 09:48:22PM -0500, Jeff King wrote:\n\n> > I think that we probably could just use ALLOC_GROW() as you suggest.\n> > Funny enough, reading through GitHub's chat logs, apparently this is\n> > something that Peff and I talked about. So, 16 probably came from\n> > alloc_nr(), but we probably stopped short of realizing that we could\n> > just use ALLOC_GROW as-is.\n> \n> That would probably be OK. It's a bit more aggressive, which could\n> matter if you have a large number of very small bitmaps. My original\n> goal of the \"grow less aggressively\" patch was to keep memory usage\n> down, knowing that I was going to be holding a lot of bitmaps in memory\n> at once. But even with micro-optimizations like this, it turned out to\n> be far too big in practice (and hence Stolee's work on top to reduce the\n> total number we hold at once).\n\nOh, sorry, I was mixing this patch up with patches 6 and 7, which touch\nbuffer_grow(). This is a totally separate spot, and this is a pure\nbug-fix.\n\nI think the main reason we didn't use ALLOC_GROW() here in the beginning\nis that the ewah code was originally designed to be a separate library\n(a port of the java ewah library), and didn't depend on Git code.\n\nThese days we pull in xmalloc, etc, so we should be fine to use\nALLOC_GROW().\n\nLikewise...\n\n> I think the real test would be measuring the peak heap of the series as\n> you posted it in v2, and this version replacing this patch (and the\n> \"grow less aggressively\" one) with ALLOC_GROW(). On something big, like\n> repacking all of the torvalds/linux or git/git fork networks.\n> \n> If there's no appreciable difference, then definitely I think it's worth\n> the simplicity of reusing ALLOC_GROW().\n\nAll of this is nonsense (though it does apply to the question of using\nALLOC_GROW() in bitmap_grow(), via patch 7).\n\nWe have many fewer ewah bitmaps in memory at one time, so I don't think\nit's worth micro-managing a few extra bytes of growth. Using\nALLOC_GROW() for this case would be fine.\n\n-Peff\n"},{"id":"410627","messageId":"X7x3WtCItVGhQ57O@coredump.intra.peff.net","threadId":"54622","inReplyTo":"X7voLUlevHygqFg/@nand.local","subject":"Re: [PATCH 07/23] ewah: make bitmap growth less aggressive","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-11-24T03:00:42Z","receivedAt":"2020-11-24T03:01:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 23, 2020 at 11:49:49AM -0500, Taylor Blau wrote:\n\n> On Sun, Nov 22, 2020 at 12:32:01PM -0800, Junio C Hamano wrote:\n> > Taylor Blau writes:\n> >\n> > > - a geometric increase in existing size; we'll switch to 3/2 instead of\n> > > 2 here. That's less aggressive and may help avoid fragmenting memory\n> > > (N + 3N/2 > 9N/4, so old chunks can be reused as we scale up).\n> >\n> > I am sure this is something obvious to bitmap folks, but where does\n> > 9N/4 come from (I get that the left-hand-side of the comparison is\n> > the memory necessary to hold both the old and the new copy while\n> > reallocating the words[] array)?\n> \n> I thought that I was in the group of \"bitmap folks\", but since it's not\n> obvious to me either, I guess I'll have to hand in my bitmap folks\n> membership card ;).\n> \n> Peff: where does 9N/4 come from?\n\nit is not a bitmap thing at all. We are growing a buffer, so if we\ncontinually multiply it by 3/2, then our sequence of sizes is:\n\n - before growth: N\n - after 1 growth: 3N/2\n - after 2 growths: 9N/4\n\nMeaning we can fit the third chunk into the memory vacated by the second\ntwo. Whereas with a factor of, say 2:\n\n - before growth: N\n - after 1 growth: 2N\n - after 2 growth: 4N\n\nwhich does not fit, and fragments your memory.\n\nThere's a slight lie there, which is that you'll typically still hold\nthe growth G-1 while doing growth G (after all, that is where you will\ncopy the data from). But it still works out that you eventually get to\nuse old chunks. The breakeven point is actually the golden ratio, but a)\nit's irrational and b) it probably makes sense to give some slop for\nmalloc chunk overhead. 1.6 would probably be fine, too, though. :)\n\n> On a similar note: we could certainly\n> use ALLOC_GROW here, too, but it would change the behavior slightly (by\n> using alloc_nr()'s \"add-16-first\" behavior). Maybe we should be using\n> it, but I'll defer to your judgement.\n\nThat would be OK, modulo the measurement question I asked in the other\n(wrong) part of the thread.\n\n-Peff\n"},{"id":"410636","messageId":"20201124060738.762751-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"5262daa3300114fbaccdbc7393882c5435f95f4f.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 18/24] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-24T06:07:38Z","receivedAt":"2020-11-24T06:07:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I think this is the \"big patch\" mentioned in IRC [1]? I'll review just\nthe commit message of this one first, then go back and review from patch\n10 (\"pack-bitmap-write: reimplement bitmap writing\") up to this\none inclusive.\n\n[1] https://colabti.org/irclogger/irclogger_log/git-devel?date=2020-11-23\n\n> From: Derrick Stolee \n> \n> The bitmap_writer_build() method calls bitmap_builder_init() to\n> construct a list of commits reachable from the selected commits along\n> with a \"reverse graph\". This reverse graph has edges pointing from a\n> commit to other commits that can reach that commit. After computing a\n> reachability bitmap for a commit, the values in that bitmap are then\n> copied to the reachability bitmaps across the edges in the reverse\n> graph.\n> \n> We can now relax the role of the reverse graph to greatly reduce the\n> number of intermediate reachability bitmaps we compute during this\n> reverse walk. The end result is that we walk objects the same number of\n> times as before when constructing the reachability bitmaps, but we also\n> spend much less time copying bits between bitmaps and have much lower\n> memory pressure in the process.\n\nOK - as I have seen in the previous patches, and as said here, the edges\nof the graph were previously parent to immediate child, but I believe\nthat patch 12 (\"pack-bitmap-write: fill bitmap with commit history\")\nmakes it so that the edges don't need to be direct parent-child\nrelationships. That patch does not, however, decide what the vertices\nand edges should thus be, so that ability could not be used. This patch\nseems to do so, and thus makes use of that ability.\n\n> The core idea is to select a set of \"important\" commits based on\n> interactions among the sets of commits reachable from each selected commit.\n\nMakes sense.\n\n> The first technical concept is to create a new 'commit_mask' member in the\n> bb_commit struct. Note that the selected commits are provided in an\n> ordered array. The first thing to do is to mark the ith bit in the\n> commit_mask for the ith selected commit. \n\nOK - so this commit_mask is like the bitmaps in Git. There is an array,\ninitially populated with the list of selected commits (which are\nselected using another algorithm, which is a separate concern from this\npatch set), and each bit in commit_mask corresponds to the corresponding\nentry in that array.\n\nFrom this, I assume that the commit_mask values in the selected\nbb_commit structs will start with a nonzero value (written in binary,\nhaving 1 bit set), and the other commit_mask values will start with\nzero.\n\n> As we walk the commit-graph, we\n> copy the bits in a commit's commit_mask to its parents. At the end of\n> the walk, the ith bit in the commit_mask for a commit C stores a boolean\n> representing \"The ith selected commit can reach C.\"\n\nThe walk is done in topological order - visiting children before\nparents. Copying makes sense - if a commit can reach me, it can reach my\nparents as well.\n\n> As we walk, we will discover non-selected commits that are important. We\n> will get into this later, but those important commits must also receive\n> bit positions, growing the width of the bitmasks as we walk. At the true\n> end of the walk, the ith bit means \"the ith _important_ commit can reach\n> C.\"\n\nOK - so the initial array, initially populated with the list of selected\ncommits, can be grown and will include other important commits as well.\nThis is similar to the bitmap revwalk algorithm - the bitmaps in that\nalgorithm can be grown to include other objects as well.\n\n> MAXIMAL COMMITS\n> ---------------\n> \n> We use a new 'maximal' bit in the bb_commit struct to represent whether\n> a commit is important or not. The term \"maximal\" comes from the\n> partially-ordered set of commits in the commit-graph where C >= P if P\n> is a parent of C, and then extending the relationship transitively.\n\nI had to look up what \"maximal\" means in a partial order. :-P An element\nof a partially ordered set is \"maximal\" if there is no other element\nthat is \"greater\" than it. Here, all descendants are \"greater\" than\ntheir ancestors.\n\n> Instead of taking the maximal commits across the entire commit-graph, \n\nI was wondering about this :-)\n\n> we\n> instead focus on selecting each commit that is maximal among commits\n> with the same bits on in their commit_mask. \n\nAh, OK. Two commits will have the same commit_mask if the exact same set\nof important commits can reach them.\n\n> This definition is\n> important, so let's consider an example.\n> \n> Suppose we have three selected commits A, B, and C. These are assigned\n> bitmasks 100, 010, and 001 to start. Each of these can be marked as\n> maximal immediately because they each will be the uniquely maximal\n> commit that contains their own bit. \n\nThat is correct. To further elaborate on this explanation, let's say we\nhave a selected commit C (and since it is selected, the commit_mask in\neach commit will have a bit corresponding to whether C can reach it).\nEach other commit is either an ancestor, a descendant, or unrelated.\n\n - C cannot reach descendants.\n - C cannot reach unrelated commits.\n - C can reach all ancestors, but in the partial order, C compares\n \"greater\" to them anyway.\n\nSo every other commit cannot affect C's maximal status.\n\n> Keep in mind that that these commits\n> may have different bitmasks after the walk; for example, if B can reach\n> C but A cannot, then the final bitmask for C is 011. Even in these\n> cases, C would still be a maximal commit among all commits with the\n> third bit on in their masks.\n\nYes.\n\n> Now define sets X, Y, and Z to be the sets of commits reachable from A,\n> B, and C, respectively. The intersections of these sets correspond to\n> different bitmasks:\n> \n> * 100: X - (Y union Z)\n> * 010: Y - (X union Z)\n> * 001: Z - (X union Y)\n> * 110: (X intersect Y) - Z\n> * 101: (X intersect Z) - Y\n> * 011: (Y intersect Z) - X\n> * 111: X intersect Y intersect Z\n> \n> This can be visualized with the following Hasse diagram:\n> \n> \t100 010 001\n> | \\ / \\ / |\n> | \\/ \\/ |\n> | /\\ /\\ |\n> | / \\ / \\ |\n> 110 101 011\n> \\___ | ___/\n> \\ | /\n> 111\n> \n> Some of these bitmasks may not be represented, depending on the topology\n> of the commit-graph. In fact, we are counting on it, since the number of\n> possible bitmasks is exponential in the number of selected commits, but\n> is also limited by the total number of commits. In practice, very few\n> bitmasks are possible because most commits converge on a common \"trunk\"\n> in the commit history.\n\nThis section wasn't very useful to me - but I would appreciate it if\nothers chimed in to say it was useful to them.\n\n> With this three-bit example, we wish to find commits that are maximal\n> for each bitmask. How can we identify this as we are walking?\n\nOK - now we come to the algorithm. I presume the algorithm doesn't only\nfind commits that are maximal for each bitmask, but also updates the\nlist of important commits (and thus increasing the size of the bitmask)?\nReading below, I see that the answer to my question is yes. Ah...it\nwasn't clear to me that the purpose of finding the maximal commits is\nalso to add to the list of important commits, but perhaps it will be\nobvious to other reasons.\n\nI'll work through the algorithm using the butterfly example below,\nreproduced here:\n\n> I J\n> |\\ /|\n> | \\/ |\n> | /\\ |\n> |/ \\|\n> M N\n> \\ /\n> |/\n> Q\n\nI was going to suggest that we suppose that there are no selected\ncommits, but it looks like the algorithm would optimize itself out\n(meaning that it won't make any commit maximal - which makes sense, I\nguess). The example below had I and J as selected commits (which I know\nbecause the commit_mask values for I and J are \"0b10\" and \"0b01\"\nrespectively), so let's go with that.\n\n> As we walk, we visit a commit C. Since we are walking the commits in\n> topo-order, we know that C is visited after all of its children are\n> visited. Thus, when we get C from the revision walk we inspect the\n> 'maximal' property of its bb_data and use that to determine if C is truly\n> important. Its commit_mask is also nearly final. \n\nOK - when a commit is visited, we would already know its \"maximal\"\nstatus because when we had visited its parents, we already modified\n\"maximal\" (because we update a commit's children when we visit it -\ndetails about this are to follow, presumably).\n\n> If C is not one of the\n> originally-selected commits, then assign a bit position to C (by\n> incrementing num_maximal) and set that bit on in commit_mask. See\n> \"MULTIPLE MAXIMAL COMMITS\" below for more detail on this.\n\nI presume we only assign a bit position to C if it is \"maximal\"?\n\n> Now that the commit C is known to be maximal or not, consider each\n> parent P of C. Compute two new values:\n> \n> * c_not_p : true if and only if the commit_mask for C contains a bit\n> that is not contained in the commit_mask for P.\n> \n> * p_not_c : true if and only if the commit_mask for P contains a bit\n> that is not contained in the commit_mask for P.\n\nOK, let's try this with I. I'll use the same : notation as the one the\ncommit author uses below to indicate the commit_mask of a commit. We\nhave I:10 with 2 parents M:00 and N:00, so for both parents, c_not_p is\ntrue and p_not_c is false.\n\n> If c_not_p is false, then P already has all of the bits that C would\n> provide to its commit_mask. In this case, move on to other parents as C\n> has nothing to contribute to P's state that was not already provided by\n> other children of P.\n\nTo emphasize, we \"move on\" regardless of what p_not_c is. In our\nexample, this is not true in I's case, so let's read on.\n\nAfter the analysis below, I see why we can \"move on\".\n\n> We continue with the case that c_not_p is true. This means there are\n> bits in C's commit_mask to copy to P's commit_mask, so use bitmap_or()\n> to add those bits.\n\nOK. So we have I:10 (unchanged), M:10, N:10. Which as I said above,\nmakes sense, since if a commit can reach I, it can reach M and N.\n\n> If p_not_c is also true, then set the maximal bit for P to one. This means\n> that if no other commit has P as a parent, then P is definitely maximal.\n> This is because no child had the same bitmask. It is important to think\n> about the maximal bit for P at this point as a temporary state: \"P is\n> maximal based on current information.\"\n> \n> In contrast, if p_not_c is false, then set the maximal bit for P to\n> zero.\n\nAt first it was confusing that (1) the last maximal bit is used when\nthere is no guarantee that the children are iterated in any particular\norder, and (2) the maximal bit is never updated when c_not_p is false.\nSo I was thinking of a counterexample, but couldn't think of one. So\nthis algorithm looks correct so far.\n\nFor (2), I think of it this way:\n\n C1 C2 C3\n \\ |/\n P\n\nLet's say that C1 has a commit_mask, and C2 and C3 have the exact same\ncommit_mask. P's maximal bit will be the exact same in the following\nsituation:\n\n C1 C2\n \\ |\n P\n\nSo skipping over C3 is correct. And C3 must be skipped because the\nalgorithm as written is not idempotent (during the C2 iteration,\ncommit_mask of P is updated).\n\nFor (1), I think of it as follows. The only one that counts is the very\nlast calculation (which you can see from the algorithm - the maximal bit\nis constantly being overridden). During the very last iteration, does P\nhave any information that Cx does not? If yes, P has a commit_mask that\nis unique w.r.t. all its children (since it combines unique information\nfrom its other children that Cx does not, plus some other unique\ninformation that Cx has and its other children do not) and is\ntherefore maximal. If not, all the information P got is also known by\nCx, so it is definitely not maximal (Cx shares the commit_mask with P,\nand is \"greater\" than P).\n\n> Further, clear all reverse_edges for P since any edges that were\n> previously assigned to P are no longer important. P will gain all\n> reverse edges based on C.\n\n(I read ahead to see what the reverse edges are.) Indeed, C has all the\ninformation.\n\n> The final thing we need to do is to update the reverse edges for P.\n> These reverse edges respresent \"which closest maximal commits\n> contributed bits to my commit_mask?\" Since C contributed bits to P's\n> commit_mask in this case, C must add to the reverse edges of P.\n> \n> If C is maximal, then C is a 'closest' maximal commit that contributed\n> bits to P. Add C to P's reverse_edges list.\n> \n> Otherwise, C has a list of maximal commits that contributed bits to its\n> bitmask (and this list is exactly one element). Add all of these items\n> to P's reverse_edges list. Be careful to ignore duplicates here.\n\nOK - the other end of reverse edges are always to maximal commits.\nPropagation in this way (if C is not maximal) makes sense.\n\n> After inspecting all parents P for a commit C, we can clear the\n> commit_mask for C. This reduces the memory load to be limited to the\n> \"width\" of the commit graph.\n\nOptimization - OK.\n\nI might as well finish working through the example. Let's start from the\nbeginning.\n\n I J I:10()M J:01()M\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N\n \\ /\n |/\n Q\n\n(Brackets are the destinations of reverse edges. M is maximal, ~M is not\nmaximal.)\n\nIteration starts with I. I is maximal, but it is also one of the\nselected commits, so we need not do anything else. Look at its parents,\nstarting with M. c_of_p is true, so copy the bits over. p_of_c is false,\nso the maximal bit of M is zero, clear all its reverse edges (no-op in\nthis case, since it has none), and update its reverse edges: C is\nmaximal so it will just be C. The procedure for N is exactly the same.\nSo we have:\n\n I J I:10()M J:01()M\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N M:10(I)~M N:10(I)~M\n \\ /\n |/\n Q\n\nNow onto J. J is maximal, but it is also one of the selected commits, so\nwe need not do anything else. Look at its parent M. c_of_p is true, so\ncopy the bits over. p_of_c is true this time. So set the maximal bit to\ntrue. (Indeed, M has information that is independent of J - the stuff\nthat it got from I.) We need not clear any reverse edges, but must still\nupdate them: J is maximal so we add J. The procedure for N is exactly\nthe same. So we have:\n\n I J I:10()M J:01()M\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N M:11(I,J)M N:11(I,J)M\n \\ /\n |/\n Q\n\nLet's go to M. M is maximal, and it is not one of the selected commits,\nso widen the commit_mask and set the corresponding bit on M. Look at its\nonly parent Q. c_of_p is true, so copy the bits over. p_of_c is false,\nso the maximal bit of Q is zero, and update its reverse edges as usual.\n\n I J I:10()M J:01()M\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N M:111(I,J)M N:11(I,J)M\n \\ /\n |/\n Q Q:111(M)~M\n\nNow, N. N is maximal, and it is not one of the selected commits, so\nwiden the commit_mask and set the corresponding bit on N. Look at its\nonly parent Q. c_of_p is true; copy bits; but this time p_of_c is true,\nso set the maximal bit to true. Don't clear reverse edges; N is maximal\nso add it.\n\n I J I:10()M J:01()M\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N M:111(I,J)M N:1101(I,J)M\n \\ /\n |/\n Q Q:1111(M,N)M\n\nChecking the answer below, it looks like the commit author and I agree.\n\n> Consider our ABC/XYZ example from earlier and let's inspect the state of\n> the commits for an interesting bitmask, say 011. Suppose that D is the\n> only maximal commit with this bitmask (in the first three bits). All\n> other commits with bitmask 011 have D as the only entry in their\n> reverse_edges list. D's reverse_edges list contains B and C.\n\nYes, this makes sense.\n\nLet me write about \"D's reverse_edges list contains B and C\" first: The\nfact that D has a bitmask of 011 shows no important commits can reach it\nother than B or C. Any intermediate commits between B and D would not be\nmaximal (because B is \"greater\" than such an intermediate commit, and we\nalready established that no other important commit can reach D, and\ntherefore no other important commit can reach any of the commits on the\npath between B and D). Same analysis applies for C instead of B. So the\npropagated reverse edges would just be B and C.\n\nNow \"all other commits with bitmask 011 have D as the only entry in\ntheir reverse_edges list\". Since D is the only maximal commit with 011,\nany other commit that has 011 (1) must have got it from D, and (2)\ncannot be reached by any other important commit. A similar analysis as\nin the previous paragraph shows why the reverse_edges list for all these\ncommits would only contain D.\n\n> COMPUTING REACHABILITY BITMAPS\n> ------------------------------\n> \n> Now that we have our definition, let's zoom out and consider what\n> happens with our new reverse graph when computing reachability bitmaps.\n> We walk the reverse graph in reverse-topo-order, so we visit commits\n> with largest commit_masks first. After we compute the reachability\n> bitmap for a commit C, we push the bits in that bitmap to each commit D\n> in the reverse edge list for C. \n\nThat makes sense - the reachability bitmap for D is the reachability\nbitmap for C + the reachability bitmap of (D-C). Here we have the first\noperand.\n\n> Then, when we finally visit D we already\n> have the bits for everything reachable from maximal commits that D can\n> reach and we only need to walk the objects in the set-difference.\n\nWalking the objects in the set-difference gives us the second operand.\nMakes sense.\n\n> In our ABC/XYZ example, when we finally walk for the commit A we only\n> need to walk commits with bitmask equal to A's bitmask. If that bitmask\n> is 100, then we are only walking commits in X - (Y union Z) because the\n> bitmap already contains the bits for objects reachable from (X intersect\n> Y) union (X intersect Z) (i.e. the bits from the reachability bitmaps\n> for the maximal commits with bitmasks 110 and 101).\n\nThis is probably correct, but I've lost track of what X, Y, and Z are so\nI'll just skip this paragraph.\n\n> The behavior is intended to walk each commit (and the trees that commit\n> introduces) at most once while allocating and copying fewer reachability\n> bitmaps. \n\nYes, I can see how this algorithm causes this behavior.\n\n> There is one caveat: what happens when there are multiple\n> maximal commits with the same bitmask, with respect to the initial set\n> of selected commits?\n\nI'm going to be lazy here and ask, is this possible? As described\nbelow in \"MULTIPLE MAXIMAL COMMITS\", if a non-selected commit turns out\nto be maximal, it will have its very own bit, and thus become the\n\"progenitor\" of all commits with that bit set. (This is not a true\nprogenitor, because this bit propagates from the children to the\nparents and not the other way round - unlike a gene.)\n\n> MULTIPLE MAXIMAL COMMITS\n> ------------------------\n\nI think I discussed everything here earlier, so [skip].\n\n> PERFORMANCE MEASUREMENTS\n> ------------------------\n\nNumbers look good. [skip]\n\nAs discussed above, I'll not review the code this round. [skip code]\n\nPhew...this took longer than expected. I'll see if I can review the rest\nof the patches tomorrow.\n"},{"id":"410700","messageId":"xmqqh7pejz07.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"X7x3WtCItVGhQ57O@coredump.intra.peff.net","subject":"Re: [PATCH 07/23] ewah: make bitmap growth less aggressive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-24T20:11:52Z","receivedAt":"2020-11-24T20:12:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Mon, Nov 23, 2020 at 11:49:49AM -0500, Taylor Blau wrote:\n>\n>> On Sun, Nov 22, 2020 at 12:32:01PM -0800, Junio C Hamano wrote:\n>> > Taylor Blau writes:\n>> >\n>> > > - a geometric increase in existing size; we'll switch to 3/2 instead of\n>> > > 2 here. That's less aggressive and may help avoid fragmenting memory\n>> > > (N + 3N/2 > 9N/4, so old chunks can be reused as we scale up).\n>> >\n>> > I am sure this is something obvious to bitmap folks, but where does\n>> > 9N/4 come from (I get that the left-hand-side of the comparison is\n>> > the memory necessary to hold both the old and the new copy while\n>> > reallocating the words[] array)?\n>> \n>> I thought that I was in the group of \"bitmap folks\", but since it's not\n>> obvious to me either, I guess I'll have to hand in my bitmap folks\n>> membership card ;).\n>> \n>> Peff: where does 9N/4 come from?\n>\n> it is not a bitmap thing at all. We are growing a buffer, so if we\n> continually multiply it by 3/2, then our sequence of sizes is:\n>\n> - before growth: N\n> - after 1 growth: 3N/2\n> - after 2 growths: 9N/4\n\nAH, OK. I feel stupid not to have thought of this myself X-<.\n\nThanks.\n"},{"id":"410740","messageId":"20201125005344.935924-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"c9512067293c082ad3082262e50dfd04f1bc1648.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/24] pack-bitmap-write: reimplement bitmap writing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-25T00:53:44Z","receivedAt":"2020-11-25T00:54:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"[snip commit message]\n\nThanks for the very clear commit message explaining the new algorithm.\n\n> +struct bb_commit {\n> +\tstruct commit_list *children;\n> +\tstruct bitmap *bitmap;\n> +\tunsigned selected:1;\n> +\tunsigned idx; /* within selected array */\n> +};\n> +\n> +define_commit_slab(bb_data, struct bb_commit);\n> +\n> +struct bitmap_builder {\n> +\tstruct bb_data data;\n> +\tstruct commit **commits;\n> +\tsize_t commits_nr, commits_alloc;\n> +};\n> +\n> +static void bitmap_builder_init(struct bitmap_builder *bb,\n> +\t\t\t\tstruct bitmap_writer *writer)\n> {\n> \tstruct rev_info revs;\n> +\tstruct commit *commit;\n> +\tunsigned int i;\n> +\n> +\tmemset(bb, 0, sizeof(*bb));\n> +\tinit_bb_data(&bb->data);\n> +\n> +\treset_revision_walk();\n> +\trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n> +\trevs.topo_order = 1;\n> +\n> +\tfor (i = 0; i < writer->selected_nr; i++) {\n> +\t\tstruct commit *c = writer->selected[i].commit;\n> +\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n> +\t\tent->selected = 1;\n> +\t\tent->idx = i;\n> +\t\tadd_pending_object(&revs, &c->object, \"\");\n> +\t}\n> +\n> +\tif (prepare_revision_walk(&revs))\n> +\t\tdie(\"revision walk setup failed\");\n> +\n> +\twhile ((commit = get_revision(&revs))) {\n> +\t\tstruct commit_list *p;\n> +\n> +\t\tparse_commit_or_die(commit);\n> +\n> +\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n> +\t\tbb->commits[bb->commits_nr++] = commit;\n> +\n> +\t\tfor (p = commit->parents; p; p = p->next) {\n> +\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n> +\t\t\tcommit_list_insert(commit, &ent->children);\n> +\t\t}\n> +\t}\n> +}\n\nLooks straightforward.\n\n> +static void bitmap_builder_clear(struct bitmap_builder *bb)\n> +{\n> +\tclear_bb_data(&bb->data);\n> +\tfree(bb->commits);\n> +\tbb->commits_nr = bb->commits_alloc = 0;\n> +}\n\nI was wondering why the commit list and the children in struct bb_commit\nweren't cleared, but that's because they are cleared during the\nalgorithm. So this is fine.\n\n> +static void fill_bitmap_tree(struct bitmap *bitmap,\n> +\t\t\t struct tree *tree)\n> +{\n> +\tuint32_t pos;\n> +\tstruct tree_desc desc;\n> +\tstruct name_entry entry;\n> +\n> +\t/*\n> +\t * If our bit is already set, then there is nothing to do. Both this\n> +\t * tree and all of its children will be set.\n> +\t */\n> +\tpos = find_object_pos(&tree->object.oid);\n> +\tif (bitmap_get(bitmap, pos))\n> +\t\treturn;\n> +\tbitmap_set(bitmap, pos);\n> +\n> +\tif (parse_tree(tree) < 0)\n> +\t\tdie(\"unable to load tree object %s\",\n> +\t\t oid_to_hex(&tree->object.oid));\n> +\tinit_tree_desc(&desc, tree->buffer, tree->size);\n> +\n> +\twhile (tree_entry(&desc, &entry)) {\n> +\t\tswitch (object_type(entry.mode)) {\n> +\t\tcase OBJ_TREE:\n> +\t\t\tfill_bitmap_tree(bitmap,\n> +\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n> +\t\t\tbreak;\n> +\t\tcase OBJ_BLOB:\n> +\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n> +\t\t\tbreak;\n> +\t\tdefault:\n> +\t\t\t/* Gitlink, etc; not reachable */\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n> +\tfree_tree_buffer(tree);\n> +}\n\nLooks straightforward.\n\n> +static void fill_bitmap_commit(struct bb_commit *ent,\n> +\t\t\t struct commit *commit)\n> +{\n> +\tif (!ent->bitmap)\n> +\t\tent->bitmap = bitmap_new();\n> +\n> +\t/*\n> +\t * mark ourselves, but do not bother with parents; their values\n> +\t * will already have been propagated to us\n> +\t */\n> +\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n> +\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n> +}\n\nOK - when filling the bitmap for a commit, we only set the specific bit\nfor the commit itself, and all the bits for the commit's tree and the\ntree's descendants. This is consistent with the explanation of the\nalgorithm in the commit message.\n\n> +static void store_selected(struct bb_commit *ent, struct commit *commit)\n> +{\n> +\tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n> +\tkhiter_t hash_pos;\n> +\tint hash_ret;\n> +\n> +\t/*\n> +\t * the \"reuse bitmaps\" phase may have stored something here, but\n> +\t * our new algorithm doesn't use it. Drop it.\n> +\t */\n> +\tif (stored->bitmap)\n> +\t\tewah_free(stored->bitmap);\n\nI tried to figure out how the \"reuse bitmaps\" phase stores things in\nthis field, but that led me down a rabbit hole that I didn't pursue.\nBut anyway, the new bitmap is correctly generated, so clearing the old\nbitmap is safe (except, possibly, wasting time, but I see that in a\nsubsequent patch, existing bitmaps will be reused in a new wawy).\n\n> +\n> +\tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n> +\n> +\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n> +\tif (hash_ret == 0)\n> +\t\tdie(\"Duplicate entry when writing index: %s\",\n> +\t\t oid_to_hex(&commit->object.oid));\n> +\tkh_value(writer.bitmaps, hash_pos) = stored;\n> +}\n> +\n> +void bitmap_writer_build(struct packing_data *to_pack)\n> +{\n> +\tstruct bitmap_builder bb;\n> +\tsize_t i;\n> +\tint nr_stored = 0; /* for progress */\n> \n> \twriter.bitmaps = kh_init_oid_map();\n> \twriter.to_pack = to_pack;\n> \n> \tif (writer.show_progress)\n> \t\twriter.progress = start_progress(\"Building bitmaps\", writer.selected_nr);\n> +\ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n> +\t\tthe_repository);\n> +\n> +\tbitmap_builder_init(&bb, &writer);\n> +\tfor (i = bb.commits_nr; i > 0; i--) {\n> +\t\tstruct commit *commit = bb.commits[i-1];\n> +\t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n> +\t\tstruct commit *child;\n> +\n> +\t\tfill_bitmap_commit(ent, commit);\n> +\n> +\t\tif (ent->selected) {\n> +\t\t\tstore_selected(ent, commit);\n> +\t\t\tnr_stored++;\n> +\t\t\tdisplay_progress(writer.progress, nr_stored);\n> +\t\t}\n> +\n> +\t\twhile ((child = pop_commit(&ent->children))) {\n\nHere the children (specifically, the struct commit_list) are freed (one\nby one).\n\n> +\t\t\tstruct bb_commit *child_ent =\n> +\t\t\t\tbb_data_at(&bb.data, child);\n> +\n> +\t\t\tif (child_ent->bitmap)\n> +\t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n> +\t\t\telse\n> +\t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n> +\t\t}\n> +\t\tbitmap_free(ent->bitmap);\n> +\t\tent->bitmap = NULL;\n\nHere the bitmap is freed.\n\n> \t}\n> +\tbitmap_builder_clear(&bb);\n> \n> \tstop_progress(&writer.progress);\n> \n> \tcompute_xor_offsets();\n\nThanks - overall this looks straightforward.\n"},{"id":"410741","messageId":"20201125010022.938266-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"466dd3036a8ca7dc9718a53f17cf87e0eb22c066.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-25T01:00:22Z","receivedAt":"2020-11-25T01:00:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index f2f0b6b2c2..d2d46ff5f4 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -333,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \t\tstruct commit *commit = bb.commits[i-1];\n> \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n> \t\tstruct commit *child;\n> +\t\tint reused = 0;\n> \n> \t\tfill_bitmap_commit(ent, commit);\n> \n\nBefore the following chunk is the start of a loop:\n\n\"while ((child = pop_commit(&ent->children))) {\"\n\n> @@ -348,10 +349,15 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \n> \t\t\tif (child_ent->bitmap)\n> \t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n> -\t\t\telse\n> +\t\t\telse if (reused)\n> \t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n> +\t\t\telse {\n> +\t\t\t\tchild_ent->bitmap = ent->bitmap;\n> +\t\t\t\treused = 1;\n> +\t\t\t}\n> \t\t}\n> -\t\tbitmap_free(ent->bitmap);\n> +\t\tif (!reused)\n> +\t\t\tbitmap_free(ent->bitmap);\n> \t\tent->bitmap = NULL;\n> \t}\n> \tbitmap_builder_clear(&bb);\n> -- \n> 2.29.2.312.gabc4d358d8\n\nSo this is clearly correct.\n\nI asked myself if this optimization is worth it when we're going to\ndrastically reduce the number of steps in patch 18, but I think that the\nanswer is still yes.\n"},{"id":"410742","messageId":"20201125011409.941639-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"8e5607929d66a3c808dbe3a06c312d0cda1ef568.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-25T01:14:09Z","receivedAt":"2020-11-25T01:14:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> The fill_bitmap_commit() method assumes that every parent of the given\n> commit is already part of the current bitmap. Instead of making that\n> assumption, let's walk parents until we reach commits already part of\n> the bitmap. Set the value for that parent immediately after querying to\n> save time doing double calls to find_object_pos() and to avoid inserting\n> the parent into the queue multiple times.\n\nI see from the later patches that this has no effect until the part\nwhere we can skip commits, but as Junio says [1], it's worth mentioning\nit here. Maybe something like:\n\n The fill_bitmap_commit() method assumes that every parent of the given\n commit is already part of the current bitmap. This is currently\n correct, but a subsequent patch will change the nature of the edges of\n the graph from parent-child to ancestor-descendant. In preparation for\n that, let's walk parents...\n\n> static void fill_bitmap_commit(struct bb_commit *ent,\n> -\t\t\t struct commit *commit)\n> +\t\t\t struct commit *commit,\n> +\t\t\t struct prio_queue *queue)\n\nAs far as I can see, this function expects an empty queue and always\nends with the queue empty, and the only reason why we don't instantiate\na new queue every time is so that we can save on the internal array\nallocation/deallocation. Maybe add a comment to that effect.\n\n> {\n> \tif (!ent->bitmap)\n> \t\tent->bitmap = bitmap_new();\n> \n> -\t/*\n> -\t * mark ourselves, but do not bother with parents; their values\n> -\t * will already have been propagated to us\n> -\t */\n> \tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n> -\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n> +\tprio_queue_put(queue, commit);\n> +\n> +\twhile (queue->nr) {\n> +\t\tstruct commit_list *p;\n> +\t\tstruct commit *c = prio_queue_get(queue);\n> +\n> +\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n> +\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n> +\n> +\t\tfor (p = c->parents; p; p = p->next) {\n> +\t\t\tint pos = find_object_pos(&p->item->object.oid);\n> +\t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n> +\t\t\t\tbitmap_set(ent->bitmap, pos);\n> +\t\t\t\tprio_queue_put(queue, p->item);\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> }\n\n[snip rest of code]\n\nEverything else makes sense.\n"},{"id":"410743","messageId":"20201125011748.944638-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"8b5d23933301988e42ddd57687e0535f8749367f.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 15/24] t5310: add branch-based checks","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-25T01:17:48Z","receivedAt":"2020-11-25T01:17:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> The current rev-list tests that check the bitmap data only work on HEAD\n> instead of multiple branches. Expand the test cases to handle both\n> 'master' and 'other' branches.\n\n[snip]\n\n> +rev_list_tests () {\n> +\tstate=$1\n> +\n> +\tfor branch in \"master\" \"other\"\n\nWould it be worth including \"HEAD\" in the list here? It would make more\nsense with the commit message saying \"exstend\" instead of \"replace\".\n\n[snip rest]\n\nThe 2 prior patches (13 and 14) look good to me.\n"},{"id":"410744","messageId":"20201125014633.951649-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"5262daa3300114fbaccdbc7393882c5435f95f4f.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 18/24] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-25T01:46:33Z","receivedAt":"2020-11-25T01:46:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I've reviewed the commit message already [1]; now let's look at the code.\n\n[1] https://lore.kernel.org/git/20201124060738.762751-1-jonathantanmy@google.com/\n\n> struct bb_commit {\n> \tstruct commit_list *reverse_edges;\n> +\tstruct bitmap *commit_mask;\n> \tstruct bitmap *bitmap;\n> -\tunsigned selected:1;\n> +\tunsigned selected:1,\n> +\t\t maximal:1;\n\nThe code itself probably should contain comments about the algorithm,\nbut I'm not sure of the best way to do it. (E.g. I would say that\n\"maximal\" should be \"When iteration in bitmap_builder_init() reaches\nthis bb_commit, this is true iff none of its descendants has or will\never have the exact same commit_mask\" - but then when do we explain why\nthe commit_mask matters?)\n\n> \tunsigned idx; /* within selected array */\n> };\n> \n> @@ -198,7 +200,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n> {\n> \tstruct rev_info revs;\n> \tstruct commit *commit;\n> -\tunsigned int i;\n> +\tunsigned int i, num_maximal;\n> \n> \tmemset(bb, 0, sizeof(*bb));\n> \tinit_bb_data(&bb->data);\n> @@ -210,27 +212,85 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n> \tfor (i = 0; i < writer->selected_nr; i++) {\n> \t\tstruct commit *c = writer->selected[i].commit;\n> \t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n> +\n> \t\tent->selected = 1;\n> +\t\tent->maximal = 1;\n> \t\tent->idx = i;\n> +\n> +\t\tent->commit_mask = bitmap_new();\n> +\t\tbitmap_set(ent->commit_mask, i);\n> +\n> \t\tadd_pending_object(&revs, &c->object, \"\");\n> \t}\n> +\tnum_maximal = writer->selected_nr;\n> \n> \tif (prepare_revision_walk(&revs))\n> \t\tdie(\"revision walk setup failed\");\n> \n> \twhile ((commit = get_revision(&revs))) {\n> \t\tstruct commit_list *p;\n> +\t\tstruct bb_commit *c_ent;\n> \n> \t\tparse_commit_or_die(commit);\n> \n> -\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n> -\t\tbb->commits[bb->commits_nr++] = commit;\n> +\t\tc_ent = bb_data_at(&bb->data, commit);\n> +\n> +\t\tif (c_ent->maximal) {\n> +\t\t\tif (!c_ent->selected) {\n> +\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n> +\t\t\t\tnum_maximal++;\n> +\t\t\t}\n> +\n> +\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n> +\t\t\tbb->commits[bb->commits_nr++] = commit;\n\nSo the order of bit assignments in the commit_mask and the order of\ncommits in bb->commits are not the same. In the commit_mask, bits are\nfirst assigned for selected commits and then the rest for commits we\ndiscover to be maximal. But in bb->commits, the order follows the\ntopologically-sorted iteration. This is fine, but might be worth a\ncomment (to add to the already big comment burden...)\n\n> +\t\t}\n> \n> \t\tfor (p = commit->parents; p; p = p->next) {\n> -\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n> -\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n> +\t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n> +\t\t\tint c_not_p, p_not_c;\n> +\n> +\t\t\tif (!p_ent->commit_mask) {\n> +\t\t\t\tp_ent->commit_mask = bitmap_new();\n> +\t\t\t\tc_not_p = 1;\n> +\t\t\t\tp_not_c = 0;\n> +\t\t\t} else {\n> +\t\t\t\tc_not_p = bitmap_diff_nonzero(c_ent->commit_mask, p_ent->commit_mask);\n> +\t\t\t\tp_not_c = bitmap_diff_nonzero(p_ent->commit_mask, c_ent->commit_mask);\n> +\t\t\t}\n> +\n> +\t\t\tif (!c_not_p)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tbitmap_or(p_ent->commit_mask, c_ent->commit_mask);\n> +\n> +\t\t\tif (p_not_c)\n> +\t\t\t\tp_ent->maximal = 1;\n> +\t\t\telse {\n> +\t\t\t\tp_ent->maximal = 0;\n> +\t\t\t\tfree_commit_list(p_ent->reverse_edges);\n> +\t\t\t\tp_ent->reverse_edges = NULL;\n> +\t\t\t}\n> +\n> +\t\t\tif (c_ent->maximal) {\n> +\t\t\t\tcommit_list_insert(commit, &p_ent->reverse_edges);\n> +\t\t\t} else {\n> +\t\t\t\tstruct commit_list *cc = c_ent->reverse_edges;\n> +\n> +\t\t\t\tfor (; cc; cc = cc->next) {\n> +\t\t\t\t\tif (!commit_list_contains(cc->item, p_ent->reverse_edges))\n> +\t\t\t\t\t\tcommit_list_insert(cc->item, &p_ent->reverse_edges);\n> +\t\t\t\t}\n> +\t\t\t}\n> \t\t}\n> +\n> +\t\tbitmap_free(c_ent->commit_mask);\n> +\t\tc_ent->commit_mask = NULL;\n> \t}\n> +\n> +\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n> +\t\t\t \"num_selected_commits\", writer->selected_nr);\n> +\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n> +\t\t\t \"num_maximal_commits\", num_maximal);\n> }\n\nThe rest looks like a faithful implementation of the algorithm.\n\nNow let's look at the tests.\n\n> +# To ensure the logic for \"maximal commits\" is exercised, make\n> +# the repository a bit more complicated.\n> +#\n> +# other master\n> +# * *\n> +# (99 commits) (99 commits)\n> +# * *\n> +# |\\ /|\n> +# | * octo-other octo-master * |\n> +# |/|\\_________ ____________/|\\|\n> +# | \\ \\/ __________/ |\n> +# | | ________/\\ / |\n> +# * |/ * merge-right *\n> +# | _|__________/ \\____________ |\n> +# |/ | \\|\n> +# (l1) * * merge-left * (r1)\n> +# | / \\________________________ |\n> +# |/ \\|\n> +# (l2) * * (r2)\n> +# \\____________...____________ |\n\nWhat does the ... represent? If a certain number of commits, it would be\nclearer to write that there.\n\n> +# \\|\n> +# * (base)\n\nOK - some of the crosses are unclear, but from the bitmask given below,\nI know where the lines should go.\n\n> +#\n> +# The important part for the maximal commit algorithm is how\n> +# the bitmasks are extended. Assuming starting bit positions\n> +# for master (bit 0) and other (bit 1), and some flexibility\n> +# in the order that merge bases are visited, the bitmasks at\n> +# the end should be:\n> +#\n> +# master: 1 (maximal, selected)\n> +# other: 01 (maximal, selected)\n> +# octo-master: 1\n> +# octo-other: 01\n> +# merge-right: 111 (maximal)\n> +# (l1): 111\n> +# (r1): 111\n> +# merge-left: 1101 (maximal)\n> +# (l2): 11111 (maximal)\n> +# (r2): 111101 (maximal)\n> +# (base): 1111111 (maximal)\n\nThis makes sense. (l1) and (r1) are not maximal because everything that\ncan reach merge-right can also reach them.\n\n[snip]\n\n> test_expect_success 'full repack creates bitmaps' '\n> -\tgit repack -ad &&\n> +\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n> +\t\tgit repack -ad &&\n> \tls .git/objects/pack/ | grep bitmap >output &&\n> -\ttest_line_count = 1 output\n> +\ttest_line_count = 1 output &&\n> +\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n> +\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n\nFrom the diagram and bit masks, I see that the important number for\n\"maximal\" is 7. Could this test be run twice - one without the crosses\nand one with, and we can verify that the difference between the maximal\ncommits is 7? As it is, this 111 number is hard to verify.\n"},{"id":"410953","messageId":"X8KIm7siBMKR2W5S@nand.local","threadId":"54622","inReplyTo":"20201125005344.935924-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 10/24] pack-bitmap-write: reimplement bitmap writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-28T17:27:55Z","receivedAt":"2020-11-28T22:17:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 24, 2020 at 04:53:44PM -0800, Jonathan Tan wrote:\n> > +static void store_selected(struct bb_commit *ent, struct commit *commit)\n> > +{\n> > +\tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n> > +\tkhiter_t hash_pos;\n> > +\tint hash_ret;\n> > +\n> > +\t/*\n> > +\t * the \"reuse bitmaps\" phase may have stored something here, but\n> > +\t * our new algorithm doesn't use it. Drop it.\n> > +\t */\n> > +\tif (stored->bitmap)\n> > +\t\tewah_free(stored->bitmap);\n>\n> I tried to figure out how the \"reuse bitmaps\" phase stores things in\n> this field, but that led me down a rabbit hole that I didn't pursue.\n> But anyway, the new bitmap is correctly generated, so clearing the old\n> bitmap is safe (except, possibly, wasting time, but I see that in a\n> subsequent patch, existing bitmaps will be reused in a new wawy).\n\nYep. The existing reuse mechanism is thrown out in a later patch.\n\n> Thanks - overall this looks straightforward.\n\nThanks for taking a look! Very much appreciated.\n\nTaylor\n"},{"id":"410954","messageId":"X8KJPwwG4eD+9TGW@nand.local","threadId":"54622","inReplyTo":"20201125011748.944638-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 15/24] t5310: add branch-based checks","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-28T17:30:39Z","receivedAt":"2020-11-28T22:19:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 24, 2020 at 05:17:48PM -0800, Jonathan Tan wrote:\n> > From: Derrick Stolee \n> >\n> > The current rev-list tests that check the bitmap data only work on HEAD\n> > instead of multiple branches. Expand the test cases to handle both\n> > 'master' and 'other' branches.\n>\n> [snip]\n>\n> > +rev_list_tests () {\n> > +\tstate=$1\n> > +\n> > +\tfor branch in \"master\" \"other\"\n>\n> Would it be worth including \"HEAD\" in the list here? It would make more\n> sense with the commit message saying \"exstend\" instead of \"replace\".\n\nI don't think so. These tests were run with master checked out, so\ntesting HEAD would be no different than including \"master\" in the list\nof branches here, so the commit message is correct that this is an\nextension, too.\n\n> [snip rest]\n>\n> The 2 prior patches (13 and 14) look good to me.\n\nThanks for taking a look.\n\nThanks,\nTaylor\n"},{"id":"410955","messageId":"X8KHEMl87sU5cIGK@nand.local","threadId":"54622","inReplyTo":"20201125011409.941639-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-11-28T17:21:20Z","receivedAt":"2020-11-28T22:19:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 24, 2020 at 05:14:09PM -0800, Jonathan Tan wrote:\n> > From: Derrick Stolee \n> >\n> > The fill_bitmap_commit() method assumes that every parent of the given\n> > commit is already part of the current bitmap. Instead of making that\n> > assumption, let's walk parents until we reach commits already part of\n> > the bitmap. Set the value for that parent immediately after querying to\n> > save time doing double calls to find_object_pos() and to avoid inserting\n> > the parent into the queue multiple times.\n>\n> I see from the later patches that this has no effect until the part\n> where we can skip commits, but as Junio says [1], it's worth mentioning\n> it here. Maybe something like:\n>\n> The fill_bitmap_commit() method assumes that every parent of the given\n> commit is already part of the current bitmap. This is currently\n> correct, but a subsequent patch will change the nature of the edges of\n> the graph from parent-child to ancestor-descendant. In preparation for\n> that, let's walk parents...\n\nThanks. Stolee and I worked a little on revising this last week, and I\nthink that the current log message is more along these lines. Here's\nwhat we wrote:\n\n pack-bitmap-write: fill bitmap with commit history\n\n The current implementation of bitmap_writer_build() creates a\n reachability bitmap for every walked commit. After computing a bitmap\n for a commit, those bits are pushed to an in-progress bitmap for its\n children.\n\n fill_bitmap_commit() assumes the bits corresponding to objects\n reachable from the parents of a commit are already set. This means that\n when visiting a new commit, we only have to walk the objects reachable\n between it and any of its parents.\n\n A future change to bitmap_writer_build() will relax this condition so\n not all parents have their reachable objects set in the in-progress\n bitmap. Prepare for that by having 'fill_bitmap_commit()' walk\n parents until reaching commits whose bits are already set. Then, walk\n the trees for these commits as well.\n\n This has no functional change with the current implementation of\n bitmap_writer_build().\n\n> > static void fill_bitmap_commit(struct bb_commit *ent,\n> > -\t\t\t struct commit *commit)\n> > +\t\t\t struct commit *commit,\n> > +\t\t\t struct prio_queue *queue)\n>\n> As far as I can see, this function expects an empty queue and always\n> ends with the queue empty, and the only reason why we don't instantiate\n> a new queue every time is so that we can save on the internal array\n> allocation/deallocation. Maybe add a comment to that effect.\n\nSure. Would you find a comment like that more helpful above\n'fill_bitmap_commit()', or above the declaration of 'queue' (in\n'bitmap_writer_build()') itself?\n\nThanks,\nTaylor\n"},{"id":"411004","messageId":"20201130183358.2965225-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"X8KHEMl87sU5cIGK@nand.local","subject":"Re: [PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-30T18:33:58Z","receivedAt":"2020-11-30T18:34:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Thanks. Stolee and I worked a little on revising this last week, and I\n> think that the current log message is more along these lines. Here's\n> what we wrote:\n> \n> pack-bitmap-write: fill bitmap with commit history\n> \n> The current implementation of bitmap_writer_build() creates a\n> reachability bitmap for every walked commit. After computing a bitmap\n> for a commit, those bits are pushed to an in-progress bitmap for its\n> children.\n> \n> fill_bitmap_commit() assumes the bits corresponding to objects\n> reachable from the parents of a commit are already set. This means that\n> when visiting a new commit, we only have to walk the objects reachable\n> between it and any of its parents.\n> \n> A future change to bitmap_writer_build() will relax this condition so\n> not all parents have their reachable objects set in the in-progress\n\nI would write \"not all parents have their bits set\" instead, but this is\nfine too.\n\n> bitmap. Prepare for that by having 'fill_bitmap_commit()' walk\n> parents until reaching commits whose bits are already set. Then, walk\n> the trees for these commits as well.\n> \n> This has no functional change with the current implementation of\n> bitmap_writer_build().\n> \n> > > static void fill_bitmap_commit(struct bb_commit *ent,\n> > > -\t\t\t struct commit *commit)\n> > > +\t\t\t struct commit *commit,\n> > > +\t\t\t struct prio_queue *queue)\n> >\n> > As far as I can see, this function expects an empty queue and always\n> > ends with the queue empty, and the only reason why we don't instantiate\n> > a new queue every time is so that we can save on the internal array\n> > allocation/deallocation. Maybe add a comment to that effect.\n> \n> Sure. Would you find a comment like that more helpful above\n> 'fill_bitmap_commit()', or above the declaration of 'queue' (in\n> 'bitmap_writer_build()') itself?\n\nI think it's better with fill_bitmap_commit(), as it's the one in\ncontrol of how \"queue\" will be used.\n"},{"id":"411005","messageId":"ea0c8c5d-6bc3-0dca-4fa1-fb461ed7ccb9@gmail.com","threadId":"54622","inReplyTo":"20201125014633.951649-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 18/24] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-30T18:41:44Z","receivedAt":"2020-11-30T18:42:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/24/2020 8:46 PM, Jonathan Tan wrote:\n> I've reviewed the commit message already [1]; now let's look at the code.\n> \n> [1] https://lore.kernel.org/git/20201124060738.762751-1-jonathantanmy@google.com/\n> \n>> struct bb_commit {\n>> \tstruct commit_list *reverse_edges;\n>> +\tstruct bitmap *commit_mask;\n>> \tstruct bitmap *bitmap;\n>> -\tunsigned selected:1;\n>> +\tunsigned selected:1,\n>> +\t\t maximal:1;\n> \n> The code itself probably should contain comments about the algorithm,\n> but I'm not sure of the best way to do it. (E.g. I would say that\n> \"maximal\" should be \"When iteration in bitmap_builder_init() reaches\n> this bb_commit, this is true iff none of its descendants has or will\n> ever have the exact same commit_mask\" - but then when do we explain why\n> the commit_mask matters?)\n\nComments are tricky, as they are likely to go stale. In fact,\nthis algorithm changes dramatically later in this very series!\n\nHow much can we expect a reader to inspect the commit history to\ndiscover the lengthy commit message? The message explains the\nalgorithm and its many subtleties versus reading comments that may\nbe too specific to this initial version.\n\nAt this point, \"maximal\" is a property that doesn't mean much\nwithout inspecting the places where we set or check that bit.\n\n>> +\t\tif (c_ent->maximal) {\n>> +\t\t\tif (!c_ent->selected) {\n>> +\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n>> +\t\t\t\tnum_maximal++;\n>> +\t\t\t}\n>> +\n>> +\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n>> +\t\t\tbb->commits[bb->commits_nr++] = commit;\n> \n> So the order of bit assignments in the commit_mask and the order of\n> commits in bb->commits are not the same. In the commit_mask, bits are\n> first assigned for selected commits and then the rest for commits we\n> discover to be maximal. But in bb->commits, the order follows the\n> topologically-sorted iteration. This is fine, but might be worth a\n> comment (to add to the already big comment burden...)\n\nThis one I waffled on a bit. There _is_ a difference between the\nbitmask order and this list order. This isn't a problem as long as\nno one attempts to use the bitmasks to navigate into this list.\n\nFurther, it is entirely possible that the tests to never demonstrate\nthat difference, so pointing out may help a future developer from\nmaking that mistake. Could be good to add this comment over the\nbb->commits definition:\n\n\t/*\n\t * 'commits' stores the list of maximal commits, in visited\n\t * order. This can be different than the bitmask order for\n\t * the selected commits.\n\t */\n\n>> +# To ensure the logic for \"maximal commits\" is exercised, make\n>> +# the repository a bit more complicated.\n>> +#\n>> +# other master\n>> +# * *\n>> +# (99 commits) (99 commits)\n>> +# * *\n>> +# |\\ /|\n>> +# | * octo-other octo-master * |\n>> +# |/|\\_________ ____________/|\\|\n>> +# | \\ \\/ __________/ |\n>> +# | | ________/\\ / |\n>> +# * |/ * merge-right *\n>> +# | _|__________/ \\____________ |\n>> +# |/ | \\|\n>> +# (l1) * * merge-left * (r1)\n>> +# | / \\________________________ |\n>> +# |/ \\|\n>> +# (l2) * * (r2)\n>> +# \\____________...____________ |\n> \n> What does the ... represent? If a certain number of commits, it would be\n> clearer to write that there.\n\nThe ... are unnecessary and should be ___ instead. Thanks.\n \n>> +# \\|\n>> +# * (base)\n> \n> OK - some of the crosses are unclear, but from the bitmask given below,\n> I know where the lines should go.\n> \n>> +#\n>> +# The important part for the maximal commit algorithm is how\n>> +# the bitmasks are extended. Assuming starting bit positions\n>> +# for master (bit 0) and other (bit 1), and some flexibility\n>> +# in the order that merge bases are visited, the bitmasks at\n>> +# the end should be:\n>> +#\n>> +# master: 1 (maximal, selected)\n>> +# other: 01 (maximal, selected)\n>> +# octo-master: 1\n>> +# octo-other: 01\n>> +# merge-right: 111 (maximal)\n>> +# (l1): 111\n>> +# (r1): 111\n>> +# merge-left: 1101 (maximal)\n>> +# (l2): 11111 (maximal)\n>> +# (r2): 111101 (maximal)\n>> +# (base): 1111111 (maximal)\n> \n> This makes sense. (l1) and (r1) are not maximal because everything that\n> can reach merge-right can also reach them.\n> \n> [snip]\n> \n>> test_expect_success 'full repack creates bitmaps' '\n>> -\tgit repack -ad &&\n>> +\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n>> +\t\tgit repack -ad &&\n>> \tls .git/objects/pack/ | grep bitmap >output &&\n>> -\ttest_line_count = 1 output\n>> +\ttest_line_count = 1 output &&\n>> +\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n>> +\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n> \n> From the diagram and bit masks, I see that the important number for\n> \"maximal\" is 7. Could this test be run twice - one without the crosses\n> and one with, and we can verify that the difference between the maximal\n> commits is 7? As it is, this 111 number is hard to verify.\n\nThis number _is_ hard to verify. It is fragile to many behaviors inside\nthe code of Git, including the selection algorithm and some hard-coded\nlimits (there's a reason we insert 100 commits on top of each side).\nFurther, this number changes as the algorithm is modified.\n\nPerhaps the best way to recognize this number is that it changes by adding\n5 to the previous number (these are the 5 \"newly maximal\" commits, since\ntwo are already selected as tips).\n\nThis number changes again later, and the difference is justified by\nthe number of maximal commits dropping by 4.\n\nThanks,\n-Stolee\n"},{"id":"411115","messageId":"X8bKJUypb9WViUdt@nand.local","threadId":"54622","inReplyTo":"X7x1PciN+bVDkJhr@coredump.intra.peff.net","subject":"Re: [PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-01T22:56:37Z","receivedAt":"2020-12-01T22:57:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Nov 23, 2020 at 09:51:41PM -0500, Jeff King wrote:\n> On Mon, Nov 23, 2020 at 09:48:22PM -0500, Jeff King wrote:\n>\n> > > I think that we probably could just use ALLOC_GROW() as you suggest.\n> > > Funny enough, reading through GitHub's chat logs, apparently this is\n> > > something that Peff and I talked about. So, 16 probably came from\n> > > alloc_nr(), but we probably stopped short of realizing that we could\n> > > just use ALLOC_GROW as-is.\n> >\n> > That would probably be OK. It's a bit more aggressive, which could\n> > matter if you have a large number of very small bitmaps. My original\n> > goal of the \"grow less aggressively\" patch was to keep memory usage\n> > down, knowing that I was going to be holding a lot of bitmaps in memory\n> > at once. But even with micro-optimizations like this, it turned out to\n> > be far too big in practice (and hence Stolee's work on top to reduce the\n> > total number we hold at once).\n>\n> Oh, sorry, I was mixing this patch up with patches 6 and 7, which touch\n> buffer_grow(). This is a totally separate spot, and this is a pure\n> bug-fix.\n>\n> I think the main reason we didn't use ALLOC_GROW() here in the beginning\n> is that the ewah code was originally designed to be a separate library\n> (a port of the java ewah library), and didn't depend on Git code.\n>\n> These days we pull in xmalloc, etc, so we should be fine to use\n> ALLOC_GROW().\n>\n> Likewise...\n>\n> > I think the real test would be measuring the peak heap of the series as\n> > you posted it in v2, and this version replacing this patch (and the\n> > \"grow less aggressively\" one) with ALLOC_GROW(). On something big, like\n> > repacking all of the torvalds/linux or git/git fork networks.\n> >\n> > If there's no appreciable difference, then definitely I think it's worth\n> > the simplicity of reusing ALLOC_GROW().\n>\n> All of this is nonsense (though it does apply to the question of using\n> ALLOC_GROW() in bitmap_grow(), via patch 7).\n\nYou and I timed this a week or two ago, but I only just returned to this\ntopic today. Switching to ALLOC_GROW() doesn't affect the final memory\nusage at all, so I changed patch 7 up to use that instead of more or\nless open-coding alloc_nr().\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"411118","messageId":"X8bL7baFdXufk2jj@nand.local","threadId":"54622","inReplyTo":"X7xzWClGr3bM3wcg@coredump.intra.peff.net","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-01T23:04:13Z","receivedAt":"2020-12-01T23:05:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Nov 23, 2020 at 09:43:36PM -0500, Jeff King wrote:\n> On Sat, Nov 21, 2020 at 09:31:50PM -0500, Taylor Blau wrote:\n>\n> > > There was SZEDER's comment on that last patch in v2, where future\n> > > readers of that patch will have to wonder why it does s/256/270/ in a\n> > > test. I agree with SZEDER that the change should be mentioned in the\n> > > commit message, even if it's just \"unfortunately, we have some magicness\n> > > here, plus we want to pass both with SHA-1 and SHA-256; turns out 270\n> > > hits the problem we want to test for\".\n> >\n> > Thanks for reviewing it, and noticing a couple of problems in the\n> > earlier patches, too. If folks are happy with the replacement that I\n> > sent [1], then I am too :-).\n> >\n> > I don't think that the \"big\" patch generated a ton of review on the\n> > list, but maybe that's OK. Peff, Stolee, and I all reviewed that patch\n> > extensively when deploying it at GitHub (where it has been running since\n> > late Summer).\n>\n> Hrm. I thought you were going to integrate the extra checks I suggested\n> for load_bitmap_entries_v1(). Which is looks like you did in patch 17.\n> After that, the s/256/270/ hack should not be necessary anymore (if it\n> is, then we should keep fixing more spots).\n\nOops. I even wrote down a big \"S/256/270\" in my notebook after you and I\ntalked last about it, and then promptly forgot about it before sending\nv2.\n\nIn any case, I have all of that fixed up, as well as other comments and\nsuggestions from review, all of which were very helpful. (Thanks\neverybody who took a look at this monstrously large series, and\napologies in advance for more to come ;-)).\n\nI think I would like a little more clarification on the discussion in\n[1]. From my reading, Jonathan Tan wants comments about the algorithm in\nthe code, but Stolee would rather rely on the commits, especially since\nthe algorithm changes later on in the series relative to the patch that\nthis is downthread from.\n\nOnce we can reach a good decision there, I'll send a v3 (which currently\nlives in my fork[2]).\n\n> -Peff\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/ea0c8c5d-6bc3-0dca-4fa1-fb461ed7ccb9@gmail.com/\n[2]: https://github.com/ttaylorr/git/compare/tb/bitmap-build-fast-for-upstream\n"},{"id":"411122","messageId":"20201201233725.3396453-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"X8bL7baFdXufk2jj@nand.local","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-01T23:37:25Z","receivedAt":"2020-12-01T23:38:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> I think I would like a little more clarification on the discussion in\n> [1]. From my reading, Jonathan Tan wants comments about the algorithm in\n> the code, but Stolee would rather rely on the commits, especially since\n> the algorithm changes later on in the series relative to the patch that\n> this is downthread from.\n> \n> Once we can reach a good decision there, I'll send a v3 (which currently\n> lives in my fork[2]).\n\nI did, but Stolee has a point that the algorithm will change later on.\nI'm OK with the parts I reviewed (patches 10 to 18).\n"},{"id":"411123","messageId":"X8bVHQ1eyDkv2CLT@nand.local","threadId":"54622","inReplyTo":"20201201233725.3396453-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-01T23:43:25Z","receivedAt":"2020-12-01T23:44:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 01, 2020 at 03:37:25PM -0800, Jonathan Tan wrote:\n> > Once we can reach a good decision there, I'll send a v3 (which currently\n> > lives in my fork[2]).\n>\n> I did, but Stolee has a point that the algorithm will change later on.\n> I'm OK with the parts I reviewed (patches 10 to 18).\n\nAh, good to know. I'll hold off on a v3, then, until you have had a\nchance to look through the remaining handful of patches (if you were\nplanning on doing that). I haven't touched those locally, so it'll be\ngood to hear any comments you might have before sending another version.\n\nThanks for all of your very helpful review :-).\n\nTaylor\n"},{"id":"411142","messageId":"20201202071359.3466347-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"a206f486140097c1554ce92ebcfa7190554dd19f.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T07:13:59Z","receivedAt":"2020-12-02T07:14:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Just because a commit happened to be bitmapped last time does not make\n> it a good candidate for having a bitmap this time. In particular, we may\n> choose bitmaps based on how recent they are in history, or whether a ref\n> tip points to them, and those things will change. We're better off\n> re-considering fresh which commits are good candidates.\n> \n> Reusing the existing bitmap _is_ a reasonable thing to do to save\n> computation. But only reusing exact bitmaps is a weak form of this. If\n> we have an old bitmap for A and now want a new bitmap for its child, we\n> should be able to compute that only by looking at trees and that are new\n> to the child. But this code would consider only exact reuse (which is\n> perhaps why it was eager to select those commits in the first place).\n\nMakes sense.\n\n> -int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n> -\t\t\t struct packing_data *mapping,\n> -\t\t\t kh_oid_map_t *reused_bitmaps,\n> -\t\t\t int show_progress)\n> +uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n> +\t\t\t\tstruct packing_data *mapping)\n\n[snip body of function]\n\nHere, a lot of the function is deleted, and only the part that creates\nthe mapping from old indices to new indices remains - hence, the\nrenaming of the function. OK.\n\nOverall this looks good. I was wondering if there would be any functions\nnow unused, but looking at the deleted lines, that doesn't seem to be\nthe case.\n"},{"id":"411143","messageId":"20201202071753.3468604-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"9928b3c7da33edd8d4beae006a74dd682daf5fa5.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 20/24] pack-bitmap: factor out 'bitmap_for_commit()'","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T07:17:53Z","receivedAt":"2020-12-02T07:18:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n> +\t\t\t\t struct commit *commit)\n> +{\n> +\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n> +\t\t\t\t\t commit->object.oid);\n> +\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n> +\t\treturn NULL;\n> +\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n> +}\n\nThe new function.\n\n> static int add_to_include_set(struct bitmap_index *bitmap_git,\n> \t\t\t struct include_data *data,\n> -\t\t\t const struct object_id *oid,\n> +\t\t\t struct commit *commit,\n> \t\t\t int bitmap_pos)\n> {\n> -\tkhiter_t hash_pos;\n> +\tstruct ewah_bitmap *partial;\n> \n> \tif (data->seen && bitmap_get(data->seen, bitmap_pos))\n> \t\treturn 0;\n> @@ -476,10 +486,9 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n> \tif (bitmap_get(data->base, bitmap_pos))\n> \t\treturn 0;\n> \n> -\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, *oid);\n> -\tif (hash_pos < kh_end(bitmap_git->bitmaps)) {\n> -\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, hash_pos);\n> -\t\tbitmap_or_ewah(data->base, lookup_stored_bitmap(st));\n> +\tpartial = bitmap_for_commit(bitmap_git, commit);\n> +\tif (partial) {\n> +\t\tbitmap_or_ewah(data->base, partial);\n> \t\treturn 0;\n> \t}\n\nA straightforward mechanical change. The function invocation replaces\nconversion from commit to oid (which is why add_to_include_set() now\ntakes a struct commit * instead of a struct object_id *) and all the\nother deleted lines here.\n\n> @@ -1297,12 +1305,9 @@ void test_bitmap_walk(struct rev_info *revs)\n> \t\tbitmap_git->version, bitmap_git->entry_count);\n> \n> \troot = revs->pending.objects[0].item;\n> -\tpos = kh_get_oid_map(bitmap_git->bitmaps, root->oid);\n> -\n> -\tif (pos < kh_end(bitmap_git->bitmaps)) {\n> -\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n> -\t\tstruct ewah_bitmap *bm = lookup_stored_bitmap(st);\n> +\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n> \n> +\tif (bm) {\n> \t\tfprintf(stderr, \"Found bitmap for %s. %d bits / %08x checksum\\n\",\n> \t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n> \n\nSame here. LGTM.\n"},{"id":"411144","messageId":"20201202072040.3471532-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"f40a39a48a834443f76015821c0e56021b58fc9a.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()'","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T07:20:40Z","receivedAt":"2020-12-02T07:21:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n> +\t\t\t\tstruct bitmap **base,\n> +\t\t\t\tstruct commit *commit)\n> +{\n> +\tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n> +\n> +\tif (!or_with)\n> +\t\treturn 0;\n> +\n> +\tif (*base == NULL)\n> +\t\t*base = ewah_to_bitmap(or_with);\n> +\telse\n> +\t\tbitmap_or_ewah(*base, or_with);\n> +\n> +\treturn 1;\n> +}\n> +\n> static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n> \t\t\t\t struct rev_info *revs,\n> \t\t\t\t struct object_list *roots,\n> @@ -544,21 +561,10 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n> \t\tstruct object *object = roots->item;\n> \t\troots = roots->next;\n> \n> -\t\tif (object->type == OBJ_COMMIT) {\n> -\t\t\tkhiter_t pos = kh_get_oid_map(bitmap_git->bitmaps, object->oid);\n> -\n> -\t\t\tif (pos < kh_end(bitmap_git->bitmaps)) {\n> -\t\t\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n> -\t\t\t\tstruct ewah_bitmap *or_with = lookup_stored_bitmap(st);\n\nThe code from kh_get_oid_map() to lookup_stored_bitmap() here now\nexists, in add_commit_to_bitmap(), in the form of an invocation to\nbitmap_for_commit(). Which is correct - that is exactly what\nbitmap_for_commit() does.\n\nLooks good.\n"},{"id":"411145","messageId":"20201202072811.3474340-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"4bf5e78a54dfdcbe13dd66ba4c5955a159ea181d.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 22/24] pack-bitmap-write: use existing bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T07:28:11Z","receivedAt":"2020-12-02T07:28:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> When constructing new bitmaps, we perform a commit and tree walk in\n> fill_bitmap_commit() and fill_bitmap_tree(). This walk would benefit\n> from using existing bitmaps when available. We must track the existing\n> bitmaps and translate them into the new object order, but this is\n> generally faster than parsing trees.\n\nMakes sense.\n\n> In fill_bitmap_commit(), we must reorder thing somewhat. The priority\n> queue walks commits from newest-to-oldest, which means we correctly stop\n> walking when reaching a commit with a bitmap. \n\nMakes sense.\n\n> However, if we walk trees\n> from top to bottom, then we might be parsing trees that are actually\n> part of a re-used bitmap. \n\nIsn't the issue that we shouldn't walk trees at all before exhausting\nour commit search, not the direction that we walk the trees in (top to\nbottom or bottom to top or whatever)?\n\n> To avoid over-walking trees, add them to a\n> LIFO queue and walk them from bottom-to-top after exploring commits\n> completely.\n\nJust to clarify - would it work just as well with a FIFO queue (not LIFO\nqueue)? It seems to me that the most important part is doing this after\nexploring commits completely.\n\n> On git.git, this reduces a second immediate bitmap computation from 2.0s\n> to 1.0s. On linux.git, we go from 32s to 22s. On chromium's fork\n> network, we go from 227s to 198s.\n\nNice timings.\n\n> static void fill_bitmap_commit(struct bb_commit *ent,\n> \t\t\t struct commit *commit,\n> -\t\t\t struct prio_queue *queue)\n> +\t\t\t struct prio_queue *queue,\n> +\t\t\t struct prio_queue *tree_queue,\n> +\t\t\t struct bitmap_index *old_bitmap,\n> +\t\t\t const uint32_t *mapping)\n> {\n> \tif (!ent->bitmap)\n> \t\tent->bitmap = bitmap_new();\n> \n> -\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n> \tprio_queue_put(queue, commit);\n> \n> \twhile (queue->nr) {\n> \t\tstruct commit_list *p;\n> \t\tstruct commit *c = prio_queue_get(queue);\n> \n> +\t\t/*\n> +\t\t * If this commit has an old bitmap, then translate that\n> +\t\t * bitmap and add its bits to this one. No need to walk\n> +\t\t * parents or the tree for this commit.\n> +\t\t */\n\nThis comment should be right before \"if (old && ...\", I think. Here, it\nis a bit misleading. It leads me to think that \"this commit has an old\nbitmap\" means old_bitmap != NULL, but it is actually old != NULL.\n\n> +\t\tif (old_bitmap && mapping) {\n\nThis is defensive in that if we somehow calculate old_bitmap without\nmapping (or the other way around) (which is a bug), things just slow\ndown instead of breaking. I'm OK with this, but I still wanted to call\nit out.\n\n> +\t\t\tstruct ewah_bitmap *old;\n> +\n> +\t\t\told = bitmap_for_commit(old_bitmap, c);\n> +\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap))\n> +\t\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\t/*\n> +\t\t * Mark ourselves and queue our tree. The commit\n> +\t\t * walk ensures we cover all parents.\n> +\t\t */\n> \t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n> -\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n> +\t\tprio_queue_put(tree_queue, get_commit_tree(c));\n> \n> \t\tfor (p = c->parents; p; p = p->next) {\n> \t\t\tint pos = find_object_pos(&p->item->object.oid);\n\n[snip]\n\n> @@ -386,6 +408,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \tsize_t i;\n> \tint nr_stored = 0; /* for progress */\n> \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> +\tstruct prio_queue tree_queue = { NULL };\n\nNULL here does mean LIFO queue. OK.\n\n> @@ -395,6 +420,12 @@ void bitmap_writer_build(struct packing_data *to_pack)\n> \ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n> \t\tthe_repository);\n> \n> +\told_bitmap = prepare_bitmap_git(to_pack->repo);\n> +\tif (old_bitmap)\n> +\t\tmapping = create_bitmap_mapping(old_bitmap, to_pack);\n> +\telse\n> +\t\tmapping = NULL;\n\nHere, we prepare the old_bitmap and mapping arguments. OK.\n"},{"id":"411146","messageId":"20201202074439.3478090-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"1da4fa0fb85fe848aa86987e767b33d296f8f878.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T07:44:39Z","receivedAt":"2020-12-02T07:45:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> However, there was a (significant) drawback: wide histories with many\n> refs had an explosion of memory costs to compute the commit bitmasks\n> during the exploration that discovers these intermediate commits. Since\n> these wide histories are unlikely to repeat walking objects, the benefit\n> of walking objects multiple times was not expensive before. But now, the\n> commit walk *before computing bitmaps* is incredibly expensive.\n\nDo you have numbers of how large the commit bitmasks are?\n\n> In an effort to discover a happy medium, this change reduces the walk\n> for intermediate commits to only the first-parent history. This focuses\n> the walk on how the histories converge, which still has significant\n> reduction in repeat object walks. It is still possible to create\n> quadratic behavior in this version, but it is probably less likely in\n> realistic data shapes.\n\nWould this work? I agree that the width of the commit bitmasks would go\ndown (and there would also be fewer commit bitmasks generated, further\nincreasing the memory savings). But intuitively, if there is a commit\nthat is selected and only accessible through non-1st-parent links, then\nany bitmaps generated for it cannot be contributed to its descendants\n(since there was no descendant-to-ancestor walk that could reach it in\norder to form the reverse edge).\n\n> Here is some data taken on a fresh clone of the kernel:\n> \n> | runtime (sec) | peak heap (GB) |\n> | | |\n> | from | with | from | with |\n> | scratch | existing | scratch | existing |\n> -----------+---------+----------+---------+-----------\n> original | 64.044 | 83.241 | 2.088 | 2.194 |\n> last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n> this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n\nHmm...the jump from 44 to 100 seems rather large.\n"},{"id":"411147","messageId":"20201202080808.3482917-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"42399a1c2e52e1d055a2d0ad96af2ca4dce6b1a0.1605649533.git.me@ttaylorr.com","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T08:08:08Z","receivedAt":"2020-12-02T08:09:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index b0493d971d..3ac90ae410 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -195,7 +195,8 @@ struct bitmap_builder {\n> };\n> \n> static void bitmap_builder_init(struct bitmap_builder *bb,\n> -\t\t\t\tstruct bitmap_writer *writer)\n> +\t\t\t\tstruct bitmap_writer *writer,\n> +\t\t\t\tstruct bitmap_index *old_bitmap)\n> {\n> \tstruct rev_info revs;\n> \tstruct commit *commit;\n> @@ -234,12 +235,26 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n> \n> \t\tc_ent = bb_data_at(&bb->data, commit);\n> \n> +\t\tif (old_bitmap && bitmap_for_commit(old_bitmap, commit)) {\n> +\t\t\t/*\n> +\t\t\t * This commit has an existing bitmap, so we can\n> +\t\t\t * get its bits immediately without an object\n> +\t\t\t * walk. There is no need to continue walking\n> +\t\t\t * beyond this commit.\n> +\t\t\t */\n\nOK - as far as I understand, the reason for continuing the walk would be\nto find reverse edges that connect this commit and its ancestors so that\nthis commit's ancestors can contribute bitmaps to this commit, but we do\nnot need such contributions, so we do not need to continue the walk.\nMakes sense.\n\n> +\t\t\tc_ent->maximal = 1;\n> +\t\t\tp = NULL;\n\nHere, we're setting maximal without also setting a bit in this commit's\ncommit_mask. This is fine because we're not propagating this commit's\ncommit_mask to any parents (we're not continuing the walk from this\ncommit), but it seems like a code smell. Suggested fix is below.\n\n> +\t\t}\n> +\n> \t\tif (c_ent->maximal) {\n> \t\t\tnum_maximal++;\n> \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n> \t\t\tbb->commits[bb->commits_nr++] = commit;\n> \t\t}\n\nAs far as I can tell, this means that this commit occupies a bit\nposition in the commit mask that it doesn't need. Could this go into a\nseparate list instead, to be appended to bb->commits at the very end?\n\nWe could even skip the whole maximal stuff (for commits with existing\nbitmaps) and replace \"c_ent->maximal = 1;\" above with \"add to list that\nwe're going to append to bb->commits at the very end\". That has the\nadvantage of not having to redefine \"maximal\".\n\n> \n> +\t\tif (!c_ent->commit_mask)\n> +\t\t\tcontinue;\n\nI think this should be moved as far up as possible (right after\nthe call to bb_data_at()) and commented, something like:\n\n If there is no commit_mask, there is no reason to iterate over this\n commit; it is not selected (if it were, it would not have a blank\n commit mask) and all its children have existing bitmaps (see the\n comment starting with \"This commit has an existing bitmap\" below), so\n it does not contribute anything to the final bitmap file or its\n descendants.\n"},{"id":"411148","messageId":"20201202081114.3485460-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"X8bVHQ1eyDkv2CLT@nand.local","subject":"Re: [PATCH v2 00/24] pack-bitmap: bitmap generation improvements","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-02T08:11:14Z","receivedAt":"2020-12-02T08:12:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Tue, Dec 01, 2020 at 03:37:25PM -0800, Jonathan Tan wrote:\n> > > Once we can reach a good decision there, I'll send a v3 (which currently\n> > > lives in my fork[2]).\n> >\n> > I did, but Stolee has a point that the algorithm will change later on.\n> > I'm OK with the parts I reviewed (patches 10 to 18).\n> \n> Ah, good to know. I'll hold off on a v3, then, until you have had a\n> chance to look through the remaining handful of patches (if you were\n> planning on doing that). I haven't touched those locally, so it'll be\n> good to hear any comments you might have before sending another version.\n> \n> Thanks for all of your very helpful review :-).\n> \n> Taylor\n\nYou're welcome! I've gone ahead and reviewed all the patches subsequent\nto 18. I think others have looked at the patches prior to 10, so I'm not\nplanning to review the others.\n"},{"id":"411169","messageId":"X8e+0476g+TtfGHx@nand.local","threadId":"54622","inReplyTo":"20201202072811.3474340-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 22/24] pack-bitmap-write: use existing bitmaps","fromName":"Taylor Blau","fromEmail":"ttaylorr@github.com","sentAt":"2020-12-02T16:21:22Z","receivedAt":"2020-12-02T16:22:20Z","isPatch":true,"sender":{"key":"ttaylorr@github.com","avatar":"https://gravatar.com/avatar/d5f3476f26b6f99cbb6b467e7ed7482f5762c8157bc73f569196e428bdcbea25?d=mp&s=160"},"body":"On Tue, Dec 01, 2020 at 11:28:11PM -0800, Jonathan Tan wrote:\n> > However, if we walk trees\n> > from top to bottom, then we might be parsing trees that are actually\n> > part of a re-used bitmap.\n>\n> Isn't the issue that we shouldn't walk trees at all before exhausting\n> our commit search, not the direction that we walk the trees in (top to\n> bottom or bottom to top or whatever)?\n\nRight, the direction that we explore trees in isn't important: what\nmatters is that we consider them after the commits. I've clarified the\ncommit message to reflect this.\n\n> > To avoid over-walking trees, add them to a\n> > LIFO queue and walk them from bottom-to-top after exploring commits\n> > completely.\n>\n> Just to clarify - would it work just as well with a FIFO queue (not LIFO\n> queue)? It seems to me that the most important part is doing this after\n> exploring commits completely.\n\nYup, see above.\n\n> > static void fill_bitmap_commit(struct bb_commit *ent,\n> > \t\t\t struct commit *commit,\n> > -\t\t\t struct prio_queue *queue)\n> > +\t\t\t struct prio_queue *queue,\n> > +\t\t\t struct prio_queue *tree_queue,\n> > +\t\t\t struct bitmap_index *old_bitmap,\n> > +\t\t\t const uint32_t *mapping)\n> > {\n> > \tif (!ent->bitmap)\n> > \t\tent->bitmap = bitmap_new();\n> >\n> > -\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n> > \tprio_queue_put(queue, commit);\n> >\n> > \twhile (queue->nr) {\n> > \t\tstruct commit_list *p;\n> > \t\tstruct commit *c = prio_queue_get(queue);\n> >\n> > +\t\t/*\n> > +\t\t * If this commit has an old bitmap, then translate that\n> > +\t\t * bitmap and add its bits to this one. No need to walk\n> > +\t\t * parents or the tree for this commit.\n> > +\t\t */\n>\n> This comment should be right before \"if (old && ...\", I think. Here, it\n> is a bit misleading. It leads me to think that \"this commit has an old\n> bitmap\" means old_bitmap != NULL, but it is actually old != NULL.\n\nYup, the comment is much more clear when placed there, thanks.\n\n> > +\t\tif (old_bitmap && mapping) {\n>\n> This is defensive in that if we somehow calculate old_bitmap without\n> mapping (or the other way around) (which is a bug), things just slow\n> down instead of breaking. I'm OK with this, but I still wanted to call\n> it out.\n\nRight, we should never have one without the other, so in that sense this\nis a defensive check. IOW, this could easily be written as `if\n(old_bitmap)` or `if (mapping)`, but being extra defensive here doesn't\nhurt.\n\nThanks,\nTaylor\n"},{"id":"411171","messageId":"X8fBHz2A82hxUzV8@nand.local","threadId":"54622","inReplyTo":"20201202074439.3478090-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-02T16:30:23Z","receivedAt":"2020-12-02T16:31:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 01, 2020 at 11:44:39PM -0800, Jonathan Tan wrote:\n> Do you have numbers of how large the commit bitmasks are?\n\nNo, I didn't measure the size of the commit bitmasks directly, but they\nare captured in the peak heap measurements that I took below.\n\n> > In an effort to discover a happy medium, this change reduces the walk\n> > for intermediate commits to only the first-parent history. This focuses\n> > the walk on how the histories converge, which still has significant\n> > reduction in repeat object walks. It is still possible to create\n> > quadratic behavior in this version, but it is probably less likely in\n> > realistic data shapes.\n>\n> Would this work? I agree that the width of the commit bitmasks would go\n> down (and there would also be fewer commit bitmasks generated, further\n> increasing the memory savings). But intuitively, if there is a commit\n> that is selected and only accessible through non-1st-parent links, then\n> any bitmaps generated for it cannot be contributed to its descendants\n> (since there was no descendant-to-ancestor walk that could reach it in\n> order to form the reverse edge).\n\ns/bitmaps/bitmasks. We'll select commits independent of their first\nparent histories, and so in the situation that you're describing, if C\nreaches A only through non-1st-parent history, then A's bitmask will not\ncontain the bits from C.\n\nBut when generating the reachability bitmap for C, we'll still find that\nwe've generated a bitmap for A, and we can copy its bits directly. If\nthis differs from an ancestor P that _is_ in the first-parent history,\nthen P pushed its bits to C before calling fill_bitmap_commit() through\nthe reverse edges.\n\n> > Here is some data taken on a fresh clone of the kernel:\n> >\n> > | runtime (sec) | peak heap (GB) |\n> > | | |\n> > | from | with | from | with |\n> > | scratch | existing | scratch | existing |\n> > -----------+---------+----------+---------+-----------\n> > original | 64.044 | 83.241 | 2.088 | 2.194 |\n> > last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n> > this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n>\n> Hmm...the jump from 44 to 100 seems rather large.\n\nIndeed. It's ameliorated a little bit in the later patches. We are\nover-walking some objects (as in we are walking them multiple times),\nbut the return we get is reducing the peak heap usage from what it was\nin the last patch.\n\nIn the \"unfathomably large\" category, this makes things tractable.\n\nThanks,\nTaylor\n"},{"id":"411172","messageId":"X8fCViBtnDek6oAI@nand.local","threadId":"54622","inReplyTo":"20201202080808.3482917-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-02T16:35:34Z","receivedAt":"2020-12-02T16:36:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Dec 02, 2020 at 12:08:08AM -0800, Jonathan Tan wrote:\n> > +\t\t\tc_ent->maximal = 1;\n> > +\t\t\tp = NULL;\n>\n> Here, we're setting maximal without also setting a bit in this commit's\n> commit_mask. This is fine because we're not propagating this commit's\n> commit_mask to any parents (we're not continuing the walk from this\n> commit), but it seems like a code smell. Suggested fix is below.\n>\n> > +\t\t}\n> > +\n> > \t\tif (c_ent->maximal) {\n> > \t\t\tnum_maximal++;\n> > \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n> > \t\t\tbb->commits[bb->commits_nr++] = commit;\n> > \t\t}\n>\n> As far as I can tell, this means that this commit occupies a bit\n> position in the commit mask that it doesn't need. Could this go into a\n> separate list instead, to be appended to bb->commits at the very end?\n>\n> We could even skip the whole maximal stuff (for commits with existing\n> bitmaps) and replace \"c_ent->maximal = 1;\" above with \"add to list that\n> we're going to append to bb->commits at the very end\". That has the\n> advantage of not having to redefine \"maximal\".\n\nHmm. I'd trust Stolee's opinion over mine here, so I'll be curious what\nhe has to say.\n\n> >\n> > +\t\tif (!c_ent->commit_mask)\n> > +\t\t\tcontinue;\n>\n> I think this should be moved as far up as possible (right after\n> the call to bb_data_at()) and commented, something like:\n>\n> If there is no commit_mask, there is no reason to iterate over this\n> commit; it is not selected (if it were, it would not have a blank\n> commit mask) and all its children have existing bitmaps (see the\n> comment starting with \"This commit has an existing bitmap\" below), so\n> it does not contribute anything to the final bitmap file or its\n> descendants.\n\nGood suggestion, thanks.\n\nThanks,\nTaylor\n"},{"id":"411174","messageId":"39441f40-f496-af81-87c1-9d7e04fdef20@gmail.com","threadId":"54622","inReplyTo":"X8fCViBtnDek6oAI@nand.local","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-12-02T18:22:27Z","receivedAt":"2020-12-02T18:23:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/2/2020 11:35 AM, Taylor Blau wrote:\n> On Wed, Dec 02, 2020 at 12:08:08AM -0800, Jonathan Tan wrote:\n>>> +\t\t\tc_ent->maximal = 1;\n>>> +\t\t\tp = NULL;\n>>\n>> Here, we're setting maximal without also setting a bit in this commit's\n>> commit_mask. This is fine because we're not propagating this commit's\n>> commit_mask to any parents (we're not continuing the walk from this\n>> commit), but it seems like a code smell. Suggested fix is below.\n>>\n>>> +\t\t}\n>>> +\n>>> \t\tif (c_ent->maximal) {\n>>> \t\t\tnum_maximal++;\n>>> \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n>>> \t\t\tbb->commits[bb->commits_nr++] = commit;\n>>> \t\t}\n>>\n>> As far as I can tell, this means that this commit occupies a bit\n>> position in the commit mask that it doesn't need. Could this go into a\n>> separate list instead, to be appended to bb->commits at the very end?\n\nI don't see any value in having a second list here. That only makes\nthings more complicated.\n\n>> We could even skip the whole maximal stuff (for commits with existing\n>> bitmaps) and replace \"c_ent->maximal = 1;\" above with \"add to list that\n>> we're going to append to bb->commits at the very end\". That has the\n>> advantage of not having to redefine \"maximal\".\n> \n> Hmm. I'd trust Stolee's opinion over mine here, so I'll be curious what\n> he has to say.\n\nIt would be equivalent to add it to the list and then continuing the\nloop instead of piggy-backing on the if (c_ent->maximal) block, followed\nby a trivial loop over the (nullified) parents.\n\n>>>\n>>> +\t\tif (!c_ent->commit_mask)\n>>> +\t\t\tcontinue;\n>>\n>> I think this should be moved as far up as possible (right after\n>> the call to bb_data_at()) and commented, something like:\n>>\n>> If there is no commit_mask, there is no reason to iterate over this\n>> commit; it is not selected (if it were, it would not have a blank\n>> commit mask) and all its children have existing bitmaps (see the\n>> comment starting with \"This commit has an existing bitmap\" below), so\n>> it does not contribute anything to the final bitmap file or its\n>> descendants.\n> \n> Good suggestion, thanks.\n\nYeah, makes sense to me.\n\nThanks,\n-Stolee\n"},{"id":"411175","messageId":"X8fcAZU30P5MdfRR@nand.local","threadId":"54622","inReplyTo":"39441f40-f496-af81-87c1-9d7e04fdef20@gmail.com","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-02T18:25:05Z","receivedAt":"2020-12-02T18:25:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Dec 02, 2020 at 01:22:27PM -0500, Derrick Stolee wrote:\n> >> We could even skip the whole maximal stuff (for commits with existing\n> >> bitmaps) and replace \"c_ent->maximal = 1;\" above with \"add to list that\n> >> we're going to append to bb->commits at the very end\". That has the\n> >> advantage of not having to redefine \"maximal\".\n> >\n> > Hmm. I'd trust Stolee's opinion over mine here, so I'll be curious what\n> > he has to say.\n>\n> It would be equivalent to add it to the list and then continuing the\n> loop instead of piggy-backing on the if (c_ent->maximal) block, followed\n> by a trivial loop over the (nullified) parents.\n\nJonathan: does that seem OK to you to leave it as-is? If you don't have\nstrong objections, I'll go ahead with sending v3 a little later today.\n\nThanks,\nTaylor\n"},{"id":"411610","messageId":"20201207181909.3032039-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"X8fBHz2A82hxUzV8@nand.local","subject":"Re: [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-07T18:19:09Z","receivedAt":"2020-12-07T18:20:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > > In an effort to discover a happy medium, this change reduces the walk\n> > > for intermediate commits to only the first-parent history. This focuses\n> > > the walk on how the histories converge, which still has significant\n> > > reduction in repeat object walks. It is still possible to create\n> > > quadratic behavior in this version, but it is probably less likely in\n> > > realistic data shapes.\n> >\n> > Would this work? I agree that the width of the commit bitmasks would go\n> > down (and there would also be fewer commit bitmasks generated, further\n> > increasing the memory savings). But intuitively, if there is a commit\n> > that is selected and only accessible through non-1st-parent links, then\n> > any bitmaps generated for it cannot be contributed to its descendants\n> > (since there was no descendant-to-ancestor walk that could reach it in\n> > order to form the reverse edge).\n> \n> s/bitmaps/bitmasks. \n\nI do mean bitmaps there - bitmasks are contributed to parents, but\nbitmaps are contributed to descendants, if I remember correctly.\n\n> We'll select commits independent of their first\n> parent histories, and so in the situation that you're describing, if C\n> reaches A only through non-1st-parent history, then A's bitmask will not\n> contain the bits from C.\n\nC is the descendant and A is the ancestor. Yes, A's bitmask will not\ncontain the bits from C.\n\n> But when generating the reachability bitmap for C, we'll still find that\n> we've generated a bitmap for A, and we can copy its bits directly. \n\nHere is my contention - this can happen only if there is a reverse edge\nfrom A to C, as far as I can tell, but such a reverse edge has not been\nformed.\n\n> If\n> this differs from an ancestor P that _is_ in the first-parent history,\n> then P pushed its bits to C before calling fill_bitmap_commit() through\n> the reverse edges.\n> \n> > > Here is some data taken on a fresh clone of the kernel:\n> > >\n> > > | runtime (sec) | peak heap (GB) |\n> > > | | |\n> > > | from | with | from | with |\n> > > | scratch | existing | scratch | existing |\n> > > -----------+---------+----------+---------+-----------\n> > > original | 64.044 | 83.241 | 2.088 | 2.194 |\n> > > last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n> > > this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n> >\n> > Hmm...the jump from 44 to 100 seems rather large.\n> \n> Indeed. It's ameliorated a little bit in the later patches. We are\n> over-walking some objects (as in we are walking them multiple times),\n> but the return we get is reducing the peak heap usage from what it was\n> in the last patch.\n> \n> In the \"unfathomably large\" category, this makes things tractable.\n\nQuoting from the next patch [1]:\n\n> | runtime (sec) | peak heap (GB) |\n> | | |\n> | from | with | from | with |\n> | scratch | existing | scratch | existing |\n> -----------+---------+----------+---------+-----------\n> last patch | 100.641 | 35.560 | 2.152 | 2.224 |\n> this patch | 99.720 | 11.696 | 2.152 | 2.217 |\n\nThat is true, but it is not ameliorated much :-(\n\nIf you have steps to generate these timings, I would like to try\ncomparing the performance between all patches and all-except-23.\n\n[1] https://lore.kernel.org/git/42399a1c2e52e1d055a2d0ad96af2ca4dce6b1a0.1605649533.git.me@ttaylorr.com/\n"},{"id":"411611","messageId":"20201207182418.3034961-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"39441f40-f496-af81-87c1-9d7e04fdef20@gmail.com","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-07T18:24:18Z","receivedAt":"2020-12-07T18:25:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On 12/2/2020 11:35 AM, Taylor Blau wrote:\n> > On Wed, Dec 02, 2020 at 12:08:08AM -0800, Jonathan Tan wrote:\n> >>> +\t\t\tc_ent->maximal = 1;\n> >>> +\t\t\tp = NULL;\n> >>\n> >> Here, we're setting maximal without also setting a bit in this commit's\n> >> commit_mask. This is fine because we're not propagating this commit's\n> >> commit_mask to any parents (we're not continuing the walk from this\n> >> commit), but it seems like a code smell. Suggested fix is below.\n> >>\n> >>> +\t\t}\n> >>> +\n> >>> \t\tif (c_ent->maximal) {\n> >>> \t\t\tnum_maximal++;\n> >>> \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n> >>> \t\t\tbb->commits[bb->commits_nr++] = commit;\n> >>> \t\t}\n> >>\n> >> As far as I can tell, this means that this commit occupies a bit\n> >> position in the commit mask that it doesn't need. Could this go into a\n> >> separate list instead, to be appended to bb->commits at the very end?\n> \n> I don't see any value in having a second list here. That only makes\n> things more complicated.\n\nIt does make things more complicated, but it could help shrink commit\nbitmasks (which seem to be a concern, according to patch 23).\n\nSuppose num_maximal was 3 and we encountered such a commit (not\nselected, but has an old bitmap). So we increment num_maximal. Then, we\nencounter a selected commit. That commit would then have a bitmask of\n???01. If we had not incremented num_maximal (which would require a\nsecond list), then the bitmask would be ???1.\n\n> >> We could even skip the whole maximal stuff (for commits with existing\n> >> bitmaps) and replace \"c_ent->maximal = 1;\" above with \"add to list that\n> >> we're going to append to bb->commits at the very end\". That has the\n> >> advantage of not having to redefine \"maximal\".\n> > \n> > Hmm. I'd trust Stolee's opinion over mine here, so I'll be curious what\n> > he has to say.\n> \n> It would be equivalent to add it to the list and then continuing the\n> loop instead of piggy-backing on the if (c_ent->maximal) block, followed\n> by a trivial loop over the (nullified) parents.\n\nThat is true. This suggestion was for code clarity, not for correctness.\n"},{"id":"411612","messageId":"20201207182633.3036948-1-jonathantanmy@google.com","threadId":"54622","inReplyTo":"X8fcAZU30P5MdfRR@nand.local","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-12-07T18:26:33Z","receivedAt":"2020-12-07T18:27:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Wed, Dec 02, 2020 at 01:22:27PM -0500, Derrick Stolee wrote:\n> > >> We could even skip the whole maximal stuff (for commits with existing\n> > >> bitmaps) and replace \"c_ent->maximal = 1;\" above with \"add to list that\n> > >> we're going to append to bb->commits at the very end\". That has the\n> > >> advantage of not having to redefine \"maximal\".\n> > >\n> > > Hmm. I'd trust Stolee's opinion over mine here, so I'll be curious what\n> > > he has to say.\n> >\n> > It would be equivalent to add it to the list and then continuing the\n> > loop instead of piggy-backing on the if (c_ent->maximal) block, followed\n> > by a trivial loop over the (nullified) parents.\n> \n> Jonathan: does that seem OK to you to leave it as-is? If you don't have\n> strong objections, I'll go ahead with sending v3 a little later today.\n\nLike I (just) said in [1], I think that my comment stands, but this is a\nminor and local issue that does not affect the functionality of the\noverall patch set so I think you can go ahead and send v3.\n\n[1] https://lore.kernel.org/git/20201207182418.3034961-1-jonathantanmy@google.com/\n"},{"id":"411616","messageId":"2f0540e6-b4f4-ea9f-bac0-ecf92c7b764d@gmail.com","threadId":"54622","inReplyTo":"20201207181909.3032039-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-12-07T18:43:51Z","receivedAt":"2020-12-07T18:44:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/7/2020 1:19 PM, Jonathan Tan wrote:\n>>>> In an effort to discover a happy medium, this change reduces the walk\n>>>> for intermediate commits to only the first-parent history. This focuses\n>>>> the walk on how the histories converge, which still has significant\n>>>> reduction in repeat object walks. It is still possible to create\n>>>> quadratic behavior in this version, but it is probably less likely in\n>>>> realistic data shapes.\n>>>\n>>> Would this work? I agree that the width of the commit bitmasks would go\n>>> down (and there would also be fewer commit bitmasks generated, further\n>>> increasing the memory savings). But intuitively, if there is a commit\n>>> that is selected and only accessible through non-1st-parent links, then\n>>> any bitmaps generated for it cannot be contributed to its descendants\n>>> (since there was no descendant-to-ancestor walk that could reach it in\n>>> order to form the reverse edge).\n>>\n>> s/bitmaps/bitmasks. \n> \n> I do mean bitmaps there - bitmasks are contributed to parents, but\n> bitmaps are contributed to descendants, if I remember correctly.\n\nAh, the confusion is related around the word \"contributed\".\n\nYes, without walking all the parents, we will not populate the\nreverse edges with all of the possible connections. Thus, the\nstep that pushes reachability bitmap bits along the reverse edges\nwill not be as effective.\n\nAnd this is the whole point: the reverse-edges existed to get us\ninto a state of _never_ walking an object multiple times, but that\nended up being too expensive to guarantee. This change relaxes that\ncondition in a way that still works for large, linear histories.\n\nSince \"pack-bitmap-write: fill bitmap with commit history\" changed\nfill_bitmap_commit() to walk commits until reaching those already in\nthe precomputed reachability bitmap, it will correctly walk far\nenough to compute the reachability bitmap for that commit. It might\njust walk objects that are part of _another_, already computed bitmap\nthat is not reachable via the first-parent history.\n\nThe very next patch \"pack-bitmap-write: better reuse bitmaps\" fixes\nthis problem by checking for computed bitmaps during the walk in\nfill_bitmap_commit().\n\n>> We'll select commits independent of their first\n>> parent histories, and so in the situation that you're describing, if C\n>> reaches A only through non-1st-parent history, then A's bitmask will not\n>> contain the bits from C.\n> \n> C is the descendant and A is the ancestor. Yes, A's bitmask will not\n> contain the bits from C.\n> \n>> But when generating the reachability bitmap for C, we'll still find that\n>> we've generated a bitmap for A, and we can copy its bits directly. \n> \n> Here is my contention - this can happen only if there is a reverse edge\n> from A to C, as far as I can tell, but such a reverse edge has not been\n> formed.\n\nSee above. This patch is completely correct given the changes to\nfill_bitmap_commit() from earlier. It just needs a tweak (in the\nnext patch) to recover some of the performance.\n\n>> If\n>> this differs from an ancestor P that _is_ in the first-parent history,\n>> then P pushed its bits to C before calling fill_bitmap_commit() through\n>> the reverse edges.\n>>\n>>>> Here is some data taken on a fresh clone of the kernel:\n>>>>\n>>>> | runtime (sec) | peak heap (GB) |\n>>>> | | |\n>>>> | from | with | from | with |\n>>>> | scratch | existing | scratch | existing |\n>>>> -----------+---------+----------+---------+-----------\n>>>> original | 64.044 | 83.241 | 2.088 | 2.194 |\n>>>> last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n>>>> this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n>>>\n>>> Hmm...the jump from 44 to 100 seems rather large.\n>>\n>> Indeed. It's ameliorated a little bit in the later patches. We are\n>> over-walking some objects (as in we are walking them multiple times),\n>> but the return we get is reducing the peak heap usage from what it was\n>> in the last patch.\n>>\n>> In the \"unfathomably large\" category, this makes things tractable.\n> \n> Quoting from the next patch [1]:\n> \n>> | runtime (sec) | peak heap (GB) |\n>> | | |\n>> | from | with | from | with |\n>> | scratch | existing | scratch | existing |\n>> -----------+---------+----------+---------+-----------\n>> last patch | 100.641 | 35.560 | 2.152 | 2.224 |\n>> this patch | 99.720 | 11.696 | 2.152 | 2.217 |\n> \n> That is true, but it is not ameliorated much :-(\n> \n> If you have steps to generate these timings, I would like to try\n> comparing the performance between all patches and all-except-23.\n> \n> [1] https://lore.kernel.org/git/42399a1c2e52e1d055a2d0ad96af2ca4dce6b1a0.1605649533.git.me@ttaylorr.com/\n\nThe biggest problem is that all-except-23 is an unnacceptable\nfinal state, since it has a performance blowout on super-wide\nrepos such as the git/git fork network. Perhaps Taylor could\ninclude some performance numbers on that, but I'm pretty sure\nthat the calculation literally OOMs instead of completing. It\nmight be worth an explicit mention in the patch.\n\nIt might also be better to always include a baseline from the\nstart of the series to ensure that the final state is better\nthan the initial state. With only the last/this comparison,\nit doesn't look great when we backtrack in performance (even\nwhen it is necessary to do so).\n\nThanks,\n-Stolee\n"},{"id":"411617","messageId":"4bcb8636-54f3-e98e-d4f3-c86f15edc0c9@gmail.com","threadId":"54622","inReplyTo":"2f0540e6-b4f4-ea9f-bac0-ecf92c7b764d@gmail.com","subject":"Re: [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-12-07T18:45:39Z","receivedAt":"2020-12-07T18:46:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/7/2020 1:43 PM, Derrick Stolee wrote:\n> The very next patch \"pack-bitmap-write: better reuse bitmaps\" fixes\n> this problem by checking for computed bitmaps during the walk in\n> fill_bitmap_commit().\n\nOf course I got confused and instead I meant to refer to the _previous_\npatch, \"pack-bitmap-write: use existing bitmaps\".\n\n-Stolee\n\n"},{"id":"411619","messageId":"X855DX0CE//Cndwb@coredump.intra.peff.net","threadId":"54622","inReplyTo":"20201207181909.3032039-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-12-07T18:48:45Z","receivedAt":"2020-12-07T18:49:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Dec 07, 2020 at 10:19:09AM -0800, Jonathan Tan wrote:\n\n> Quoting from the next patch [1]:\n> \n> > | runtime (sec) | peak heap (GB) |\n> > | | |\n> > | from | with | from | with |\n> > | scratch | existing | scratch | existing |\n> > -----------+---------+----------+---------+-----------\n> > last patch | 100.641 | 35.560 | 2.152 | 2.224 |\n> > this patch | 99.720 | 11.696 | 2.152 | 2.217 |\n> \n> That is true, but it is not ameliorated much :-(\n> \n> If you have steps to generate these timings, I would like to try\n> comparing the performance between all patches and all-except-23.\n\nYes, the drop in CPU performance is disappointing. And there may be a\nbetter way of selecting the commits that recovers some of it.\n\nBut all-except-23 is not workable from a memory usage perspective.\nOriginally we did not have that commit at all, and a full repack of our\ngit/git fork network (i.e., all forks stuffed into one alternates repo)\nwent from 16GB to OOM-ing after growing 80+GB (I don't know how large it\nwould have gone on a bigger machine).\n\n-Peff\n"},{"id":"411635","messageId":"1cd3f001-389f-6e2e-90fd-5e8b4b17119a@gmail.com","threadId":"54622","inReplyTo":"20201207182418.3034961-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-12-07T19:20:54Z","receivedAt":"2020-12-07T19:21:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/7/2020 1:24 PM, Jonathan Tan wrote:\n>> On 12/2/2020 11:35 AM, Taylor Blau wrote:\n>>> On Wed, Dec 02, 2020 at 12:08:08AM -0800, Jonathan Tan wrote:\n>>>>> +\t\t\tc_ent->maximal = 1;\n>>>>> +\t\t\tp = NULL;\n>>>>\n>>>> Here, we're setting maximal without also setting a bit in this commit's\n>>>> commit_mask. This is fine because we're not propagating this commit's\n>>>> commit_mask to any parents (we're not continuing the walk from this\n>>>> commit), but it seems like a code smell. Suggested fix is below.\n>>>>\n>>>>> +\t\t}\n>>>>> +\n>>>>> \t\tif (c_ent->maximal) {\n>>>>> \t\t\tnum_maximal++;\n>>>>> \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n>>>>> \t\t\tbb->commits[bb->commits_nr++] = commit;\n>>>>> \t\t}\n>>>>\n>>>> As far as I can tell, this means that this commit occupies a bit\n>>>> position in the commit mask that it doesn't need. Could this go into a\n>>>> separate list instead, to be appended to bb->commits at the very end?\n>>\n>> I don't see any value in having a second list here. That only makes\n>> things more complicated.\n> \n> It does make things more complicated, but it could help shrink commit\n> bitmasks (which seem to be a concern, according to patch 23).\n> \n> Suppose num_maximal was 3 and we encountered such a commit (not\n> selected, but has an old bitmap). So we increment num_maximal. Then, we\n> encounter a selected commit. That commit would then have a bitmask of\n> ???01. If we had not incremented num_maximal (which would require a\n> second list), then the bitmask would be ???1.\n\nOK, I see the value. The value is bounded, since the number of\nthese \"0\" gaps is bounded by the number of selected commits _and_\nreduces the possible number of maximal commits.\n\nHowever, that seems like enough justification to create the second\nlist.\n\nThanks,\n-Stolee\n"},{"id":"411657","messageId":"0b25ba4ca7133be1f7b763ba9c3d18dbdd03f37a.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 01/24] ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:12Z","receivedAt":"2020-12-08T00:05:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'ewah/ewah_bitmap.c:buffer_grow()' is responsible for growing the buffer\nused to store the bits of an EWAH bitmap. It is essentially doing the\nsame task as the 'ALLOC_GROW()' macro, so use that instead.\n\nThis simplifies the callers of 'buffer_grow()', who no longer have to\nask for a specific size, but rather specify how much of the buffer they\nneed. They also no longer need to guard 'buffer_grow()' behind an if\nstatement, since 'ALLOC_GROW()' (and, by extension, 'buffer_grow()') is\na noop if the buffer is already large enough.\n\nBut, the most significant change is that this fixes a bug when calling\nbuffer_grow() with both 'alloc_size' and 'new_size' set to 1. In this\ncase, truncating integer math will leave the new size set to 1, causing\nthe buffer to never grow.\n\nInstead, let alloc_nr() handle this, which asks for '(new_size + 16) * 3\n/ 2' instead of 'new_size * 3 / 2'.\n\nSigned-off-by: Taylor Blau \n---\n ewah/ewah_bitmap.c | 15 ++++-----------\n 1 file changed, 4 insertions(+), 11 deletions(-)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex d59b1afe3d..2a8c7c5c33 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -19,6 +19,7 @@\n #include \"git-compat-util.h\"\n #include \"ewok.h\"\n #include \"ewok_rlw.h\"\n+#include \"cache.h\"\n \n static inline size_t min_size(size_t a, size_t b)\n {\n@@ -33,20 +34,13 @@ static inline size_t max_size(size_t a, size_t b)\n static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n {\n \tsize_t rlw_offset = (uint8_t *)self->rlw - (uint8_t *)self->buffer;\n-\n-\tif (self->alloc_size >= new_size)\n-\t\treturn;\n-\n-\tself->alloc_size = new_size;\n-\tREALLOC_ARRAY(self->buffer, self->alloc_size);\n+\tALLOC_GROW(self->buffer, new_size, self->alloc_size);\n \tself->rlw = self->buffer + (rlw_offset / sizeof(eword_t));\n }\n \n static inline void buffer_push(struct ewah_bitmap *self, eword_t value)\n {\n-\tif (self->buffer_size + 1 >= self->alloc_size)\n-\t\tbuffer_grow(self, self->buffer_size * 3 / 2);\n-\n+\tbuffer_grow(self, self->buffer_size + 1);\n \tself->buffer[self->buffer_size++] = value;\n }\n \n@@ -137,8 +131,7 @@ void ewah_add_dirty_words(\n \n \t\trlw_set_literal_words(self->rlw, literals + can_add);\n \n-\t\tif (self->buffer_size + can_add >= self->alloc_size)\n-\t\t\tbuffer_grow(self, (self->buffer_size + can_add) * 3 / 2);\n+\t\tbuffer_grow(self, self->buffer_size + can_add);\n \n \t\tif (negate) {\n \t\t\tsize_t i;\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411658","messageId":"cover.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH v3 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:06Z","receivedAt":"2020-12-08T00:05:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here's an updated v3 of mine, Stolee, and Peff's series to improve the\nCPU performance of generating reachability bitmaps.\n\nNot a great deal has changed since last time, though this version does\nincorporate feedback from Jonathan Tan and Junio (thanks, both, for your\nreview!). A range-diff is below for convenience, but the major\nhighlights are:\n\n - ALLOC_GROW() is now used in more places (this didn't measurably\n affect peak-heap usage, so it's a pure nicetie to avoid duplicating\n that logic throughout the ewah code)\n\n - Some later commits have been reworded to add additional clarity\n\n - bitmap_diff_nonzero() was replaced with bitmap_is_subset(), and the\n implementation amended to follow Junio's suggestion\n\n - The final patches have been slightly modified to avoid allocating\n extra bits in the bitmasks for cases where reachability bitmaps have\n already been generated for those patches (suggestion courtesy of\n Jonathan Tan)\n\nI'm hopeful that this will be in good shape for queuing up, since\nJonathan had a chance to review the whole series. Thanks!\n\nDerrick Stolee (9):\n pack-bitmap-write: fill bitmap with commit history\n bitmap: implement bitmap_is_subset()\n commit: implement commit_list_contains()\n t5310: add branch-based checks\n pack-bitmap-write: rename children to reverse_edges\n pack-bitmap-write: build fewer intermediate bitmaps\n pack-bitmap-write: use existing bitmaps\n pack-bitmap-write: relax unique rewalk condition\n pack-bitmap-write: better reuse bitmaps\n\nJeff King (11):\n pack-bitmap: fix header size check\n pack-bitmap: bounds-check size of cache extension\n t5310: drop size of truncated ewah bitmap\n rev-list: die when --test-bitmap detects a mismatch\n ewah: factor out bitmap growth\n ewah: make bitmap growth less aggressive\n ewah: implement bitmap_or()\n ewah: add bitmap_dup() function\n pack-bitmap-write: reimplement bitmap writing\n pack-bitmap-write: pass ownership of intermediate bitmaps\n pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n\nTaylor Blau (4):\n ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW()\n pack-bitmap.c: check reads more aggressively when loading\n pack-bitmap: factor out 'bitmap_for_commit()'\n pack-bitmap: factor out 'add_commit_to_bitmap()'\n\n builtin/pack-objects.c | 1 -\n commit.c | 11 +\n commit.h | 2 +\n ewah/bitmap.c | 54 ++++-\n ewah/ewah_bitmap.c | 15 +-\n ewah/ewok.h | 3 +-\n pack-bitmap-write.c | 474 ++++++++++++++++++++++++++--------------\n pack-bitmap.c | 139 ++++++------\n pack-bitmap.h | 8 +-\n t/t5310-pack-bitmaps.sh | 164 +++++++++++---\n 10 files changed, 576 insertions(+), 295 deletions(-)\n\nRange-diff against v2:\n 1: 07054ff8ee < -: ---------- ewah/ewah_bitmap.c: grow buffer past 1\n -: ---------- > 1: 0b25ba4ca7 ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW()\n 2: 74a13b4a6e = 2: b455b248e4 pack-bitmap: fix header size check\n 3: db11116dac = 3: 7322427444 pack-bitmap: bounds-check size of cache extension\n 4: f779e76f82 = 4: 055bc1fe66 t5310: drop size of truncated ewah bitmap\n 5: 1a9ac1c4ae = 5: c99cacea67 rev-list: die when --test-bitmap detects a mismatch\n 6: 9bb1ea3b19 = 6: b79360383e ewah: factor out bitmap growth\n 7: f8426c7e8b < -: ---------- ewah: make bitmap growth less aggressive\n -: ---------- > 7: 4b56f12932 ewah: make bitmap growth less aggressive\n 8: 674e31f98e ! 8: 34137a7f35 ewah: implement bitmap_or()\n @@ Commit message\n\n Interestingly, we have a public header declaration going back to\n e1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\n - function was never implemented.\n + function was never implemented. That was all OK since there were no\n + users of 'bitmap_or()', but a first caller will be added in a couple of\n + patches.\n\n Signed-off-by: Jeff King \n Signed-off-by: Taylor Blau \n 9: a903c949d8 ! 9: fe89f87716 ewah: add bitmap_dup() function\n @@ ewah/bitmap.c: struct bitmap *bitmap_new(void)\n +\n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n - \tif (word_alloc > self->word_alloc) {\n + \tsize_t old_size = self->word_alloc;\n\n ## ewah/ewok.h ##\n @@ ewah/ewok.h: struct bitmap {\n10: c951206729 ! 10: 91cd8b1a49 pack-bitmap-write: reimplement bitmap writing\n @@ pack-bitmap-write.c: static void compute_xor_offsets(void)\n -\t\tkh_value(writer.bitmaps, hash_pos) = stored;\n -\t\tdisplay_progress(writer.progress, writer.selected_nr - i);\n +\ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n -+\t\tthe_repository);\n ++\t\t\t the_repository);\n +\n +\tbitmap_builder_init(&bb, &writer);\n +\tfor (i = bb.commits_nr; i > 0; i--) {\n @@ pack-bitmap-write.c: static void compute_xor_offsets(void)\n +\t\tent->bitmap = NULL;\n \t}\n +\tbitmap_builder_clear(&bb);\n ++\n ++\ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n ++\t\t\t the_repository);\n\n -\tbitmap_free(base);\n \tstop_progress(&writer.progress);\n11: 466dd3036a = 11: 64598024ec pack-bitmap-write: pass ownership of intermediate bitmaps\n12: 8e5607929d ! 12: 93fc437a3c pack-bitmap-write: fill bitmap with commit history\n @@ Metadata\n ## Commit message ##\n pack-bitmap-write: fill bitmap with commit history\n\n - The fill_bitmap_commit() method assumes that every parent of the given\n - commit is already part of the current bitmap. Instead of making that\n - assumption, let's walk parents until we reach commits already part of\n - the bitmap. Set the value for that parent immediately after querying to\n - save time doing double calls to find_object_pos() and to avoid inserting\n - the parent into the queue multiple times.\n + The current implementation of bitmap_writer_build() creates a\n + reachability bitmap for every walked commit. After computing a bitmap\n + for a commit, those bits are pushed to an in-progress bitmap for its\n + children.\n +\n + fill_bitmap_commit() assumes the bits corresponding to objects\n + reachable from the parents of a commit are already set. This means that\n + when visiting a new commit, we only have to walk the objects reachable\n + between it and any of its parents.\n +\n + A future change to bitmap_writer_build() will relax this condition so\n + not all parents have their bits set. Prepare for that by having\n + 'fill_bitmap_commit()' walk parents until reaching commits whose bits\n + are already set. Then, walk the trees for these commits as well.\n +\n + This has no functional change with the current implementation of\n + bitmap_writer_build().\n\n Signed-off-by: Derrick Stolee \n Signed-off-by: Taylor Blau \n @@ pack-bitmap-write.c: void bitmap_writer_build(struct packing_data *to_pack)\n +\tclear_prio_queue(&queue);\n \tbitmap_builder_clear(&bb);\n\n - \tstop_progress(&writer.progress);\n + \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n13: 4840c64c51 ! 13: 0d5213ba44 bitmap: add bitmap_diff_nonzero()\n @@ Metadata\n Author: Derrick Stolee \n\n ## Commit message ##\n - bitmap: add bitmap_diff_nonzero()\n + bitmap: implement bitmap_is_subset()\n\n - The bitmap_diff_nonzero() checks if the 'self' bitmap contains any bits\n - that are not on in the 'other' bitmap.\n -\n - Also, delete the declaration of bitmap_is_subset() as it is not used or\n - implemented.\n + The bitmap_is_subset() function checks if the 'self' bitmap contains any\n + bitmaps that are not on in the 'other' bitmap. Up until this patch, it\n + had a declaration, but no implementation or callers. A subsequent patch\n + will want this function, so implement it here.\n\n + Helped-by: Junio C Hamano \n Signed-off-by: Derrick Stolee \n Signed-off-by: Taylor Blau \n\n @@ ewah/bitmap.c: int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n\n -+int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other)\n ++int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n +{\n -+\tstruct bitmap *small;\n -+\tsize_t i;\n ++\tsize_t common_size, i;\n +\n -+\tif (self->word_alloc < other->word_alloc) {\n -+\t\tsmall = self;\n -+\t} else {\n -+\t\tsmall = other;\n -+\n -+\t\tfor (i = other->word_alloc; i < self->word_alloc; i++) {\n -+\t\t\tif (self->words[i] != 0)\n ++\tif (self->word_alloc < other->word_alloc)\n ++\t\tcommon_size = self->word_alloc;\n ++\telse {\n ++\t\tcommon_size = other->word_alloc;\n ++\t\tfor (i = common_size; i < self->word_alloc; i++) {\n ++\t\t\tif (self->words[i])\n +\t\t\t\treturn 1;\n +\t\t}\n +\t}\n +\n -+\tfor (i = 0; i < small->word_alloc; i++) {\n -+\t\tif ((self->words[i] & ~other->words[i]))\n ++\tfor (i = 0; i < common_size; i++) {\n ++\t\tif (self->words[i] & ~other->words[i])\n +\t\t\treturn 1;\n +\t}\n -+\n +\treturn 0;\n +}\n +\n @@ ewah/ewok.h: int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n -int bitmap_is_subset(struct bitmap *self, struct bitmap *super);\n -+int bitmap_diff_nonzero(struct bitmap *self, struct bitmap *other);\n ++int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n\n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n14: 63e846f4e8 = 14: 72e745fed8 commit: implement commit_list_contains()\n15: 8b5d239333 = 15: c2cae4a8d0 t5310: add branch-based checks\n16: 60a46091bb = 16: c0e2b6f5d9 pack-bitmap-write: rename children to reverse_edges\n17: 8f7bb2dd2e = 17: 37f9636098 pack-bitmap.c: check reads more aggressively when loading\n18: 5262daa330 ! 18: e520c8fdc4 pack-bitmap-write: build fewer intermediate bitmaps\n @@ pack-bitmap-write.c: static void bitmap_builder_init(struct bitmap_builder *bb,\n +\t\t\t\tc_not_p = 1;\n +\t\t\t\tp_not_c = 0;\n +\t\t\t} else {\n -+\t\t\t\tc_not_p = bitmap_diff_nonzero(c_ent->commit_mask, p_ent->commit_mask);\n -+\t\t\t\tp_not_c = bitmap_diff_nonzero(p_ent->commit_mask, c_ent->commit_mask);\n ++\t\t\t\tc_not_p = bitmap_is_subset(c_ent->commit_mask, p_ent->commit_mask);\n ++\t\t\t\tp_not_c = bitmap_is_subset(p_ent->commit_mask, c_ent->commit_mask);\n +\t\t\t}\n +\n +\t\t\tif (!c_not_p)\n @@ t/t5310-pack-bitmaps.sh: has_any () {\n +# | / \\________________________ |\n +# |/ \\|\n +# (l2) * * (r2)\n -+# \\____________...____________ |\n ++# \\___________________________ |\n +# \\|\n +# * (base)\n +#\n @@ t/t5310-pack-bitmaps.sh: test_expect_success 'setup repo with moderate-sized his\n '\n\n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n -@@ t/t5310-pack-bitmaps.sh: test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n - \tgit rev-list --use-bitmap-index --count --all >expect &&\n - \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n - \ttest_when_finished \"rm -f $bitmap\" &&\n --\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n -+\ttest_copy_bytes 270 <$bitmap >$bitmap.tmp &&\n - \tmv -f $bitmap.tmp $bitmap &&\n - \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n - \ttest_cmp expect actual &&\n19: a206f48614 = 19: c3975fcf78 pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n20: 9928b3c7da = 20: d5ef2c7f81 pack-bitmap: factor out 'bitmap_for_commit()'\n21: f40a39a48a = 21: f0500190f0 pack-bitmap: factor out 'add_commit_to_bitmap()'\n22: 4bf5e78a54 ! 22: c6fde2b0c4 pack-bitmap-write: use existing bitmaps\n @@ Commit message\n In fill_bitmap_commit(), we must reorder thing somewhat. The priority\n queue walks commits from newest-to-oldest, which means we correctly stop\n walking when reaching a commit with a bitmap. However, if we walk trees\n - from top to bottom, then we might be parsing trees that are actually\n - part of a re-used bitmap. To avoid over-walking trees, add them to a\n - LIFO queue and walk them from bottom-to-top after exploring commits\n - completely.\n + interleaved with the commits, then we might be parsing trees that are\n + actually part of a re-used bitmap. To avoid over-walking trees, add them\n + to a LIFO queue and walk them after exploring commits completely.\n\n On git.git, this reduces a second immediate bitmap computation from 2.0s\n to 1.0s. On linux.git, we go from 32s to 22s. On chromium's fork\n @@ pack-bitmap-write.c: static void fill_bitmap_tree(struct bitmap *bitmap,\n \t\tstruct commit_list *p;\n \t\tstruct commit *c = prio_queue_get(queue);\n\n -+\t\t/*\n -+\t\t * If this commit has an old bitmap, then translate that\n -+\t\t * bitmap and add its bits to this one. No need to walk\n -+\t\t * parents or the tree for this commit.\n -+\t\t */\n +\t\tif (old_bitmap && mapping) {\n -+\t\t\tstruct ewah_bitmap *old;\n -+\n -+\t\t\told = bitmap_for_commit(old_bitmap, c);\n ++\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n ++\t\t\t/*\n ++\t\t\t * If this commit has an old bitmap, then translate that\n ++\t\t\t * bitmap and add its bits to this one. No need to walk\n ++\t\t\t * parents or the tree for this commit.\n ++\t\t\t */\n +\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap))\n +\t\t\t\tcontinue;\n +\t\t}\n @@ pack-bitmap-write.c: void bitmap_writer_build(struct packing_data *to_pack)\n \twriter.to_pack = to_pack;\n @@ pack-bitmap-write.c: void bitmap_writer_build(struct packing_data *to_pack)\n \ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n - \t\tthe_repository);\n + \t\t\t the_repository);\n\n +\told_bitmap = prepare_bitmap_git(to_pack->repo);\n +\tif (old_bitmap)\n @@ pack-bitmap-write.c: void bitmap_writer_build(struct packing_data *to_pack)\n \tbitmap_builder_clear(&bb);\n +\tfree(mapping);\n\n - \tstop_progress(&writer.progress);\n -\n + \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n + \t\t\t the_repository);\n23: 1da4fa0fb8 ! 23: 50d2031deb pack-bitmap-write: relax unique rewalk condition\n @@ Commit message\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n original | 64.044 | 83.241 | 2.088 | 2.194 |\n - last patch | 44.811 | 27.828 | 2.289 | 2.358 |\n - this patch | 100.641 | 35.560 | 2.152 | 2.224 |\n + last patch | 45.049 | 37.624 | 2.267 | 2.334 |\n + this patch | 88.478 | 53.218 | 2.157 | 2.224 |\n\n Signed-off-by: Derrick Stolee \n Signed-off-by: Taylor Blau \n24: 42399a1c2e < -: ---------- pack-bitmap-write: better reuse bitmaps\n -: ---------- > 24: 6b9950771e pack-bitmap-write: better reuse bitmaps\n--\n2.29.2.533.g07db1f5344\n"},{"id":"411659","messageId":"b455b248e4e12d50a7fd1ef2cac34af7f6c53bed.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 02/24] pack-bitmap: fix header size check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:17Z","receivedAt":"2020-12-08T00:05:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWhen we parse a .bitmap header, we first check that we have enough bytes\nto make a valid header. We do that based on sizeof(struct\nbitmap_disk_header). However, as of 0f4d6cada8 (pack-bitmap: make bitmap\nheader handling hash agnostic, 2019-02-19), that struct oversizes its\nchecksum member to GIT_MAX_RAWSZ. That means we need to adjust for the\ndifference between that constant and the size of the actual hash we're\nusing. That commit adjusted the code which moves our pointer forward,\nbut forgot to update the size check.\n\nThis meant we were overly strict about the header size (requiring room\nfor a 32-byte worst-case hash, when sha1 is only 20 bytes). But in\npractice it didn't matter because bitmap files tend to have at least 12\nbytes of actual data anyway, so it was unlikely for a valid file to be\ncaught by this.\n\nLet's fix it by pulling the header size into a separate variable and\nusing it in both spots. That fixes the bug and simplifies the code to make\nit harder to have a mismatch like this in the future. It will also come\nin handy in the next patch for more bounds checking.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4077e731e8..fe5647e72e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -138,9 +138,10 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n+\tsize_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n \n-\tif (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n-\t\treturn error(\"Corrupted bitmap index (missing header data)\");\n+\tif (index->map_size < header_size + the_hash_algo->rawsz)\n+\t\treturn error(\"Corrupted bitmap index (too small)\");\n \n \tif (memcmp(header->magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)) != 0)\n \t\treturn error(\"Corrupted bitmap index file (wrong header)\");\n@@ -164,7 +165,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-\tindex->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n+\tindex->map_pos += header_size;\n \treturn 0;\n }\n \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411660","messageId":"73224274444ea1385bb98b26710725bbb66a9128.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 03/24] pack-bitmap: bounds-check size of cache extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:22Z","receivedAt":"2020-12-08T00:05:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nA .bitmap file may have a \"name hash cache\" extension, which puts a\nsequence of uint32_t values (one per object) at the end of the file.\nWhen we see a flag indicating this extension, we blindly subtract the\nappropriate number of bytes from our available length. However, if the\n.bitmap file is too short, we'll underflow our length variable and wrap\naround, thinking we have a very large length. This can lead to reading\nout-of-bounds bytes while loading individual ewah bitmaps.\n\nWe can fix this by checking the number of available bytes when we parse\nthe header. The existing \"truncated bitmap\" test is now split into two\ntests: one where we don't have this extension at all (and hence actually\ndo try to read a truncated ewah bitmap) and one where we realize\nup-front that we can't even fit in the cache structure. We'll check\nstderr in each case to make sure we hit the error we're expecting.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 8 ++++++--\n t/t5310-pack-bitmaps.sh | 17 +++++++++++++++--\n 2 files changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fe5647e72e..074d9ac8f2 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -153,14 +153,18 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n+\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n \t\t\treturn error(\"Unsupported options for bitmap index file \"\n \t\t\t\t\"(Git requires BITMAP_OPT_FULL_DAG)\");\n \n \t\tif (flags & BITMAP_OPT_HASH_CACHE) {\n-\t\t\tunsigned char *end = index->map + index->map_size - the_hash_algo->rawsz;\n-\t\t\tindex->hashes = ((uint32_t *)end) - index->pack->num_objects;\n+\t\t\tif (cache_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit hash cache)\");\n+\t\t\tindex->hashes = (void *)(index_end - cache_size);\n+\t\t\tindex_end -= cache_size;\n \t\t}\n \t}\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 1d40fcad39..dbe1ffc88a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -343,7 +343,8 @@ test_expect_success 'pack reuse respects --incremental' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_success 'truncated bitmap fails gracefully' '\n+test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\ttest_config pack.writebitmaphashcache false &&\n \tgit repack -ad &&\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -352,7 +353,19 @@ test_expect_success 'truncated bitmap fails gracefully' '\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt stderr\n+\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\tgit repack -ad &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n # have_delta \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411661","messageId":"c99cacea6792fab685f821ef19ae4e20b91f5875.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 05/24] rev-list: die when --test-bitmap detects a mismatch","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:32Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nYou can use \"git rev-list --test-bitmap HEAD\" to check that bitmaps\nproduce the same answer we'd get from a regular traversal. But if we\ndetect an error, we only print \"mismatch\", and still exit with a\nsuccessful error code.\n\nThat makes the uses of --test-bitmap in the test suite (e.g., in t5310)\nmostly pointless: even if we saw an error, the tests wouldn't notice.\nLet's instead call die(), which will let these tests work as designed,\nand alert us if the bitmaps are bogus.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 074d9ac8f2..4431f9f120 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1328,7 +1328,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (bitmap_equals(result, tdata.base))\n \t\tfprintf(stderr, \"OK!\\n\");\n \telse\n-\t\tfprintf(stderr, \"Mismatch!\\n\");\n+\t\tdie(\"mismatch in bitmap results\");\n \n \tfree_bitmap_index(bitmap_git);\n }\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411662","messageId":"055bc1fe66d415bc8e6dcef3e7201d007608e27f.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 04/24] t5310: drop size of truncated ewah bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:27Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe truncate the .bitmap file to 512 bytes and expect to run into\nproblems reading an individual ewah file. But this length is somewhat\narbitrary, and just happened to work when the test was added in\n9d2e330b17 (ewah_read_mmap: bounds-check mmap reads, 2018-06-14).\n\nAn upcoming commit will change the size of the history we create in the\ntest repo, which will cause this test to fail. We can future-proof it a\nbit more by reducing the size of the truncated bitmap file.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex dbe1ffc88a..8a2a3b2114 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -349,7 +349,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411663","messageId":"34137a7f35835c923fe9db39049a124a41ca0839.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 08/24] ewah: implement bitmap_or()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:47Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe have a function to bitwise-OR an ewah into an uncompressed bitmap,\nbut not to OR two uncompressed bitmaps. Let's add it.\n\nInterestingly, we have a public header declaration going back to\ne1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\nfunction was never implemented. That was all OK since there were no\nusers of 'bitmap_or()', but a first caller will be added in a couple of\npatches.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 6f9e5c529b..0a3502603f 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -122,6 +122,15 @@ void bitmap_and_not(struct bitmap *self, struct bitmap *other)\n \t\tself->words[i] &= ~other->words[i];\n }\n \n+void bitmap_or(struct bitmap *self, const struct bitmap *other)\n+{\n+\tsize_t i;\n+\n+\tbitmap_grow(self, other->word_alloc);\n+\tfor (i = 0; i < other->word_alloc; i++)\n+\t\tself->words[i] |= other->words[i];\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411664","messageId":"4b56f12932c0fd9e47a82a1adbeb4080a894013f.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 07/24] ewah: make bitmap growth less aggressive","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:42Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nIf you ask to set a bit in the Nth word and we haven't yet allocated\nthat many slots in our array, we'll increase the bitmap size to 2*N.\nThis means we might frequently end up with bitmaps that are twice the\nnecessary size (as soon as you ask for the biggest bit, we'll size up to\ntwice that).\n\nBut if we just allocate as many words as were asked for, we may not grow\nfast enough. The worst case there is setting bit 0, then 1, etc. Each\ntime we grow we'd just extend by one more word, giving us linear\nreallocations (and quadratic memory copies).\n\nA middle ground is relying on alloc_nr(), which causes us to grow by a\nfactor of roughly 3/2 instead of 2. That's less aggressive than\ndoubling, and it may help avoid fragmenting memory. (If we start with N,\nthen grow twice, our total is N*(3/2)^2 = 9N/4. After growing twice,\nthat array of size 9N/4 can fit into the space vacated by the original\narray and first growth, N+3N/2 = 10N/4 > 9N/4, leading to less\nfragmentation in memory).\n\nOur worst case is still 3/2N wasted bits (you set bit N-1, then setting\nbit N causes us to grow by 3/2), but our average should be much better.\n\nThis isn't usually that big a deal, but it will matter as we shift the\nreachability bitmap generation code to store more bitmaps in memory.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 7c1ecfa6fd..6f9e5c529b 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -37,13 +37,10 @@ struct bitmap *bitmap_new(void)\n \n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n-\tif (word_alloc > self->word_alloc) {\n-\t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = word_alloc * 2;\n-\t\tREALLOC_ARRAY(self->words, self->word_alloc);\n-\t\tmemset(self->words + old_size, 0x0,\n-\t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n-\t}\n+\tsize_t old_size = self->word_alloc;\n+\tALLOC_GROW(self->words, word_alloc, self->word_alloc);\n+\tmemset(self->words + old_size, 0x0,\n+\t (self->word_alloc - old_size) * sizeof(eword_t));\n }\n \n void bitmap_set(struct bitmap *self, size_t pos)\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411665","messageId":"fe89f87716739a9a65dca06a9eae6ae1fbfeefea.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 09/24] ewah: add bitmap_dup() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:51Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThere's no easy way to make a copy of a bitmap. Obviously a caller can\niterate over the bits and set them one by one in a new bitmap, but we\ncan go much faster by copying whole words with memcpy().\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 7 +++++++\n ewah/ewok.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 0a3502603f..b5f6376282 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,6 +35,13 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n+struct bitmap *bitmap_dup(const struct bitmap *src)\n+{\n+\tstruct bitmap *dst = bitmap_word_alloc(src->word_alloc);\n+\tCOPY_ARRAY(dst->words, src->words, src->word_alloc);\n+\treturn dst;\n+}\n+\n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n \tsize_t old_size = self->word_alloc;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 011852bef1..1fc555e672 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -173,6 +173,7 @@ struct bitmap {\n \n struct bitmap *bitmap_new(void);\n struct bitmap *bitmap_word_alloc(size_t word_alloc);\n+struct bitmap *bitmap_dup(const struct bitmap *src);\n void bitmap_set(struct bitmap *self, size_t pos);\n void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411666","messageId":"64598024ecfe60428df800caabb4e2c6efc0f905.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:00Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nOur algorithm to generate reachability bitmaps walks through the commit\ngraph from the bottom up, passing bitmap data from each commit to its\ndescendants. For a linear stretch of history like:\n\n A -- B -- C\n\nour sequence of steps is:\n\n - compute the bitmap for A by walking its trees, etc\n\n - duplicate A's bitmap as a starting point for B; we can now free A's\n bitmap, since we only needed it as an intermediate result\n\n - OR in any extra objects that B can reach into its bitmap\n\n - duplicate B's bitmap as a starting point for C; likewise, free B's\n bitmap\n\n - OR in objects for C, and so on...\n\nRather than duplicating bitmaps and immediately freeing the original, we\ncan just pass ownership from commit to commit. Note that this doesn't\nalways work:\n\n - the recipient may be a merge which already has an intermediate\n bitmap from its other ancestor. In that case we have to OR our\n result into it. Note that the first ancestor to reach the merge does\n get to pass ownership, though.\n\n - we may have multiple children; we can only pass ownership to one of\n them\n\nHowever, it happens often enough and copying bitmaps is expensive enough\nthat this provides a noticeable speedup. On a clone of linux.git, this\nreduces the time to generate bitmaps from 205s to 70s. This is about the\nsame amount of time it took to generate bitmaps using our old \"many\ntraversals\" algorithm (the previous commit measures the identical\nscenario as taking 63s). It unfortunately provides only a very modest\nreduction in the peak memory usage, though.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex bcd059ccd9..1eb9615df8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -333,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n \t\tstruct commit *child;\n+\t\tint reused = 0;\n \n \t\tfill_bitmap_commit(ent, commit);\n \n@@ -348,10 +349,15 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \t\t\tif (child_ent->bitmap)\n \t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n-\t\t\telse\n+\t\t\telse if (reused)\n \t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t\telse {\n+\t\t\t\tchild_ent->bitmap = ent->bitmap;\n+\t\t\t\treused = 1;\n+\t\t\t}\n \t\t}\n-\t\tbitmap_free(ent->bitmap);\n+\t\tif (!reused)\n+\t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n \tbitmap_builder_clear(&bb);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411667","messageId":"b79360383e298051a26e2dc353548f9799f43424.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 06/24] ewah: factor out bitmap growth","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:37Z","receivedAt":"2020-12-08T00:05:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe auto-grow bitmaps when somebody asks to set a bit whose position is\noutside of our currently allocated range. Other operations besides\nsingle bit-setting might need to do this, too, so let's pull it into its\nown function.\n\nNote that we change the semantics a little: you now ask for the number\nof words you'd like to have, not the id of the block you'd like to write\nto.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 +++++++++-----\n 1 file changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d8cec585af..7c1ecfa6fd 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,18 +35,22 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n-void bitmap_set(struct bitmap *self, size_t pos)\n+static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n-\tsize_t block = EWAH_BLOCK(pos);\n-\n-\tif (block >= self->word_alloc) {\n+\tif (word_alloc > self->word_alloc) {\n \t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = block ? block * 2 : 1;\n+\t\tself->word_alloc = word_alloc * 2;\n \t\tREALLOC_ARRAY(self->words, self->word_alloc);\n \t\tmemset(self->words + old_size, 0x0,\n \t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n \t}\n+}\n \n+void bitmap_set(struct bitmap *self, size_t pos)\n+{\n+\tsize_t block = EWAH_BLOCK(pos);\n+\n+\tbitmap_grow(self, block + 1);\n \tself->words[block] |= EWAH_MASK(pos);\n }\n \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411668","messageId":"93fc437a3c1b4ac3bdf1c241b2178281348ba561.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:05Z","receivedAt":"2020-12-08T00:05:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe current implementation of bitmap_writer_build() creates a\nreachability bitmap for every walked commit. After computing a bitmap\nfor a commit, those bits are pushed to an in-progress bitmap for its\nchildren.\n\nfill_bitmap_commit() assumes the bits corresponding to objects\nreachable from the parents of a commit are already set. This means that\nwhen visiting a new commit, we only have to walk the objects reachable\nbetween it and any of its parents.\n\nA future change to bitmap_writer_build() will relax this condition so\nnot all parents have their bits set. Prepare for that by having\n'fill_bitmap_commit()' walk parents until reaching commits whose bits\nare already set. Then, walk the trees for these commits as well.\n\nThis has no functional change with the current implementation of\nbitmap_writer_build().\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 30 +++++++++++++++++++++++-------\n 1 file changed, 23 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 1eb9615df8..957639241e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -12,6 +12,7 @@\n #include \"sha1-lookup.h\"\n #include \"pack-objects.h\"\n #include \"commit-reach.h\"\n+#include \"prio-queue.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -279,17 +280,30 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n }\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit)\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\t/*\n-\t * mark ourselves, but do not bother with parents; their values\n-\t * will already have been propagated to us\n-\t */\n \tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n-\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+\tprio_queue_put(queue, commit);\n+\n+\twhile (queue->nr) {\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = prio_queue_get(queue);\n+\n+\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n+\t\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\t\tprio_queue_put(queue, p->item);\n+\t\t\t}\n+\t\t}\n+\t}\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -319,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct bitmap_builder bb;\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -335,7 +350,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit);\n+\t\tfill_bitmap_commit(ent, commit, &queue);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -360,6 +375,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n+\tclear_prio_queue(&queue);\n \tbitmap_builder_clear(&bb);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411671","messageId":"91cd8b1a49290095ba55955d86f29cf3afd5e1ce.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 10/24] pack-bitmap-write: reimplement bitmap writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:04:56Z","receivedAt":"2020-12-08T00:05:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe bitmap generation code works by iterating over the set of commits\nfor which we plan to write bitmaps, and then for each one performing a\ntraditional traversal over the reachable commits and trees, filling in\nthe bitmap. Between two traversals, we can often reuse the previous\nbitmap result as long as the first commit is an ancestor of the second.\nHowever, our worst case is that we may end up doing \"n\" complete\ncomplete traversals to the root in order to create \"n\" bitmaps.\n\nIn a real-world case (the shared-storage repo consisting of all GitHub\nforks of chromium/chromium), we perform very poorly: generating bitmaps\ntakes ~3 hours, whereas we can walk the whole object graph in ~3\nminutes.\n\nThis commit completely rewrites the algorithm, with the goal of\naccessing each object only once. It works roughly like this:\n\n - generate a list of commits in topo-order using a single traversal\n\n - invert the edges of the graph (so have parents point at their\n children)\n\n - make one pass in reverse topo-order, generating a bitmap for each\n commit and passing the result along to child nodes\n\nWe generate correct results because each node we visit has already had\nall of its ancestors added to the bitmap. And we make only two linear\npasses over the commits.\n\nWe also visit each tree usually only once. When filling in a bitmap, we\ndon't bother to recurse into trees whose bit is already set in the\nbitmap (since we know we've already done so when setting their bit).\nThat means that if commit A references tree T, none of its descendants\nwill need to open T again. I say \"usually\", though, because it is\npossible for a given tree to be mentioned in unrelated parts of history\n(e.g., cherry-picking to a parallel branch).\n\nSo we've accomplished our goal, and the resulting algorithm is pretty\nsimple to understand. But there are some downsides, at least with this\ninitial implementation:\n\n - we no longer reuse the results of any on-disk bitmaps when\n generating. So we'd expect to sometimes be slower than the original\n when bitmaps already exist. However, this is something we'll be able\n to add back in later.\n\n - we use much more memory. Instead of keeping one bitmap in memory at\n a time, we're passing them up through the graph. So our memory use\n should scale with the graph width (times the size of a bitmap).\n\nSo how does it perform?\n\nFor a clone of linux.git, generating bitmaps from scratch with the old\nalgorithm took 63s. Using this algorithm it takes 205s. Which is much\nworse, but _might_ be acceptable if it behaved linearly as the size\ngrew. It also increases peak heap usage by ~1G. That's not impossibly\nlarge, but not encouraging.\n\nOn the complete fork-network of torvalds/linux, it increases the peak\nRAM usage by 40GB. Yikes. (I forgot to record the time it took, but the\nmemory usage was too much to consider this reasonable anyway).\n\nOn the complete fork-network of chromium/chromium, I ran out of memory\nbefore succeeding. Some back-of-the-envelope calculations indicate it\nwould need 80+GB to complete.\n\nSo at this stage, we've managed to make things much worse. But because\nof the way this new algorithm is structured, there are a lot of\nopportunities for optimization on top. We'll start implementing those in\nthe follow-on patches.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 306 +++++++++++++++++++++++++-------------------\n 1 file changed, 172 insertions(+), 134 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 5e998bdaa7..bcd059ccd9 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -110,8 +110,6 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n /**\n * Compute the actual bitmaps\n */\n-static struct object **seen_objects;\n-static unsigned int seen_objects_nr, seen_objects_alloc;\n \n static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n {\n@@ -127,21 +125,6 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \twriter.selected_nr++;\n }\n \n-static inline void mark_as_seen(struct object *object)\n-{\n-\tALLOC_GROW(seen_objects, seen_objects_nr + 1, seen_objects_alloc);\n-\tseen_objects[seen_objects_nr++] = object;\n-}\n-\n-static inline void reset_all_seen(void)\n-{\n-\tunsigned int i;\n-\tfor (i = 0; i < seen_objects_nr; ++i) {\n-\t\tseen_objects[i]->flags &= ~(SEEN | ADDED | SHOWN);\n-\t}\n-\tseen_objects_nr = 0;\n-}\n-\n static uint32_t find_object_pos(const struct object_id *oid)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n@@ -154,60 +137,6 @@ static uint32_t find_object_pos(const struct object_id *oid)\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n-static void show_object(struct object *object, const char *name, void *data)\n-{\n-\tstruct bitmap *base = data;\n-\tbitmap_set(base, find_object_pos(&object->oid));\n-\tmark_as_seen(object);\n-}\n-\n-static void show_commit(struct commit *commit, void *data)\n-{\n-\tmark_as_seen((struct object *)commit);\n-}\n-\n-static int\n-add_to_include_set(struct bitmap *base, struct commit *commit)\n-{\n-\tkhiter_t hash_pos;\n-\tuint32_t bitmap_pos = find_object_pos(&commit->object.oid);\n-\n-\tif (bitmap_get(base, bitmap_pos))\n-\t\treturn 0;\n-\n-\thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n-\tif (hash_pos < kh_end(writer.bitmaps)) {\n-\t\tstruct bitmapped_commit *bc = kh_value(writer.bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(base, bc->bitmap);\n-\t\treturn 0;\n-\t}\n-\n-\tbitmap_set(base, bitmap_pos);\n-\treturn 1;\n-}\n-\n-static int\n-should_include(struct commit *commit, void *_data)\n-{\n-\tstruct bitmap *base = _data;\n-\n-\tif (!add_to_include_set(base, commit)) {\n-\t\tstruct commit_list *parent = commit->parents;\n-\n-\t\tmark_as_seen((struct object *)commit);\n-\n-\t\twhile (parent) {\n-\t\t\tparent->item->object.flags |= SEEN;\n-\t\t\tmark_as_seen((struct object *)parent->item);\n-\t\t\tparent = parent->next;\n-\t\t}\n-\n-\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n static void compute_xor_offsets(void)\n {\n \tstatic const int MAX_XOR_OFFSET_SEARCH = 10;\n@@ -248,79 +177,188 @@ static void compute_xor_offsets(void)\n \t}\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+struct bb_commit {\n+\tstruct commit_list *children;\n+\tstruct bitmap *bitmap;\n+\tunsigned selected:1;\n+\tunsigned idx; /* within selected array */\n+};\n+\n+define_commit_slab(bb_data, struct bb_commit);\n+\n+struct bitmap_builder {\n+\tstruct bb_data data;\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+};\n+\n+static void bitmap_builder_init(struct bitmap_builder *bb,\n+\t\t\t\tstruct bitmap_writer *writer)\n {\n-\tstatic const double REUSE_BITMAP_THRESHOLD = 0.2;\n-\n-\tint i, reuse_after, need_reset;\n-\tstruct bitmap *base = bitmap_new();\n \tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tunsigned int i;\n+\n+\tmemset(bb, 0, sizeof(*bb));\n+\tinit_bb_data(&bb->data);\n+\n+\treset_revision_walk();\n+\trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n+\trevs.topo_order = 1;\n+\n+\tfor (i = 0; i < writer->selected_nr; i++) {\n+\t\tstruct commit *c = writer->selected[i].commit;\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tent->selected = 1;\n+\t\tent->idx = i;\n+\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(\"revision walk setup failed\");\n+\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tstruct commit_list *p;\n+\n+\t\tparse_commit_or_die(commit);\n+\n+\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\tbb->commits[bb->commits_nr++] = commit;\n+\n+\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n+\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t}\n+\t}\n+}\n+\n+static void bitmap_builder_clear(struct bitmap_builder *bb)\n+{\n+\tclear_bb_data(&bb->data);\n+\tfree(bb->commits);\n+\tbb->commits_nr = bb->commits_alloc = 0;\n+}\n+\n+static void fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n+{\n+\tuint32_t pos;\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\t/*\n+\t * If our bit is already set, then there is nothing to do. Both this\n+\t * tree and all of its children will be set.\n+\t */\n+\tpos = find_object_pos(&tree->object.oid);\n+\tif (bitmap_get(bitmap, pos))\n+\t\treturn;\n+\tbitmap_set(bitmap, pos);\n+\n+\tif (parse_tree(tree) < 0)\n+\t\tdie(\"unable to load tree object %s\",\n+\t\t oid_to_hex(&tree->object.oid));\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tfill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Gitlink, etc; not reachable */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n+static void fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit)\n+{\n+\tif (!ent->bitmap)\n+\t\tent->bitmap = bitmap_new();\n+\n+\t/*\n+\t * mark ourselves, but do not bother with parents; their values\n+\t * will already have been propagated to us\n+\t */\n+\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n+\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+}\n+\n+static void store_selected(struct bb_commit *ent, struct commit *commit)\n+{\n+\tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n+\tkhiter_t hash_pos;\n+\tint hash_ret;\n+\n+\t/*\n+\t * the \"reuse bitmaps\" phase may have stored something here, but\n+\t * our new algorithm doesn't use it. Drop it.\n+\t */\n+\tif (stored->bitmap)\n+\t\tewah_free(stored->bitmap);\n+\n+\tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n+\n+\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n+\tif (hash_ret == 0)\n+\t\tdie(\"Duplicate entry when writing index: %s\",\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer.bitmaps, hash_pos) = stored;\n+}\n+\n+void bitmap_writer_build(struct packing_data *to_pack)\n+{\n+\tstruct bitmap_builder bb;\n+\tsize_t i;\n+\tint nr_stored = 0; /* for progress */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n \n \tif (writer.show_progress)\n \t\twriter.progress = start_progress(\"Building bitmaps\", writer.selected_nr);\n-\n-\trepo_init_revisions(to_pack->repo, &revs, NULL);\n-\trevs.tag_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.blob_objects = 1;\n-\trevs.no_walk = 0;\n-\n-\trevs.include_check = should_include;\n-\treset_revision_walk();\n-\n-\treuse_after = writer.selected_nr * REUSE_BITMAP_THRESHOLD;\n-\tneed_reset = 0;\n-\n-\tfor (i = writer.selected_nr - 1; i >= 0; --i) {\n-\t\tstruct bitmapped_commit *stored;\n-\t\tstruct object *object;\n-\n-\t\tkhiter_t hash_pos;\n-\t\tint hash_ret;\n-\n-\t\tstored = &writer.selected[i];\n-\t\tobject = (struct object *)stored->commit;\n-\n-\t\tif (stored->bitmap == NULL) {\n-\t\t\tif (i < writer.selected_nr - 1 &&\n-\t\t\t (need_reset ||\n-\t\t\t !in_merge_bases(writer.selected[i + 1].commit,\n-\t\t\t\t\t stored->commit))) {\n-\t\t\t bitmap_reset(base);\n-\t\t\t reset_all_seen();\n-\t\t\t}\n-\n-\t\t\tadd_pending_object(&revs, object, \"\");\n-\t\t\trevs.include_check_data = base;\n-\n-\t\t\tif (prepare_revision_walk(&revs))\n-\t\t\t\tdie(\"revision walk setup failed\");\n-\n-\t\t\ttraverse_commit_list(&revs, show_commit, show_object, base);\n-\n-\t\t\tobject_array_clear(&revs.pending);\n-\n-\t\t\tstored->bitmap = bitmap_to_ewah(base);\n-\t\t\tneed_reset = 0;\n-\t\t} else\n-\t\t\tneed_reset = 1;\n-\n-\t\tif (i >= reuse_after)\n-\t\t\tstored->flags |= BITMAP_FLAG_REUSE;\n-\n-\t\thash_pos = kh_put_oid_map(writer.bitmaps, object->oid, &hash_ret);\n-\t\tif (hash_ret == 0)\n-\t\t\tdie(\"Duplicate entry when writing index: %s\",\n-\t\t\t oid_to_hex(&object->oid));\n-\n-\t\tkh_value(writer.bitmaps, hash_pos) = stored;\n-\t\tdisplay_progress(writer.progress, writer.selected_nr - i);\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n+\t\t\t the_repository);\n+\n+\tbitmap_builder_init(&bb, &writer);\n+\tfor (i = bb.commits_nr; i > 0; i--) {\n+\t\tstruct commit *commit = bb.commits[i-1];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n+\t\tstruct commit *child;\n+\n+\t\tfill_bitmap_commit(ent, commit);\n+\n+\t\tif (ent->selected) {\n+\t\t\tstore_selected(ent, commit);\n+\t\t\tnr_stored++;\n+\t\t\tdisplay_progress(writer.progress, nr_stored);\n+\t\t}\n+\n+\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\t\tstruct bb_commit *child_ent =\n+\t\t\t\tbb_data_at(&bb.data, child);\n+\n+\t\t\tif (child_ent->bitmap)\n+\t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n+\t\t\telse\n+\t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t}\n+\t\tbitmap_free(ent->bitmap);\n+\t\tent->bitmap = NULL;\n \t}\n+\tbitmap_builder_clear(&bb);\n+\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n+\t\t\t the_repository);\n \n-\tbitmap_free(base);\n \tstop_progress(&writer.progress);\n \n \tcompute_xor_offsets();\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411669","messageId":"0d5213ba44351df71d4f1405683c3d0729031f14.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 13/24] bitmap: implement bitmap_is_subset()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:09Z","receivedAt":"2020-12-08T00:05:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_is_subset() function checks if the 'self' bitmap contains any\nbitmaps that are not on in the 'other' bitmap. Up until this patch, it\nhad a declaration, but no implementation or callers. A subsequent patch\nwill want this function, so implement it here.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 21 +++++++++++++++++++++\n ewah/ewok.h | 2 +-\n 2 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex b5f6376282..0d31cdc866 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -195,6 +195,27 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n+{\n+\tsize_t common_size, i;\n+\n+\tif (self->word_alloc < other->word_alloc)\n+\t\tcommon_size = self->word_alloc;\n+\telse {\n+\t\tcommon_size = other->word_alloc;\n+\t\tfor (i = common_size; i < self->word_alloc; i++) {\n+\t\t\tif (self->words[i])\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < common_size; i++) {\n+\t\tif (self->words[i] & ~other->words[i])\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n void bitmap_reset(struct bitmap *bitmap)\n {\n \tmemset(bitmap->words, 0x0, bitmap->word_alloc * sizeof(eword_t));\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 1fc555e672..66920965da 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -180,7 +180,7 @@ int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_reset(struct bitmap *self);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n-int bitmap_is_subset(struct bitmap *self, struct bitmap *super);\n+int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411670","messageId":"72e745fed8e357e8af6725ab8a0929752c0d2593.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 14/24] commit: implement commit_list_contains()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:13Z","receivedAt":"2020-12-08T00:05:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to check if a commit_list contains a commit. Use\npointer equality, assuming lookup_commit() was used.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit.c | 11 +++++++++++\n commit.h | 2 ++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/commit.c b/commit.c\nindex fe1fa3dc41..9a785bf906 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -544,6 +544,17 @@ struct commit_list *commit_list_insert(struct commit *item, struct commit_list *\n \treturn new_list;\n }\n \n+int commit_list_contains(struct commit *item, struct commit_list *list)\n+{\n+\twhile (list) {\n+\t\tif (list->item == item)\n+\t\t\treturn 1;\n+\t\tlist = list->next;\n+\t}\n+\n+\treturn 0;\n+}\n+\n unsigned commit_list_count(const struct commit_list *l)\n {\n \tunsigned c = 0;\ndiff --git a/commit.h b/commit.h\nindex 5467786c7b..742a6de460 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -167,6 +167,8 @@ int find_commit_subject(const char *commit_buffer, const char **subject);\n \n struct commit_list *commit_list_insert(struct commit *item,\n \t\t\t\t\tstruct commit_list **list);\n+int commit_list_contains(struct commit *item,\n+\t\t\t struct commit_list *list);\n struct commit_list **commit_list_append(struct commit *commit,\n \t\t\t\t\tstruct commit_list **next);\n unsigned commit_list_count(const struct commit_list *l);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411672","messageId":"c2cae4a8d0a000b1c42f1617c9872b6cbf00babd.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 15/24] t5310: add branch-based checks","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:17Z","receivedAt":"2020-12-08T00:06:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe current rev-list tests that check the bitmap data only work on HEAD\ninstead of multiple branches. Expand the test cases to handle both\n'master' and 'other' branches.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 61 +++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 8a2a3b2114..b1248f1cc8 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -41,63 +41,70 @@ test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n \tgit rev-list --test-bitmap HEAD\n '\n \n-rev_list_tests() {\n-\tstate=$1\n-\n-\ttest_expect_success \"counting commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD >actual &&\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting partial commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD~5..HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD~5..HEAD >actual &&\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limit ($state)\" '\n-\t\tgit rev-list --count -n 1 HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 HEAD >actual &&\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting non-linear history ($state)\" '\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n \t\tgit rev-list --count other...master >expect &&\n \t\tgit rev-list --use-bitmap-index --count other...master >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limiting ($state)\" '\n-\t\tgit rev-list --count HEAD -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD -- 1.t >actual &&\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting objects via bitmap ($state)\" '\n-\t\tgit rev-list --count --objects HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects HEAD >actual &&\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"enumerate commits ($state)\" '\n-\t\tgit rev-list --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list HEAD >expect &&\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n \t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n \t'\n \n-\ttest_expect_success \"enumerate --objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list --objects HEAD >expect &&\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n \t\ttest_bitmap_traversal expect actual\n \t'\n \n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD tagged-blob >actual &&\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n \t\tgrep $blob actual\n \t'\n }\n \n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"master\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n rev_list_tests 'full bitmap'\n \n test_expect_success 'clone from bitmapped repository' '\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411673","messageId":"37f96360983557f6653af85da45383dc16d8b00c.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 17/24] pack-bitmap.c: check reads more aggressively when loading","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:27Z","receivedAt":"2020-12-08T00:06:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Before 'load_bitmap_entries_v1()' reads an actual EWAH bitmap, it should\ncheck that it can safely do so by ensuring that there are at least 6\nbytes available to be read (four for the commit's index position, and\nthen two more for the xor offset and flags, respectively).\n\nLikewise, it should check that the commit index it read refers to a\nlegitimate object in the pack.\n\nThe first fix catches a truncation bug that was exposed when testing,\nand the second is purely precautionary.\n\nThere are some possible future improvements, not pursued here. They are:\n\n - Computing the correct boundary of the bitmap itself in the caller\n and ensuring that we don't read past it. This may or may not be\n worth it, since in a truncation situation, all bets are off: (is the\n trailer still there and the bitmap entries malformed, or is the\n trailer truncated?). The best we can do is try to read what's there\n as if it's correct data (and protect ourselves when it's obviously\n bogus).\n\n - Avoid the magic \"6\" by teaching read_be32() and read_u8() (both of\n which are custom helpers for this function) to check sizes before\n advancing the pointers.\n\n - Adding more tests in this area. Testing these truncation situations\n are remarkably fragile to even subtle changes in the bitmap\n generation. So, the resulting tests are likely to be quite brittle.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4431f9f120..60c781d100 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -229,11 +229,16 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\tuint32_t commit_idx_pos;\n \t\tstruct object_id oid;\n \n+\t\tif (index->map_size - index->map_pos < 6)\n+\t\t\treturn error(\"corrupt ewah bitmap: truncated header for entry %d\", i);\n+\n \t\tcommit_idx_pos = read_be32(index->map, &index->map_pos);\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tnth_packed_object_id(&oid, index->pack, commit_idx_pos);\n+\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n+\t\t\t\t (unsigned)commit_idx_pos);\n \n \t\tbitmap = read_bitmap_1(index);\n \t\tif (!bitmap)\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411674","messageId":"c0e2b6f5d9eaa4c20047287637a1f6f0e3000ab9.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 16/24] pack-bitmap-write: rename children to reverse_edges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:22Z","receivedAt":"2020-12-08T00:06:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_builder_init() method walks the reachable commits in\ntopological order and constructs a \"reverse graph\" along the way. At the\nmoment, this reverse graph contains an edge from commit A to commit B if\nand only if A is a parent of B. Thus, the name \"children\" is appropriate\nfor for this reverse graph.\n\nIn the next change, we will repurpose the reverse graph to not be\ndirectly-adjacent commits in the commit-graph, but instead a more\nabstract relationship. The previous changes have already incorporated\nthe necessary updates to fill_bitmap_commit() that allow these edges to\nnot be immediate children.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 957639241e..7e218d02a6 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -179,7 +179,7 @@ static void compute_xor_offsets(void)\n }\n \n struct bb_commit {\n-\tstruct commit_list *children;\n+\tstruct commit_list *reverse_edges;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1;\n \tunsigned idx; /* within selected array */\n@@ -228,7 +228,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n \t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n \t\t}\n \t}\n }\n@@ -358,7 +358,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tdisplay_progress(writer.progress, nr_stored);\n \t\t}\n \n-\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\twhile ((child = pop_commit(&ent->reverse_edges))) {\n \t\t\tstruct bb_commit *child_ent =\n \t\t\t\tbb_data_at(&bb.data, child);\n \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411675","messageId":"c6fde2b0c4d47b5f460b5b8d8ca84e155da37414.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 22/24] pack-bitmap-write: use existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:49Z","receivedAt":"2020-12-08T00:06:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nWhen constructing new bitmaps, we perform a commit and tree walk in\nfill_bitmap_commit() and fill_bitmap_tree(). This walk would benefit\nfrom using existing bitmaps when available. We must track the existing\nbitmaps and translate them into the new object order, but this is\ngenerally faster than parsing trees.\n\nIn fill_bitmap_commit(), we must reorder thing somewhat. The priority\nqueue walks commits from newest-to-oldest, which means we correctly stop\nwalking when reaching a commit with a bitmap. However, if we walk trees\ninterleaved with the commits, then we might be parsing trees that are\nactually part of a re-used bitmap. To avoid over-walking trees, add them\nto a LIFO queue and walk them after exploring commits completely.\n\nOn git.git, this reduces a second immediate bitmap computation from 2.0s\nto 1.0s. On linux.git, we go from 32s to 22s. On chromium's fork\nnetwork, we go from 227s to 198s.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 40 ++++++++++++++++++++++++++++++++++++----\n 1 file changed, 36 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 333058854d..76c8236f94 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -340,20 +340,37 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue)\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n \tprio_queue_put(queue, commit);\n \n \twhile (queue->nr) {\n \t\tstruct commit_list *p;\n \t\tstruct commit *c = prio_queue_get(queue);\n \n+\t\tif (old_bitmap && mapping) {\n+\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\t/*\n+\t\t\t * If this commit has an old bitmap, then translate that\n+\t\t\t * bitmap and add its bits to this one. No need to walk\n+\t\t\t * parents or the tree for this commit.\n+\t\t\t */\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Mark ourselves and queue our tree. The commit\n+\t\t * walk ensures we cover all parents.\n+\t\t */\n \t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n-\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tint pos = find_object_pos(&p->item->object.oid);\n@@ -363,6 +380,9 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t}\n \t\t}\n \t}\n+\n+\twhile (tree_queue->nr)\n+\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -386,6 +406,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n+\tstruct prio_queue tree_queue = { NULL };\n+\tstruct bitmap_index *old_bitmap;\n+\tuint32_t *mapping;\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -395,6 +418,12 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n \t\t\t the_repository);\n \n+\told_bitmap = prepare_bitmap_git(to_pack->repo);\n+\tif (old_bitmap)\n+\t\tmapping = create_bitmap_mapping(old_bitmap, to_pack);\n+\telse\n+\t\tmapping = NULL;\n+\n \tbitmap_builder_init(&bb, &writer);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n@@ -402,7 +431,8 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue);\n+\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -428,7 +458,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tent->bitmap = NULL;\n \t}\n \tclear_prio_queue(&queue);\n+\tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n \t\t\t the_repository);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411676","messageId":"e520c8fdc4b09d52a9bef8956361f8a6e72f7856.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 18/24] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:31Z","receivedAt":"2020-12-08T00:06:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_writer_build() method calls bitmap_builder_init() to\nconstruct a list of commits reachable from the selected commits along\nwith a \"reverse graph\". This reverse graph has edges pointing from a\ncommit to other commits that can reach that commit. After computing a\nreachability bitmap for a commit, the values in that bitmap are then\ncopied to the reachability bitmaps across the edges in the reverse\ngraph.\n\nWe can now relax the role of the reverse graph to greatly reduce the\nnumber of intermediate reachability bitmaps we compute during this\nreverse walk. The end result is that we walk objects the same number of\ntimes as before when constructing the reachability bitmaps, but we also\nspend much less time copying bits between bitmaps and have much lower\nmemory pressure in the process.\n\nThe core idea is to select a set of \"important\" commits based on\ninteractions among the sets of commits reachable from each selected commit.\n\nThe first technical concept is to create a new 'commit_mask' member in the\nbb_commit struct. Note that the selected commits are provided in an\nordered array. The first thing to do is to mark the ith bit in the\ncommit_mask for the ith selected commit. As we walk the commit-graph, we\ncopy the bits in a commit's commit_mask to its parents. At the end of\nthe walk, the ith bit in the commit_mask for a commit C stores a boolean\nrepresenting \"The ith selected commit can reach C.\"\n\nAs we walk, we will discover non-selected commits that are important. We\nwill get into this later, but those important commits must also receive\nbit positions, growing the width of the bitmasks as we walk. At the true\nend of the walk, the ith bit means \"the ith _important_ commit can reach\nC.\"\n\nMAXIMAL COMMITS\n---------------\n\nWe use a new 'maximal' bit in the bb_commit struct to represent whether\na commit is important or not. The term \"maximal\" comes from the\npartially-ordered set of commits in the commit-graph where C >= P if P\nis a parent of C, and then extending the relationship transitively.\nInstead of taking the maximal commits across the entire commit-graph, we\ninstead focus on selecting each commit that is maximal among commits\nwith the same bits on in their commit_mask. This definition is\nimportant, so let's consider an example.\n\nSuppose we have three selected commits A, B, and C. These are assigned\nbitmasks 100, 010, and 001 to start. Each of these can be marked as\nmaximal immediately because they each will be the uniquely maximal\ncommit that contains their own bit. Keep in mind that that these commits\nmay have different bitmasks after the walk; for example, if B can reach\nC but A cannot, then the final bitmask for C is 011. Even in these\ncases, C would still be a maximal commit among all commits with the\nthird bit on in their masks.\n\nNow define sets X, Y, and Z to be the sets of commits reachable from A,\nB, and C, respectively. The intersections of these sets correspond to\ndifferent bitmasks:\n\n * 100: X - (Y union Z)\n * 010: Y - (X union Z)\n * 001: Z - (X union Y)\n * 110: (X intersect Y) - Z\n * 101: (X intersect Z) - Y\n * 011: (Y intersect Z) - X\n * 111: X intersect Y intersect Z\n\nThis can be visualized with the following Hasse diagram:\n\n\t100 010 001\n | \\ / \\ / |\n | \\/ \\/ |\n | /\\ /\\ |\n | / \\ / \\ |\n 110 101 011\n \\___ | ___/\n \\ | /\n 111\n\nSome of these bitmasks may not be represented, depending on the topology\nof the commit-graph. In fact, we are counting on it, since the number of\npossible bitmasks is exponential in the number of selected commits, but\nis also limited by the total number of commits. In practice, very few\nbitmasks are possible because most commits converge on a common \"trunk\"\nin the commit history.\n\nWith this three-bit example, we wish to find commits that are maximal\nfor each bitmask. How can we identify this as we are walking?\n\nAs we walk, we visit a commit C. Since we are walking the commits in\ntopo-order, we know that C is visited after all of its children are\nvisited. Thus, when we get C from the revision walk we inspect the\n'maximal' property of its bb_data and use that to determine if C is truly\nimportant. Its commit_mask is also nearly final. If C is not one of the\noriginally-selected commits, then assign a bit position to C (by\nincrementing num_maximal) and set that bit on in commit_mask. See\n\"MULTIPLE MAXIMAL COMMITS\" below for more detail on this.\n\nNow that the commit C is known to be maximal or not, consider each\nparent P of C. Compute two new values:\n\n * c_not_p : true if and only if the commit_mask for C contains a bit\n that is not contained in the commit_mask for P.\n\n * p_not_c : true if and only if the commit_mask for P contains a bit\n that is not contained in the commit_mask for P.\n\nIf c_not_p is false, then P already has all of the bits that C would\nprovide to its commit_mask. In this case, move on to other parents as C\nhas nothing to contribute to P's state that was not already provided by\nother children of P.\n\nWe continue with the case that c_not_p is true. This means there are\nbits in C's commit_mask to copy to P's commit_mask, so use bitmap_or()\nto add those bits.\n\nIf p_not_c is also true, then set the maximal bit for P to one. This means\nthat if no other commit has P as a parent, then P is definitely maximal.\nThis is because no child had the same bitmask. It is important to think\nabout the maximal bit for P at this point as a temporary state: \"P is\nmaximal based on current information.\"\n\nIn contrast, if p_not_c is false, then set the maximal bit for P to\nzero. Further, clear all reverse_edges for P since any edges that were\npreviously assigned to P are no longer important. P will gain all\nreverse edges based on C.\n\nThe final thing we need to do is to update the reverse edges for P.\nThese reverse edges respresent \"which closest maximal commits\ncontributed bits to my commit_mask?\" Since C contributed bits to P's\ncommit_mask in this case, C must add to the reverse edges of P.\n\nIf C is maximal, then C is a 'closest' maximal commit that contributed\nbits to P. Add C to P's reverse_edges list.\n\nOtherwise, C has a list of maximal commits that contributed bits to its\nbitmask (and this list is exactly one element). Add all of these items\nto P's reverse_edges list. Be careful to ignore duplicates here.\n\nAfter inspecting all parents P for a commit C, we can clear the\ncommit_mask for C. This reduces the memory load to be limited to the\n\"width\" of the commit graph.\n\nConsider our ABC/XYZ example from earlier and let's inspect the state of\nthe commits for an interesting bitmask, say 011. Suppose that D is the\nonly maximal commit with this bitmask (in the first three bits). All\nother commits with bitmask 011 have D as the only entry in their\nreverse_edges list. D's reverse_edges list contains B and C.\n\nCOMPUTING REACHABILITY BITMAPS\n------------------------------\n\nNow that we have our definition, let's zoom out and consider what\nhappens with our new reverse graph when computing reachability bitmaps.\nWe walk the reverse graph in reverse-topo-order, so we visit commits\nwith largest commit_masks first. After we compute the reachability\nbitmap for a commit C, we push the bits in that bitmap to each commit D\nin the reverse edge list for C. Then, when we finally visit D we already\nhave the bits for everything reachable from maximal commits that D can\nreach and we only need to walk the objects in the set-difference.\n\nIn our ABC/XYZ example, when we finally walk for the commit A we only\nneed to walk commits with bitmask equal to A's bitmask. If that bitmask\nis 100, then we are only walking commits in X - (Y union Z) because the\nbitmap already contains the bits for objects reachable from (X intersect\nY) union (X intersect Z) (i.e. the bits from the reachability bitmaps\nfor the maximal commits with bitmasks 110 and 101).\n\nThe behavior is intended to walk each commit (and the trees that commit\nintroduces) at most once while allocating and copying fewer reachability\nbitmaps. There is one caveat: what happens when there are multiple\nmaximal commits with the same bitmask, with respect to the initial set\nof selected commits?\n\nMULTIPLE MAXIMAL COMMITS\n------------------------\n\nEarlier, we mentioned that when we discover a new maximal commit, we\nassign a new bit position to that commit and set that bit position to\none for that commit. This is absolutely important for interesting\ncommit-graphs such as git/git and torvalds/linux. The reason is due to\nthe existence of \"butterflies\" in the commit-graph partial order.\n\nHere is an example of four commits forming a butterfly:\n\n I J\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N\n \\ /\n |/\n Q\n\nHere, I and J both have parents M and N. In general, these do not need\nto be exact parent relationships, but reachability relationships. The\nmost important part is that M and N cannot reach each other, so they are\nindependent in the partial order. If I had commit_mask 10 and J had\ncommit_mask 01, then M and N would both be assigned commit_mask 11 and\nbe maximal commits with the bitmask 11. Then, what happens when M and N\ncan both reach a commit Q? If Q is also assigned the bitmask 11, then it\nis not maximal but is reachable from both M and N.\n\nWhile this is not necessarily a deal-breaker for our abstract definition\nof finding maximal commits according to a given bitmask, we have a few\nissues that can come up in our larger picture of constructing\nreachability bitmaps.\n\nIn particular, if we do not also consider Q to be a \"maximal\" commit,\nthen we will walk commits reachable from Q twice: once when computing\nthe reachability bitmap for M and another time when computing the\nreachability bitmap for N. This becomes much worse if the topology\ncontinues this pattern with multiple butterflies.\n\nThe solution has already been mentioned: each of M and N are assigned\ntheir own bits to the bitmask and hence they become uniquely maximal for\ntheir bitmasks. Finally, Q also becomes maximal and thus we do not need\nto walk its commits multiple times. The final bitmasks for these commits\nare as follows:\n\n I:10 J:01\n |\\ /|\n | \\ _____/ |\n | /\\____ |\n |/ \\ |\n M:111 N:1101\n \\ /\n Q:1111\n\nFurther, Q's reverse edge list is { M, N }, while M and N both have\nreverse edge list { I, J }.\n\nPERFORMANCE MEASUREMENTS\n------------------------\n\nNow that we've spent a LOT of time on the theory of this algorithm,\nlet's show that this is actually worth all that effort.\n\nTo test the performance, use GIT_TRACE2_PERF=1 when running\n'git repack -abd' in a repository with no existing reachability bitmaps.\nThis avoids any issues with keeping existing bitmaps to skew the\nnumbers.\n\nInspect the \"building_bitmaps_total\" region in the trace2 output to\nfocus on the portion of work that is affected by this change. Here are\nthe performance comparisons for a few repositories. The timings are for\nthe following versions of Git: \"multi\" is the timing from before any\nreverse graph is constructed, where we might perform multiple\ntraversals. \"reverse\" is for the previous change where the reverse graph\nhas every reachable commit. Finally \"maximal\" is the version introduced\nhere where the reverse graph only contains the maximal commits.\n\n Repository: git/git\n multi: 2.628 sec\n reverse: 2.344 sec\n maximal: 2.047 sec\n\n Repository: torvalds/linux\n multi: 64.7 sec\n reverse: 205.3 sec\n maximal: 44.7 sec\n\nSo in all cases we've not only recovered any time lost to switching to\nthe reverse-edge algorithm, but we come out ahead of \"multi\" in all\ncases. Likewise, peak heap has gone back to something reasonable:\n\n Repository: torvalds/linux\n multi: 2.087 GB\n reverse: 3.141 GB\n maximal: 2.288 GB\n\nWhile I do not have access to full fork networks on GitHub, Peff has run\nthis algorithm on the chromium/chromium fork network and reported a\nchange from 3 hours to ~233 seconds. That network is particularly\nbeneficial for this approach because it has a long, linear history along\nwith many tags. The \"multi\" approach was obviously quadratic and the new\napproach is linear.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++---\n t/t5310-pack-bitmaps.sh | 85 +++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 148 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 7e218d02a6..0af93193d8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -180,8 +180,10 @@ static void compute_xor_offsets(void)\n \n struct bb_commit {\n \tstruct commit_list *reverse_edges;\n+\tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n-\tunsigned selected:1;\n+\tunsigned selected:1,\n+\t\t maximal:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -198,7 +200,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i;\n+\tunsigned int i, num_maximal;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -210,27 +212,85 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n \t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\n \t\tent->selected = 1;\n+\t\tent->maximal = 1;\n \t\tent->idx = i;\n+\n+\t\tent->commit_mask = bitmap_new();\n+\t\tbitmap_set(ent->commit_mask, i);\n+\n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n+\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n \t\tstruct commit_list *p;\n+\t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n \n-\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n-\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\tc_ent = bb_data_at(&bb->data, commit);\n+\n+\t\tif (c_ent->maximal) {\n+\t\t\tif (!c_ent->selected) {\n+\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n+\t\t\t\tnum_maximal++;\n+\t\t\t}\n+\n+\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\t}\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n-\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n+\t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n+\t\t\tint c_not_p, p_not_c;\n+\n+\t\t\tif (!p_ent->commit_mask) {\n+\t\t\t\tp_ent->commit_mask = bitmap_new();\n+\t\t\t\tc_not_p = 1;\n+\t\t\t\tp_not_c = 0;\n+\t\t\t} else {\n+\t\t\t\tc_not_p = bitmap_is_subset(c_ent->commit_mask, p_ent->commit_mask);\n+\t\t\t\tp_not_c = bitmap_is_subset(p_ent->commit_mask, c_ent->commit_mask);\n+\t\t\t}\n+\n+\t\t\tif (!c_not_p)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_or(p_ent->commit_mask, c_ent->commit_mask);\n+\n+\t\t\tif (p_not_c)\n+\t\t\t\tp_ent->maximal = 1;\n+\t\t\telse {\n+\t\t\t\tp_ent->maximal = 0;\n+\t\t\t\tfree_commit_list(p_ent->reverse_edges);\n+\t\t\t\tp_ent->reverse_edges = NULL;\n+\t\t\t}\n+\n+\t\t\tif (c_ent->maximal) {\n+\t\t\t\tcommit_list_insert(commit, &p_ent->reverse_edges);\n+\t\t\t} else {\n+\t\t\t\tstruct commit_list *cc = c_ent->reverse_edges;\n+\n+\t\t\t\tfor (; cc; cc = cc->next) {\n+\t\t\t\t\tif (!commit_list_contains(cc->item, p_ent->reverse_edges))\n+\t\t\t\t\t\tcommit_list_insert(cc->item, &p_ent->reverse_edges);\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n+\n+\t\tbitmap_free(c_ent->commit_mask);\n+\t\tc_ent->commit_mask = NULL;\n \t}\n+\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_selected_commits\", writer->selected_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_maximal_commits\", num_maximal);\n }\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex b1248f1cc8..4c928221be 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -20,11 +20,87 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other master\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-master * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for master (bit 0) and other (bit 1), and some flexibility\n+# in the order that merge bases are visited, the bitmasks at\n+# the end should be:\n+#\n+# master: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# octo-master: 1\n+# octo-other: 01\n+# merge-right: 111 (maximal)\n+# (l1): 111\n+# (r1): 111\n+# merge-left: 1101 (maximal)\n+# (l2): 11111 (maximal)\n+# (r2): 111101 (maximal)\n+# (base): 1111111 (maximal)\n+\n test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 100 &&\n+\ttest_commit_bulk --id=file 10 &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n+\n+\t# add complicated history setup, including merges and\n+\t# ambiguous merge-bases\n+\n+\tgit checkout -b merge-left other~2 &&\n+\tgit merge master~2 -m \"merge-left\" &&\n+\n+\tgit checkout -b merge-right master~1 &&\n+\tgit merge other~1 -m \"merge-right\" &&\n+\n+\tgit checkout -b octo-master master &&\n+\tgit merge merge-left merge-right -m \"octopus-master\" &&\n+\n+\tgit checkout -b octo-other other &&\n+\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\tgit checkout other &&\n+\tgit merge octo-other -m \"pull octopus\" &&\n+\n \tgit checkout master &&\n+\tgit merge octo-master -m \"pull octopus\" &&\n+\n+\t# Remove these branches so they are not selected\n+\t# as bitmap tips\n+\tgit branch -D merge-left &&\n+\tgit branch -D merge-right &&\n+\tgit branch -D octo-other &&\n+\tgit branch -D octo-master &&\n+\n+\t# add padding to make these merges less interesting\n+\t# and avoid having them selected for bitmaps\n+\ttest_commit_bulk --id=file 100 &&\n+\tgit checkout other &&\n+\ttest_commit_bulk --id=side 100 &&\n+\tgit checkout master &&\n+\n \tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n@@ -32,9 +108,12 @@ test_expect_success 'setup repo with moderate-sized history' '\n '\n \n test_expect_success 'full repack creates bitmaps' '\n-\tgit repack -ad &&\n+\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n+\ttest_line_count = 1 output &&\n+\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411677","messageId":"d5ef2c7f81de5bdba3a013c926a73a48d3208fc5.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 20/24] pack-bitmap: factor out 'bitmap_for_commit()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:40Z","receivedAt":"2020-12-08T00:06:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A couple of callers within pack-bitmap.c duplicate logic to lookup a\ngiven object id in the bitamps khash. Factor this out into a new\nfunction, 'bitmap_for_commit()' to reduce some code duplication.\n\nMake this new function non-static, since it will be used in later\ncommits from outside of pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 33 +++++++++++++++++++--------------\n pack-bitmap.h | 2 ++\n 2 files changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d1368b69bb..5efb8af121 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -380,6 +380,16 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit)\n+{\n+\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n+\t\t\t\t\t commit->object.oid);\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n+\t\treturn NULL;\n+\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t const struct object_id *oid)\n {\n@@ -465,10 +475,10 @@ static void show_commit(struct commit *commit, void *data)\n \n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n-\t\t\t const struct object_id *oid,\n+\t\t\t struct commit *commit,\n \t\t\t int bitmap_pos)\n {\n-\tkhiter_t hash_pos;\n+\tstruct ewah_bitmap *partial;\n \n \tif (data->seen && bitmap_get(data->seen, bitmap_pos))\n \t\treturn 0;\n@@ -476,10 +486,9 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \tif (bitmap_get(data->base, bitmap_pos))\n \t\treturn 0;\n \n-\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, *oid);\n-\tif (hash_pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(data->base, lookup_stored_bitmap(st));\n+\tpartial = bitmap_for_commit(bitmap_git, commit);\n+\tif (partial) {\n+\t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n@@ -498,8 +507,7 @@ static int should_include(struct commit *commit, void *_data)\n \t\t\t\t\t\t (struct object *)commit,\n \t\t\t\t\t\t NULL);\n \n-\tif (!add_to_include_set(data->bitmap_git, data, &commit->object.oid,\n-\t\t\t\tbitmap_pos)) {\n+\tif (!add_to_include_set(data->bitmap_git, data, commit, bitmap_pos)) {\n \t\tstruct commit_list *parent = commit->parents;\n \n \t\twhile (parent) {\n@@ -1282,10 +1290,10 @@ void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n-\tkhiter_t pos;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n \tstruct bitmap_index *bitmap_git;\n+\tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n \t\tdie(\"failed to load bitmap indexes\");\n@@ -1297,12 +1305,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tbitmap_git->version, bitmap_git->entry_count);\n \n \troot = revs->pending.objects[0].item;\n-\tpos = kh_get_oid_map(bitmap_git->bitmaps, root->oid);\n-\n-\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\tstruct ewah_bitmap *bm = lookup_stored_bitmap(st);\n+\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n \n+\tif (bm) {\n \t\tfprintf(stderr, \"Found bitmap for %s. %d bits / %08x checksum\\n\",\n \t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex afa4115136..25dfcf5615 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,6 +78,8 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest);\n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411678","messageId":"c3975fcf78f13a17c1326e002d3321e826f0b2cd.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:36Z","receivedAt":"2020-12-08T00:06:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe on-disk bitmap format has a flag to mark a bitmap to be \"reused\".\nThis is a rather curious feature, and works like this:\n\n - a run of pack-objects would decide to mark the last 80% of the\n bitmaps it generates with the reuse flag\n\n - the next time we generate bitmaps, we'd see those reuse flags from\n the last run, and mark those commits as special:\n\n - we'd be more likely to select those commits to get bitmaps in\n the new output\n\n - when generating the bitmap for a selected commit, we'd reuse the\n old bitmap as-is (rearranging the bits to match the new pack, of\n course)\n\nHowever, neither of these behaviors particularly makes sense.\n\nJust because a commit happened to be bitmapped last time does not make\nit a good candidate for having a bitmap this time. In particular, we may\nchoose bitmaps based on how recent they are in history, or whether a ref\ntip points to them, and those things will change. We're better off\nre-considering fresh which commits are good candidates.\n\nReusing the existing bitmap _is_ a reasonable thing to do to save\ncomputation. But only reusing exact bitmaps is a weak form of this. If\nwe have an old bitmap for A and now want a new bitmap for its child, we\nshould be able to compute that only by looking at trees and that are new\nto the child. But this code would consider only exact reuse (which is\nperhaps why it was eager to select those commits in the first place).\n\nFurthermore, the recent switch to the reverse-edge algorithm for\ngenerating bitmaps dropped this optimization entirely (and yet still\nperforms better).\n\nSo let's do a few cleanups:\n\n - drop the whole \"reusing bitmaps\" phase of generating bitmaps. It's\n not helping anything, and is mostly unused code (or worse, code that\n is using CPU but not doing anything useful)\n\n - drop the use of the on-disk reuse flag to select commits to bitmap\n\n - stop setting the on-disk reuse flag in bitmaps we generate (since\n nothing respects it anymore)\n\nWe will keep a few innards of the reuse code, which will help us\nimplement a more capable version of the \"reuse\" optimization:\n\n - simplify rebuild_existing_bitmaps() into a function that only builds\n the mapping of bits between the old and new orders, but doesn't\n actually convert any bitmaps\n\n - make rebuild_bitmap() public; we'll call it lazily to convert bitmaps\n as we traverse (using the mapping created above)\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 -\n pack-bitmap-write.c | 50 +++++-------------------------------------\n pack-bitmap.c | 46 +++++---------------------------------\n pack-bitmap.h | 6 ++++-\n 4 files changed, 16 insertions(+), 87 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5617c01b5a..2a00358f34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1104,7 +1104,6 @@ static void write_pack_file(void)\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n-\t\t\t\tbitmap_writer_reuse_bitmaps(&to_pack);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 0af93193d8..333058854d 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -30,7 +30,6 @@ struct bitmap_writer {\n \tstruct ewah_bitmap *tags;\n \n \tkh_oid_map_t *bitmaps;\n-\tkh_oid_map_t *reused;\n \tstruct packing_data *to_pack;\n \n \tstruct bitmapped_commit *selected;\n@@ -112,7 +111,7 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n+static inline void push_bitmapped_commit(struct commit *commit)\n {\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n@@ -120,7 +119,7 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \t}\n \n \twriter.selected[writer.selected_nr].commit = commit;\n-\twriter.selected[writer.selected_nr].bitmap = reused;\n+\twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n \n \twriter.selected_nr++;\n@@ -372,13 +371,6 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkhiter_t hash_pos;\n \tint hash_ret;\n \n-\t/*\n-\t * the \"reuse bitmaps\" phase may have stored something here, but\n-\t * our new algorithm doesn't use it. Drop it.\n-\t */\n-\tif (stored->bitmap)\n-\t\tewah_free(stored->bitmap);\n-\n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n \thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n@@ -480,35 +472,6 @@ static int date_compare(const void *_a, const void *_b)\n \treturn (long)b->date - (long)a->date;\n }\n \n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack)\n-{\n-\tstruct bitmap_index *bitmap_git;\n-\tif (!(bitmap_git = prepare_bitmap_git(to_pack->repo)))\n-\t\treturn;\n-\n-\twriter.reused = kh_init_oid_map();\n-\trebuild_existing_bitmaps(bitmap_git, to_pack, writer.reused,\n-\t\t\t\t writer.show_progress);\n-\t/*\n-\t * NEEDSWORK: rebuild_existing_bitmaps() makes writer.reused reference\n-\t * some bitmaps in bitmap_git, so we can't free the latter.\n-\t */\n-}\n-\n-static struct ewah_bitmap *find_reused_bitmap(const struct object_id *oid)\n-{\n-\tkhiter_t hash_pos;\n-\n-\tif (!writer.reused)\n-\t\treturn NULL;\n-\n-\thash_pos = kh_get_oid_map(writer.reused, *oid);\n-\tif (hash_pos >= kh_end(writer.reused))\n-\t\treturn NULL;\n-\n-\treturn kh_value(writer.reused, hash_pos);\n-}\n-\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr,\n \t\t\t\t int max_bitmaps)\n@@ -522,12 +485,11 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(indexed_commits[i], NULL);\n+\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n \tfor (;;) {\n-\t\tstruct ewah_bitmap *reused_bitmap = NULL;\n \t\tstruct commit *chosen = NULL;\n \n \t\tnext = next_commit_index(i);\n@@ -542,15 +504,13 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \t\tif (next == 0) {\n \t\t\tchosen = indexed_commits[i];\n-\t\t\treused_bitmap = find_reused_bitmap(&chosen->object.oid);\n \t\t} else {\n \t\t\tchosen = indexed_commits[i + next];\n \n \t\t\tfor (j = 0; j <= next; ++j) {\n \t\t\t\tstruct commit *cm = indexed_commits[i + j];\n \n-\t\t\t\treused_bitmap = find_reused_bitmap(&cm->object.oid);\n-\t\t\t\tif (reused_bitmap || (cm->object.flags & NEEDS_BITMAP) != 0) {\n+\t\t\t\tif ((cm->object.flags & NEEDS_BITMAP) != 0) {\n \t\t\t\t\tchosen = cm;\n \t\t\t\t\tbreak;\n \t\t\t\t}\n@@ -560,7 +520,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(chosen, reused_bitmap);\n+\t\tpush_bitmapped_commit(chosen);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer.progress, i);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 60c781d100..d1368b69bb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1338,9 +1338,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \tfree_bitmap_index(bitmap_git);\n }\n \n-static int rebuild_bitmap(uint32_t *reposition,\n-\t\t\t struct ewah_bitmap *source,\n-\t\t\t struct bitmap *dest)\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest)\n {\n \tuint32_t pos = 0;\n \tstruct ewah_iterator it;\n@@ -1369,19 +1369,11 @@ static int rebuild_bitmap(uint32_t *reposition,\n \treturn 0;\n }\n \n-int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n-\t\t\t struct packing_data *mapping,\n-\t\t\t kh_oid_map_t *reused_bitmaps,\n-\t\t\t int show_progress)\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping)\n {\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n-\tstruct bitmap *rebuild;\n-\tstruct stored_bitmap *stored;\n-\tstruct progress *progress = NULL;\n-\n-\tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tnum_objects = bitmap_git->pack->num_objects;\n \treposition = xcalloc(num_objects, sizeof(uint32_t));\n@@ -1399,33 +1391,7 @@ int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n \t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n-\trebuild = bitmap_new();\n-\ti = 0;\n-\n-\tif (show_progress)\n-\t\tprogress = start_progress(\"Reusing bitmaps\", 0);\n-\n-\tkh_foreach_value(bitmap_git->bitmaps, stored, {\n-\t\tif (stored->flags & BITMAP_FLAG_REUSE) {\n-\t\t\tif (!rebuild_bitmap(reposition,\n-\t\t\t\t\t lookup_stored_bitmap(stored),\n-\t\t\t\t\t rebuild)) {\n-\t\t\t\thash_pos = kh_put_oid_map(reused_bitmaps,\n-\t\t\t\t\t\t\t stored->oid,\n-\t\t\t\t\t\t\t &hash_ret);\n-\t\t\t\tkh_value(reused_bitmaps, hash_pos) =\n-\t\t\t\t\tbitmap_to_ewah(rebuild);\n-\t\t\t}\n-\t\t\tbitmap_reset(rebuild);\n-\t\t\tdisplay_progress(progress, ++i);\n-\t\t}\n-\t});\n-\n-\tstop_progress(&progress);\n-\n-\tfree(reposition);\n-\tbitmap_free(rebuild);\n-\treturn 0;\n+\treturn reposition;\n }\n \n void free_bitmap_index(struct bitmap_index *b)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 1203120c43..afa4115136 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,7 +73,11 @@ void bitmap_writer_set_checksum(unsigned char *sha1);\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr);\n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping);\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411679","messageId":"f0500190f02643aa5b88d58efe72f826bf616ade.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:44Z","receivedAt":"2020-12-08T00:06:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'find_objects()' currently needs to interact with the bitmaps khash\npretty closely. To make 'find_objects()' read a little more\nstraightforwardly, remove some of the khash-level details into a new\nfunction that describes what it does: 'add_commit_to_bitmap()'.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 36 +++++++++++++++++++++---------------\n 1 file changed, 21 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5efb8af121..d88745fb02 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -521,6 +521,23 @@ static int should_include(struct commit *commit, void *_data)\n \treturn 1;\n }\n \n+static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct bitmap **base,\n+\t\t\t\tstruct commit *commit)\n+{\n+\tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n+\n+\tif (!or_with)\n+\t\treturn 0;\n+\n+\tif (*base == NULL)\n+\t\t*base = ewah_to_bitmap(or_with);\n+\telse\n+\t\tbitmap_or_ewah(*base, or_with);\n+\n+\treturn 1;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -544,21 +561,10 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (object->type == OBJ_COMMIT) {\n-\t\t\tkhiter_t pos = kh_get_oid_map(bitmap_git->bitmaps, object->oid);\n-\n-\t\t\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\t\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\t\t\tstruct ewah_bitmap *or_with = lookup_stored_bitmap(st);\n-\n-\t\t\t\tif (base == NULL)\n-\t\t\t\t\tbase = ewah_to_bitmap(or_with);\n-\t\t\t\telse\n-\t\t\t\t\tbitmap_or_ewah(base, or_with);\n-\n-\t\t\t\tobject->flags |= SEEN;\n-\t\t\t\tcontinue;\n-\t\t\t}\n+\t\tif (object->type == OBJ_COMMIT &&\n+\t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n+\t\t\tobject->flags |= SEEN;\n+\t\t\tcontinue;\n \t\t}\n \n \t\tobject_list_insert(object, ¬_mapped);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411680","messageId":"50d2031debdc8e5fb627851738d2a5b0c4d75266.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 23/24] pack-bitmap-write: relax unique rewalk condition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:53Z","receivedAt":"2020-12-08T00:06:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe previous commits improved the bitmap computation process for very\nlong, linear histories with many refs by removing quadratic growth in\nhow many objects were walked. The strategy of computing \"intermediate\ncommits\" using bitmasks for which refs can reach those commits\npartitioned the poset of reachable objects so each part could be walked\nexactly once. This was effective for linear histories.\n\nHowever, there was a (significant) drawback: wide histories with many\nrefs had an explosion of memory costs to compute the commit bitmasks\nduring the exploration that discovers these intermediate commits. Since\nthese wide histories are unlikely to repeat walking objects, the benefit\nof walking objects multiple times was not expensive before. But now, the\ncommit walk *before computing bitmaps* is incredibly expensive.\n\nIn an effort to discover a happy medium, this change reduces the walk\nfor intermediate commits to only the first-parent history. This focuses\nthe walk on how the histories converge, which still has significant\nreduction in repeat object walks. It is still possible to create\nquadratic behavior in this version, but it is probably less likely in\nrealistic data shapes.\n\nHere is some data taken on a fresh clone of the kernel:\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n original | 64.044 | 83.241 | 2.088 | 2.194 |\n last patch | 45.049 | 37.624 | 2.267 | 2.334 |\n this patch | 88.478 | 53.218 | 2.157 | 2.224 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 14 +++++---------\n t/t5310-pack-bitmaps.sh | 27 ++++++++++++++-------------\n 2 files changed, 19 insertions(+), 22 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 76c8236f94..d2af4a974f 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -199,7 +199,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i, num_maximal;\n+\tunsigned int i, num_maximal = 0;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -207,6 +207,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \treset_revision_walk();\n \trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n \trevs.topo_order = 1;\n+\trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n@@ -221,13 +222,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n-\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n-\t\tstruct commit_list *p;\n+\t\tstruct commit_list *p = commit->parents;\n \t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n@@ -235,16 +235,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n \t\tif (c_ent->maximal) {\n-\t\t\tif (!c_ent->selected) {\n-\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n-\t\t\t\tnum_maximal++;\n-\t\t\t}\n-\n+\t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n \t\t\tbb->commits[bb->commits_nr++] = commit;\n \t\t}\n \n-\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\tif (p) {\n \t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n \t\t\tint c_not_p, p_not_c;\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 4c928221be..332af446a8 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -43,23 +43,24 @@ has_any () {\n # \\|\n # * (base)\n #\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n # The important part for the maximal commit algorithm is how\n # the bitmasks are extended. Assuming starting bit positions\n-# for master (bit 0) and other (bit 1), and some flexibility\n-# in the order that merge bases are visited, the bitmasks at\n-# the end should be:\n+# for master (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n #\n # master: 1 (maximal, selected)\n # other: 01 (maximal, selected)\n-# octo-master: 1\n-# octo-other: 01\n-# merge-right: 111 (maximal)\n-# (l1): 111\n-# (r1): 111\n-# merge-left: 1101 (maximal)\n-# (l2): 11111 (maximal)\n-# (r2): 111101 (maximal)\n-# (base): 1111111 (maximal)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n \n test_expect_success 'setup repo with moderate-sized history' '\n \ttest_commit_bulk --id=file 10 &&\n@@ -113,7 +114,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411681","messageId":"6b9950771e57bd89dadcddaee7f27269e6ded535.1607385833.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"[PATCH v3 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T00:05:57Z","receivedAt":"2020-12-08T00:06:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIf the old bitmap file contains a bitmap for a given commit, then that\ncommit does not need help from intermediate commits in its history to\ncompute its final bitmap. Eject that commit from the walk and insert it\ninto a separate list of reusable commits that are eventually stored in\nthe list of commits for computing bitmaps.\n\nThis helps the repeat bitmap computation task, even if the selected\ncommits shift drastically. This helps when a previously-bitmapped commit\nexists in the first-parent history of a newly-selected commit. Since we\nstop the walk at these commits and we use a first-parent walk, it is\nharder to walk \"around\" these bitmapped commits. It's not impossible,\nbut we can greatly reduce the computation time for many selected\ncommits.\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n last patch | 88.478 | 53.218 | 2.157 | 2.224 |\n this patch | 86.681 | 16.164 | 2.157 | 2.222 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 40 ++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 38 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d2af4a974f..cc5ead9990 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -195,10 +195,13 @@ struct bitmap_builder {\n };\n \n static void bitmap_builder_init(struct bitmap_builder *bb,\n-\t\t\t\tstruct bitmap_writer *writer)\n+\t\t\t\tstruct bitmap_writer *writer,\n+\t\t\t\tstruct bitmap_index *old_bitmap)\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n+\tstruct commit_list *reusable = NULL;\n+\tstruct commit_list *r;\n \tunsigned int i, num_maximal = 0;\n \n \tmemset(bb, 0, sizeof(*bb));\n@@ -234,6 +237,31 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n+\t\t/*\n+\t\t * If there is no commit_mask, there is no reason to iterate\n+\t\t * over this commit; it is not selected (if it were, it would\n+\t\t * not have a blank commit mask) and all its children have\n+\t\t * existing bitmaps (see the comment starting with \"This commit\n+\t\t * has an existing bitmap\" below), so it does not contribute\n+\t\t * anything to the final bitmap file or its descendants.\n+\t\t */\n+\t\tif (!c_ent->commit_mask)\n+\t\t\tcontinue;\n+\n+\t\tif (old_bitmap && bitmap_for_commit(old_bitmap, commit)) {\n+\t\t\t/*\n+\t\t\t * This commit has an existing bitmap, so we can\n+\t\t\t * get its bits immediately without an object\n+\t\t\t * walk. That is, it is reusable as-is and there is no\n+\t\t\t * need to continue walking beyond it.\n+\t\t\t *\n+\t\t\t * Mark it as such and add it to bb->commits separately\n+\t\t\t * to avoid allocating a position in the commit mask.\n+\t\t\t */\n+\t\t\tcommit_list_insert(commit, &reusable);\n+\t\t\tgoto next;\n+\t\t}\n+\n \t\tif (c_ent->maximal) {\n \t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n@@ -278,14 +306,22 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \t\t\t}\n \t\t}\n \n+next:\n \t\tbitmap_free(c_ent->commit_mask);\n \t\tc_ent->commit_mask = NULL;\n \t}\n \n+\tfor (r = reusable; r; r = r->next) {\n+\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\tbb->commits[bb->commits_nr++] = r->item;\n+\t}\n+\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"num_selected_commits\", writer->selected_nr);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"num_maximal_commits\", num_maximal);\n+\n+\tfree_commit_list(reusable);\n }\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\n@@ -420,7 +456,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \telse\n \t\tmapping = NULL;\n \n-\tbitmap_builder_init(&bb, &writer);\n+\tbitmap_builder_init(&bb, &writer, old_bitmap);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n-- \n2.29.2.533.g07db1f5344\n"},{"id":"411742","messageId":"xmqqmtyo6mqi.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"cover.1607385833.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/24] pack-bitmap: bitmap generation improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-08T20:56:05Z","receivedAt":"2020-12-08T20:57:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here's an updated v3 of mine, Stolee, and Peff's series to improve the\n> CPU performance of generating reachability bitmaps.\n\nHas the \"avoid having to assume the default branch name is 'master',\nby naming the initial branch we create our history to use in testing\n'second'\" fix-up by Dscho, which has been queued in 'seen' on top of\nthe previous round of this topic, incorporated to this round? \n\nI think [4/24] and [15/24] can be adjusted by adding this piece from\nDscho to the set-up procedure and ...\n\n@@ -64,6 +64,7 @@ has_any () {\n \n test_expect_success 'setup repo with moderate-sized history' '\n \ttest_commit_bulk --id=file 10 &&\n+\tgit branch -M second &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n \n... fixing the remainder of the test script by adjusting for the\nfallout from the 'master' that is now called 'second'.\n\nThanks.\n\n"},{"id":"411746","messageId":"X8/qG70Wgd7xInq+@nand.local","threadId":"54622","inReplyTo":"xmqqmtyo6mqi.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T21:03:23Z","receivedAt":"2020-12-08T21:04:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 08, 2020 at 12:56:05PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Here's an updated v3 of mine, Stolee, and Peff's series to improve the\n> > CPU performance of generating reachability bitmaps.\n>\n> Has the \"avoid having to assume the default branch name is 'master',\n> by naming the initial branch we create our history to use in testing\n> 'second'\" fix-up by Dscho, which has been queued in 'seen' on top of\n> the previous round of this topic, incorporated to this round?\n\nUnfortunately, no. I wrote you an email a little earlier today, but it's\npossible that our emails may have crossed (vger seems to be rather slow\ntoday...).\n\n> I think [4/24] and [15/24] can be adjusted by adding this piece from\n> Dscho to the set-up procedure and ...\n>\n> @@ -64,6 +64,7 @@ has_any () {\n>\n> test_expect_success 'setup repo with moderate-sized history' '\n> \ttest_commit_bulk --id=file 10 &&\n> +\tgit branch -M second &&\n> \tgit checkout -b other HEAD~5 &&\n> \ttest_commit_bulk --id=side 10 &&\n>\n> ... fixing the remainder of the test script by adjusting for the\n> fallout from the 'master' that is now called 'second'.\n\nThat seems reasonable. Another approach would be to leave these patches\nuntouched and apply Dscho's fixup on the end, but I'm not sure which\nyou'd prefer.\n\nIf the latter, then I think you have everything you need. If the former,\nwould you like a re-submission of this series? Either is fine with me.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"411754","messageId":"cover.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1605123652.git.me@ttaylorr.com","subject":"[PATCH v4 00/24] pack-bitmap: bitmap generation improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:08Z","receivedAt":"2020-12-08T22:04:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here's v4 as requested from [1, 2], but I think we can safely call this\n\"the improved v3\", since all we're doing here is removing new instances\nof \"master\" as part of the ongoing default branch transition.\n\nA range-diff that shows this is included below, but it counts v4's 4/24\nas a complete replacement of v3's, so careful readers are encouraged to\nlook at the inter-diff there.\n\nI also snuck a typo-fix into 23/24 (which has existed unnoticed since\nv1) changing \"rewalk\" to \"revwalk\".\n\nSorry for all of the shuffling around, hopefully this one should do the\ntrick.\n\n[1]: https://lore.kernel.org/git/xmqqmtyo6mqi.fsf@gitster.c.googlers.com/\n[2]: https://lore.kernel.org/git/pull.809.git.1607260623935.gitgitgadget@gmail.com/\n\nDerrick Stolee (9):\n pack-bitmap-write: fill bitmap with commit history\n bitmap: implement bitmap_is_subset()\n commit: implement commit_list_contains()\n t5310: add branch-based checks\n pack-bitmap-write: rename children to reverse_edges\n pack-bitmap-write: build fewer intermediate bitmaps\n pack-bitmap-write: use existing bitmaps\n pack-bitmap-write: relax unique revwalk condition\n pack-bitmap-write: better reuse bitmaps\n\nJeff King (11):\n pack-bitmap: fix header size check\n pack-bitmap: bounds-check size of cache extension\n t5310: drop size of truncated ewah bitmap\n rev-list: die when --test-bitmap detects a mismatch\n ewah: factor out bitmap growth\n ewah: make bitmap growth less aggressive\n ewah: implement bitmap_or()\n ewah: add bitmap_dup() function\n pack-bitmap-write: reimplement bitmap writing\n pack-bitmap-write: pass ownership of intermediate bitmaps\n pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n\nTaylor Blau (4):\n ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW()\n pack-bitmap.c: check reads more aggressively when loading\n pack-bitmap: factor out 'bitmap_for_commit()'\n pack-bitmap: factor out 'add_commit_to_bitmap()'\n\n builtin/pack-objects.c | 1 -\n commit.c | 11 +\n commit.h | 2 +\n ewah/bitmap.c | 54 ++++-\n ewah/ewah_bitmap.c | 15 +-\n ewah/ewok.h | 3 +-\n pack-bitmap-write.c | 474 ++++++++++++++++++++++++++--------------\n pack-bitmap.c | 139 ++++++------\n pack-bitmap.h | 8 +-\n t/t5310-pack-bitmaps.sh | 177 +++++++++++----\n 10 files changed, 583 insertions(+), 301 deletions(-)\n\nRange-diff against v3:\n 1: 0b25ba4ca7 = 1: e72f85f82f ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW()\n 2: b455b248e4 = 2: b24395e4b0 pack-bitmap: fix header size check\n 3: 7322427444 = 3: 97533dba27 pack-bitmap: bounds-check size of cache extension\n 4: 055bc1fe66 < -: ---------- t5310: drop size of truncated ewah bitmap\n -: ---------- > 4: 2e7454d7b9 t5310: drop size of truncated ewah bitmap\n 5: c99cacea67 = 5: 3cb4156372 rev-list: die when --test-bitmap detects a mismatch\n 6: b79360383e = 6: 570bf22425 ewah: factor out bitmap growth\n 7: 4b56f12932 = 7: 48a1949ee6 ewah: make bitmap growth less aggressive\n 8: 34137a7f35 = 8: 04bf0de474 ewah: implement bitmap_or()\n 9: fe89f87716 = 9: c8bd4ed5fa ewah: add bitmap_dup() function\n10: 91cd8b1a49 = 10: bbeb87a95d pack-bitmap-write: reimplement bitmap writing\n11: 64598024ec = 11: f87c11700b pack-bitmap-write: pass ownership of intermediate bitmaps\n12: 93fc437a3c = 12: c466dda576 pack-bitmap-write: fill bitmap with commit history\n13: 0d5213ba44 = 13: 0cfa932b71 bitmap: implement bitmap_is_subset()\n14: 72e745fed8 = 14: 033fb2ed55 commit: implement commit_list_contains()\n15: c2cae4a8d0 ! 15: 76071f9f4e t5310: add branch-based checks\n @@ Commit message\n 'master' and 'other' branches.\n\n Signed-off-by: Derrick Stolee \n + Helped-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n\n ## t/t5310-pack-bitmaps.sh ##\n @@ t/t5310-pack-bitmaps.sh: test_expect_success 'rev-list --test-bitmap verifies bi\n\n -\ttest_expect_success \"counting non-linear history ($state)\" '\n +\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n - \t\tgit rev-list --count other...master >expect &&\n - \t\tgit rev-list --use-bitmap-index --count other...master >actual &&\n + \t\tgit rev-list --count other...second >expect &&\n + \t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n \t\ttest_cmp expect actual\n \t'\n\n @@ t/t5310-pack-bitmaps.sh: test_expect_success 'rev-list --test-bitmap verifies bi\n +rev_list_tests () {\n +\tstate=$1\n +\n -+\tfor branch in \"master\" \"other\"\n ++\tfor branch in \"second\" \"other\"\n +\tdo\n +\t\trev_list_tests_head\n +\tdone\n16: c0e2b6f5d9 = 16: d8c6f0f0bc pack-bitmap-write: rename children to reverse_edges\n17: 37f9636098 = 17: 2e08243706 pack-bitmap.c: check reads more aggressively when loading\n18: e520c8fdc4 ! 18: b4c5d2c3df pack-bitmap-write: build fewer intermediate bitmaps\n @@ Commit message\n\n Helped-by: Jeff King \n Signed-off-by: Derrick Stolee \n + Helped-by: Johannes Schindelin \n Signed-off-by: Taylor Blau \n\n ## pack-bitmap-write.c ##\n @@ t/t5310-pack-bitmaps.sh: has_any () {\n test_expect_success 'setup repo with moderate-sized history' '\n -\ttest_commit_bulk --id=file 100 &&\n +\ttest_commit_bulk --id=file 10 &&\n + \tgit branch -M second &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n +\n @@ t/t5310-pack-bitmaps.sh: has_any () {\n +\t# ambiguous merge-bases\n +\n +\tgit checkout -b merge-left other~2 &&\n -+\tgit merge master~2 -m \"merge-left\" &&\n ++\tgit merge second~2 -m \"merge-left\" &&\n +\n -+\tgit checkout -b merge-right master~1 &&\n ++\tgit checkout -b merge-right second~1 &&\n +\tgit merge other~1 -m \"merge-right\" &&\n +\n -+\tgit checkout -b octo-master master &&\n -+\tgit merge merge-left merge-right -m \"octopus-master\" &&\n ++\tgit checkout -b octo-second second &&\n ++\tgit merge merge-left merge-right -m \"octopus-second\" &&\n +\n +\tgit checkout -b octo-other other &&\n +\tgit merge merge-left merge-right -m \"octopus-other\" &&\n @@ t/t5310-pack-bitmaps.sh: has_any () {\n +\tgit checkout other &&\n +\tgit merge octo-other -m \"pull octopus\" &&\n +\n - \tgit checkout master &&\n -+\tgit merge octo-master -m \"pull octopus\" &&\n + \tgit checkout second &&\n ++\tgit merge octo-second -m \"pull octopus\" &&\n +\n +\t# Remove these branches so they are not selected\n +\t# as bitmap tips\n +\tgit branch -D merge-left &&\n +\tgit branch -D merge-right &&\n +\tgit branch -D octo-other &&\n -+\tgit branch -D octo-master &&\n ++\tgit branch -D octo-second &&\n +\n +\t# add padding to make these merges less interesting\n +\t# and avoid having them selected for bitmaps\n +\ttest_commit_bulk --id=file 100 &&\n +\tgit checkout other &&\n +\ttest_commit_bulk --id=side 100 &&\n -+\tgit checkout master &&\n ++\tgit checkout second &&\n +\n - \tbitmaptip=$(git rev-parse master) &&\n + \tbitmaptip=$(git rev-parse second) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n @@ t/t5310-pack-bitmaps.sh: test_expect_success 'setup repo with moderate-sized history' '\n19: c3975fcf78 = 19: d973cf240d pack-bitmap-write: ignore BITMAP_FLAG_REUSE\n20: d5ef2c7f81 = 20: 4d7a4184ac pack-bitmap: factor out 'bitmap_for_commit()'\n21: f0500190f0 = 21: bd3a16088b pack-bitmap: factor out 'add_commit_to_bitmap()'\n22: c6fde2b0c4 = 22: e0d989b98f pack-bitmap-write: use existing bitmaps\n23: 50d2031deb ! 23: 8f9fdb0f43 pack-bitmap-write: relax unique rewalk condition\n @@ Metadata\n Author: Derrick Stolee \n\n ## Commit message ##\n - pack-bitmap-write: relax unique rewalk condition\n + pack-bitmap-write: relax unique revwalk condition\n\n The previous commits improved the bitmap computation process for very\n long, linear histories with many refs by removing quadratic growth in\n @@ Commit message\n this patch | 88.478 | 53.218 | 2.157 | 2.224 |\n\n Signed-off-by: Derrick Stolee \n + Helped-by: Johannes Schindelin \n Signed-off-by: Taylor Blau \n\n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: static void bitmap_builder_init(struct bitmap_builder *bb,\n\n\n ## t/t5310-pack-bitmaps.sh ##\n +@@ t/t5310-pack-bitmaps.sh: has_any () {\n + # To ensure the logic for \"maximal commits\" is exercised, make\n + # the repository a bit more complicated.\n + #\n +-# other master\n ++# other second\n + # * *\n + # (99 commits) (99 commits)\n + # * *\n + # |\\ /|\n +-# | * octo-other octo-master * |\n ++# | * octo-other octo-second * |\n + # |/|\\_________ ____________/|\\|\n + # | \\ \\/ __________/ |\n + # | | ________/\\ / |\n @@ t/t5310-pack-bitmaps.sh: has_any () {\n # \\|\n # * (base)\n @@ t/t5310-pack-bitmaps.sh: has_any () {\n -# for master (bit 0) and other (bit 1), and some flexibility\n -# in the order that merge bases are visited, the bitmasks at\n -# the end should be:\n -+# for master (bit 0) and other (bit 1), the bitmasks at the\n ++# for second (bit 0) and other (bit 1), the bitmasks at the\n +# end should be:\n #\n - # master: 1 (maximal, selected)\n +-# master: 1 (maximal, selected)\n ++# second: 1 (maximal, selected)\n # other: 01 (maximal, selected)\n -# octo-master: 1\n -# octo-other: 01\n24: 6b9950771e = 24: 720b6e0dc7 pack-bitmap-write: better reuse bitmaps\n--\n2.29.2.533.g07db1f5344\n"},{"id":"411755","messageId":"b24395e4b0c1a8643371616cdae4499edf88939d.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 02/24] pack-bitmap: fix header size check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:19Z","receivedAt":"2020-12-08T22:04:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWhen we parse a .bitmap header, we first check that we have enough bytes\nto make a valid header. We do that based on sizeof(struct\nbitmap_disk_header). However, as of 0f4d6cada8 (pack-bitmap: make bitmap\nheader handling hash agnostic, 2019-02-19), that struct oversizes its\nchecksum member to GIT_MAX_RAWSZ. That means we need to adjust for the\ndifference between that constant and the size of the actual hash we're\nusing. That commit adjusted the code which moves our pointer forward,\nbut forgot to update the size check.\n\nThis meant we were overly strict about the header size (requiring room\nfor a 32-byte worst-case hash, when sha1 is only 20 bytes). But in\npractice it didn't matter because bitmap files tend to have at least 12\nbytes of actual data anyway, so it was unlikely for a valid file to be\ncaught by this.\n\nLet's fix it by pulling the header size into a separate variable and\nusing it in both spots. That fixes the bug and simplifies the code to make\nit harder to have a mismatch like this in the future. It will also come\nin handy in the next patch for more bounds checking.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4077e731e8..fe5647e72e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -138,9 +138,10 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n static int load_bitmap_header(struct bitmap_index *index)\n {\n \tstruct bitmap_disk_header *header = (void *)index->map;\n+\tsize_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n \n-\tif (index->map_size < sizeof(*header) + the_hash_algo->rawsz)\n-\t\treturn error(\"Corrupted bitmap index (missing header data)\");\n+\tif (index->map_size < header_size + the_hash_algo->rawsz)\n+\t\treturn error(\"Corrupted bitmap index (too small)\");\n \n \tif (memcmp(header->magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)) != 0)\n \t\treturn error(\"Corrupted bitmap index file (wrong header)\");\n@@ -164,7 +165,7 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-\tindex->map_pos += sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz;\n+\tindex->map_pos += header_size;\n \treturn 0;\n }\n \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411756","messageId":"e72f85f82ff66d5b8904e36e9571db1a0d52455c.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 01/24] ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:14Z","receivedAt":"2020-12-08T22:04:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'ewah/ewah_bitmap.c:buffer_grow()' is responsible for growing the buffer\nused to store the bits of an EWAH bitmap. It is essentially doing the\nsame task as the 'ALLOC_GROW()' macro, so use that instead.\n\nThis simplifies the callers of 'buffer_grow()', who no longer have to\nask for a specific size, but rather specify how much of the buffer they\nneed. They also no longer need to guard 'buffer_grow()' behind an if\nstatement, since 'ALLOC_GROW()' (and, by extension, 'buffer_grow()') is\na noop if the buffer is already large enough.\n\nBut, the most significant change is that this fixes a bug when calling\nbuffer_grow() with both 'alloc_size' and 'new_size' set to 1. In this\ncase, truncating integer math will leave the new size set to 1, causing\nthe buffer to never grow.\n\nInstead, let alloc_nr() handle this, which asks for '(new_size + 16) * 3\n/ 2' instead of 'new_size * 3 / 2'.\n\nSigned-off-by: Taylor Blau \n---\n ewah/ewah_bitmap.c | 15 ++++-----------\n 1 file changed, 4 insertions(+), 11 deletions(-)\n\ndiff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c\nindex d59b1afe3d..2a8c7c5c33 100644\n--- a/ewah/ewah_bitmap.c\n+++ b/ewah/ewah_bitmap.c\n@@ -19,6 +19,7 @@\n #include \"git-compat-util.h\"\n #include \"ewok.h\"\n #include \"ewok_rlw.h\"\n+#include \"cache.h\"\n \n static inline size_t min_size(size_t a, size_t b)\n {\n@@ -33,20 +34,13 @@ static inline size_t max_size(size_t a, size_t b)\n static inline void buffer_grow(struct ewah_bitmap *self, size_t new_size)\n {\n \tsize_t rlw_offset = (uint8_t *)self->rlw - (uint8_t *)self->buffer;\n-\n-\tif (self->alloc_size >= new_size)\n-\t\treturn;\n-\n-\tself->alloc_size = new_size;\n-\tREALLOC_ARRAY(self->buffer, self->alloc_size);\n+\tALLOC_GROW(self->buffer, new_size, self->alloc_size);\n \tself->rlw = self->buffer + (rlw_offset / sizeof(eword_t));\n }\n \n static inline void buffer_push(struct ewah_bitmap *self, eword_t value)\n {\n-\tif (self->buffer_size + 1 >= self->alloc_size)\n-\t\tbuffer_grow(self, self->buffer_size * 3 / 2);\n-\n+\tbuffer_grow(self, self->buffer_size + 1);\n \tself->buffer[self->buffer_size++] = value;\n }\n \n@@ -137,8 +131,7 @@ void ewah_add_dirty_words(\n \n \t\trlw_set_literal_words(self->rlw, literals + can_add);\n \n-\t\tif (self->buffer_size + can_add >= self->alloc_size)\n-\t\t\tbuffer_grow(self, (self->buffer_size + can_add) * 3 / 2);\n+\t\tbuffer_grow(self, self->buffer_size + can_add);\n \n \t\tif (negate) {\n \t\t\tsize_t i;\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411758","messageId":"2e7454d7b9bf2f1953bee54d578434e6831632cd.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 04/24] t5310: drop size of truncated ewah bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:28Z","receivedAt":"2020-12-08T22:04:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe truncate the .bitmap file to 512 bytes and expect to run into\nproblems reading an individual ewah file. But this length is somewhat\narbitrary, and just happened to work when the test was added in\n9d2e330b17 (ewah_read_mmap: bounds-check mmap reads, 2018-06-14).\n\nAn upcoming commit will change the size of the history we create in the\ntest repo, which will cause this test to fail. We can future-proof it a\nbit more by reducing the size of the truncated bitmap file.\n\nSigned-off-by: Jeff King \nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex dbe1ffc88a..bf094cfe42 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -22,10 +22,11 @@ has_any () {\n \n test_expect_success 'setup repo with moderate-sized history' '\n \ttest_commit_bulk --id=file 100 &&\n+\tgit branch -M second &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n-\tgit checkout master &&\n-\tbitmaptip=$(git rev-parse master) &&\n+\tgit checkout second &&\n+\tbitmaptip=$(git rev-parse second) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true\n@@ -63,8 +64,8 @@ rev_list_tests() {\n \t'\n \n \ttest_expect_success \"counting non-linear history ($state)\" '\n-\t\tgit rev-list --count other...master >expect &&\n-\t\tgit rev-list --use-bitmap-index --count other...master >actual &&\n+\t\tgit rev-list --count other...second >expect &&\n+\t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n@@ -128,7 +129,7 @@ test_expect_success 'setup further non-bitmapped commits' '\n rev_list_tests 'partial bitmap'\n \n test_expect_success 'fetch (partial bitmap)' '\n-\tgit --git-dir=clone.git fetch origin master:master &&\n+\tgit --git-dir=clone.git fetch origin second:second &&\n \tgit rev-parse HEAD >expect &&\n \tgit --git-dir=clone.git rev-parse HEAD >actual &&\n \ttest_cmp expect actual\n@@ -230,7 +231,7 @@ test_expect_success 'full repack, reusing previous bitmaps' '\n '\n \n test_expect_success 'fetch (full bitmap)' '\n-\tgit --git-dir=clone.git fetch origin master:master &&\n+\tgit --git-dir=clone.git fetch origin second:second &&\n \tgit rev-parse HEAD >expect &&\n \tgit --git-dir=clone.git rev-parse HEAD >actual &&\n \ttest_cmp expect actual\n@@ -349,7 +350,7 @@ test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n \ttest_when_finished \"rm -f $bitmap\" &&\n-\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\ttest_copy_bytes 256 <$bitmap >$bitmap.tmp &&\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411757","messageId":"97533dba2720f7709699aaf42afede65a2cb540b.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 03/24] pack-bitmap: bounds-check size of cache extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:24Z","receivedAt":"2020-12-08T22:04:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nA .bitmap file may have a \"name hash cache\" extension, which puts a\nsequence of uint32_t values (one per object) at the end of the file.\nWhen we see a flag indicating this extension, we blindly subtract the\nappropriate number of bytes from our available length. However, if the\n.bitmap file is too short, we'll underflow our length variable and wrap\naround, thinking we have a very large length. This can lead to reading\nout-of-bounds bytes while loading individual ewah bitmaps.\n\nWe can fix this by checking the number of available bytes when we parse\nthe header. The existing \"truncated bitmap\" test is now split into two\ntests: one where we don't have this extension at all (and hence actually\ndo try to read a truncated ewah bitmap) and one where we realize\nup-front that we can't even fit in the cache structure. We'll check\nstderr in each case to make sure we hit the error we're expecting.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 8 ++++++--\n t/t5310-pack-bitmaps.sh | 17 +++++++++++++++--\n 2 files changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fe5647e72e..074d9ac8f2 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -153,14 +153,18 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t/* Parse known bitmap format options */\n \t{\n \t\tuint32_t flags = ntohs(header->options);\n+\t\tsize_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));\n+\t\tunsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;\n \n \t\tif ((flags & BITMAP_OPT_FULL_DAG) == 0)\n \t\t\treturn error(\"Unsupported options for bitmap index file \"\n \t\t\t\t\"(Git requires BITMAP_OPT_FULL_DAG)\");\n \n \t\tif (flags & BITMAP_OPT_HASH_CACHE) {\n-\t\t\tunsigned char *end = index->map + index->map_size - the_hash_algo->rawsz;\n-\t\t\tindex->hashes = ((uint32_t *)end) - index->pack->num_objects;\n+\t\t\tif (cache_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(\"corrupted bitmap index file (too short to fit hash cache)\");\n+\t\t\tindex->hashes = (void *)(index_end - cache_size);\n+\t\t\tindex_end -= cache_size;\n \t\t}\n \t}\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 1d40fcad39..dbe1ffc88a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -343,7 +343,8 @@ test_expect_success 'pack reuse respects --incremental' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_success 'truncated bitmap fails gracefully' '\n+test_expect_success 'truncated bitmap fails gracefully (ewah)' '\n+\ttest_config pack.writebitmaphashcache false &&\n \tgit repack -ad &&\n \tgit rev-list --use-bitmap-index --count --all >expect &&\n \tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n@@ -352,7 +353,19 @@ test_expect_success 'truncated bitmap fails gracefully' '\n \tmv -f $bitmap.tmp $bitmap &&\n \tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n \ttest_cmp expect actual &&\n-\ttest_i18ngrep corrupt stderr\n+\ttest_i18ngrep corrupt.ewah.bitmap stderr\n+'\n+\n+test_expect_success 'truncated bitmap fails gracefully (cache)' '\n+\tgit repack -ad &&\n+\tgit rev-list --use-bitmap-index --count --all >expect &&\n+\tbitmap=$(ls .git/objects/pack/*.bitmap) &&\n+\ttest_when_finished \"rm -f $bitmap\" &&\n+\ttest_copy_bytes 512 <$bitmap >$bitmap.tmp &&\n+\tmv -f $bitmap.tmp $bitmap &&\n+\tgit rev-list --use-bitmap-index --count --all >actual 2>stderr &&\n+\ttest_cmp expect actual &&\n+\ttest_i18ngrep corrupted.bitmap.index stderr\n '\n \n # have_delta \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411759","messageId":"3cb41563725a16f87f65eb75c5c4f61b79abe1cc.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 05/24] rev-list: die when --test-bitmap detects a mismatch","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:33Z","receivedAt":"2020-12-08T22:04:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nYou can use \"git rev-list --test-bitmap HEAD\" to check that bitmaps\nproduce the same answer we'd get from a regular traversal. But if we\ndetect an error, we only print \"mismatch\", and still exit with a\nsuccessful error code.\n\nThat makes the uses of --test-bitmap in the test suite (e.g., in t5310)\nmostly pointless: even if we saw an error, the tests wouldn't notice.\nLet's instead call die(), which will let these tests work as designed,\nand alert us if the bitmaps are bogus.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 074d9ac8f2..4431f9f120 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1328,7 +1328,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \tif (bitmap_equals(result, tdata.base))\n \t\tfprintf(stderr, \"OK!\\n\");\n \telse\n-\t\tfprintf(stderr, \"Mismatch!\\n\");\n+\t\tdie(\"mismatch in bitmap results\");\n \n \tfree_bitmap_index(bitmap_git);\n }\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411760","messageId":"570bf22425b1b9d9088846e39b26b07265079ece.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 06/24] ewah: factor out bitmap growth","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:38Z","receivedAt":"2020-12-08T22:04:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe auto-grow bitmaps when somebody asks to set a bit whose position is\noutside of our currently allocated range. Other operations besides\nsingle bit-setting might need to do this, too, so let's pull it into its\nown function.\n\nNote that we change the semantics a little: you now ask for the number\nof words you'd like to have, not the id of the block you'd like to write\nto.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 +++++++++-----\n 1 file changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d8cec585af..7c1ecfa6fd 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,18 +35,22 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n-void bitmap_set(struct bitmap *self, size_t pos)\n+static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n-\tsize_t block = EWAH_BLOCK(pos);\n-\n-\tif (block >= self->word_alloc) {\n+\tif (word_alloc > self->word_alloc) {\n \t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = block ? block * 2 : 1;\n+\t\tself->word_alloc = word_alloc * 2;\n \t\tREALLOC_ARRAY(self->words, self->word_alloc);\n \t\tmemset(self->words + old_size, 0x0,\n \t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n \t}\n+}\n \n+void bitmap_set(struct bitmap *self, size_t pos)\n+{\n+\tsize_t block = EWAH_BLOCK(pos);\n+\n+\tbitmap_grow(self, block + 1);\n \tself->words[block] |= EWAH_MASK(pos);\n }\n \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411761","messageId":"xmqqo8j45516.fsf@gitster.c.googlers.com","threadId":"54622","inReplyTo":"X8/qG70Wgd7xInq+@nand.local","subject":"Re: [PATCH v3 00/24] pack-bitmap: bitmap generation improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-08T22:03:49Z","receivedAt":"2020-12-08T22:04:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> That seems reasonable. Another approach would be to leave these patches\n> untouched and apply Dscho's fixup on the end, but I'm not sure which\n> you'd prefer.\n\nI'd prefer not to see known breakages that are found before the\ntopic is not yet in 'next' left in the topic, and fix them at the\nsource before the topic gets merged.\n\n"},{"id":"411762","messageId":"48a1949ee62113fdb187287de4e3584c45f444e8.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 07/24] ewah: make bitmap growth less aggressive","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:42Z","receivedAt":"2020-12-08T22:04:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nIf you ask to set a bit in the Nth word and we haven't yet allocated\nthat many slots in our array, we'll increase the bitmap size to 2*N.\nThis means we might frequently end up with bitmaps that are twice the\nnecessary size (as soon as you ask for the biggest bit, we'll size up to\ntwice that).\n\nBut if we just allocate as many words as were asked for, we may not grow\nfast enough. The worst case there is setting bit 0, then 1, etc. Each\ntime we grow we'd just extend by one more word, giving us linear\nreallocations (and quadratic memory copies).\n\nA middle ground is relying on alloc_nr(), which causes us to grow by a\nfactor of roughly 3/2 instead of 2. That's less aggressive than\ndoubling, and it may help avoid fragmenting memory. (If we start with N,\nthen grow twice, our total is N*(3/2)^2 = 9N/4. After growing twice,\nthat array of size 9N/4 can fit into the space vacated by the original\narray and first growth, N+3N/2 = 10N/4 > 9N/4, leading to less\nfragmentation in memory).\n\nOur worst case is still 3/2N wasted bits (you set bit N-1, then setting\nbit N causes us to grow by 3/2), but our average should be much better.\n\nThis isn't usually that big a deal, but it will matter as we shift the\nreachability bitmap generation code to store more bitmaps in memory.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 7c1ecfa6fd..6f9e5c529b 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -37,13 +37,10 @@ struct bitmap *bitmap_new(void)\n \n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n-\tif (word_alloc > self->word_alloc) {\n-\t\tsize_t old_size = self->word_alloc;\n-\t\tself->word_alloc = word_alloc * 2;\n-\t\tREALLOC_ARRAY(self->words, self->word_alloc);\n-\t\tmemset(self->words + old_size, 0x0,\n-\t\t\t(self->word_alloc - old_size) * sizeof(eword_t));\n-\t}\n+\tsize_t old_size = self->word_alloc;\n+\tALLOC_GROW(self->words, word_alloc, self->word_alloc);\n+\tmemset(self->words + old_size, 0x0,\n+\t (self->word_alloc - old_size) * sizeof(eword_t));\n }\n \n void bitmap_set(struct bitmap *self, size_t pos)\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411763","messageId":"c466dda57646b636dd7c6071d9dfad9ea876022f.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 12/24] pack-bitmap-write: fill bitmap with commit history","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:03Z","receivedAt":"2020-12-08T22:05:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe current implementation of bitmap_writer_build() creates a\nreachability bitmap for every walked commit. After computing a bitmap\nfor a commit, those bits are pushed to an in-progress bitmap for its\nchildren.\n\nfill_bitmap_commit() assumes the bits corresponding to objects\nreachable from the parents of a commit are already set. This means that\nwhen visiting a new commit, we only have to walk the objects reachable\nbetween it and any of its parents.\n\nA future change to bitmap_writer_build() will relax this condition so\nnot all parents have their bits set. Prepare for that by having\n'fill_bitmap_commit()' walk parents until reaching commits whose bits\nare already set. Then, walk the trees for these commits as well.\n\nThis has no functional change with the current implementation of\nbitmap_writer_build().\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 30 +++++++++++++++++++++++-------\n 1 file changed, 23 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 1eb9615df8..957639241e 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -12,6 +12,7 @@\n #include \"sha1-lookup.h\"\n #include \"pack-objects.h\"\n #include \"commit-reach.h\"\n+#include \"prio-queue.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -279,17 +280,30 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n }\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n-\t\t\t struct commit *commit)\n+\t\t\t struct commit *commit,\n+\t\t\t struct prio_queue *queue)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\t/*\n-\t * mark ourselves, but do not bother with parents; their values\n-\t * will already have been propagated to us\n-\t */\n \tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n-\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+\tprio_queue_put(queue, commit);\n+\n+\twhile (queue->nr) {\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = prio_queue_get(queue);\n+\n+\t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n+\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tint pos = find_object_pos(&p->item->object.oid);\n+\t\t\tif (!bitmap_get(ent->bitmap, pos)) {\n+\t\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\t\tprio_queue_put(queue, p->item);\n+\t\t\t}\n+\t\t}\n+\t}\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -319,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tstruct bitmap_builder bb;\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -335,7 +350,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit);\n+\t\tfill_bitmap_commit(ent, commit, &queue);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -360,6 +375,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n+\tclear_prio_queue(&queue);\n \tbitmap_builder_clear(&bb);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411764","messageId":"bbeb87a95dd1fee4ed5c8f0311ab488f1021b5df.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 10/24] pack-bitmap-write: reimplement bitmap writing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:55Z","receivedAt":"2020-12-08T22:05:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe bitmap generation code works by iterating over the set of commits\nfor which we plan to write bitmaps, and then for each one performing a\ntraditional traversal over the reachable commits and trees, filling in\nthe bitmap. Between two traversals, we can often reuse the previous\nbitmap result as long as the first commit is an ancestor of the second.\nHowever, our worst case is that we may end up doing \"n\" complete\ncomplete traversals to the root in order to create \"n\" bitmaps.\n\nIn a real-world case (the shared-storage repo consisting of all GitHub\nforks of chromium/chromium), we perform very poorly: generating bitmaps\ntakes ~3 hours, whereas we can walk the whole object graph in ~3\nminutes.\n\nThis commit completely rewrites the algorithm, with the goal of\naccessing each object only once. It works roughly like this:\n\n - generate a list of commits in topo-order using a single traversal\n\n - invert the edges of the graph (so have parents point at their\n children)\n\n - make one pass in reverse topo-order, generating a bitmap for each\n commit and passing the result along to child nodes\n\nWe generate correct results because each node we visit has already had\nall of its ancestors added to the bitmap. And we make only two linear\npasses over the commits.\n\nWe also visit each tree usually only once. When filling in a bitmap, we\ndon't bother to recurse into trees whose bit is already set in the\nbitmap (since we know we've already done so when setting their bit).\nThat means that if commit A references tree T, none of its descendants\nwill need to open T again. I say \"usually\", though, because it is\npossible for a given tree to be mentioned in unrelated parts of history\n(e.g., cherry-picking to a parallel branch).\n\nSo we've accomplished our goal, and the resulting algorithm is pretty\nsimple to understand. But there are some downsides, at least with this\ninitial implementation:\n\n - we no longer reuse the results of any on-disk bitmaps when\n generating. So we'd expect to sometimes be slower than the original\n when bitmaps already exist. However, this is something we'll be able\n to add back in later.\n\n - we use much more memory. Instead of keeping one bitmap in memory at\n a time, we're passing them up through the graph. So our memory use\n should scale with the graph width (times the size of a bitmap).\n\nSo how does it perform?\n\nFor a clone of linux.git, generating bitmaps from scratch with the old\nalgorithm took 63s. Using this algorithm it takes 205s. Which is much\nworse, but _might_ be acceptable if it behaved linearly as the size\ngrew. It also increases peak heap usage by ~1G. That's not impossibly\nlarge, but not encouraging.\n\nOn the complete fork-network of torvalds/linux, it increases the peak\nRAM usage by 40GB. Yikes. (I forgot to record the time it took, but the\nmemory usage was too much to consider this reasonable anyway).\n\nOn the complete fork-network of chromium/chromium, I ran out of memory\nbefore succeeding. Some back-of-the-envelope calculations indicate it\nwould need 80+GB to complete.\n\nSo at this stage, we've managed to make things much worse. But because\nof the way this new algorithm is structured, there are a lot of\nopportunities for optimization on top. We'll start implementing those in\nthe follow-on patches.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 306 +++++++++++++++++++++++++-------------------\n 1 file changed, 172 insertions(+), 134 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 5e998bdaa7..bcd059ccd9 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -110,8 +110,6 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n /**\n * Compute the actual bitmaps\n */\n-static struct object **seen_objects;\n-static unsigned int seen_objects_nr, seen_objects_alloc;\n \n static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n {\n@@ -127,21 +125,6 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \twriter.selected_nr++;\n }\n \n-static inline void mark_as_seen(struct object *object)\n-{\n-\tALLOC_GROW(seen_objects, seen_objects_nr + 1, seen_objects_alloc);\n-\tseen_objects[seen_objects_nr++] = object;\n-}\n-\n-static inline void reset_all_seen(void)\n-{\n-\tunsigned int i;\n-\tfor (i = 0; i < seen_objects_nr; ++i) {\n-\t\tseen_objects[i]->flags &= ~(SEEN | ADDED | SHOWN);\n-\t}\n-\tseen_objects_nr = 0;\n-}\n-\n static uint32_t find_object_pos(const struct object_id *oid)\n {\n \tstruct object_entry *entry = packlist_find(writer.to_pack, oid);\n@@ -154,60 +137,6 @@ static uint32_t find_object_pos(const struct object_id *oid)\n \treturn oe_in_pack_pos(writer.to_pack, entry);\n }\n \n-static void show_object(struct object *object, const char *name, void *data)\n-{\n-\tstruct bitmap *base = data;\n-\tbitmap_set(base, find_object_pos(&object->oid));\n-\tmark_as_seen(object);\n-}\n-\n-static void show_commit(struct commit *commit, void *data)\n-{\n-\tmark_as_seen((struct object *)commit);\n-}\n-\n-static int\n-add_to_include_set(struct bitmap *base, struct commit *commit)\n-{\n-\tkhiter_t hash_pos;\n-\tuint32_t bitmap_pos = find_object_pos(&commit->object.oid);\n-\n-\tif (bitmap_get(base, bitmap_pos))\n-\t\treturn 0;\n-\n-\thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n-\tif (hash_pos < kh_end(writer.bitmaps)) {\n-\t\tstruct bitmapped_commit *bc = kh_value(writer.bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(base, bc->bitmap);\n-\t\treturn 0;\n-\t}\n-\n-\tbitmap_set(base, bitmap_pos);\n-\treturn 1;\n-}\n-\n-static int\n-should_include(struct commit *commit, void *_data)\n-{\n-\tstruct bitmap *base = _data;\n-\n-\tif (!add_to_include_set(base, commit)) {\n-\t\tstruct commit_list *parent = commit->parents;\n-\n-\t\tmark_as_seen((struct object *)commit);\n-\n-\t\twhile (parent) {\n-\t\t\tparent->item->object.flags |= SEEN;\n-\t\t\tmark_as_seen((struct object *)parent->item);\n-\t\t\tparent = parent->next;\n-\t\t}\n-\n-\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n static void compute_xor_offsets(void)\n {\n \tstatic const int MAX_XOR_OFFSET_SEARCH = 10;\n@@ -248,79 +177,188 @@ static void compute_xor_offsets(void)\n \t}\n }\n \n-void bitmap_writer_build(struct packing_data *to_pack)\n+struct bb_commit {\n+\tstruct commit_list *children;\n+\tstruct bitmap *bitmap;\n+\tunsigned selected:1;\n+\tunsigned idx; /* within selected array */\n+};\n+\n+define_commit_slab(bb_data, struct bb_commit);\n+\n+struct bitmap_builder {\n+\tstruct bb_data data;\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+};\n+\n+static void bitmap_builder_init(struct bitmap_builder *bb,\n+\t\t\t\tstruct bitmap_writer *writer)\n {\n-\tstatic const double REUSE_BITMAP_THRESHOLD = 0.2;\n-\n-\tint i, reuse_after, need_reset;\n-\tstruct bitmap *base = bitmap_new();\n \tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tunsigned int i;\n+\n+\tmemset(bb, 0, sizeof(*bb));\n+\tinit_bb_data(&bb->data);\n+\n+\treset_revision_walk();\n+\trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n+\trevs.topo_order = 1;\n+\n+\tfor (i = 0; i < writer->selected_nr; i++) {\n+\t\tstruct commit *c = writer->selected[i].commit;\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tent->selected = 1;\n+\t\tent->idx = i;\n+\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(\"revision walk setup failed\");\n+\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tstruct commit_list *p;\n+\n+\t\tparse_commit_or_die(commit);\n+\n+\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\tbb->commits[bb->commits_nr++] = commit;\n+\n+\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n+\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t}\n+\t}\n+}\n+\n+static void bitmap_builder_clear(struct bitmap_builder *bb)\n+{\n+\tclear_bb_data(&bb->data);\n+\tfree(bb->commits);\n+\tbb->commits_nr = bb->commits_alloc = 0;\n+}\n+\n+static void fill_bitmap_tree(struct bitmap *bitmap,\n+\t\t\t struct tree *tree)\n+{\n+\tuint32_t pos;\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\t/*\n+\t * If our bit is already set, then there is nothing to do. Both this\n+\t * tree and all of its children will be set.\n+\t */\n+\tpos = find_object_pos(&tree->object.oid);\n+\tif (bitmap_get(bitmap, pos))\n+\t\treturn;\n+\tbitmap_set(bitmap, pos);\n+\n+\tif (parse_tree(tree) < 0)\n+\t\tdie(\"unable to load tree object %s\",\n+\t\t oid_to_hex(&tree->object.oid));\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tfill_bitmap_tree(bitmap,\n+\t\t\t\t\t lookup_tree(the_repository, &entry.oid));\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tbitmap_set(bitmap, find_object_pos(&entry.oid));\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Gitlink, etc; not reachable */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n+static void fill_bitmap_commit(struct bb_commit *ent,\n+\t\t\t struct commit *commit)\n+{\n+\tif (!ent->bitmap)\n+\t\tent->bitmap = bitmap_new();\n+\n+\t/*\n+\t * mark ourselves, but do not bother with parents; their values\n+\t * will already have been propagated to us\n+\t */\n+\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n+\tfill_bitmap_tree(ent->bitmap, get_commit_tree(commit));\n+}\n+\n+static void store_selected(struct bb_commit *ent, struct commit *commit)\n+{\n+\tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n+\tkhiter_t hash_pos;\n+\tint hash_ret;\n+\n+\t/*\n+\t * the \"reuse bitmaps\" phase may have stored something here, but\n+\t * our new algorithm doesn't use it. Drop it.\n+\t */\n+\tif (stored->bitmap)\n+\t\tewah_free(stored->bitmap);\n+\n+\tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n+\n+\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n+\tif (hash_ret == 0)\n+\t\tdie(\"Duplicate entry when writing index: %s\",\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer.bitmaps, hash_pos) = stored;\n+}\n+\n+void bitmap_writer_build(struct packing_data *to_pack)\n+{\n+\tstruct bitmap_builder bb;\n+\tsize_t i;\n+\tint nr_stored = 0; /* for progress */\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n \n \tif (writer.show_progress)\n \t\twriter.progress = start_progress(\"Building bitmaps\", writer.selected_nr);\n-\n-\trepo_init_revisions(to_pack->repo, &revs, NULL);\n-\trevs.tag_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.blob_objects = 1;\n-\trevs.no_walk = 0;\n-\n-\trevs.include_check = should_include;\n-\treset_revision_walk();\n-\n-\treuse_after = writer.selected_nr * REUSE_BITMAP_THRESHOLD;\n-\tneed_reset = 0;\n-\n-\tfor (i = writer.selected_nr - 1; i >= 0; --i) {\n-\t\tstruct bitmapped_commit *stored;\n-\t\tstruct object *object;\n-\n-\t\tkhiter_t hash_pos;\n-\t\tint hash_ret;\n-\n-\t\tstored = &writer.selected[i];\n-\t\tobject = (struct object *)stored->commit;\n-\n-\t\tif (stored->bitmap == NULL) {\n-\t\t\tif (i < writer.selected_nr - 1 &&\n-\t\t\t (need_reset ||\n-\t\t\t !in_merge_bases(writer.selected[i + 1].commit,\n-\t\t\t\t\t stored->commit))) {\n-\t\t\t bitmap_reset(base);\n-\t\t\t reset_all_seen();\n-\t\t\t}\n-\n-\t\t\tadd_pending_object(&revs, object, \"\");\n-\t\t\trevs.include_check_data = base;\n-\n-\t\t\tif (prepare_revision_walk(&revs))\n-\t\t\t\tdie(\"revision walk setup failed\");\n-\n-\t\t\ttraverse_commit_list(&revs, show_commit, show_object, base);\n-\n-\t\t\tobject_array_clear(&revs.pending);\n-\n-\t\t\tstored->bitmap = bitmap_to_ewah(base);\n-\t\t\tneed_reset = 0;\n-\t\t} else\n-\t\t\tneed_reset = 1;\n-\n-\t\tif (i >= reuse_after)\n-\t\t\tstored->flags |= BITMAP_FLAG_REUSE;\n-\n-\t\thash_pos = kh_put_oid_map(writer.bitmaps, object->oid, &hash_ret);\n-\t\tif (hash_ret == 0)\n-\t\t\tdie(\"Duplicate entry when writing index: %s\",\n-\t\t\t oid_to_hex(&object->oid));\n-\n-\t\tkh_value(writer.bitmaps, hash_pos) = stored;\n-\t\tdisplay_progress(writer.progress, writer.selected_nr - i);\n+\ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n+\t\t\t the_repository);\n+\n+\tbitmap_builder_init(&bb, &writer);\n+\tfor (i = bb.commits_nr; i > 0; i--) {\n+\t\tstruct commit *commit = bb.commits[i-1];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n+\t\tstruct commit *child;\n+\n+\t\tfill_bitmap_commit(ent, commit);\n+\n+\t\tif (ent->selected) {\n+\t\t\tstore_selected(ent, commit);\n+\t\t\tnr_stored++;\n+\t\t\tdisplay_progress(writer.progress, nr_stored);\n+\t\t}\n+\n+\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\t\tstruct bb_commit *child_ent =\n+\t\t\t\tbb_data_at(&bb.data, child);\n+\n+\t\t\tif (child_ent->bitmap)\n+\t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n+\t\t\telse\n+\t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t}\n+\t\tbitmap_free(ent->bitmap);\n+\t\tent->bitmap = NULL;\n \t}\n+\tbitmap_builder_clear(&bb);\n+\n+\ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n+\t\t\t the_repository);\n \n-\tbitmap_free(base);\n \tstop_progress(&writer.progress);\n \n \tcompute_xor_offsets();\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411765","messageId":"c8bd4ed5fa83212d119e2cac21fda1bd93fb1561.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 09/24] ewah: add bitmap_dup() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:50Z","receivedAt":"2020-12-08T22:05:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThere's no easy way to make a copy of a bitmap. Obviously a caller can\niterate over the bits and set them one by one in a new bitmap, but we\ncan go much faster by copying whole words with memcpy().\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 7 +++++++\n ewah/ewok.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 0a3502603f..b5f6376282 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -35,6 +35,13 @@ struct bitmap *bitmap_new(void)\n \treturn bitmap_word_alloc(32);\n }\n \n+struct bitmap *bitmap_dup(const struct bitmap *src)\n+{\n+\tstruct bitmap *dst = bitmap_word_alloc(src->word_alloc);\n+\tCOPY_ARRAY(dst->words, src->words, src->word_alloc);\n+\treturn dst;\n+}\n+\n static void bitmap_grow(struct bitmap *self, size_t word_alloc)\n {\n \tsize_t old_size = self->word_alloc;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 011852bef1..1fc555e672 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -173,6 +173,7 @@ struct bitmap {\n \n struct bitmap *bitmap_new(void);\n struct bitmap *bitmap_word_alloc(size_t word_alloc);\n+struct bitmap *bitmap_dup(const struct bitmap *src);\n void bitmap_set(struct bitmap *self, size_t pos);\n void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411766","messageId":"04bf0de4742386b2c5bce7bbfdb07cbaefc637c8.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 08/24] ewah: implement bitmap_or()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:46Z","receivedAt":"2020-12-08T22:05:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nWe have a function to bitwise-OR an ewah into an uncompressed bitmap,\nbut not to OR two uncompressed bitmaps. Let's add it.\n\nInterestingly, we have a public header declaration going back to\ne1273106f6 (ewah: compressed bitmap implementation, 2013-11-14), but the\nfunction was never implemented. That was all OK since there were no\nusers of 'bitmap_or()', but a first caller will be added in a couple of\npatches.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 6f9e5c529b..0a3502603f 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -122,6 +122,15 @@ void bitmap_and_not(struct bitmap *self, struct bitmap *other)\n \t\tself->words[i] &= ~other->words[i];\n }\n \n+void bitmap_or(struct bitmap *self, const struct bitmap *other)\n+{\n+\tsize_t i;\n+\n+\tbitmap_grow(self, other->word_alloc);\n+\tfor (i = 0; i < other->word_alloc; i++)\n+\t\tself->words[i] |= other->words[i];\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411767","messageId":"f87c11700b1cada168a50864f14eb8d33b193443.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:03:59Z","receivedAt":"2020-12-08T22:05:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nOur algorithm to generate reachability bitmaps walks through the commit\ngraph from the bottom up, passing bitmap data from each commit to its\ndescendants. For a linear stretch of history like:\n\n A -- B -- C\n\nour sequence of steps is:\n\n - compute the bitmap for A by walking its trees, etc\n\n - duplicate A's bitmap as a starting point for B; we can now free A's\n bitmap, since we only needed it as an intermediate result\n\n - OR in any extra objects that B can reach into its bitmap\n\n - duplicate B's bitmap as a starting point for C; likewise, free B's\n bitmap\n\n - OR in objects for C, and so on...\n\nRather than duplicating bitmaps and immediately freeing the original, we\ncan just pass ownership from commit to commit. Note that this doesn't\nalways work:\n\n - the recipient may be a merge which already has an intermediate\n bitmap from its other ancestor. In that case we have to OR our\n result into it. Note that the first ancestor to reach the merge does\n get to pass ownership, though.\n\n - we may have multiple children; we can only pass ownership to one of\n them\n\nHowever, it happens often enough and copying bitmaps is expensive enough\nthat this provides a noticeable speedup. On a clone of linux.git, this\nreduces the time to generate bitmaps from 205s to 70s. This is about the\nsame amount of time it took to generate bitmaps using our old \"many\ntraversals\" algorithm (the previous commit measures the identical\nscenario as taking 63s). It unfortunately provides only a very modest\nreduction in the peak memory usage, though.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex bcd059ccd9..1eb9615df8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -333,6 +333,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n \t\tstruct commit *child;\n+\t\tint reused = 0;\n \n \t\tfill_bitmap_commit(ent, commit);\n \n@@ -348,10 +349,15 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \n \t\t\tif (child_ent->bitmap)\n \t\t\t\tbitmap_or(child_ent->bitmap, ent->bitmap);\n-\t\t\telse\n+\t\t\telse if (reused)\n \t\t\t\tchild_ent->bitmap = bitmap_dup(ent->bitmap);\n+\t\t\telse {\n+\t\t\t\tchild_ent->bitmap = ent->bitmap;\n+\t\t\t\treused = 1;\n+\t\t\t}\n \t\t}\n-\t\tbitmap_free(ent->bitmap);\n+\t\tif (!reused)\n+\t\t\tbitmap_free(ent->bitmap);\n \t\tent->bitmap = NULL;\n \t}\n \tbitmap_builder_clear(&bb);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411768","messageId":"0cfa932b7175ab2863f42c195c2eea6c711f6553.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 13/24] bitmap: implement bitmap_is_subset()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:08Z","receivedAt":"2020-12-08T22:05:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_is_subset() function checks if the 'self' bitmap contains any\nbitmaps that are not on in the 'other' bitmap. Up until this patch, it\nhad a declaration, but no implementation or callers. A subsequent patch\nwill want this function, so implement it here.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 21 +++++++++++++++++++++\n ewah/ewok.h | 2 +-\n 2 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex b5f6376282..0d31cdc866 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -195,6 +195,27 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n+{\n+\tsize_t common_size, i;\n+\n+\tif (self->word_alloc < other->word_alloc)\n+\t\tcommon_size = self->word_alloc;\n+\telse {\n+\t\tcommon_size = other->word_alloc;\n+\t\tfor (i = common_size; i < self->word_alloc; i++) {\n+\t\t\tif (self->words[i])\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < common_size; i++) {\n+\t\tif (self->words[i] & ~other->words[i])\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n void bitmap_reset(struct bitmap *bitmap)\n {\n \tmemset(bitmap->words, 0x0, bitmap->word_alloc * sizeof(eword_t));\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 1fc555e672..66920965da 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -180,7 +180,7 @@ int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_reset(struct bitmap *self);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n-int bitmap_is_subset(struct bitmap *self, struct bitmap *super);\n+int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411769","messageId":"76071f9f4e04c67bf8b4191df880aa5bfd3348c8.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 15/24] t5310: add branch-based checks","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:17Z","receivedAt":"2020-12-08T22:05:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe current rev-list tests that check the bitmap data only work on HEAD\ninstead of multiple branches. Expand the test cases to handle both\n'master' and 'other' branches.\n\nSigned-off-by: Derrick Stolee \nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/t5310-pack-bitmaps.sh | 61 +++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex bf094cfe42..8bf02336d9 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -42,63 +42,70 @@ test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n \tgit rev-list --test-bitmap HEAD\n '\n \n-rev_list_tests() {\n-\tstate=$1\n-\n-\ttest_expect_success \"counting commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD >actual &&\n+rev_list_tests_head () {\n+\ttest_expect_success \"counting commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting partial commits via bitmap ($state)\" '\n-\t\tgit rev-list --count HEAD~5..HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD~5..HEAD >actual &&\n+\ttest_expect_success \"counting partial commits via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count $branch~5..$branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch~5..$branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limit ($state)\" '\n-\t\tgit rev-list --count -n 1 HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count -n 1 HEAD >actual &&\n+\ttest_expect_success \"counting commits with limit ($state, $branch)\" '\n+\t\tgit rev-list --count -n 1 $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count -n 1 $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting non-linear history ($state)\" '\n+\ttest_expect_success \"counting non-linear history ($state, $branch)\" '\n \t\tgit rev-list --count other...second >expect &&\n \t\tgit rev-list --use-bitmap-index --count other...second >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting commits with limiting ($state)\" '\n-\t\tgit rev-list --count HEAD -- 1.t >expect &&\n-\t\tgit rev-list --use-bitmap-index --count HEAD -- 1.t >actual &&\n+\ttest_expect_success \"counting commits with limiting ($state, $branch)\" '\n+\t\tgit rev-list --count $branch -- 1.t >expect &&\n+\t\tgit rev-list --use-bitmap-index --count $branch -- 1.t >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"counting objects via bitmap ($state)\" '\n-\t\tgit rev-list --count --objects HEAD >expect &&\n-\t\tgit rev-list --use-bitmap-index --count --objects HEAD >actual &&\n+\ttest_expect_success \"counting objects via bitmap ($state, $branch)\" '\n+\t\tgit rev-list --count --objects $branch >expect &&\n+\t\tgit rev-list --use-bitmap-index --count --objects $branch >actual &&\n \t\ttest_cmp expect actual\n \t'\n \n-\ttest_expect_success \"enumerate commits ($state)\" '\n-\t\tgit rev-list --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list HEAD >expect &&\n+\ttest_expect_success \"enumerate commits ($state, $branch)\" '\n+\t\tgit rev-list --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list $branch >expect &&\n \t\ttest_bitmap_traversal --no-confirm-bitmaps expect actual\n \t'\n \n-\ttest_expect_success \"enumerate --objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD >actual &&\n-\t\tgit rev-list --objects HEAD >expect &&\n+\ttest_expect_success \"enumerate --objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch >actual &&\n+\t\tgit rev-list --objects $branch >expect &&\n \t\ttest_bitmap_traversal expect actual\n \t'\n \n-\ttest_expect_success \"bitmap --objects handles non-commit objects ($state)\" '\n-\t\tgit rev-list --objects --use-bitmap-index HEAD tagged-blob >actual &&\n+\ttest_expect_success \"bitmap --objects handles non-commit objects ($state, $branch)\" '\n+\t\tgit rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&\n \t\tgrep $blob actual\n \t'\n }\n \n+rev_list_tests () {\n+\tstate=$1\n+\n+\tfor branch in \"second\" \"other\"\n+\tdo\n+\t\trev_list_tests_head\n+\tdone\n+}\n+\n rev_list_tests 'full bitmap'\n \n test_expect_success 'clone from bitmapped repository' '\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411770","messageId":"d8c6f0f0bced8267575fad2045b45ee726554952.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 16/24] pack-bitmap-write: rename children to reverse_edges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:22Z","receivedAt":"2020-12-08T22:05:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_builder_init() method walks the reachable commits in\ntopological order and constructs a \"reverse graph\" along the way. At the\nmoment, this reverse graph contains an edge from commit A to commit B if\nand only if A is a parent of B. Thus, the name \"children\" is appropriate\nfor for this reverse graph.\n\nIn the next change, we will repurpose the reverse graph to not be\ndirectly-adjacent commits in the commit-graph, but instead a more\nabstract relationship. The previous changes have already incorporated\nthe necessary updates to fill_bitmap_commit() that allow these edges to\nnot be immediate children.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 957639241e..7e218d02a6 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -179,7 +179,7 @@ static void compute_xor_offsets(void)\n }\n \n struct bb_commit {\n-\tstruct commit_list *children;\n+\tstruct commit_list *reverse_edges;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1;\n \tunsigned idx; /* within selected array */\n@@ -228,7 +228,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n \t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->children);\n+\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n \t\t}\n \t}\n }\n@@ -358,7 +358,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\t\tdisplay_progress(writer.progress, nr_stored);\n \t\t}\n \n-\t\twhile ((child = pop_commit(&ent->children))) {\n+\t\twhile ((child = pop_commit(&ent->reverse_edges))) {\n \t\t\tstruct bb_commit *child_ent =\n \t\t\t\tbb_data_at(&bb.data, child);\n \n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411771","messageId":"2e082437060c43b7a6410be1f9ddd2eeb104e4bc.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 17/24] pack-bitmap.c: check reads more aggressively when loading","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:26Z","receivedAt":"2020-12-08T22:06:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Before 'load_bitmap_entries_v1()' reads an actual EWAH bitmap, it should\ncheck that it can safely do so by ensuring that there are at least 6\nbytes available to be read (four for the commit's index position, and\nthen two more for the xor offset and flags, respectively).\n\nLikewise, it should check that the commit index it read refers to a\nlegitimate object in the pack.\n\nThe first fix catches a truncation bug that was exposed when testing,\nand the second is purely precautionary.\n\nThere are some possible future improvements, not pursued here. They are:\n\n - Computing the correct boundary of the bitmap itself in the caller\n and ensuring that we don't read past it. This may or may not be\n worth it, since in a truncation situation, all bets are off: (is the\n trailer still there and the bitmap entries malformed, or is the\n trailer truncated?). The best we can do is try to read what's there\n as if it's correct data (and protect ourselves when it's obviously\n bogus).\n\n - Avoid the magic \"6\" by teaching read_be32() and read_u8() (both of\n which are custom helpers for this function) to check sizes before\n advancing the pointers.\n\n - Adding more tests in this area. Testing these truncation situations\n are remarkably fragile to even subtle changes in the bitmap\n generation. So, the resulting tests are likely to be quite brittle.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4431f9f120..60c781d100 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -229,11 +229,16 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)\n \t\tuint32_t commit_idx_pos;\n \t\tstruct object_id oid;\n \n+\t\tif (index->map_size - index->map_pos < 6)\n+\t\t\treturn error(\"corrupt ewah bitmap: truncated header for entry %d\", i);\n+\n \t\tcommit_idx_pos = read_be32(index->map, &index->map_pos);\n \t\txor_offset = read_u8(index->map, &index->map_pos);\n \t\tflags = read_u8(index->map, &index->map_pos);\n \n-\t\tnth_packed_object_id(&oid, index->pack, commit_idx_pos);\n+\t\tif (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)\n+\t\t\treturn error(\"corrupt ewah bitmap: commit index %u out of range\",\n+\t\t\t\t (unsigned)commit_idx_pos);\n \n \t\tbitmap = read_bitmap_1(index);\n \t\tif (!bitmap)\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411772","messageId":"d973cf240d7ce28c20b9eb2a0207edd1cb2e1968.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:34Z","receivedAt":"2020-12-08T22:06:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThe on-disk bitmap format has a flag to mark a bitmap to be \"reused\".\nThis is a rather curious feature, and works like this:\n\n - a run of pack-objects would decide to mark the last 80% of the\n bitmaps it generates with the reuse flag\n\n - the next time we generate bitmaps, we'd see those reuse flags from\n the last run, and mark those commits as special:\n\n - we'd be more likely to select those commits to get bitmaps in\n the new output\n\n - when generating the bitmap for a selected commit, we'd reuse the\n old bitmap as-is (rearranging the bits to match the new pack, of\n course)\n\nHowever, neither of these behaviors particularly makes sense.\n\nJust because a commit happened to be bitmapped last time does not make\nit a good candidate for having a bitmap this time. In particular, we may\nchoose bitmaps based on how recent they are in history, or whether a ref\ntip points to them, and those things will change. We're better off\nre-considering fresh which commits are good candidates.\n\nReusing the existing bitmap _is_ a reasonable thing to do to save\ncomputation. But only reusing exact bitmaps is a weak form of this. If\nwe have an old bitmap for A and now want a new bitmap for its child, we\nshould be able to compute that only by looking at trees and that are new\nto the child. But this code would consider only exact reuse (which is\nperhaps why it was eager to select those commits in the first place).\n\nFurthermore, the recent switch to the reverse-edge algorithm for\ngenerating bitmaps dropped this optimization entirely (and yet still\nperforms better).\n\nSo let's do a few cleanups:\n\n - drop the whole \"reusing bitmaps\" phase of generating bitmaps. It's\n not helping anything, and is mostly unused code (or worse, code that\n is using CPU but not doing anything useful)\n\n - drop the use of the on-disk reuse flag to select commits to bitmap\n\n - stop setting the on-disk reuse flag in bitmaps we generate (since\n nothing respects it anymore)\n\nWe will keep a few innards of the reuse code, which will help us\nimplement a more capable version of the \"reuse\" optimization:\n\n - simplify rebuild_existing_bitmaps() into a function that only builds\n the mapping of bits between the old and new orders, but doesn't\n actually convert any bitmaps\n\n - make rebuild_bitmap() public; we'll call it lazily to convert bitmaps\n as we traverse (using the mapping created above)\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 -\n pack-bitmap-write.c | 50 +++++-------------------------------------\n pack-bitmap.c | 46 +++++---------------------------------\n pack-bitmap.h | 6 ++++-\n 4 files changed, 16 insertions(+), 87 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5617c01b5a..2a00358f34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1104,7 +1104,6 @@ static void write_pack_file(void)\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n-\t\t\t\tbitmap_writer_reuse_bitmaps(&to_pack);\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 0af93193d8..333058854d 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -30,7 +30,6 @@ struct bitmap_writer {\n \tstruct ewah_bitmap *tags;\n \n \tkh_oid_map_t *bitmaps;\n-\tkh_oid_map_t *reused;\n \tstruct packing_data *to_pack;\n \n \tstruct bitmapped_commit *selected;\n@@ -112,7 +111,7 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitmap *reused)\n+static inline void push_bitmapped_commit(struct commit *commit)\n {\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n@@ -120,7 +119,7 @@ static inline void push_bitmapped_commit(struct commit *commit, struct ewah_bitm\n \t}\n \n \twriter.selected[writer.selected_nr].commit = commit;\n-\twriter.selected[writer.selected_nr].bitmap = reused;\n+\twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n \n \twriter.selected_nr++;\n@@ -372,13 +371,6 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \tkhiter_t hash_pos;\n \tint hash_ret;\n \n-\t/*\n-\t * the \"reuse bitmaps\" phase may have stored something here, but\n-\t * our new algorithm doesn't use it. Drop it.\n-\t */\n-\tif (stored->bitmap)\n-\t\tewah_free(stored->bitmap);\n-\n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n \thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n@@ -480,35 +472,6 @@ static int date_compare(const void *_a, const void *_b)\n \treturn (long)b->date - (long)a->date;\n }\n \n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack)\n-{\n-\tstruct bitmap_index *bitmap_git;\n-\tif (!(bitmap_git = prepare_bitmap_git(to_pack->repo)))\n-\t\treturn;\n-\n-\twriter.reused = kh_init_oid_map();\n-\trebuild_existing_bitmaps(bitmap_git, to_pack, writer.reused,\n-\t\t\t\t writer.show_progress);\n-\t/*\n-\t * NEEDSWORK: rebuild_existing_bitmaps() makes writer.reused reference\n-\t * some bitmaps in bitmap_git, so we can't free the latter.\n-\t */\n-}\n-\n-static struct ewah_bitmap *find_reused_bitmap(const struct object_id *oid)\n-{\n-\tkhiter_t hash_pos;\n-\n-\tif (!writer.reused)\n-\t\treturn NULL;\n-\n-\thash_pos = kh_get_oid_map(writer.reused, *oid);\n-\tif (hash_pos >= kh_end(writer.reused))\n-\t\treturn NULL;\n-\n-\treturn kh_value(writer.reused, hash_pos);\n-}\n-\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr,\n \t\t\t\t int max_bitmaps)\n@@ -522,12 +485,11 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(indexed_commits[i], NULL);\n+\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n \tfor (;;) {\n-\t\tstruct ewah_bitmap *reused_bitmap = NULL;\n \t\tstruct commit *chosen = NULL;\n \n \t\tnext = next_commit_index(i);\n@@ -542,15 +504,13 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \t\tif (next == 0) {\n \t\t\tchosen = indexed_commits[i];\n-\t\t\treused_bitmap = find_reused_bitmap(&chosen->object.oid);\n \t\t} else {\n \t\t\tchosen = indexed_commits[i + next];\n \n \t\t\tfor (j = 0; j <= next; ++j) {\n \t\t\t\tstruct commit *cm = indexed_commits[i + j];\n \n-\t\t\t\treused_bitmap = find_reused_bitmap(&cm->object.oid);\n-\t\t\t\tif (reused_bitmap || (cm->object.flags & NEEDS_BITMAP) != 0) {\n+\t\t\t\tif ((cm->object.flags & NEEDS_BITMAP) != 0) {\n \t\t\t\t\tchosen = cm;\n \t\t\t\t\tbreak;\n \t\t\t\t}\n@@ -560,7 +520,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(chosen, reused_bitmap);\n+\t\tpush_bitmapped_commit(chosen);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer.progress, i);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 60c781d100..d1368b69bb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1338,9 +1338,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \tfree_bitmap_index(bitmap_git);\n }\n \n-static int rebuild_bitmap(uint32_t *reposition,\n-\t\t\t struct ewah_bitmap *source,\n-\t\t\t struct bitmap *dest)\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest)\n {\n \tuint32_t pos = 0;\n \tstruct ewah_iterator it;\n@@ -1369,19 +1369,11 @@ static int rebuild_bitmap(uint32_t *reposition,\n \treturn 0;\n }\n \n-int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n-\t\t\t struct packing_data *mapping,\n-\t\t\t kh_oid_map_t *reused_bitmaps,\n-\t\t\t int show_progress)\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping)\n {\n \tuint32_t i, num_objects;\n \tuint32_t *reposition;\n-\tstruct bitmap *rebuild;\n-\tstruct stored_bitmap *stored;\n-\tstruct progress *progress = NULL;\n-\n-\tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tnum_objects = bitmap_git->pack->num_objects;\n \treposition = xcalloc(num_objects, sizeof(uint32_t));\n@@ -1399,33 +1391,7 @@ int rebuild_existing_bitmaps(struct bitmap_index *bitmap_git,\n \t\t\treposition[i] = oe_in_pack_pos(mapping, oe) + 1;\n \t}\n \n-\trebuild = bitmap_new();\n-\ti = 0;\n-\n-\tif (show_progress)\n-\t\tprogress = start_progress(\"Reusing bitmaps\", 0);\n-\n-\tkh_foreach_value(bitmap_git->bitmaps, stored, {\n-\t\tif (stored->flags & BITMAP_FLAG_REUSE) {\n-\t\t\tif (!rebuild_bitmap(reposition,\n-\t\t\t\t\t lookup_stored_bitmap(stored),\n-\t\t\t\t\t rebuild)) {\n-\t\t\t\thash_pos = kh_put_oid_map(reused_bitmaps,\n-\t\t\t\t\t\t\t stored->oid,\n-\t\t\t\t\t\t\t &hash_ret);\n-\t\t\t\tkh_value(reused_bitmaps, hash_pos) =\n-\t\t\t\t\tbitmap_to_ewah(rebuild);\n-\t\t\t}\n-\t\t\tbitmap_reset(rebuild);\n-\t\t\tdisplay_progress(progress, ++i);\n-\t\t}\n-\t});\n-\n-\tstop_progress(&progress);\n-\n-\tfree(reposition);\n-\tbitmap_free(rebuild);\n-\treturn 0;\n+\treturn reposition;\n }\n \n void free_bitmap_index(struct bitmap_index *b)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 1203120c43..afa4115136 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,7 +73,11 @@ void bitmap_writer_set_checksum(unsigned char *sha1);\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr);\n-void bitmap_writer_reuse_bitmaps(struct packing_data *to_pack);\n+uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct packing_data *mapping);\n+int rebuild_bitmap(const uint32_t *reposition,\n+\t\t struct ewah_bitmap *source,\n+\t\t struct bitmap *dest);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411773","messageId":"4d7a4184ac141115ef2ede59d9af30ae47b3a387.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 20/24] pack-bitmap: factor out 'bitmap_for_commit()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:38Z","receivedAt":"2020-12-08T22:06:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A couple of callers within pack-bitmap.c duplicate logic to lookup a\ngiven object id in the bitamps khash. Factor this out into a new\nfunction, 'bitmap_for_commit()' to reduce some code duplication.\n\nMake this new function non-static, since it will be used in later\ncommits from outside of pack-bitmap.c.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 33 +++++++++++++++++++--------------\n pack-bitmap.h | 2 ++\n 2 files changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d1368b69bb..5efb8af121 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -380,6 +380,16 @@ struct include_data {\n \tstruct bitmap *seen;\n };\n \n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit)\n+{\n+\tkhiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,\n+\t\t\t\t\t commit->object.oid);\n+\tif (hash_pos >= kh_end(bitmap_git->bitmaps))\n+\t\treturn NULL;\n+\treturn lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));\n+}\n+\n static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n \t\t\t\t\t const struct object_id *oid)\n {\n@@ -465,10 +475,10 @@ static void show_commit(struct commit *commit, void *data)\n \n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n-\t\t\t const struct object_id *oid,\n+\t\t\t struct commit *commit,\n \t\t\t int bitmap_pos)\n {\n-\tkhiter_t hash_pos;\n+\tstruct ewah_bitmap *partial;\n \n \tif (data->seen && bitmap_get(data->seen, bitmap_pos))\n \t\treturn 0;\n@@ -476,10 +486,9 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \tif (bitmap_get(data->base, bitmap_pos))\n \t\treturn 0;\n \n-\thash_pos = kh_get_oid_map(bitmap_git->bitmaps, *oid);\n-\tif (hash_pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, hash_pos);\n-\t\tbitmap_or_ewah(data->base, lookup_stored_bitmap(st));\n+\tpartial = bitmap_for_commit(bitmap_git, commit);\n+\tif (partial) {\n+\t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n@@ -498,8 +507,7 @@ static int should_include(struct commit *commit, void *_data)\n \t\t\t\t\t\t (struct object *)commit,\n \t\t\t\t\t\t NULL);\n \n-\tif (!add_to_include_set(data->bitmap_git, data, &commit->object.oid,\n-\t\t\t\tbitmap_pos)) {\n+\tif (!add_to_include_set(data->bitmap_git, data, commit, bitmap_pos)) {\n \t\tstruct commit_list *parent = commit->parents;\n \n \t\twhile (parent) {\n@@ -1282,10 +1290,10 @@ void test_bitmap_walk(struct rev_info *revs)\n {\n \tstruct object *root;\n \tstruct bitmap *result = NULL;\n-\tkhiter_t pos;\n \tsize_t result_popcnt;\n \tstruct bitmap_test_data tdata;\n \tstruct bitmap_index *bitmap_git;\n+\tstruct ewah_bitmap *bm;\n \n \tif (!(bitmap_git = prepare_bitmap_git(revs->repo)))\n \t\tdie(\"failed to load bitmap indexes\");\n@@ -1297,12 +1305,9 @@ void test_bitmap_walk(struct rev_info *revs)\n \t\tbitmap_git->version, bitmap_git->entry_count);\n \n \troot = revs->pending.objects[0].item;\n-\tpos = kh_get_oid_map(bitmap_git->bitmaps, root->oid);\n-\n-\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\tstruct ewah_bitmap *bm = lookup_stored_bitmap(st);\n+\tbm = bitmap_for_commit(bitmap_git, (struct commit *)root);\n \n+\tif (bm) {\n \t\tfprintf(stderr, \"Found bitmap for %s. %d bits / %08x checksum\\n\",\n \t\t\toid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex afa4115136..25dfcf5615 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,6 +78,8 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest);\n+struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tunsigned int indexed_commits_nr, int max_bitmaps);\n void bitmap_writer_build(struct packing_data *to_pack);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411774","messageId":"e0d989b98f2f3fde4c9a24a333f980b9ad624449.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 22/24] pack-bitmap-write: use existing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:05:21Z","receivedAt":"2020-12-08T22:06:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nWhen constructing new bitmaps, we perform a commit and tree walk in\nfill_bitmap_commit() and fill_bitmap_tree(). This walk would benefit\nfrom using existing bitmaps when available. We must track the existing\nbitmaps and translate them into the new object order, but this is\ngenerally faster than parsing trees.\n\nIn fill_bitmap_commit(), we must reorder thing somewhat. The priority\nqueue walks commits from newest-to-oldest, which means we correctly stop\nwalking when reaching a commit with a bitmap. However, if we walk trees\ninterleaved with the commits, then we might be parsing trees that are\nactually part of a re-used bitmap. To avoid over-walking trees, add them\nto a LIFO queue and walk them after exploring commits completely.\n\nOn git.git, this reduces a second immediate bitmap computation from 2.0s\nto 1.0s. On linux.git, we go from 32s to 22s. On chromium's fork\nnetwork, we go from 227s to 198s.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 40 ++++++++++++++++++++++++++++++++++++----\n 1 file changed, 36 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 333058854d..76c8236f94 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -340,20 +340,37 @@ static void fill_bitmap_tree(struct bitmap *bitmap,\n \n static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t struct commit *commit,\n-\t\t\t struct prio_queue *queue)\n+\t\t\t struct prio_queue *queue,\n+\t\t\t struct prio_queue *tree_queue,\n+\t\t\t struct bitmap_index *old_bitmap,\n+\t\t\t const uint32_t *mapping)\n {\n \tif (!ent->bitmap)\n \t\tent->bitmap = bitmap_new();\n \n-\tbitmap_set(ent->bitmap, find_object_pos(&commit->object.oid));\n \tprio_queue_put(queue, commit);\n \n \twhile (queue->nr) {\n \t\tstruct commit_list *p;\n \t\tstruct commit *c = prio_queue_get(queue);\n \n+\t\tif (old_bitmap && mapping) {\n+\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\t/*\n+\t\t\t * If this commit has an old bitmap, then translate that\n+\t\t\t * bitmap and add its bits to this one. No need to walk\n+\t\t\t * parents or the tree for this commit.\n+\t\t\t */\n+\t\t\tif (old && !rebuild_bitmap(mapping, old, ent->bitmap))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Mark ourselves and queue our tree. The commit\n+\t\t * walk ensures we cover all parents.\n+\t\t */\n \t\tbitmap_set(ent->bitmap, find_object_pos(&c->object.oid));\n-\t\tfill_bitmap_tree(ent->bitmap, get_commit_tree(c));\n+\t\tprio_queue_put(tree_queue, get_commit_tree(c));\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tint pos = find_object_pos(&p->item->object.oid);\n@@ -363,6 +380,9 @@ static void fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t}\n \t\t}\n \t}\n+\n+\twhile (tree_queue->nr)\n+\t\tfill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue));\n }\n \n static void store_selected(struct bb_commit *ent, struct commit *commit)\n@@ -386,6 +406,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \tsize_t i;\n \tint nr_stored = 0; /* for progress */\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n+\tstruct prio_queue tree_queue = { NULL };\n+\tstruct bitmap_index *old_bitmap;\n+\tuint32_t *mapping;\n \n \twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n@@ -395,6 +418,12 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \ttrace2_region_enter(\"pack-bitmap-write\", \"building_bitmaps_total\",\n \t\t\t the_repository);\n \n+\told_bitmap = prepare_bitmap_git(to_pack->repo);\n+\tif (old_bitmap)\n+\t\tmapping = create_bitmap_mapping(old_bitmap, to_pack);\n+\telse\n+\t\tmapping = NULL;\n+\n \tbitmap_builder_init(&bb, &writer);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n@@ -402,7 +431,8 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tstruct commit *child;\n \t\tint reused = 0;\n \n-\t\tfill_bitmap_commit(ent, commit, &queue);\n+\t\tfill_bitmap_commit(ent, commit, &queue, &tree_queue,\n+\t\t\t\t old_bitmap, mapping);\n \n \t\tif (ent->selected) {\n \t\t\tstore_selected(ent, commit);\n@@ -428,7 +458,9 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \t\tent->bitmap = NULL;\n \t}\n \tclear_prio_queue(&queue);\n+\tclear_prio_queue(&tree_queue);\n \tbitmap_builder_clear(&bb);\n+\tfree(mapping);\n \n \ttrace2_region_leave(\"pack-bitmap-write\", \"building_bitmaps_total\",\n \t\t\t the_repository);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411775","messageId":"033fb2ed55fcab150bc019c4d6b6749ab59e3274.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 14/24] commit: implement commit_list_contains()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:13Z","receivedAt":"2020-12-08T22:06:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to check if a commit_list contains a commit. Use\npointer equality, assuming lookup_commit() was used.\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit.c | 11 +++++++++++\n commit.h | 2 ++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/commit.c b/commit.c\nindex fe1fa3dc41..9a785bf906 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -544,6 +544,17 @@ struct commit_list *commit_list_insert(struct commit *item, struct commit_list *\n \treturn new_list;\n }\n \n+int commit_list_contains(struct commit *item, struct commit_list *list)\n+{\n+\twhile (list) {\n+\t\tif (list->item == item)\n+\t\t\treturn 1;\n+\t\tlist = list->next;\n+\t}\n+\n+\treturn 0;\n+}\n+\n unsigned commit_list_count(const struct commit_list *l)\n {\n \tunsigned c = 0;\ndiff --git a/commit.h b/commit.h\nindex 5467786c7b..742a6de460 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -167,6 +167,8 @@ int find_commit_subject(const char *commit_buffer, const char **subject);\n \n struct commit_list *commit_list_insert(struct commit *item,\n \t\t\t\t\tstruct commit_list **list);\n+int commit_list_contains(struct commit *item,\n+\t\t\t struct commit_list *list);\n struct commit_list **commit_list_append(struct commit *commit,\n \t\t\t\t\tstruct commit_list **next);\n unsigned commit_list_count(const struct commit_list *l);\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411776","messageId":"b4c5d2c3dfb357a40074f4729226c529ff918566.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 18/24] pack-bitmap-write: build fewer intermediate bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:04:30Z","receivedAt":"2020-12-08T22:06:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe bitmap_writer_build() method calls bitmap_builder_init() to\nconstruct a list of commits reachable from the selected commits along\nwith a \"reverse graph\". This reverse graph has edges pointing from a\ncommit to other commits that can reach that commit. After computing a\nreachability bitmap for a commit, the values in that bitmap are then\ncopied to the reachability bitmaps across the edges in the reverse\ngraph.\n\nWe can now relax the role of the reverse graph to greatly reduce the\nnumber of intermediate reachability bitmaps we compute during this\nreverse walk. The end result is that we walk objects the same number of\ntimes as before when constructing the reachability bitmaps, but we also\nspend much less time copying bits between bitmaps and have much lower\nmemory pressure in the process.\n\nThe core idea is to select a set of \"important\" commits based on\ninteractions among the sets of commits reachable from each selected commit.\n\nThe first technical concept is to create a new 'commit_mask' member in the\nbb_commit struct. Note that the selected commits are provided in an\nordered array. The first thing to do is to mark the ith bit in the\ncommit_mask for the ith selected commit. As we walk the commit-graph, we\ncopy the bits in a commit's commit_mask to its parents. At the end of\nthe walk, the ith bit in the commit_mask for a commit C stores a boolean\nrepresenting \"The ith selected commit can reach C.\"\n\nAs we walk, we will discover non-selected commits that are important. We\nwill get into this later, but those important commits must also receive\nbit positions, growing the width of the bitmasks as we walk. At the true\nend of the walk, the ith bit means \"the ith _important_ commit can reach\nC.\"\n\nMAXIMAL COMMITS\n---------------\n\nWe use a new 'maximal' bit in the bb_commit struct to represent whether\na commit is important or not. The term \"maximal\" comes from the\npartially-ordered set of commits in the commit-graph where C >= P if P\nis a parent of C, and then extending the relationship transitively.\nInstead of taking the maximal commits across the entire commit-graph, we\ninstead focus on selecting each commit that is maximal among commits\nwith the same bits on in their commit_mask. This definition is\nimportant, so let's consider an example.\n\nSuppose we have three selected commits A, B, and C. These are assigned\nbitmasks 100, 010, and 001 to start. Each of these can be marked as\nmaximal immediately because they each will be the uniquely maximal\ncommit that contains their own bit. Keep in mind that that these commits\nmay have different bitmasks after the walk; for example, if B can reach\nC but A cannot, then the final bitmask for C is 011. Even in these\ncases, C would still be a maximal commit among all commits with the\nthird bit on in their masks.\n\nNow define sets X, Y, and Z to be the sets of commits reachable from A,\nB, and C, respectively. The intersections of these sets correspond to\ndifferent bitmasks:\n\n * 100: X - (Y union Z)\n * 010: Y - (X union Z)\n * 001: Z - (X union Y)\n * 110: (X intersect Y) - Z\n * 101: (X intersect Z) - Y\n * 011: (Y intersect Z) - X\n * 111: X intersect Y intersect Z\n\nThis can be visualized with the following Hasse diagram:\n\n\t100 010 001\n | \\ / \\ / |\n | \\/ \\/ |\n | /\\ /\\ |\n | / \\ / \\ |\n 110 101 011\n \\___ | ___/\n \\ | /\n 111\n\nSome of these bitmasks may not be represented, depending on the topology\nof the commit-graph. In fact, we are counting on it, since the number of\npossible bitmasks is exponential in the number of selected commits, but\nis also limited by the total number of commits. In practice, very few\nbitmasks are possible because most commits converge on a common \"trunk\"\nin the commit history.\n\nWith this three-bit example, we wish to find commits that are maximal\nfor each bitmask. How can we identify this as we are walking?\n\nAs we walk, we visit a commit C. Since we are walking the commits in\ntopo-order, we know that C is visited after all of its children are\nvisited. Thus, when we get C from the revision walk we inspect the\n'maximal' property of its bb_data and use that to determine if C is truly\nimportant. Its commit_mask is also nearly final. If C is not one of the\noriginally-selected commits, then assign a bit position to C (by\nincrementing num_maximal) and set that bit on in commit_mask. See\n\"MULTIPLE MAXIMAL COMMITS\" below for more detail on this.\n\nNow that the commit C is known to be maximal or not, consider each\nparent P of C. Compute two new values:\n\n * c_not_p : true if and only if the commit_mask for C contains a bit\n that is not contained in the commit_mask for P.\n\n * p_not_c : true if and only if the commit_mask for P contains a bit\n that is not contained in the commit_mask for P.\n\nIf c_not_p is false, then P already has all of the bits that C would\nprovide to its commit_mask. In this case, move on to other parents as C\nhas nothing to contribute to P's state that was not already provided by\nother children of P.\n\nWe continue with the case that c_not_p is true. This means there are\nbits in C's commit_mask to copy to P's commit_mask, so use bitmap_or()\nto add those bits.\n\nIf p_not_c is also true, then set the maximal bit for P to one. This means\nthat if no other commit has P as a parent, then P is definitely maximal.\nThis is because no child had the same bitmask. It is important to think\nabout the maximal bit for P at this point as a temporary state: \"P is\nmaximal based on current information.\"\n\nIn contrast, if p_not_c is false, then set the maximal bit for P to\nzero. Further, clear all reverse_edges for P since any edges that were\npreviously assigned to P are no longer important. P will gain all\nreverse edges based on C.\n\nThe final thing we need to do is to update the reverse edges for P.\nThese reverse edges respresent \"which closest maximal commits\ncontributed bits to my commit_mask?\" Since C contributed bits to P's\ncommit_mask in this case, C must add to the reverse edges of P.\n\nIf C is maximal, then C is a 'closest' maximal commit that contributed\nbits to P. Add C to P's reverse_edges list.\n\nOtherwise, C has a list of maximal commits that contributed bits to its\nbitmask (and this list is exactly one element). Add all of these items\nto P's reverse_edges list. Be careful to ignore duplicates here.\n\nAfter inspecting all parents P for a commit C, we can clear the\ncommit_mask for C. This reduces the memory load to be limited to the\n\"width\" of the commit graph.\n\nConsider our ABC/XYZ example from earlier and let's inspect the state of\nthe commits for an interesting bitmask, say 011. Suppose that D is the\nonly maximal commit with this bitmask (in the first three bits). All\nother commits with bitmask 011 have D as the only entry in their\nreverse_edges list. D's reverse_edges list contains B and C.\n\nCOMPUTING REACHABILITY BITMAPS\n------------------------------\n\nNow that we have our definition, let's zoom out and consider what\nhappens with our new reverse graph when computing reachability bitmaps.\nWe walk the reverse graph in reverse-topo-order, so we visit commits\nwith largest commit_masks first. After we compute the reachability\nbitmap for a commit C, we push the bits in that bitmap to each commit D\nin the reverse edge list for C. Then, when we finally visit D we already\nhave the bits for everything reachable from maximal commits that D can\nreach and we only need to walk the objects in the set-difference.\n\nIn our ABC/XYZ example, when we finally walk for the commit A we only\nneed to walk commits with bitmask equal to A's bitmask. If that bitmask\nis 100, then we are only walking commits in X - (Y union Z) because the\nbitmap already contains the bits for objects reachable from (X intersect\nY) union (X intersect Z) (i.e. the bits from the reachability bitmaps\nfor the maximal commits with bitmasks 110 and 101).\n\nThe behavior is intended to walk each commit (and the trees that commit\nintroduces) at most once while allocating and copying fewer reachability\nbitmaps. There is one caveat: what happens when there are multiple\nmaximal commits with the same bitmask, with respect to the initial set\nof selected commits?\n\nMULTIPLE MAXIMAL COMMITS\n------------------------\n\nEarlier, we mentioned that when we discover a new maximal commit, we\nassign a new bit position to that commit and set that bit position to\none for that commit. This is absolutely important for interesting\ncommit-graphs such as git/git and torvalds/linux. The reason is due to\nthe existence of \"butterflies\" in the commit-graph partial order.\n\nHere is an example of four commits forming a butterfly:\n\n I J\n |\\ /|\n | \\/ |\n | /\\ |\n |/ \\|\n M N\n \\ /\n |/\n Q\n\nHere, I and J both have parents M and N. In general, these do not need\nto be exact parent relationships, but reachability relationships. The\nmost important part is that M and N cannot reach each other, so they are\nindependent in the partial order. If I had commit_mask 10 and J had\ncommit_mask 01, then M and N would both be assigned commit_mask 11 and\nbe maximal commits with the bitmask 11. Then, what happens when M and N\ncan both reach a commit Q? If Q is also assigned the bitmask 11, then it\nis not maximal but is reachable from both M and N.\n\nWhile this is not necessarily a deal-breaker for our abstract definition\nof finding maximal commits according to a given bitmask, we have a few\nissues that can come up in our larger picture of constructing\nreachability bitmaps.\n\nIn particular, if we do not also consider Q to be a \"maximal\" commit,\nthen we will walk commits reachable from Q twice: once when computing\nthe reachability bitmap for M and another time when computing the\nreachability bitmap for N. This becomes much worse if the topology\ncontinues this pattern with multiple butterflies.\n\nThe solution has already been mentioned: each of M and N are assigned\ntheir own bits to the bitmask and hence they become uniquely maximal for\ntheir bitmasks. Finally, Q also becomes maximal and thus we do not need\nto walk its commits multiple times. The final bitmasks for these commits\nare as follows:\n\n I:10 J:01\n |\\ /|\n | \\ _____/ |\n | /\\____ |\n |/ \\ |\n M:111 N:1101\n \\ /\n Q:1111\n\nFurther, Q's reverse edge list is { M, N }, while M and N both have\nreverse edge list { I, J }.\n\nPERFORMANCE MEASUREMENTS\n------------------------\n\nNow that we've spent a LOT of time on the theory of this algorithm,\nlet's show that this is actually worth all that effort.\n\nTo test the performance, use GIT_TRACE2_PERF=1 when running\n'git repack -abd' in a repository with no existing reachability bitmaps.\nThis avoids any issues with keeping existing bitmaps to skew the\nnumbers.\n\nInspect the \"building_bitmaps_total\" region in the trace2 output to\nfocus on the portion of work that is affected by this change. Here are\nthe performance comparisons for a few repositories. The timings are for\nthe following versions of Git: \"multi\" is the timing from before any\nreverse graph is constructed, where we might perform multiple\ntraversals. \"reverse\" is for the previous change where the reverse graph\nhas every reachable commit. Finally \"maximal\" is the version introduced\nhere where the reverse graph only contains the maximal commits.\n\n Repository: git/git\n multi: 2.628 sec\n reverse: 2.344 sec\n maximal: 2.047 sec\n\n Repository: torvalds/linux\n multi: 64.7 sec\n reverse: 205.3 sec\n maximal: 44.7 sec\n\nSo in all cases we've not only recovered any time lost to switching to\nthe reverse-edge algorithm, but we come out ahead of \"multi\" in all\ncases. Likewise, peak heap has gone back to something reasonable:\n\n Repository: torvalds/linux\n multi: 2.087 GB\n reverse: 3.141 GB\n maximal: 2.288 GB\n\nWhile I do not have access to full fork networks on GitHub, Peff has run\nthis algorithm on the chromium/chromium fork network and reported a\nchange from 3 hours to ~233 seconds. That network is particularly\nbeneficial for this approach because it has a long, linear history along\nwith many tags. The \"multi\" approach was obviously quadratic and the new\napproach is linear.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \nHelped-by: Johannes Schindelin \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 72 +++++++++++++++++++++++++++++++---\n t/t5310-pack-bitmaps.sh | 85 +++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 148 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 7e218d02a6..0af93193d8 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -180,8 +180,10 @@ static void compute_xor_offsets(void)\n \n struct bb_commit {\n \tstruct commit_list *reverse_edges;\n+\tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n-\tunsigned selected:1;\n+\tunsigned selected:1,\n+\t\t maximal:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -198,7 +200,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i;\n+\tunsigned int i, num_maximal;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -210,27 +212,85 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n \t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\n \t\tent->selected = 1;\n+\t\tent->maximal = 1;\n \t\tent->idx = i;\n+\n+\t\tent->commit_mask = bitmap_new();\n+\t\tbitmap_set(ent->commit_mask, i);\n+\n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n+\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n \t\tstruct commit_list *p;\n+\t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n \n-\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n-\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\tc_ent = bb_data_at(&bb->data, commit);\n+\n+\t\tif (c_ent->maximal) {\n+\t\t\tif (!c_ent->selected) {\n+\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n+\t\t\t\tnum_maximal++;\n+\t\t\t}\n+\n+\t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\t\tbb->commits[bb->commits_nr++] = commit;\n+\t\t}\n \n \t\tfor (p = commit->parents; p; p = p->next) {\n-\t\t\tstruct bb_commit *ent = bb_data_at(&bb->data, p->item);\n-\t\t\tcommit_list_insert(commit, &ent->reverse_edges);\n+\t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n+\t\t\tint c_not_p, p_not_c;\n+\n+\t\t\tif (!p_ent->commit_mask) {\n+\t\t\t\tp_ent->commit_mask = bitmap_new();\n+\t\t\t\tc_not_p = 1;\n+\t\t\t\tp_not_c = 0;\n+\t\t\t} else {\n+\t\t\t\tc_not_p = bitmap_is_subset(c_ent->commit_mask, p_ent->commit_mask);\n+\t\t\t\tp_not_c = bitmap_is_subset(p_ent->commit_mask, c_ent->commit_mask);\n+\t\t\t}\n+\n+\t\t\tif (!c_not_p)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_or(p_ent->commit_mask, c_ent->commit_mask);\n+\n+\t\t\tif (p_not_c)\n+\t\t\t\tp_ent->maximal = 1;\n+\t\t\telse {\n+\t\t\t\tp_ent->maximal = 0;\n+\t\t\t\tfree_commit_list(p_ent->reverse_edges);\n+\t\t\t\tp_ent->reverse_edges = NULL;\n+\t\t\t}\n+\n+\t\t\tif (c_ent->maximal) {\n+\t\t\t\tcommit_list_insert(commit, &p_ent->reverse_edges);\n+\t\t\t} else {\n+\t\t\t\tstruct commit_list *cc = c_ent->reverse_edges;\n+\n+\t\t\t\tfor (; cc; cc = cc->next) {\n+\t\t\t\t\tif (!commit_list_contains(cc->item, p_ent->reverse_edges))\n+\t\t\t\t\t\tcommit_list_insert(cc->item, &p_ent->reverse_edges);\n+\t\t\t\t}\n+\t\t\t}\n \t\t}\n+\n+\t\tbitmap_free(c_ent->commit_mask);\n+\t\tc_ent->commit_mask = NULL;\n \t}\n+\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_selected_commits\", writer->selected_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"num_maximal_commits\", num_maximal);\n }\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 8bf02336d9..6815fb6a4e 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -20,12 +20,88 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# To ensure the logic for \"maximal commits\" is exercised, make\n+# the repository a bit more complicated.\n+#\n+# other master\n+# * *\n+# (99 commits) (99 commits)\n+# * *\n+# |\\ /|\n+# | * octo-other octo-master * |\n+# |/|\\_________ ____________/|\\|\n+# | \\ \\/ __________/ |\n+# | | ________/\\ / |\n+# * |/ * merge-right *\n+# | _|__________/ \\____________ |\n+# |/ | \\|\n+# (l1) * * merge-left * (r1)\n+# | / \\________________________ |\n+# |/ \\|\n+# (l2) * * (r2)\n+# \\___________________________ |\n+# \\|\n+# * (base)\n+#\n+# The important part for the maximal commit algorithm is how\n+# the bitmasks are extended. Assuming starting bit positions\n+# for master (bit 0) and other (bit 1), and some flexibility\n+# in the order that merge bases are visited, the bitmasks at\n+# the end should be:\n+#\n+# master: 1 (maximal, selected)\n+# other: 01 (maximal, selected)\n+# octo-master: 1\n+# octo-other: 01\n+# merge-right: 111 (maximal)\n+# (l1): 111\n+# (r1): 111\n+# merge-left: 1101 (maximal)\n+# (l2): 11111 (maximal)\n+# (r2): 111101 (maximal)\n+# (base): 1111111 (maximal)\n+\n test_expect_success 'setup repo with moderate-sized history' '\n-\ttest_commit_bulk --id=file 100 &&\n+\ttest_commit_bulk --id=file 10 &&\n \tgit branch -M second &&\n \tgit checkout -b other HEAD~5 &&\n \ttest_commit_bulk --id=side 10 &&\n+\n+\t# add complicated history setup, including merges and\n+\t# ambiguous merge-bases\n+\n+\tgit checkout -b merge-left other~2 &&\n+\tgit merge second~2 -m \"merge-left\" &&\n+\n+\tgit checkout -b merge-right second~1 &&\n+\tgit merge other~1 -m \"merge-right\" &&\n+\n+\tgit checkout -b octo-second second &&\n+\tgit merge merge-left merge-right -m \"octopus-second\" &&\n+\n+\tgit checkout -b octo-other other &&\n+\tgit merge merge-left merge-right -m \"octopus-other\" &&\n+\n+\tgit checkout other &&\n+\tgit merge octo-other -m \"pull octopus\" &&\n+\n \tgit checkout second &&\n+\tgit merge octo-second -m \"pull octopus\" &&\n+\n+\t# Remove these branches so they are not selected\n+\t# as bitmap tips\n+\tgit branch -D merge-left &&\n+\tgit branch -D merge-right &&\n+\tgit branch -D octo-other &&\n+\tgit branch -D octo-second &&\n+\n+\t# add padding to make these merges less interesting\n+\t# and avoid having them selected for bitmaps\n+\ttest_commit_bulk --id=file 100 &&\n+\tgit checkout other &&\n+\ttest_commit_bulk --id=side 100 &&\n+\tgit checkout second &&\n+\n \tbitmaptip=$(git rev-parse second) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n@@ -33,9 +109,12 @@ test_expect_success 'setup repo with moderate-sized history' '\n '\n \n test_expect_success 'full repack creates bitmaps' '\n-\tgit repack -ad &&\n+\tGIT_TRACE2_EVENT_NESTING=4 GIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-\ttest_line_count = 1 output\n+\ttest_line_count = 1 output &&\n+\tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411777","messageId":"8f9fdb0f43ef6aa6796ef50d39a4c911259aa3db.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 23/24] pack-bitmap-write: relax unique revwalk condition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:05:26Z","receivedAt":"2020-12-08T22:06:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nThe previous commits improved the bitmap computation process for very\nlong, linear histories with many refs by removing quadratic growth in\nhow many objects were walked. The strategy of computing \"intermediate\ncommits\" using bitmasks for which refs can reach those commits\npartitioned the poset of reachable objects so each part could be walked\nexactly once. This was effective for linear histories.\n\nHowever, there was a (significant) drawback: wide histories with many\nrefs had an explosion of memory costs to compute the commit bitmasks\nduring the exploration that discovers these intermediate commits. Since\nthese wide histories are unlikely to repeat walking objects, the benefit\nof walking objects multiple times was not expensive before. But now, the\ncommit walk *before computing bitmaps* is incredibly expensive.\n\nIn an effort to discover a happy medium, this change reduces the walk\nfor intermediate commits to only the first-parent history. This focuses\nthe walk on how the histories converge, which still has significant\nreduction in repeat object walks. It is still possible to create\nquadratic behavior in this version, but it is probably less likely in\nrealistic data shapes.\n\nHere is some data taken on a fresh clone of the kernel:\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n original | 64.044 | 83.241 | 2.088 | 2.194 |\n last patch | 45.049 | 37.624 | 2.267 | 2.334 |\n this patch | 88.478 | 53.218 | 2.157 | 2.224 |\n\nSigned-off-by: Derrick Stolee \nHelped-by: Johannes Schindelin \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 14 +++++---------\n t/t5310-pack-bitmaps.sh | 33 +++++++++++++++++----------------\n 2 files changed, 22 insertions(+), 25 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 76c8236f94..d2af4a974f 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -199,7 +199,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n-\tunsigned int i, num_maximal;\n+\tunsigned int i, num_maximal = 0;\n \n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n@@ -207,6 +207,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \treset_revision_walk();\n \trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n \trevs.topo_order = 1;\n+\trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct commit *c = writer->selected[i].commit;\n@@ -221,13 +222,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tadd_pending_object(&revs, &c->object, \"\");\n \t}\n-\tnum_maximal = writer->selected_nr;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \n \twhile ((commit = get_revision(&revs))) {\n-\t\tstruct commit_list *p;\n+\t\tstruct commit_list *p = commit->parents;\n \t\tstruct bb_commit *c_ent;\n \n \t\tparse_commit_or_die(commit);\n@@ -235,16 +235,12 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n \t\tif (c_ent->maximal) {\n-\t\t\tif (!c_ent->selected) {\n-\t\t\t\tbitmap_set(c_ent->commit_mask, num_maximal);\n-\t\t\t\tnum_maximal++;\n-\t\t\t}\n-\n+\t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n \t\t\tbb->commits[bb->commits_nr++] = commit;\n \t\t}\n \n-\t\tfor (p = commit->parents; p; p = p->next) {\n+\t\tif (p) {\n \t\t\tstruct bb_commit *p_ent = bb_data_at(&bb->data, p->item);\n \t\t\tint c_not_p, p_not_c;\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 6815fb6a4e..3a2c9d2d8e 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -23,12 +23,12 @@ has_any () {\n # To ensure the logic for \"maximal commits\" is exercised, make\n # the repository a bit more complicated.\n #\n-# other master\n+# other second\n # * *\n # (99 commits) (99 commits)\n # * *\n # |\\ /|\n-# | * octo-other octo-master * |\n+# | * octo-other octo-second * |\n # |/|\\_________ ____________/|\\|\n # | \\ \\/ __________/ |\n # | | ________/\\ / |\n@@ -43,23 +43,24 @@ has_any () {\n # \\|\n # * (base)\n #\n+# We only push bits down the first-parent history, which\n+# makes some of these commits unimportant!\n+#\n # The important part for the maximal commit algorithm is how\n # the bitmasks are extended. Assuming starting bit positions\n-# for master (bit 0) and other (bit 1), and some flexibility\n-# in the order that merge bases are visited, the bitmasks at\n-# the end should be:\n+# for second (bit 0) and other (bit 1), the bitmasks at the\n+# end should be:\n #\n-# master: 1 (maximal, selected)\n+# second: 1 (maximal, selected)\n # other: 01 (maximal, selected)\n-# octo-master: 1\n-# octo-other: 01\n-# merge-right: 111 (maximal)\n-# (l1): 111\n-# (r1): 111\n-# merge-left: 1101 (maximal)\n-# (l2): 11111 (maximal)\n-# (r2): 111101 (maximal)\n-# (base): 1111111 (maximal)\n+# (base): 11 (maximal)\n+#\n+# This complicated history was important for a previous\n+# version of the walk that guarantees never walking a\n+# commit multiple times. That goal might be important\n+# again, so preserve this complicated case. For now, this\n+# test will guarantee that the bitmaps are computed\n+# correctly, even with the repeat calculations.\n \n test_expect_success 'setup repo with moderate-sized history' '\n \ttest_commit_bulk --id=file 10 &&\n@@ -114,7 +115,7 @@ test_expect_success 'full repack creates bitmaps' '\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n \tgrep \"\\\"key\\\":\\\"num_selected_commits\\\",\\\"value\\\":\\\"106\\\"\" trace &&\n-\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"111\\\"\" trace\n+\tgrep \"\\\"key\\\":\\\"num_maximal_commits\\\",\\\"value\\\":\\\"107\\\"\" trace\n '\n \n test_expect_success 'rev-list --test-bitmap verifies bitmaps' '\n-- \n2.29.2.533.g07db1f5344\n\n"},{"id":"411778","messageId":"720b6e0dc7d69bf3454c57002a67a488c25050e4.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 24/24] pack-bitmap-write: better reuse bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:05:30Z","receivedAt":"2020-12-08T22:06:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nIf the old bitmap file contains a bitmap for a given commit, then that\ncommit does not need help from intermediate commits in its history to\ncompute its final bitmap. Eject that commit from the walk and insert it\ninto a separate list of reusable commits that are eventually stored in\nthe list of commits for computing bitmaps.\n\nThis helps the repeat bitmap computation task, even if the selected\ncommits shift drastically. This helps when a previously-bitmapped commit\nexists in the first-parent history of a newly-selected commit. Since we\nstop the walk at these commits and we use a first-parent walk, it is\nharder to walk \"around\" these bitmapped commits. It's not impossible,\nbut we can greatly reduce the computation time for many selected\ncommits.\n\n | runtime (sec) | peak heap (GB) |\n | | |\n | from | with | from | with |\n | scratch | existing | scratch | existing |\n -----------+---------+----------+---------+-----------\n last patch | 88.478 | 53.218 | 2.157 | 2.224 |\n this patch | 86.681 | 16.164 | 2.157 | 2.222 |\n\nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 40 ++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 38 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d2af4a974f..cc5ead9990 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -195,10 +195,13 @@ struct bitmap_builder {\n };\n \n static void bitmap_builder_init(struct bitmap_builder *bb,\n-\t\t\t\tstruct bitmap_writer *writer)\n+\t\t\t\tstruct bitmap_writer *writer,\n+\t\t\t\tstruct bitmap_index *old_bitmap)\n {\n \tstruct rev_info revs;\n \tstruct commit *commit;\n+\tstruct commit_list *reusable = NULL;\n+\tstruct commit_list *r;\n \tunsigned int i, num_maximal = 0;\n \n \tmemset(bb, 0, sizeof(*bb));\n@@ -234,6 +237,31 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n \t\tc_ent = bb_data_at(&bb->data, commit);\n \n+\t\t/*\n+\t\t * If there is no commit_mask, there is no reason to iterate\n+\t\t * over this commit; it is not selected (if it were, it would\n+\t\t * not have a blank commit mask) and all its children have\n+\t\t * existing bitmaps (see the comment starting with \"This commit\n+\t\t * has an existing bitmap\" below), so it does not contribute\n+\t\t * anything to the final bitmap file or its descendants.\n+\t\t */\n+\t\tif (!c_ent->commit_mask)\n+\t\t\tcontinue;\n+\n+\t\tif (old_bitmap && bitmap_for_commit(old_bitmap, commit)) {\n+\t\t\t/*\n+\t\t\t * This commit has an existing bitmap, so we can\n+\t\t\t * get its bits immediately without an object\n+\t\t\t * walk. That is, it is reusable as-is and there is no\n+\t\t\t * need to continue walking beyond it.\n+\t\t\t *\n+\t\t\t * Mark it as such and add it to bb->commits separately\n+\t\t\t * to avoid allocating a position in the commit mask.\n+\t\t\t */\n+\t\t\tcommit_list_insert(commit, &reusable);\n+\t\t\tgoto next;\n+\t\t}\n+\n \t\tif (c_ent->maximal) {\n \t\t\tnum_maximal++;\n \t\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n@@ -278,14 +306,22 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \t\t\t}\n \t\t}\n \n+next:\n \t\tbitmap_free(c_ent->commit_mask);\n \t\tc_ent->commit_mask = NULL;\n \t}\n \n+\tfor (r = reusable; r; r = r->next) {\n+\t\tALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);\n+\t\tbb->commits[bb->commits_nr++] = r->item;\n+\t}\n+\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"num_selected_commits\", writer->selected_nr);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"num_maximal_commits\", num_maximal);\n+\n+\tfree_commit_list(reusable);\n }\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\n@@ -420,7 +456,7 @@ void bitmap_writer_build(struct packing_data *to_pack)\n \telse\n \t\tmapping = NULL;\n \n-\tbitmap_builder_init(&bb, &writer);\n+\tbitmap_builder_init(&bb, &writer, old_bitmap);\n \tfor (i = bb.commits_nr; i > 0; i--) {\n \t\tstruct commit *commit = bb.commits[i-1];\n \t\tstruct bb_commit *ent = bb_data_at(&bb.data, commit);\n-- \n2.29.2.533.g07db1f5344\n"},{"id":"411779","messageId":"bd3a16088b752e789b0b8b050261f0536cb2d818.1607464775.git.me@ttaylorr.com","threadId":"54622","inReplyTo":"cover.1607464775.git.me@ttaylorr.com","subject":"[PATCH v4 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-12-08T22:05:14Z","receivedAt":"2020-12-08T22:06:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'find_objects()' currently needs to interact with the bitmaps khash\npretty closely. To make 'find_objects()' read a little more\nstraightforwardly, remove some of the khash-level details into a new\nfunction that describes what it does: 'add_commit_to_bitmap()'.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 36 +++++++++++++++++++++---------------\n 1 file changed, 21 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 5efb8af121..d88745fb02 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -521,6 +521,23 @@ static int should_include(struct commit *commit, void *_data)\n \treturn 1;\n }\n \n+static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\tstruct bitmap **base,\n+\t\t\t\tstruct commit *commit)\n+{\n+\tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n+\n+\tif (!or_with)\n+\t\treturn 0;\n+\n+\tif (*base == NULL)\n+\t\t*base = ewah_to_bitmap(or_with);\n+\telse\n+\t\tbitmap_or_ewah(*base, or_with);\n+\n+\treturn 1;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -544,21 +561,10 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tstruct object *object = roots->item;\n \t\troots = roots->next;\n \n-\t\tif (object->type == OBJ_COMMIT) {\n-\t\t\tkhiter_t pos = kh_get_oid_map(bitmap_git->bitmaps, object->oid);\n-\n-\t\t\tif (pos < kh_end(bitmap_git->bitmaps)) {\n-\t\t\t\tstruct stored_bitmap *st = kh_value(bitmap_git->bitmaps, pos);\n-\t\t\t\tstruct ewah_bitmap *or_with = lookup_stored_bitmap(st);\n-\n-\t\t\t\tif (base == NULL)\n-\t\t\t\t\tbase = ewah_to_bitmap(or_with);\n-\t\t\t\telse\n-\t\t\t\t\tbitmap_or_ewah(base, or_with);\n-\n-\t\t\t\tobject->flags |= SEEN;\n-\t\t\t\tcontinue;\n-\t\t\t}\n+\t\tif (object->type == OBJ_COMMIT &&\n+\t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n+\t\t\tobject->flags |= SEEN;\n+\t\t\tcontinue;\n \t\t}\n \n \t\tobject_list_insert(object, ¬_mapped);\n-- \n2.29.2.533.g07db1f5344\n\n"}]}