{"thread":{"id":"57055","subject":"[PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","startedAt":"2021-12-08T19:26:22Z","lastAt":"2022-02-24T22:50:58Z","messageCount":87,"participants":["Taylor Blau","Derrick Stolee","Jeff King","Junio C Hamano","Jonathan Tan","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"443500","messageId":"cover.1638991570.git.me@ttaylorr.com","threadId":"57055","inReplyTo":null,"subject":"[PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-08T19:26:20Z","receivedAt":"2021-12-08T19:26:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series demonstrates and fixes a problem (which has existed since the\ninception of MIDX bitmaps) which made it possible to reliably introduce bitmap\ncorruption into a repository.\n\nThe recipe (which is explained in detail in the first patch) roughly looks like\ncreating a MIDX with some packs. Then re-generating the MIDX and its bitmap but\nunder a different object order (e.g., by changing the `--preferred-pack`).\n\nBecause we used to link the new .rev file into place, and because the object\norder is not encoded into the MIDX itself, we would inadvertently generate a\n.rev file with the same name as the existing one that contains different\ncontents, and then fail to move it into place. That causes us to read the\n.bitmap under the old object order, producing incorrect results.\n\nThis series applies a minimal fix (in the second patch), which is to include the\npack ordering in the MIDX itself. Some small thoughts here:\n\n - We could alternatively have included the entire .rev file's contents in the\n MIDX. But this ends up being kind of awkward (and is also discussed in the\n second patch).\n\n - We could cache the identity of the preferred pack (which we currently\n discover by looking up the pack for the object in the 0th position of the\n MIDX's pseudo-pack order) by just reading the first value of the new\n optional chunk.\n\nI'm certainly interested in pursuing the latter, but in a different series,\nsince I'd like to keep this fix as minimal as possible.\n\nTaylor Blau (2):\n t5326: demonstrate bitmap corruption after permutation\n midx.c: make changing the preferred pack safe\n\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 4 +++\n midx.c | 25 +++++++++++++---\n t/t5326-multi-pack-bitmaps.sh | 31 ++++++++++++++++++++\n 4 files changed, 57 insertions(+), 4 deletions(-)\n\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"443501","messageId":"ea91cebb6bfe43435820ea1ffd54271cf1d920f5.1638991570.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1638991570.git.me@ttaylorr.com","subject":"[PATCH 1/2] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-08T19:26:22Z","receivedAt":"2021-12-08T19:26:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch demonstrates a cause of bitmap corruption that can occur when\nthe contents of the multi-pack index does not change, but the underlying\nobject order does.\n\nIn this example, we have a MIDX containing two packs, each with a\ndistinct set of objects (pack A corresponds to the tree, blob, and\ncommit from the first patch, and pack B corresponds to the second\npatch).\n\nFirst, a MIDX is written where the 'A' pack is preferred. As expected,\nthe bitmaps generated there are in-tact. But then, we generate an\nidentical MIDX with a different object order: this time preferring pack\n'B'.\n\nDue to a bug which will be explained and fixed in the following commit,\nthe MIDX is updated, but the .rev file is not, causing the .bitmap file\nto be read incorrectly. Specifically, the .bitmap file will contain\ncorrect data, but the auxiliary object order in the .rev file is stale,\ncausing readers to get confused by reading the new bitmaps using the old\nobject order.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex e187f90f29..0ca2868b0b 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,4 +395,35 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n+test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\tpack-$A.idx\n+\t\tpack-$B.idx\n+\t\tEOF\n+\n+\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t--preferred-pack=pack-$A.pack \n---\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 4 ++++\n midx.c | 25 ++++++++++++++++----\n t/t5326-multi-pack-bitmaps.sh | 2 +-\n 4 files changed, 27 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex 86f40f2490..077c893ae3 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -24,6 +24,7 @@ and their offsets into multiple packfiles. It contains:\n - An offset within the jth packfile for the object.\n - If large offsets are required, we use another list of large\n offsets similar to version 2 pack-indexes.\n+- An optional list of pack IDs in bitmap-sorted order.\n \n Thus, we can provide O(log N) lookup time for any number\n of packfiles.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 8d2f42f29e..edd77fb437 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -376,6 +376,10 @@ CHUNK DATA:\n \t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n \t 8-byte offsets into large packfiles.\n \n+\t[Optional] Bitmap pack order (ID: {'P', 'O', 'R', 'D'})\n+\t A list of 4-byte pack identifiers in sorted order according to\n+\t their relative bitmap positions.\n+\n TRAILER:\n \n \tIndex checksum of the above contents.\ndiff --git a/midx.c b/midx.c\nindex 837b46b2af..abcdb92dd1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n+#define MIDX_CHUNKID_PACKORDER 0x504f5244 /* \"PORD\" */\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n@@ -833,6 +834,18 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_midx_pack_order(struct hashfile *f,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\n+\treturn 0;\n+}\n+\n struct midx_pack_order_data {\n \tuint32_t nr;\n \tuint32_t pack;\n@@ -891,7 +904,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n \t\t\t\t\tmidx_hash, WRITE_REV);\n \n-\tif (finalize_object_file(tmp_file, buf.buf))\n+\tif (rename(tmp_file, buf.buf))\n \t\tdie(_(\"cannot store reverse index file\"));\n \n \tstrbuf_release(&buf);\n@@ -1403,15 +1416,19 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n \n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_PACKORDER,\n+\t\t\t ctx.entries_nr * sizeof(uint32_t),\n+\t\t\t write_midx_pack_order);\n+\t}\n+\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n-\t\tctx.pack_order = midx_pack_order(&ctx);\n-\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 0ca2868b0b..353282310d 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,7 +395,7 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"443503","messageId":"bf100d0c-66c7-d402-4790-9c5797e8baf3@gmail.com","threadId":"57055","inReplyTo":"cover.1638991570.git.me@ttaylorr.com","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-08T19:30:17Z","receivedAt":"2021-12-08T19:30:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/8/2021 2:26 PM, Taylor Blau wrote:\n> This series demonstrates and fixes a problem (which has existed since the\n> inception of MIDX bitmaps) which made it possible to reliably introduce bitmap\n> corruption into a repository.\n> \n> The recipe (which is explained in detail in the first patch) roughly looks like\n> creating a MIDX with some packs. Then re-generating the MIDX and its bitmap but\n> under a different object order (e.g., by changing the `--preferred-pack`).\n> \n> Because we used to link the new .rev file into place, and because the object\n> order is not encoded into the MIDX itself, we would inadvertently generate a\n> .rev file with the same name as the existing one that contains different\n> contents, and then fail to move it into place. That causes us to read the\n> .bitmap under the old object order, producing incorrect results.\n> \n> This series applies a minimal fix (in the second patch), which is to include the\n> pack ordering in the MIDX itself. Some small thoughts here:\n> \n> - We could alternatively have included the entire .rev file's contents in the\n> MIDX. But this ends up being kind of awkward (and is also discussed in the\n> second patch).\n> \n> - We could cache the identity of the preferred pack (which we currently\n> discover by looking up the pack for the object in the 0th position of the\n> MIDX's pseudo-pack order) by just reading the first value of the new\n> optional chunk.\n> \n> I'm certainly interested in pursuing the latter, but in a different series,\n> since I'd like to keep this fix as minimal as possible.\n> \n> Taylor Blau (2):\n> t5326: demonstrate bitmap corruption after permutation\n> midx.c: make changing the preferred pack safe\n\nJust chiming in to say that I reviewed an earlier version of this series\nand the version in this submission looks good to me.\n\nThanks,\n-Stolee\n"},{"id":"443511","messageId":"YbENofRtY0BPMEUd@coredump.intra.peff.net","threadId":"57055","inReplyTo":"bf100d0c-66c7-d402-4790-9c5797e8baf3@gmail.com","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-12-08T19:55:13Z","receivedAt":"2021-12-08T19:55:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 08, 2021 at 02:30:17PM -0500, Derrick Stolee wrote:\n\n> > Taylor Blau (2):\n> > t5326: demonstrate bitmap corruption after permutation\n> > midx.c: make changing the preferred pack safe\n> \n> Just chiming in to say that I reviewed an earlier version of this series\n> and the version in this submission looks good to me.\n\nDitto. ;)\n\n-Peff\n"},{"id":"443828","messageId":"YbOeEjNic5ETGcdy@nand.local","threadId":"57055","inReplyTo":"YbENofRtY0BPMEUd@coredump.intra.peff.net","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-10T18:36:02Z","receivedAt":"2021-12-10T18:36:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Dec 08, 2021 at 02:55:13PM -0500, Jeff King wrote:\n> On Wed, Dec 08, 2021 at 02:30:17PM -0500, Derrick Stolee wrote:\n>\n> > > Taylor Blau (2):\n> > > t5326: demonstrate bitmap corruption after permutation\n> > > midx.c: make changing the preferred pack safe\n> >\n> > Just chiming in to say that I reviewed an earlier version of this series\n> > and the version in this submission looks good to me.\n>\n> Ditto. ;)\n\nAll three of us missed that this PORD chunk actually contains the\npsuedo-pack position for every object in the MIDX. That is OK, but it's\ndefinitely adding more than 4 bytes per pack to the MIDX (in practice,\nit's adding 4 bytes per object).\n\nI'm semi-OK with this direction, since it's tantamount to storing the\n.rev file's contents in the MIDX itself. And even though we're not\nreading from it, it is doing the thing we need it to which is causing\nthe MIDX to change its checksum when the object order changes.\n\nBut I'm curious what both of your thoughts are before moving forward.\n\nThanks,\nTaylor\n"},{"id":"443843","messageId":"YbPVP0BvYcVsfOrf@nand.local","threadId":"57055","inReplyTo":"YbOeEjNic5ETGcdy@nand.local","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-10T22:31:27Z","receivedAt":"2021-12-10T22:31:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 10, 2021 at 01:36:02PM -0500, Taylor Blau wrote:\n> On Wed, Dec 08, 2021 at 02:55:13PM -0500, Jeff King wrote:\n> > On Wed, Dec 08, 2021 at 02:30:17PM -0500, Derrick Stolee wrote:\n> >\n> > > > Taylor Blau (2):\n> > > > t5326: demonstrate bitmap corruption after permutation\n> > > > midx.c: make changing the preferred pack safe\n> > >\n> > > Just chiming in to say that I reviewed an earlier version of this series\n> > > and the version in this submission looks good to me.\n> >\n> > Ditto. ;)\n>\n> All three of us missed that this PORD chunk actually contains the\n> psuedo-pack position for every object in the MIDX. That is OK, but it's\n> definitely adding more than 4 bytes per pack to the MIDX (in practice,\n> it's adding 4 bytes per object).\n>\n> I'm semi-OK with this direction, since it's tantamount to storing the\n> .rev file's contents in the MIDX itself. And even though we're not\n> reading from it, it is doing the thing we need it to which is causing\n> the MIDX to change its checksum when the object order changes.\n>\n> But I'm curious what both of your thoughts are before moving forward.\n\nTo just add a little bit more detail before I mostly close my computer\nfor the weekend:\n\nThe key part of this bug is that the MIDX checksum could remain\nunchanged even when the object order used to write the MIDX bitmap does,\nand that's what the first patch here is demonstrating.\n\nHaving PORD contain the same data as the .rev file still accomplishes\nour original goal of preventing this bug, because it forces the checksum\nto change when the object order does. But it's definitely more invasive\nthan I had imagined.\n\nI had originally imagined that storing the preferred pack's identity\nalone would be enough to solve this bug. But that isn't quite so,\nbecause we break ties among duplicate objects first by prefered-ness,\nthen by their pack's mtime. So that could change too, and it would cause\nus to break in the same way.\n\nAt the bare minimum you need an ordering of all of the packs in the\nMIDX (like I had originally imagined here). At most, we could do\nsomething like what is unintentionally written here, which would allow\nus to get rid of MIDX .rev files entirely. I think doing the former is\nsimpler, and I am not sure if there are practical advantages to the\nlatter.\n\nBut I'm definitely curious as to what others think would be a good\ndirection to pursue.\n\nThanks,\nTaylor\n"},{"id":"443852","messageId":"YbQBZX+Wo3zlOcWc@nand.local","threadId":"57055","inReplyTo":"YbPVP0BvYcVsfOrf@nand.local","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-11T01:39:49Z","receivedAt":"2021-12-11T01:39:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 10, 2021 at 05:31:27PM -0500, Taylor Blau wrote:\n> I had originally imagined that storing the preferred pack's identity\n> alone would be enough to solve this bug. But that isn't quite so,\n> because we break ties among duplicate objects first by prefered-ness,\n> then by their pack's mtime. So that could change too, and it would cause\n> us to break in the same way.\n>\n> At the bare minimum you need an ordering of all of the packs in the\n> MIDX (like I had originally imagined here). At most, we could do\n> something like what is unintentionally written here, which would allow\n> us to get rid of MIDX .rev files entirely. I think doing the former is\n> simpler, and I am not sure if there are practical advantages to the\n> latter.\n\nThinking on it more, I don't think this \"at minimum you would need...\"\nis quite right either. It would suffice to know the identity of the\npreferred pack, and the mtimes of all of the other packs, since that\nalone is enough to reconstruct the object order.\n\nThat is pretty appealing, too, because knowing the order of packs would\nrequire some major surgery (the order of packs isn't really something\nthe MIDX code thinks about, it's inferred from the way it sorts\nobjects).\n\nSo here I'd imagine we'd have two chunks:\n\n - `PREF`: a 4-byte network order integer identifier of the preferred\n pack\n - `MTME`: an array of 4-byte network order integers containing the\n mtimes of each pack in lexical order of the pack checksums.\n\nUnless I'm missing something, I'm pretty sure that these two would be\nsufficient to uniquely identify the object order (and, importantly, to\ncause the MIDX's checksum to change when the object order does).\n\nAnd these are vastly easier to write down than the ordering of the packs\nthemselves, while also being a much more incremental step forward. So I\nthink I'm more comfortable with this direction than anything discussed\nso far.\n\nThanks,\nTaylor\n"},{"id":"443979","messageId":"6ea31bb3-3363-ece0-9947-b303d5975263@gmail.com","threadId":"57055","inReplyTo":"YbQBZX+Wo3zlOcWc@nand.local","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-13T14:00:55Z","receivedAt":"2021-12-13T14:00:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/10/2021 8:39 PM, Taylor Blau wrote:\n> On Fri, Dec 10, 2021 at 05:31:27PM -0500, Taylor Blau wrote:\n>> I had originally imagined that storing the preferred pack's identity\n>> alone would be enough to solve this bug. But that isn't quite so,\n>> because we break ties among duplicate objects first by prefered-ness,\n>> then by their pack's mtime. So that could change too, and it would cause\n>> us to break in the same way.\n>>\n>> At the bare minimum you need an ordering of all of the packs in the\n>> MIDX (like I had originally imagined here). At most, we could do\n>> something like what is unintentionally written here, which would allow\n>> us to get rid of MIDX .rev files entirely. I think doing the former is\n>> simpler, and I am not sure if there are practical advantages to the\n>> latter.\n> \n> Thinking on it more, I don't think this \"at minimum you would need...\"\n> is quite right either. It would suffice to know the identity of the\n> preferred pack, and the mtimes of all of the other packs, since that\n> alone is enough to reconstruct the object order.\n> \n> That is pretty appealing, too, because knowing the order of packs would\n> require some major surgery (the order of packs isn't really something\n> the MIDX code thinks about, it's inferred from the way it sorts\n> objects).\n\nI think the root cause is that the object order can change when the\npreferred pack changes with the same set of pack-files. Suppose we\nadded more complicated ways of deduplicating objects across the packs?\nThen whatever we include here based on preferred packs and mtimes\nwould need to be updated to match.\n\nHowever, if we store the contents of the .rev file in the MIDX itself,\nthen we don't need that extra layer of indirection.\n\nI'm leaning towards keeping the contents of the PORD chunk as-is, but\nrenaming it to something like OORD (for object order). Then, we can\ncarefully transition from using the .rev file to reading this chunk.\nWe will want to continue looking for the .rev file when this chunk does\nnot exist.\n\nThanks,\n-Stolee\n\n"},{"id":"443982","messageId":"YbdZPWlqSyfcTYhR@nand.local","threadId":"57055","inReplyTo":"6ea31bb3-3363-ece0-9947-b303d5975263@gmail.com","subject":"Re: [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-13T14:31:25Z","receivedAt":"2021-12-13T14:31:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 13, 2021 at 09:00:55AM -0500, Derrick Stolee wrote:\n> I think the root cause is that the object order can change when the\n> preferred pack changes with the same set of pack-files. Suppose we\n> added more complicated ways of deduplicating objects across the packs?\n> Then whatever we include here based on preferred packs and mtimes\n> would need to be updated to match.\n>\n> However, if we store the contents of the .rev file in the MIDX itself,\n> then we don't need that extra layer of indirection.\n\nYeah, towards the end of the weekend I started to lean towards putting\nthe whole object order inside of the MIDX.\n\nYou're definitely right that the root of the problem is the object order\nis stored separately from the MIDX itself. And we probably could get\npretty creative and figure out what the minimal amount of information\nneeded to fingerprint an object order is. But the amount of time I spent\nthinking about each proposal did not leave me with the obvious sense\nthat there still weren't ways around that could change the object order\nwithout changing the MIDX's checksum.\n\n> I'm leaning towards keeping the contents of the PORD chunk as-is, but\n> renaming it to something like OORD (for object order). Then, we can\n> carefully transition from using the .rev file to reading this chunk.\n> We will want to continue looking for the .rev file when this chunk does\n> not exist.\n\nAgreed. What do you think about a series that just writes the chunk but\ndoes not read it? That would solve the bug in the immediate-term. Then\nwe can go back and transition from the .rev file to the new chunk.\n\nReading the revindex chunk over the .rev file is pretty easy, as\ndemonstrated by the patch below.\n\nWhat I'm less certain about is how we should write tests to make sure\nthe old strategy works, too. Do we want to run the MIDX bitmap tests\ntwice in a separate mode each (one that writes .rev files and one that\nwrites chuns in the MIDX)? Something else? Thoughts welcome.\n\n--- 8< ---\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 70d0fbafcb..d59ae73198 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -301,6 +301,20 @@ int load_midx_revindex(struct multi_pack_index *m)\n \tif (m->revindex_data)\n \t\treturn 0;\n\n+\tif (m->chunk_revindex) {\n+\t\t/*\n+\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n+\t\t * the reverse index instead of trying to load a separate `.rev`\n+\t\t * file.\n+\t\t *\n+\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n+\t\t * not want to accidentally call munmap() in the middle of the\n+\t\t * MIDX.\n+\t\t */\n+\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n+\t\treturn 0;\n+\t}\n+\n \tget_midx_rev_filename(&revindex_name, m);\n\n \tret = load_revindex_from_disk(revindex_name.buf,\n"},{"id":"444057","messageId":"cover.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1638991570.git.me@ttaylorr.com","subject":"[PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:22Z","receivedAt":"2021-12-14T01:55:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by\nwhich they can become corrupt when permuting their pack order.\n\nThe problem is described in detail in the second patch. But this version goes\nfurther and stops writing the MIDX .rev file altogether, instead reading the\nsame data out of a new optional 'RIDX' chunk (falling back to the .rev file when\nnecessary).\n\nApplying just the first two patches is sufficient to fix this problem, but the\nremaining six go further to deprecate the MIDX .rev file altogether. I'm not\ncrazy about the testing strategy, though I think it was the best that I came up\nwith.\n\nThe idea is to essentially copy the core of t5326 into a new t5327. The former\nuses the RIDX chunk, and the latter uses .rev files. I would have like to just\nrun the same tests in t5326 twice, but they munge the state of their test\nrepository enough to make it sufficiently awkward to do in practice.\n\nSo I'm definitely open to suggestions there, but otherwise this series should go\na long ways towards fixing my design mistake of having the MIDX .rev file be\nseparate from the MIDX itself.\n\nTaylor Blau (8):\n t5326: demonstrate bitmap corruption after permutation\n midx.c: make changing the preferred pack safe\n pack-revindex.c: instrument loading on-disk reverse index\n t5326: drop unnecessary setup\n t5326: extract `test_rev_exists`\n t5326: move tests to t/lib-bitmap.sh\n t/lib-bitmap.sh: parameterize tests over reverse index source\n midx: read `RIDX` chunk when present\n\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 13 +-\n midx.c | 31 +++-\n midx.h | 1 +\n pack-revindex.c | 20 ++\n t/lib-bitmap.sh | 186 +++++++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 144 +-------------\n t/t5327-multi-pack-bitmaps-rev.sh | 23 +++\n t/t7700-repack.sh | 4 -\n 9 files changed, 271 insertions(+), 152 deletions(-)\n create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\nRange-diff against v1:\n1: ea91cebb6b = 1: dfbac4bc60 t5326: demonstrate bitmap corruption after permutation\n2: fa42d816f4 ! 2: 4ea52e66dd midx.c: make changing the preferred pack safe\n @@ Commit message\n or MIDX, but writing that data into the MIDX itself makes that a\n circular dependency).\n \n - Instead, make the pack order used during bitmap generation part of the\n + Instead, make the object order used during bitmap generation part of the\n MIDX itself. That means that the new test in t5326 will cause the MIDX's\n checksum to update, preventing the stale read problem.\n \n - There is no need to store the complete pack order here, since the only\n - way a fixed set of packs can result in a change in the object ordering\n - is by changing the preferred pack. That's because packs are sorted\n - according to their pack name only, so it would be impossible to induce a\n - different order by, say, touching a pack's mtime.\n -\n - But storing the complete pack order gives us some flexibility in the\n - future, and it only costs 4 bytes per pack to do. In the future, we\n - could also optimize .rev reads looking for the identity of the preferred\n - pack by consulting this list if it exists.\n + In theory, it is possible to store a \"fingerprint\" of the full object\n + order here, so long as that fingerprint changes at least as often as the\n + full object order does. Some possibilities here include storing the\n + identity of the preferred pack, along with the mtimes of the\n + non-preferred packs in a consistent order. But storing a limited part of\n + the information makes it difficult to reason about whether or not there\n + are gaps between the two that would cause us to get bitten by this bug\n + again.\n \n Signed-off-by: Taylor Blau \n \n @@ Documentation/technical/multi-pack-index.txt: and their offsets into multiple pa\n - An offset within the jth packfile for the object.\n - If large offsets are required, we use another list of large\n offsets similar to version 2 pack-indexes.\n -+- An optional list of pack IDs in bitmap-sorted order.\n ++- An optional list of objects in pseudo-pack order (used with MIDX bitmaps).\n \n Thus, we can provide O(log N) lookup time for any number\n of packfiles.\n @@ Documentation/technical/pack-format.txt: CHUNK DATA:\n \t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n \t 8-byte offsets into large packfiles.\n \n -+\t[Optional] Bitmap pack order (ID: {'P', 'O', 'R', 'D'})\n -+\t A list of 4-byte pack identifiers in sorted order according to\n -+\t their relative bitmap positions.\n ++\t[Optional] Bitmap pack order (ID: {'R', 'I', 'D', 'X'})\n ++\t A list of MIDX positions (one per object in the MIDX, num_objects in\n ++\t total, each a 4-byte unsigned integer in network byte order), sorted\n ++\t according to their relative bitmap/pseudo-pack positions.\n +\n TRAILER:\n \n \tIndex checksum of the above contents.\n +@@ Documentation/technical/pack-format.txt: In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n + objects in packs stored by the MIDX, laid out in pack order, and the\n + packs arranged in MIDX order (with the preferred pack coming first).\n + \n +-Finally, note that the MIDX's reverse index is not stored as a chunk in\n +-the multi-pack-index itself. This is done because the reverse index\n +-includes the checksum of the pack or MIDX to which it belongs, which\n +-makes it impossible to write in the MIDX. To avoid races when rewriting\n +-the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n +-filename (e.g., `multi-pack-index-xyz.rev`).\n ++The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n ++the MIDX itself.\n \n ## midx.c ##\n @@\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n -+#define MIDX_CHUNKID_PACKORDER 0x504f5244 /* \"PORD\" */\n ++#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n @@ midx.c: static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n -+static int write_midx_pack_order(struct hashfile *f,\n -+\t\t\t\t void *data)\n ++static int write_midx_revindex(struct hashfile *f,\n ++\t\t\t void *data)\n +{\n +\tstruct write_midx_context *ctx = data;\n +\tuint32_t i;\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \n +\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n +\t\tctx.pack_order = midx_pack_order(&ctx);\n -+\t\tadd_chunk(cf, MIDX_CHUNKID_PACKORDER,\n ++\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n +\t\t\t ctx.entries_nr * sizeof(uint32_t),\n -+\t\t\t write_midx_pack_order);\n ++\t\t\t write_midx_revindex);\n +\t}\n +\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n-: ---------- > 3: b630fea149 pack-revindex.c: instrument loading on-disk reverse index\n-: ---------- > 4: f430b6f2e9 t5326: drop unnecessary setup\n-: ---------- > 5: 73faab9f42 t5326: extract `test_rev_exists`\n-: ---------- > 6: bf42b116e1 t5326: move tests to t/lib-bitmap.sh\n-: ---------- > 7: fa91631024 t/lib-bitmap.sh: parameterize tests over reverse index source\n-: ---------- > 8: 993bfa8dd8 midx: read `RIDX` chunk when present\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"444058","messageId":"dfbac4bc608a3a7366a57ccaca5273cbc93045d6.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"[PATCH v2 1/8] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:28Z","receivedAt":"2021-12-14T01:55:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch demonstrates a cause of bitmap corruption that can occur when\nthe contents of the multi-pack index does not change, but the underlying\nobject order does.\n\nIn this example, we have a MIDX containing two packs, each with a\ndistinct set of objects (pack A corresponds to the tree, blob, and\ncommit from the first patch, and pack B corresponds to the second\npatch).\n\nFirst, a MIDX is written where the 'A' pack is preferred. As expected,\nthe bitmaps generated there are in-tact. But then, we generate an\nidentical MIDX with a different object order: this time preferring pack\n'B'.\n\nDue to a bug which will be explained and fixed in the following commit,\nthe MIDX is updated, but the .rev file is not, causing the .bitmap file\nto be read incorrectly. Specifically, the .bitmap file will contain\ncorrect data, but the auxiliary object order in the .rev file is stale,\ncausing readers to get confused by reading the new bitmaps using the old\nobject order.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex e187f90f29..0ca2868b0b 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,4 +395,35 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n+test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\tpack-$A.idx\n+\t\tpack-$B.idx\n+\t\tEOF\n+\n+\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t--preferred-pack=pack-$A.pack \n---\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 13 +++++-----\n midx.c | 25 ++++++++++++++++----\n t/t5326-multi-pack-bitmaps.sh | 2 +-\n 4 files changed, 30 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex 86f40f2490..6e1fb092c6 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -24,6 +24,7 @@ and their offsets into multiple packfiles. It contains:\n - An offset within the jth packfile for the object.\n - If large offsets are required, we use another list of large\n offsets similar to version 2 pack-indexes.\n+- An optional list of objects in pseudo-pack order (used with MIDX bitmaps).\n \n Thus, we can provide O(log N) lookup time for any number\n of packfiles.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 8d2f42f29e..6d3efb7d16 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -376,6 +376,11 @@ CHUNK DATA:\n \t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n \t 8-byte offsets into large packfiles.\n \n+\t[Optional] Bitmap pack order (ID: {'R', 'I', 'D', 'X'})\n+\t A list of MIDX positions (one per object in the MIDX, num_objects in\n+\t total, each a 4-byte unsigned integer in network byte order), sorted\n+\t according to their relative bitmap/pseudo-pack positions.\n+\n TRAILER:\n \n \tIndex checksum of the above contents.\n@@ -456,9 +461,5 @@ In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n objects in packs stored by the MIDX, laid out in pack order, and the\n packs arranged in MIDX order (with the preferred pack coming first).\n \n-Finally, note that the MIDX's reverse index is not stored as a chunk in\n-the multi-pack-index itself. This is done because the reverse index\n-includes the checksum of the pack or MIDX to which it belongs, which\n-makes it impossible to write in the MIDX. To avoid races when rewriting\n-the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n-filename (e.g., `multi-pack-index-xyz.rev`).\n+The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n+the MIDX itself.\ndiff --git a/midx.c b/midx.c\nindex 837b46b2af..d3179e9c02 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n+#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n@@ -833,6 +834,18 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_midx_revindex(struct hashfile *f,\n+\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\n+\treturn 0;\n+}\n+\n struct midx_pack_order_data {\n \tuint32_t nr;\n \tuint32_t pack;\n@@ -891,7 +904,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n \t\t\t\t\tmidx_hash, WRITE_REV);\n \n-\tif (finalize_object_file(tmp_file, buf.buf))\n+\tif (rename(tmp_file, buf.buf))\n \t\tdie(_(\"cannot store reverse index file\"));\n \n \tstrbuf_release(&buf);\n@@ -1403,15 +1416,19 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n \n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n+\t\t\t ctx.entries_nr * sizeof(uint32_t),\n+\t\t\t write_midx_revindex);\n+\t}\n+\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n-\t\tctx.pack_order = midx_pack_order(&ctx);\n-\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 0ca2868b0b..353282310d 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,7 +395,7 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"444060","messageId":"b630fea149f840ad049a4089047bb35d59f00c28.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"[PATCH v2 3/8] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:33Z","receivedAt":"2021-12-14T01:55:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\nsource for the reverse index's data. But it will be useful for tests to\nbe able to determine whether the reverse index was loaded from the\nseparate .rev file, or from a chunk within the MIDX.\n\nTo instrument this, add a trace2 event which the tests can look for in\norder to determine the reverse index's source.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 70d0fbafcb..bd15ebad03 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -301,6 +301,9 @@ int load_midx_revindex(struct multi_pack_index *m)\n \tif (m->revindex_data)\n \t\treturn 0;\n \n+\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n+\t\t\t \"source\", \"rev\");\n+\n \tget_midx_rev_filename(&revindex_name, m);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"444061","messageId":"f430b6f2e9a8b2c0fb84e86c353fee47a9753845.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"[PATCH v2 4/8] t5326: drop unnecessary setup","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:36Z","receivedAt":"2021-12-14T01:55:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The core.multiPackIndex config became true by default back in 18e449f86b\n(midx: enable core.multiPackIndex by default, 2020-09-25), so it is no\nlonger necessary to enable it explicitly.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 4 ----\n 1 file changed, 4 deletions(-)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 353282310d..1a9581af30 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -19,10 +19,6 @@ midx_pack_source () {\n \n setup_bitmap_history\n \n-test_expect_success 'enable core.multiPackIndex' '\n-\tgit config core.multiPackIndex true\n-'\n-\n test_expect_success 'create single-pack midx with bitmaps' '\n \tgit repack -ad &&\n \tgit multi-pack-index write --bitmap &&\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"444062","messageId":"73faab9f429221b7be88ea88ce59bc47afb57348.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"[PATCH v2 5/8] t5326: extract `test_rev_exists`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:38Z","receivedAt":"2021-12-14T01:55:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To determine which source of data is used for the MIDX's reverse index\ncache, introduce a helper which forces loading the reverse index, and\nthen looks for the special trace2 event introduced in a previous commit.\n\nFor now, this helper just looks for when the legacy MIDX .rev file was\nloaded, but in a subsequent commit will become parameterized over the\nthe reverse index's source.\n\nThis function replaces checking for the existence of the .rev file. We\ncould write a similar helper to ensure that the .rev file is cleaned up\nafter repacking, but it will make subsequent tests more difficult to\nwrite, and provides marginal value since we already check that the MIDX\n.bitmap file is removed.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 38 +++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 1a9581af30..85a91c2675 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -17,16 +17,30 @@ midx_pack_source () {\n \ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n }\n \n+test_rev_exists () {\n+\tcommit=\"$1\"\n+\n+\ttest_expect_success 'reverse index exists' '\n+\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n+\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t'\n+}\n+\n setup_bitmap_history\n \n test_expect_success 'create single-pack midx with bitmaps' '\n \tgit repack -ad &&\n \tgit multi-pack-index write --bitmap &&\n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD\n+\n basic_bitmap_tests\n \n test_expect_success 'create new additional packs' '\n@@ -52,10 +66,11 @@ test_expect_success 'create multi-pack midx with bitmaps' '\n \ttest_line_count = 25 packs &&\n \n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD\n+\n basic_bitmap_tests\n \n test_expect_success '--no-bitmap is respected when bitmaps exist' '\n@@ -66,7 +81,6 @@ test_expect_success '--no-bitmap is respected when bitmaps exist' '\n \n \ttest_path_is_file $midx &&\n \ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \tgit multi-pack-index write --no-bitmap &&\n \n@@ -206,10 +220,11 @@ test_expect_success 'setup partial bitmaps' '\n \ttest_commit loose &&\n \tgit multi-pack-index write --bitmap 2>err &&\n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD~\n+\n basic_bitmap_tests HEAD~\n \n test_expect_success 'removing a MIDX clears stale bitmaps' '\n@@ -224,7 +239,6 @@ test_expect_success 'removing a MIDX clears stale bitmaps' '\n \n \t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n \t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\tstale_rev=$midx-$(midx_checksum $objdir).rev &&\n \t\trm $midx &&\n \n \t\t# Then write a new MIDX.\n@@ -234,9 +248,7 @@ test_expect_success 'removing a MIDX clears stale bitmaps' '\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\ttest_path_is_missing $stale_bitmap &&\n-\t\ttest_path_is_missing $stale_rev\n+\t\ttest_path_is_missing $stale_bitmap\n \t)\n '\n \n@@ -257,7 +269,6 @@ test_expect_success 'pack.preferBitmapTips' '\n \t\tgit multi-pack-index write --bitmap &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \t\ttest-tool bitmap list-commits | sort >bitmaps &&\n \t\tcomm -13 bitmaps commits >before &&\n@@ -267,7 +278,6 @@ test_expect_success 'pack.preferBitmapTips' '\n \t\t\tsnapshot &&\n \n \t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n \t\trm -fr $midx &&\n \n \t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"444063","messageId":"bf42b116e1872fd69164fe15f098b7a72e938c24.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"[PATCH v2 6/8] t5326: move tests to t/lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:41Z","receivedAt":"2021-12-14T01:55:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In t5326, we have a handful of tests that we would like to run twice:\nonce using the MIDX's new `RIDX` chunk as the source of the\nreverse-index cache, and once using the separate `.rev` file.\n\nBut because these tests mutate the state of the underlying repository,\nand then make assumptions about those mutations occurring in a certain\nsequence, simply running the tests twice in the same repository is\nawkward.\n\nInstead, extract the core of interesting tests into t/lib-bitmap.sh to\nprepare for them to be run twice, each in a separate test script. This\nmeans that they can each operate on a separate repository, removing any\nconcerns about mutating state.\n\nFor now, this patch is a strict cut-and-paste of some tests from t5326.\nThe tests which did not move are not interesting with respect to the\nsource of their reverse index data.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 178 ++++++++++++++++++++++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 174 +--------------------------------\n 2 files changed, 180 insertions(+), 172 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 21d0392dda..771c41c2ea 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,6 +1,9 @@\n # Helpers for scripts testing bitmap functionality; see t5310 for\n # example usage.\n \n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -264,3 +267,178 @@ have_delta () {\n midx_checksum () {\n \ttest-tool read-midx --checksum \"$1\"\n }\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+test_rev_exists () {\n+\tcommit=\"$1\"\n+\n+\ttest_expect_success 'reverse index exists' '\n+\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n+\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t'\n+}\n+\n+midx_bitmap_core () {\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'create single-pack midx with bitmaps' '\n+\t\tgit repack -ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'create new additional packs' '\n+\t\tfor i in $(test_seq 1 16)\n+\t\tdo\n+\t\t\ttest_commit \"$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\n+\t\tgit checkout -b other2 HEAD~8 &&\n+\t\tfor i in $(test_seq 1 8)\n+\t\tdo\n+\t\t\ttest_commit \"side-$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\t\tgit checkout second\n+\t'\n+\n+\ttest_expect_success 'create multi-pack midx with bitmaps' '\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\tls $objdir/pack/pack-*.pack >packs &&\n+\t\ttest_line_count = 25 packs &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_commit respect--no-bitmap &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\tgit multi-pack-index write --no-bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n+\t'\n+\n+\ttest_expect_success 'setup midx with base from later pack' '\n+\t\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t\t# prefers to delete contents out of a base rather than add to a shorter\n+\t\t# object.\n+\t\ttest_seq 1 128 >a &&\n+\t\ttest_seq 1 130 >b &&\n+\n+\t\tgit add a b &&\n+\t\tgit commit -m \"initial commit\" &&\n+\n+\t\ta=$(git rev-parse HEAD:a) &&\n+\t\tb=$(git rev-parse HEAD:b) &&\n+\n+\t\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\t\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\t) &&\n+\n+\t\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t\t# any other object.\n+\t\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t\t$b\n+\t\t$(git rev-parse HEAD)\n+\t\t$(git rev-parse HEAD^{tree})\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit prune-packed &&\n+\t\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t\t# reuse.\n+\t\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\t\thave_delta $a $b &&\n+\t\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+\t'\n+\n+\trev_list_tests 'full bitmap with backwards delta'\n+\n+\ttest_expect_success 'clone with bitmaps enabled' '\n+\t\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\t\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\n+\t\t\ttest_commit A &&\n+\t\t\ttest_commit B &&\n+\n+\t\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\t\tpack-$A.idx\n+\t\t\tpack-$B.idx\n+\t\t\tEOF\n+\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\t--preferred-pack=pack-$A.pack err &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD~\n+\n+\tbasic_bitmap_tests HEAD~\n+}\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 85a91c2675..100ac90d15 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -9,135 +9,7 @@ test_description='exercise basic multi-pack bitmap functionality'\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n-objdir=.git/objects\n-midx=$objdir/pack/multi-pack-index\n-\n-# midx_pack_source \n-midx_pack_source () {\n-\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n-}\n-\n-test_rev_exists () {\n-\tcommit=\"$1\"\n-\n-\ttest_expect_success 'reverse index exists' '\n-\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n-\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n-\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n-\t'\n-}\n-\n-setup_bitmap_history\n-\n-test_expect_success 'create single-pack midx with bitmaps' '\n-\tgit repack -ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD\n-\n-basic_bitmap_tests\n-\n-test_expect_success 'create new additional packs' '\n-\tfor i in $(test_seq 1 16)\n-\tdo\n-\t\ttest_commit \"$i\" &&\n-\t\tgit repack -d || return 1\n-\tdone &&\n-\n-\tgit checkout -b other2 HEAD~8 &&\n-\tfor i in $(test_seq 1 8)\n-\tdo\n-\t\ttest_commit \"side-$i\" &&\n-\t\tgit repack -d || return 1\n-\tdone &&\n-\tgit checkout second\n-'\n-\n-test_expect_success 'create multi-pack midx with bitmaps' '\n-\tgit multi-pack-index write --bitmap &&\n-\n-\tls $objdir/pack/pack-*.pack >packs &&\n-\ttest_line_count = 25 packs &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD\n-\n-basic_bitmap_tests\n-\n-test_expect_success '--no-bitmap is respected when bitmaps exist' '\n-\tgit multi-pack-index write --bitmap &&\n-\n-\ttest_commit respect--no-bitmap &&\n-\tgit repack -d &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\n-\tgit multi-pack-index write --no-bitmap &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n-'\n-\n-test_expect_success 'setup midx with base from later pack' '\n-\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n-\t# prefers to delete contents out of a base rather than add to a shorter\n-\t# object.\n-\ttest_seq 1 128 >a &&\n-\ttest_seq 1 130 >b &&\n-\n-\tgit add a b &&\n-\tgit commit -m \"initial commit\" &&\n-\n-\ta=$(git rev-parse HEAD:a) &&\n-\tb=$(git rev-parse HEAD:b) &&\n-\n-\t# In the first pack, \"a\" is stored as a delta to \"b\".\n-\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n-\t$a\n-\t$b\n-\tEOF\n-\t) &&\n-\n-\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n-\t# any other object.\n-\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n-\t$b\n-\t$(git rev-parse HEAD)\n-\t$(git rev-parse HEAD^{tree})\n-\tEOF\n-\t) &&\n-\n-\tgit prune-packed &&\n-\t# Use the second pack as the preferred source, so that \"b\" occurs\n-\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n-\t# reuse.\n-\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n-\n-\thave_delta $a $b &&\n-\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n-'\n-\n-rev_list_tests 'full bitmap with backwards delta'\n-\n-test_expect_success 'clone with bitmaps enabled' '\n-\tgit clone --no-local --bare . clone-reverse-delta.git &&\n-\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n-\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+midx_bitmap_core\n \n bitmap_reuse_tests() {\n \tfrom=$1\n@@ -214,18 +86,7 @@ test_expect_success 'missing object closure fails gracefully' '\n \t)\n '\n \n-test_expect_success 'setup partial bitmaps' '\n-\ttest_commit packed &&\n-\tgit repack &&\n-\ttest_commit loose &&\n-\tgit multi-pack-index write --bitmap 2>err &&\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD~\n-\n-basic_bitmap_tests HEAD~\n+midx_bitmap_partial_tests\n \n test_expect_success 'removing a MIDX clears stale bitmaps' '\n \trm -fr repo &&\n@@ -399,35 +260,4 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit A &&\n-\t\ttest_commit B &&\n-\n-\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n-\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n-\n-\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n-\t\tpack-$A.idx\n-\t\tpack-$B.idx\n-\t\tEOF\n-\n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\t--preferred-pack=pack-$A.pack \n---\n t/lib-bitmap.sh | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 771c41c2ea..0a35daf939 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -275,18 +275,24 @@ midx_pack_source () {\n \n test_rev_exists () {\n \tcommit=\"$1\"\n+\tkind=\"$2\"\n \n \ttest_expect_success 'reverse index exists' '\n \t\tGIT_TRACE2_EVENT_NESTING=10 \\\n \t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n \t\t\tgit rev-list --test-bitmap \"$commit\" &&\n \n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t\tif test \"rev\" = \"$kind\"\n+\t\tthen\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\t\tfi &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"$kind\\\"\" event.trace\n \t'\n }\n \n midx_bitmap_core () {\n+\trev_kind=\"${1:-rev}\"\n+\n \tsetup_bitmap_history\n \n \ttest_expect_success 'create single-pack midx with bitmaps' '\n@@ -296,7 +302,7 @@ midx_bitmap_core () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD\n+\ttest_rev_exists HEAD \"$rev_kind\"\n \n \tbasic_bitmap_tests\n \n@@ -326,7 +332,7 @@ midx_bitmap_core () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD\n+\ttest_rev_exists HEAD \"$rev_kind\"\n \n \tbasic_bitmap_tests\n \n@@ -429,6 +435,8 @@ midx_bitmap_core () {\n }\n \n midx_bitmap_partial_tests () {\n+\trev_kind=\"${1:-rev}\"\n+\n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\n \t\tgit repack &&\n@@ -438,7 +446,7 @@ midx_bitmap_partial_tests () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD~\n+\ttest_rev_exists HEAD~ \"$rev_kind\"\n \n \tbasic_bitmap_tests HEAD~\n }\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"444065","messageId":"993bfa8dd8480e74d64f657539b0c518ad155e5c.1639446906.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"[PATCH v2 8/8] midx: read `RIDX` chunk when present","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-14T01:55:45Z","receivedAt":"2021-12-14T01:55:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a MIDX contains the new `RIDX` chunk, ensure that the reverse index\nis read from it instead of the on-disk .rev file. Since we need to\nencode the object order in the MIDX itself for correctness reasons,\nthere is no point in storing the same data again outside of the MIDX.\n\nSo, this patch stops writing separate .rev files, and reads it out of\nthe MIDX itself. This is possible to do with relatively little new code,\nsince the format of the RIDX chunk is identical to the data in the .rev\nfile. In other words, we can implement this by pointing the\n`revindex_data` field at the reverse index chunk of the MIDX instead of\nthe .rev file without any other changes.\n\nNote that we have two knobs that are adjusted for the new tests:\nGIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\nwhether the MIDX .rev is written at all, and the latter controls whether\nwe read the MIDX's RIDX chunk.\n\nBoth are necessary to ensure that the test added at the beginning of\nthis series continues to work. This is because we always need to write\nthe RIDX chunk in the MIDX in order to change its checksum, but we want\nto make sure reading the existing .rev file still works (since the RIDX\nchunk takes precedence by default).\n\nArguably this isn't a very interesting mode to test, because the\nprecedence rules mean that we'll always read the RIDX chunk over the\n.rev file. But it makes it impossible for a user to induce corruption in\ntheir repository by adjusting the test knobs (since if we had an\neither/or knob they could stop writing the RIDX chunk, allowing them to\ntweak the MIDX's object order without changing its checksum).\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++++-\n midx.h | 1 +\n pack-revindex.c | 17 +++++++++++++++++\n t/lib-bitmap.sh | 4 ++--\n t/t5326-multi-pack-bitmaps.sh | 5 +++++\n t/t5327-multi-pack-bitmaps-rev.sh | 23 +++++++++++++++++++++++\n t/t7700-repack.sh | 4 ----\n 7 files changed, 53 insertions(+), 7 deletions(-)\n create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\ndiff --git a/midx.c b/midx.c\nindex d3179e9c02..9aba13b5b1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -162,6 +162,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n \n+\tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n+\t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tCALLOC_ARRAY(m->pack_names, m->num_packs);\n@@ -1429,7 +1432,8 @@ static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\n \t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &ctx,\ndiff --git a/midx.h b/midx.h\nindex b7d79a515c..22e8e53288 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -36,6 +36,7 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_object_offsets;\n \tconst unsigned char *chunk_large_offsets;\n+\tconst unsigned char *chunk_revindex;\n \n \tconst char **pack_names;\n \tstruct packed_git **packs;\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex bd15ebad03..08dc160167 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -298,9 +298,26 @@ int load_midx_revindex(struct multi_pack_index *m)\n {\n \tstruct strbuf revindex_name = STRBUF_INIT;\n \tint ret;\n+\n \tif (m->revindex_data)\n \t\treturn 0;\n \n+\tif (m->chunk_revindex) {\n+\t\t/*\n+\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n+\t\t * the reverse index instead of trying to load a separate `.rev`\n+\t\t * file.\n+\t\t *\n+\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n+\t\t * not want to accidentally call munmap() in the middle of the\n+\t\t * MIDX.\n+\t\t */\n+\t\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n+\t\t\t\t \"source\", \"midx\");\n+\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n+\t\treturn 0;\n+\t}\n+\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t \"source\", \"rev\");\n \ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 0a35daf939..f5eaa9cf68 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -291,7 +291,7 @@ test_rev_exists () {\n }\n \n midx_bitmap_core () {\n-\trev_kind=\"${1:-rev}\"\n+\trev_kind=\"${1:-midx}\"\n \n \tsetup_bitmap_history\n \n@@ -435,7 +435,7 @@ midx_bitmap_core () {\n }\n \n midx_bitmap_partial_tests () {\n-\trev_kind=\"${1:-rev}\"\n+\trev_kind=\"${1:-midx}\"\n \n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 100ac90d15..8c92acb0ce 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -9,6 +9,11 @@ test_description='exercise basic multi-pack bitmap functionality'\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n+GIT_TEST_MIDX_WRITE_REV=0\n+GIT_TEST_MIDX_READ_RIDX=1\n+export GIT_TEST_MIDX_WRITE_REV\n+export GIT_TEST_MIDX_READ_RIDX\n+\n midx_bitmap_core\n \n bitmap_reuse_tests() {\ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nnew file mode 100755\nindex 0000000000..d30ba632c8\n--- /dev/null\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality (.rev files)'\n+\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+# Unlike t5326, this test exercise multi-pack bitmap functionality where the\n+# object order is stored in a separate .rev file.\n+GIT_TEST_MIDX_WRITE_REV=1\n+GIT_TEST_MIDX_READ_RIDX=0\n+export GIT_TEST_MIDX_WRITE_REV\n+export GIT_TEST_MIDX_READ_RIDX\n+\n+midx_bitmap_core rev\n+midx_bitmap_partial_tests rev\n+\n+test_done\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 0260ad6f0e..3ee56cefd3 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -311,16 +311,13 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t\tchecksum=$(midx_checksum $objdir) &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$checksum.bitmap &&\n-\t\ttest_path_is_file $midx-$checksum.rev &&\n \n \t\ttest_commit repack-3 &&\n \t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-$checksum.bitmap &&\n-\t\ttest_path_is_missing $midx-$checksum.rev &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \t\ttest_commit repack-4 &&\n \t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n@@ -353,7 +350,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '\n \t\ttest_line_count = 1 before &&\n \n \t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n \t\trm -fr $midx &&\n \n \t\t# instead of constructing the snapshot ourselves (c.f., the test\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"444163","messageId":"xmqqee6d648n.fsf@gitster.g","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-15T19:46:16Z","receivedAt":"2021-12-15T19:46:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by\n> which they can become corrupt when permuting their pack order.\n\nThis seems to depend on tb/cruft-packs that is not yet in 'next', so\nI'll redo this topic branch by forking it at 'master', merging the\nother topic in, and then queuing these 8 patches.\n\n> So I'm definitely open to suggestions there, but otherwise this series should go\n> a long ways towards fixing my design mistake of having the MIDX .rev file be\n> separate from the MIDX itself.\n\nYeah, a single file with different chunks is a good way to ensure\natomicity of update.\n\nA note to reviewers.\n\nWe need to make sure that not just we can still read .rev in\nexisting repositories (and convert it to the new chunk) correctly,\nbut also decide what to do to older versions of Git once the\nrepository is touched by this new version. Would they be upset to\nsee no .rev files or is it just the performance thing (and it is\nmore correct to recompute the reverse index on the fly)? Should the\nnew chunk be made mandatory to cause them notice that they should\nnot muck with the repository, or is it optional? Things like that.\n\nThanks.\n"},{"id":"444173","messageId":"YbpgHBcyq/2+nLJB@nand.local","threadId":"57055","inReplyTo":"xmqqee6d648n.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-15T21:37:32Z","receivedAt":"2021-12-15T21:37:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Dec 15, 2021 at 11:46:16AM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by\n> > which they can become corrupt when permuting their pack order.\n>\n> This seems to depend on tb/cruft-packs that is not yet in 'next', so\n> I'll redo this topic branch by forking it at 'master', merging the\n> other topic in, and then queuing these 8 patches.\n\nHmm. They shouldn't depend on that topic. My local copy depends on\nabe6bb3905 (The first batch to start the current cycle, 2021-11-29), but\nI'm happy to rebase things if basing on abe6bb3905 was a mistake.\n\n> > So I'm definitely open to suggestions there, but otherwise this series should go\n> > a long ways towards fixing my design mistake of having the MIDX .rev file be\n> > separate from the MIDX itself.\n>\n> Yeah, a single file with different chunks is a good way to ensure\n> atomicity of update.\n>\n> A note to reviewers.\n>\n> We need to make sure that not just we can still read .rev in\n> existing repositories (and convert it to the new chunk) correctly,\n> but also decide what to do to older versions of Git once the\n> repository is touched by this new version. Would they be upset to\n> see no .rev files or is it just the performance thing (and it is\n> more correct to recompute the reverse index on the fly)? Should the\n> new chunk be made mandatory to cause them notice that they should\n> not muck with the repository, or is it optional? Things like that.\n\nAn old client that is looking in a repository where the reverse index is\nstored as a chunk inside of the MIDX (and does not appear as a separate\nfile on disk) will not be able to read the MIDX's bitmap. We could\ncompute the reverse index on the fly (as we did for a long time with\nsingle packs) but it is slow, and older clients obviously would not know\nhow to do it.\n\nBut the failure is graceful(ish): we'll get a warning that says we\ncouldn't find a reverse index, and then carry on pretending that the\nMIDX did not have a bitmap.\n\n ~/s/git [nand] (master) $ git.compile multi-pack-index write --bitmap\n Selecting bitmap commits: 73008, done.\n Building bitmaps: 100% (319/319), done.\n ~/s/git [nand] (master) $ rm -f .git/objects/pack/multi-pack-index-*.rev\n ~/s/git [nand] (master) $ git.compile rev-list --count --objects --use-bitmap-index HEAD\n warning: multi-pack bitmap is missing required reverse index\n warning: ignoring extra bitmap file:\n .git/objects/pack/pack-00465554dc3f3d96ac89a0ecc73cb5f5abbb35a5.pack\n warning: multi-pack bitmap is missing required reverse index\n warning: ignoring extra bitmap file:\n .git/objects/pack/pack-00465554dc3f3d96ac89a0ecc73cb5f5abbb35a5.pack\n 307913\n\nAs far as whether or not the chunk is necessary, it only *needs* to be\npresent in the MIDX if there is a corresponding MIDX bitmap. We could\ngenerate it always, but this series keeps in the tradition of the .rev\nfile and only writes it when we are also writing a bitmap.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"444177","messageId":"xmqqv8zp343h.fsf@gitster.g","threadId":"57055","inReplyTo":"YbpgHBcyq/2+nLJB@nand.local","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-15T22:17:38Z","receivedAt":"2021-12-15T22:17:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Wed, Dec 15, 2021 at 11:46:16AM -0800, Junio C Hamano wrote:\n>> Taylor Blau writes:\n>>\n>> > Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by\n>> > which they can become corrupt when permuting their pack order.\n>>\n>> This seems to depend on tb/cruft-packs that is not yet in 'next', so\n>> I'll redo this topic branch by forking it at 'master', merging the\n>> other topic in, and then queuing these 8 patches.\n>\n> Hmm. They shouldn't depend on that topic. My local copy depends on\n> abe6bb3905 (The first batch to start the current cycle, 2021-11-29), but\n> I'm happy to rebase things if basing on abe6bb3905 was a mistake.\n\nNot so fast. Let me double check. My attempt to apply on the tip\nof 'master' (as of yesterday) did not work and because the only\ntopic in 'seen' that touched midx.c was that other topic, I tried to\nmerge it with 'master' before applying the patches on top and it\nworked. If the other topic has no relation to this topic, it would\nwork as well (but then it does not explain why patches from the list\ndid not apply for me).\n"},{"id":"444179","messageId":"xmqqczlx32c3.fsf@gitster.g","threadId":"57055","inReplyTo":"xmqqv8zp343h.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-15T22:55:40Z","receivedAt":"2021-12-15T22:55:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Not so fast. Let me double check. My attempt to apply on the tip\n> of 'master' (as of yesterday) did not work and because the only\n> topic in 'seen' that touched midx.c was that other topic, I tried to\n> merge it with 'master' before applying the patches on top and it\n> worked. If the other topic has no relation to this topic, it would\n> work as well (but then it does not explain why patches from the list\n> did not apply for me).\n\nInteresting. The posted patches do apply cleanly to the tip of\n'master' as of the first batch abe6bb39 (The first batch to start\nthe current cycle, 2021-11-29), but since then a few topics have\ntouched Documentation/technical/multi-pack-index.txt and made the\npatch application fail to the tip of 'master' as of the second batch\ne773545c (The second batch, 2021-12-10).\n\nEven more interestingly, after merging the topic tb/cruft-packs,\nwhich does not even touch the multi-pack-index.txt file, the patches\nseemed to apply OK with \"-3\". Perhaps I should have tried \"am -3\"\nwhen I applied them to the 'master' with the second batch of topics.\nI dunno.\n\nThanks for sanity checking.\n"},{"id":"444180","messageId":"xmqqzgp11nnk.fsf@gitster.g","threadId":"57055","inReplyTo":"cover.1639446906.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-15T22:58:07Z","receivedAt":"2021-12-15T22:58:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Documentation/technical/multi-pack-index.txt | 1 +\n> Documentation/technical/pack-format.txt | 13 +-\n> midx.c | 31 +++-\n> midx.h | 1 +\n> pack-revindex.c | 20 ++\n> t/lib-bitmap.sh | 186 +++++++++++++++++++\n> t/t5326-multi-pack-bitmaps.sh | 144 +-------------\n> t/t5327-multi-pack-bitmaps-rev.sh | 23 +++\n> t/t7700-repack.sh | 4 -\n> 9 files changed, 271 insertions(+), 152 deletions(-)\n> create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\nAs 5327 is taken by the tb/cruft-packs topic, I'll move it to 5328.\n\nThanks.\n\n"},{"id":"444182","messageId":"Ybpzv+jVba6wafFN@nand.local","threadId":"57055","inReplyTo":"xmqqzgp11nnk.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-15T23:01:19Z","receivedAt":"2021-12-15T23:01:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Dec 15, 2021 at 02:58:07PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Documentation/technical/multi-pack-index.txt | 1 +\n> > Documentation/technical/pack-format.txt | 13 +-\n> > midx.c | 31 +++-\n> > midx.h | 1 +\n> > pack-revindex.c | 20 ++\n> > t/lib-bitmap.sh | 186 +++++++++++++++++++\n> > t/t5326-multi-pack-bitmaps.sh | 144 +-------------\n> > t/t5327-multi-pack-bitmaps-rev.sh | 23 +++\n> > t/t7700-repack.sh | 4 -\n> > 9 files changed, 271 insertions(+), 152 deletions(-)\n> > create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n>\n> As 5327 is taken by the tb/cruft-packs topic, I'll move it to 5328.\n\nAh, thanks. It should be a straightforward move. Alternatively feel free\nto eject the cruft packs topic and I can update that series to use\nt5328 instead.\n\nSounds like you were able to get the patches to apply, though, so I'll\nrefrain from sending anymore until there is a compelling reason to do\nso.\n\nThanks,\nTaylor\n"},{"id":"444514","messageId":"dbc75e29-aceb-051a-8235-e5c912172aed@gmail.com","threadId":"57055","inReplyTo":"73faab9f429221b7be88ea88ce59bc47afb57348.1639446906.git.me@ttaylorr.com","subject":"Re: [PATCH v2 5/8] t5326: extract `test_rev_exists`","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-20T18:33:32Z","receivedAt":"2021-12-20T18:33:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/2021 8:55 PM, Taylor Blau wrote:\n> To determine which source of data is used for the MIDX's reverse index\n> cache, introduce a helper which forces loading the reverse index, and\n> then looks for the special trace2 event introduced in a previous commit.\n> \n> For now, this helper just looks for when the legacy MIDX .rev file was\n> loaded, but in a subsequent commit will become parameterized over the\n> the reverse index's source.\n> \n> This function replaces checking for the existence of the .rev file. We\n> could write a similar helper to ensure that the .rev file is cleaned up\n> after repacking, but it will make subsequent tests more difficult to\n> write, and provides marginal value since we already check that the MIDX\n> .bitmap file is removed.\n\n...\n\n> +test_rev_exists () {\n> +\tcommit=\"$1\"\n> +\n> +\ttest_expect_success 'reverse index exists' '\n> +\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n\nVery recently, b8de3d6 (test-lib.sh: set GIT_TRACE2_EVENT_NESTING,\n2021-11-29) made it to master and sets this to 100 across the test\nsuite, so you don't need this line.\n\n> +\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n> +\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n\nThis use of $commit has me worried. Do the tests become too flaky\nto changes in how we choose commits for the bitmaps? Does that\nrequire callers to be too aware of the refstate when creating the\nbitmaps?\n\nPerhaps just `git rev-list --use-bitmap-index [--objects] HEAD`\nwould suffice to generate the trace event?\n\n> +\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n> +\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n> +\t'\n> +}\n> +\n> setup_bitmap_history\n> \n> test_expect_success 'create single-pack midx with bitmaps' '\n> \tgit repack -ad &&\n> \tgit multi-pack-index write --bitmap &&\n> \ttest_path_is_file $midx &&\n> -\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> -\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n> +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n> '\n> \n> +test_rev_exists HEAD\n> +\n\nPerhaps this helper would be more appropriate as a helper method\nwithin a test, rather than creating a test of its own? I think\nit looks better to include it next to the setup lines, something\nlike\n\ntest_expect_success 'create single-pack midx with bitmaps' '\n\tgit repack -ad &&\n\tgit multi-pack-index write --bitmap &&\n\ttest_path_is_file $midx &&\n\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n\ttest_rev_exists HEAD\n'\n\nThis would lead to a failure within 'create single-pack midx\nwith bitmaps' which is easier to find in the file.\n\nThanks,\n-Stolee\n"},{"id":"444516","messageId":"92d50859-ecfa-e7c5-d68b-d90a11579cd0@gmail.com","threadId":"57055","inReplyTo":"993bfa8dd8480e74d64f657539b0c518ad155e5c.1639446906.git.me@ttaylorr.com","subject":"Re: [PATCH v2 8/8] midx: read `RIDX` chunk when present","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-20T18:42:24Z","receivedAt":"2021-12-20T18:42:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/2021 8:55 PM, Taylor Blau wrote:\n...\n> Note that we have two knobs that are adjusted for the new tests:\n> GIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\n> whether the MIDX .rev is written at all, and the latter controls whether\n> we read the MIDX's RIDX chunk.\n\n...\n\n> +\tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n> +\t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex);\n> +\n\nSkip reading if GIT_TEST_MIDX_READ_RIDX=0 (do read if true or unset). Good.\n\n> -\tif (flags & MIDX_WRITE_REV_INDEX)\n> +\tif (flags & MIDX_WRITE_REV_INDEX &&\n> +\t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n> \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n\nOnly write if GIT_TEST_MIDX_WRITE_REV=1 (skip if false or unset). Good.\n\n> +\tif (m->chunk_revindex) {\n> +\t\t/*\n> +\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n> +\t\t * the reverse index instead of trying to load a separate `.rev`\n> +\t\t * file.\n> +\t\t *\n> +\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n> +\t\t * not want to accidentally call munmap() in the middle of the\n> +\t\t * MIDX.\n> +\t\t */\n> +\t\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> +\t\t\t\t \"source\", \"midx\");\n> +\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n> +\t\treturn 0;\n> +\t}\n> +\n> \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> \t\t\t \"source\", \"rev\");\n\nA natural way to do this.\n\n> \n> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> index 0a35daf939..f5eaa9cf68 100644\n> --- a/t/lib-bitmap.sh\n> +++ b/t/lib-bitmap.sh\n> @@ -291,7 +291,7 @@ test_rev_exists () {\n> }\n> \n> midx_bitmap_core () {\n> -\trev_kind=\"${1:-rev}\"\n> +\trev_kind=\"${1:-midx}\"\n> \n> \tsetup_bitmap_history\n> \n> @@ -435,7 +435,7 @@ midx_bitmap_core () {\n> }\n> \n> midx_bitmap_partial_tests () {\n> -\trev_kind=\"${1:-rev}\"\n> +\trev_kind=\"${1:-midx}\"\n\nAnd I'm glad we can just update the default here.\n\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> index 100ac90d15..8c92acb0ce 100755\n> --- a/t/t5326-multi-pack-bitmaps.sh\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -9,6 +9,11 @@ test_description='exercise basic multi-pack bitmap functionality'\n> GIT_TEST_MULTI_PACK_INDEX=0\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> \n> +GIT_TEST_MIDX_WRITE_REV=0\n> +GIT_TEST_MIDX_READ_RIDX=1\n> +export GIT_TEST_MIDX_WRITE_REV\n> +export GIT_TEST_MIDX_READ_RIDX\n\nTechnically, we could unset these variables, right? (\"...=\")\n\n> diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\n...\n> +# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n> +# automatic ones.\n> +GIT_TEST_MULTI_PACK_INDEX=0\n> +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> +\n> +# Unlike t5326, this test exercise multi-pack bitmap functionality where the\n> +# object order is stored in a separate .rev file.\n> +GIT_TEST_MIDX_WRITE_REV=1\n> +GIT_TEST_MIDX_READ_RIDX=0\n> +export GIT_TEST_MIDX_WRITE_REV\n> +export GIT_TEST_MIDX_READ_RIDX\n> +\n> +midx_bitmap_core rev\n> +midx_bitmap_partial_tests rev\n> +\n> +test_done\n\nNice that now we get the payoff of the test refactor. This new script exists\nonly to verify that we can still read the old .rev files, which is important\nfor compatibility.\n\nForgive me for \"speaking out loud\" throughout this patch. I don't see any\nneed for changes but it is the crux of making this series work.\n\nThanks,\n-Stolee\n"},{"id":"444517","messageId":"886da3f3-aa6a-ac74-2aa2-26abdf832ebd@gmail.com","threadId":"57055","inReplyTo":"xmqqee6d648n.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-20T18:51:22Z","receivedAt":"2021-12-20T18:51:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/15/2021 2:46 PM, Junio C Hamano wrote:\n> Taylor Blau writes:\n> \n>> Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by\n>> which they can become corrupt when permuting their pack order.\n...\n>> So I'm definitely open to suggestions there, but otherwise this series should go\n>> a long ways towards fixing my design mistake of having the MIDX .rev file be\n>> separate from the MIDX itself.\n> \n> Yeah, a single file with different chunks is a good way to ensure\n> atomicity of update.\n\nI just reviewed this series for the first time. Sorry for being so\nlate getting to it.\n\nI had a few minor recommendations but they are mostly nitpicks and\ndon't deserve holding up the series if there are no other major\ncomments.\n\n> A note to reviewers.\n> \n> We need to make sure that not just we can still read .rev in\n> existing repositories (and convert it to the new chunk) correctly,\n> but also decide what to do to older versions of Git once the\n> repository is touched by this new version. Would they be upset to\n> see no .rev files or is it just the performance thing (and it is\n> more correct to recompute the reverse index on the fly)? Should the\n> new chunk be made mandatory to cause them notice that they should\n> not muck with the repository, or is it optional? Things like that.\n\n1. Can we still read a .rev?\n\nThe new test script specifically verifies that existing repositories\nwill continue to read their .rev upon upgrade. Their .rev files will\nbe replaced with the chunk during the next write.\n\n2. What if they downgrade after the RIDX chunk is in place?\n\nThe .rev file is missing and the repo has a performance issue because\nthey can't use bitmaps. Correctness is not a problem. Anyone using\n.rev files for server use (where bitmaps are most useful) is hopefully\nalready careful about downgrading Git versions.\n\n3. Should the chunk be made mandatory?\n\nUnfortunately, the chunk format did not follow the index format's\nexample of making lowercase chunk IDs required. Instead, the chunks\nthat are necessary for v1 are necessary forever and all other chunks\nare deemed optional. Changing this would require something more\ndrastic like updating the version number or giving some grace period\nwhere released versions start treating lowercase chunk IDs as required\nbefore creating a new \"required\" chunk.\n\nThis does mean that if there is a version incompatibility, the RIDX\nchunk will just be ignored by the older version of Git.\n\nIn terms of making this a safe format upgrade, I think Taylor has\nachieved that.\n\nThe only thing I can think is that server operators might want to\ndeploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so\nany need to downgrade would not suffer a performance penalty for a\nmissing .rev file. If that is a planned way to safely deploy this\nchange, then it might be worth adding a test that we safely delete\na .rev file after writing both a .rev file and a RIDX chunk. (The\nRIDX chunk will be preferred, so maybe the previous .rev file hits\nsome logic that would skip its deletion?)\n\nThanks,\n-Stolee\n"},{"id":"444531","messageId":"YcDe9wl4XwH7PjX1@nand.local","threadId":"57055","inReplyTo":"886da3f3-aa6a-ac74-2aa2-26abdf832ebd@gmail.com","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-12-20T19:52:23Z","receivedAt":"2021-12-20T19:52:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 20, 2021 at 01:51:22PM -0500, Derrick Stolee wrote:\n> On 12/15/2021 2:46 PM, Junio C Hamano wrote:\n> > Taylor Blau writes:\n> >\n> >> Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by\n> >> which they can become corrupt when permuting their pack order.\n> ...\n> >> So I'm definitely open to suggestions there, but otherwise this series should go\n> >> a long ways towards fixing my design mistake of having the MIDX .rev file be\n> >> separate from the MIDX itself.\n> >\n> > Yeah, a single file with different chunks is a good way to ensure\n> > atomicity of update.\n>\n> I just reviewed this series for the first time. Sorry for being so\n> late getting to it.\n\nThanks for your review. I think that the upgrade plan is sane, but I can\ncomment a little bit more about that below.\n\nIn the meantime, some thoughts on all of the combinations of the new\nGIT_TEST_ variables:\n\n - GIT_TEST_MIDX_READ_RIDX=0 GIT_TEST_MIDX_WRITE_REV=0: this means that\n we won't use bitmaps at all, since we won't have a .rev file to\n read, and we will pretend that the RIDX chunk does not exist.\n\n - GIT_TEST_MIDX_READ_RIDX=0 GIT_TEST_MIDX_WRITE_REV=1: this is the\n status-quo of how things work today.\n\n - GIT_TEST_MIDX_READ_RIDX=1 GIT_TEST_MIDX_WRITE_REV=0: this is the\n status-quo of of how things will work after this patch series.\n\n - GIT_TEST_MIDX_READ_RIDX=1 GIT_TEST_MIDX_WRITE_REV=1: this is useful\n for testing that the RIDX chunk is preferred over reading the .rev\n file.\n\nSo all but the (0, 0) combination make sense. Perhaps we should ban that\ncombination entirely, since nobody would ever set it for a good reason.\nBut I think that none of these combinations would allow us to propagate\nthe corruption, since we will always *write* the new RIDX chunk, which\ncauses the MIDX's checksum to change when we changes its object order.\n\nI kind of hate these runtime checks that are only useful for testing. If\nanybody has better ideas of how we should go about this that still\nprovides comprehensive coverage of .rev files, then I'm all ears.\n\n(The nuclear option might be to just stop supporting .rev files for\nMIDX's altogether, since this is such a new feature. But that seems like\na pretty drastic step, and definitely isn't backwards compatible. So I'd\nbe wary of going in that direction).\n\n> 1. Can we still read a .rev?\n>\n> The new test script specifically verifies that existing repositories\n> will continue to read their .rev upon upgrade. Their .rev files will\n> be replaced with the chunk during the next write.\n\nYes, exactly.\n\n> 2. What if they downgrade after the RIDX chunk is in place?\n>\n> The .rev file is missing and the repo has a performance issue because\n> they can't use bitmaps. Correctness is not a problem. Anyone using\n> .rev files for server use (where bitmaps are most useful) is hopefully\n> already careful about downgrading Git versions.\n\nYes, and we gracefully degrade here (I think that I demonstrated this in\nmy previous response), so in this scenario the worst an operator would\nencounter is a performance regression.\n\n> 3. Should the chunk be made mandatory?\n>\n> Unfortunately, the chunk format did not follow the index format's\n> example of making lowercase chunk IDs required. Instead, the chunks\n> that are necessary for v1 are necessary forever and all other chunks\n> are deemed optional. Changing this would require something more\n> drastic like updating the version number or giving some grace period\n> where released versions start treating lowercase chunk IDs as required\n> before creating a new \"required\" chunk.\n>\n> This does mean that if there is a version incompatibility, the RIDX\n> chunk will just be ignored by the older version of Git.\n>\n> In terms of making this a safe format upgrade, I think Taylor has\n> achieved that.\n\nThanks. And yeah, the chunk should (and is) mandatory when writing a\nMIDX bitmap. But if we ran `git multi-pack-index write` without\n`--bitmap`, then we would be free to not write the RIDX chunk (and\nindeed that is what we do).\n\n> The only thing I can think is that server operators might want to\n> deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so\n> any need to downgrade would not suffer a performance penalty for a\n> missing .rev file. If that is a planned way to safely deploy this\n> change, then it might be worth adding a test that we safely delete\n> a .rev file after writing both a .rev file and a RIDX chunk. (The\n> RIDX chunk will be preferred, so maybe the previous .rev file hits\n> some logic that would skip its deletion?)\n\nThat logic (that we delete auxiliary files--including the .rev file--not\nmatching the checksum of the MIDX we just wrote) is unchanged. So I\nthink we should be good there since we have existing coverage.\n\nThanks,\nTaylor\n"},{"id":"444534","messageId":"c729fbc5-9f59-27d3-5f1b-0b339446174e@gmail.com","threadId":"57055","inReplyTo":"YcDe9wl4XwH7PjX1@nand.local","subject":"Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-20T20:09:25Z","receivedAt":"2021-12-20T20:09:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/20/2021 2:52 PM, Taylor Blau wrote:\n> On Mon, Dec 20, 2021 at 01:51:22PM -0500, Derrick Stolee wrote:\n\n>> 3. Should the chunk be made mandatory?\n>>\n>> Unfortunately, the chunk format did not follow the index format's\n>> example of making lowercase chunk IDs required. Instead, the chunks\n>> that are necessary for v1 are necessary forever and all other chunks\n>> are deemed optional. Changing this would require something more\n>> drastic like updating the version number or giving some grace period\n>> where released versions start treating lowercase chunk IDs as required\n>> before creating a new \"required\" chunk.\n>>\n>> This does mean that if there is a version incompatibility, the RIDX\n>> chunk will just be ignored by the older version of Git.\n>>\n>> In terms of making this a safe format upgrade, I think Taylor has\n>> achieved that.\n> \n> Thanks. And yeah, the chunk should (and is) mandatory when writing a\n> MIDX bitmap. But if we ran `git multi-pack-index write` without\n> `--bitmap`, then we would be free to not write the RIDX chunk (and\n> indeed that is what we do).\n\nI just want to be careful of the language here, with respect to how\nchunks are listed as required or optional in\nDocumentation/technical/pack-format.txt. \"Required\" means every MIDX\nneeds one or is invalid. \"Optional\" means that all versions of Git\nshould ignore the chunk if it does not recognize the ID.\n\nSo here, the new RIDX chunk will be ignored by older versions and\nwill be written only when we care about bitmaps.\n\n(Nothing I say here is in conflict with what you said, but I\nanticipate confusion with your use of the word \"mandatory\".)\n\n>> The only thing I can think is that server operators might want to\n>> deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so\n>> any need to downgrade would not suffer a performance penalty for a\n>> missing .rev file. If that is a planned way to safely deploy this\n>> change, then it might be worth adding a test that we safely delete\n>> a .rev file after writing both a .rev file and a RIDX chunk. (The\n>> RIDX chunk will be preferred, so maybe the previous .rev file hits\n>> some logic that would skip its deletion?)\n> \n> That logic (that we delete auxiliary files--including the .rev file--not\n> matching the checksum of the MIDX we just wrote) is unchanged. So I\n> think we should be good there since we have existing coverage.\n\nSounds good.\n\nThanks,\n-Stolee\n"},{"id":"445424","messageId":"YdRmFRHNeOGcSwJ0@nand.local","threadId":"57055","inReplyTo":"92d50859-ecfa-e7c5-d68b-d90a11579cd0@gmail.com","subject":"Re: [PATCH v2 8/8] midx: read `RIDX` chunk when present","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T15:21:57Z","receivedAt":"2022-01-04T15:22:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 20, 2021 at 01:42:24PM -0500, Derrick Stolee wrote:\n> > diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> > index 100ac90d15..8c92acb0ce 100755\n> > --- a/t/t5326-multi-pack-bitmaps.sh\n> > +++ b/t/t5326-multi-pack-bitmaps.sh\n> > @@ -9,6 +9,11 @@ test_description='exercise basic multi-pack bitmap functionality'\n> > GIT_TEST_MULTI_PACK_INDEX=0\n> > GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> >\n> > +GIT_TEST_MIDX_WRITE_REV=0\n> > +GIT_TEST_MIDX_READ_RIDX=1\n> > +export GIT_TEST_MIDX_WRITE_REV\n> > +export GIT_TEST_MIDX_READ_RIDX\n>\n> Technically, we could unset these variables, right? (\"...=\")\n\nWe absolutely could, and I can't think of any reason not to (other than\nthese tests would be fragile with respect to the default values of these\nspecial \"GIT_TEST_\" environment variables, but these are unlikely to\nchange ever).\n\n> > diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\n> ...\n> > +# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n> > +# automatic ones.\n> > +GIT_TEST_MULTI_PACK_INDEX=0\n> > +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> > +\n> > +# Unlike t5326, this test exercise multi-pack bitmap functionality where the\n> > +# object order is stored in a separate .rev file.\n> > +GIT_TEST_MIDX_WRITE_REV=1\n> > +GIT_TEST_MIDX_READ_RIDX=0\n> > +export GIT_TEST_MIDX_WRITE_REV\n> > +export GIT_TEST_MIDX_READ_RIDX\n> > +\n> > +midx_bitmap_core rev\n> > +midx_bitmap_partial_tests rev\n> > +\n> > +test_done\n>\n> Nice that now we get the payoff of the test refactor. This new script exists\n> only to verify that we can still read the old .rev files, which is important\n> for compatibility.\n\nExactly. Though I certainly wavered on testing the legacy .rev behavior\nas much as I did. But that had much more to do with how awkward the\ntests came out. That behavior is important enough IMHO that it's worth\nthe awkwardness.\n\nThanks,\nTaylor\n"},{"id":"445425","messageId":"YdRo1B0Y+ZBvjspa@nand.local","threadId":"57055","inReplyTo":"dbc75e29-aceb-051a-8235-e5c912172aed@gmail.com","subject":"Re: [PATCH v2 5/8] t5326: extract `test_rev_exists`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T15:33:40Z","receivedAt":"2022-01-04T15:33:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 20, 2021 at 01:33:32PM -0500, Derrick Stolee wrote:\n> On 12/13/2021 8:55 PM, Taylor Blau wrote:\n> > To determine which source of data is used for the MIDX's reverse index\n> > cache, introduce a helper which forces loading the reverse index, and\n> > then looks for the special trace2 event introduced in a previous commit.\n> >\n> > For now, this helper just looks for when the legacy MIDX .rev file was\n> > loaded, but in a subsequent commit will become parameterized over the\n> > the reverse index's source.\n> >\n> > This function replaces checking for the existence of the .rev file. We\n> > could write a similar helper to ensure that the .rev file is cleaned up\n> > after repacking, but it will make subsequent tests more difficult to\n> > write, and provides marginal value since we already check that the MIDX\n> > .bitmap file is removed.\n>\n> ...\n>\n> > +test_rev_exists () {\n> > +\tcommit=\"$1\"\n> > +\n> > +\ttest_expect_success 'reverse index exists' '\n> > +\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n>\n> Very recently, b8de3d6 (test-lib.sh: set GIT_TRACE2_EVENT_NESTING,\n> 2021-11-29) made it to master and sets this to 100 across the test\n> suite, so you don't need this line.\n\nThanks; I had written it with this in place in case this topic was going\nto reach master before the topic containing b8de3d6. But it didn't, and\nyou're right that setting GIT_TRACE2_EVENT_NESTING explicitly here is no\nlonger necessary.\n\n> > +\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n> > +\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n>\n> This use of $commit has me worried. Do the tests become too flaky\n> to changes in how we choose commits for the bitmaps? Does that\n> require callers to be too aware of the refstate when creating the\n> bitmaps?\n>\n> Perhaps just `git rev-list --use-bitmap-index [--objects] HEAD`\n> would suffice to generate the trace event?\n\nIt's necessary for the group of tests which exercise partial bitmap\ncoverage (see the \"setup partial bitmaps\" test and below for more). In\nthose tests, we don't have bitmap coverage of HEAD, only HEAD~, so we\nneed to be able to say things like:\n\n test_rev_exists HEAD~\n\nsince just asking for the tip isn't guaranteed to work always.\n\nWe could make the argument optional (i.e., `git rev-list --test-bitmap\n\"${1:-HEAD}\"`), but I generally find that it makes the callers more\ndifficult to read rather than easier.\n\nOn the \"--test-bitmap\" vs \"--objects\" thing, I don't think it matters\neither way. They are both doing basically the same traversal, but\n\"--test-bitmap\" does some additional integrity checks on top.\n\n> > test_expect_success 'create single-pack midx with bitmaps' '\n> > \tgit repack -ad &&\n> > \tgit multi-pack-index write --bitmap &&\n> > \ttest_path_is_file $midx &&\n> > -\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n> > -\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n> > +\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n> > '\n> >\n> > +test_rev_exists HEAD\n> > +\n>\n> Perhaps this helper would be more appropriate as a helper method\n> within a test, rather than creating a test of its own? I think\n> it looks better to include it next to the setup lines, something\n> like\n\nEh, sure. I won't plan on picking this up, but I'm happy to if you feel\nstrongly about it.\n\nThanks,\nTaylor\n"},{"id":"445444","messageId":"cover.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1638991570.git.me@ttaylorr.com","subject":"[PATCH v3 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:15:38Z","receivedAt":"2022-01-04T18:15:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a small reroll of my series which fixes a serious problem with MIDX\nbitmaps by which they can become corrupt when permuting their pack order.\n\nThis reroll cleans up some of the tests by removing a couple of places where we\nexplicitly set GIT_TRACE2_EVENT_NESTING in the environment, which is no longer\nneeded after 62329d336f (Merge branch 'ds/trace2-regions-in-tests', 2021-12-15).\n\nIt also adds one new patch on top, which ensure that we gracefully fall back\nwhen MIDX bitmaps cannot be read in certain cases.\n\nIt is prepared on the tip of master (which is dcc0cd074f at the time of\nwriting).\n\nTaylor Blau (9):\n t5326: demonstrate bitmap corruption after permutation\n midx.c: make changing the preferred pack safe\n pack-revindex.c: instrument loading on-disk reverse index\n t5326: drop unnecessary setup\n t5326: extract `test_rev_exists`\n t5326: move tests to t/lib-bitmap.sh\n t/lib-bitmap.sh: parameterize tests over reverse index source\n midx: read `RIDX` chunk when present\n pack-bitmap.c: gracefully fallback after opening pack/MIDX\n\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 13 +-\n midx.c | 31 +++-\n midx.h | 1 +\n pack-bitmap.c | 4 +\n pack-revindex.c | 20 ++\n t/lib-bitmap.sh | 185 +++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 28 +++\n t/t5326-multi-pack-bitmaps.sh | 164 +++-------------\n t/t5327-multi-pack-bitmaps-rev.sh | 23 +++\n t/t7700-repack.sh | 4 -\n 11 files changed, 322 insertions(+), 152 deletions(-)\n create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\nRange-diff against v2:\n 1: dfbac4bc60 = 1: babce7d29a t5326: demonstrate bitmap corruption after permutation\n 2: 4ea52e66dd ! 2: 7d20c13f8b midx.c: make changing the preferred pack safe\n @@ Commit message\n \n ## Documentation/technical/multi-pack-index.txt ##\n @@ Documentation/technical/multi-pack-index.txt: and their offsets into multiple packfiles. It contains:\n - - An offset within the jth packfile for the object.\n - - If large offsets are required, we use another list of large\n + ** An offset within the jth packfile for the object.\n + * If large offsets are required, we use another list of large\n offsets similar to version 2 pack-indexes.\n +- An optional list of objects in pseudo-pack order (used with MIDX bitmaps).\n \n 3: b630fea149 = 3: 3279e2eb9b pack-revindex.c: instrument loading on-disk reverse index\n 4: f430b6f2e9 = 4: 5818621ea8 t5326: drop unnecessary setup\n 5: 73faab9f42 ! 5: 33502d6a17 t5326: extract `test_rev_exists`\n @@ t/t5326-multi-pack-bitmaps.sh: midx_pack_source () {\n +\tcommit=\"$1\"\n +\n +\ttest_expect_success 'reverse index exists' '\n -+\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n +\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n +\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n +\n 6: bf42b116e1 ! 6: 76e23cae0f t5326: move tests to t/lib-bitmap.sh\n @@ t/lib-bitmap.sh: have_delta () {\n +\tcommit=\"$1\"\n +\n +\ttest_expect_success 'reverse index exists' '\n -+\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n +\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n +\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n +\n @@ t/t5326-multi-pack-bitmaps.sh: test_description='exercise basic multi-pack bitma\n -\tcommit=\"$1\"\n -\n -\ttest_expect_success 'reverse index exists' '\n --\t\tGIT_TRACE2_EVENT_NESTING=10 \\\n -\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n -\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n -\n 7: fa91631024 ! 7: 7ce3dc60f9 t/lib-bitmap.sh: parameterize tests over reverse index source\n @@ t/lib-bitmap.sh: midx_pack_source () {\n +\tkind=\"$2\"\n \n \ttest_expect_success 'reverse index exists' '\n - \t\tGIT_TRACE2_EVENT_NESTING=10 \\\n \t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n \t\t\tgit rev-list --test-bitmap \"$commit\" &&\n \n 8: 993bfa8dd8 ! 8: 55aa69de12 midx: read `RIDX` chunk when present\n @@ t/t5326-multi-pack-bitmaps.sh: test_description='exercise basic multi-pack bitma\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n -+GIT_TEST_MIDX_WRITE_REV=0\n -+GIT_TEST_MIDX_READ_RIDX=1\n -+export GIT_TEST_MIDX_WRITE_REV\n -+export GIT_TEST_MIDX_READ_RIDX\n ++# This test exercise multi-pack bitmap functionality where the object order is\n ++# stored and read from a special chunk within the MIDX, so use the default\n ++# behavior here.\n ++sane_unset GIT_TEST_MIDX_WRITE_REV\n ++sane_unset GIT_TEST_MIDX_READ_RIDX\n +\n midx_bitmap_core\n \n -: ---------- > 9: 9707d5ea44 pack-bitmap.c: gracefully fallback after opening pack/MIDX\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"445445","messageId":"babce7d29a85df0da54cb651433111bc33097a4e.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:15:41Z","receivedAt":"2022-01-04T18:15:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch demonstrates a cause of bitmap corruption that can occur when\nthe contents of the multi-pack index does not change, but the underlying\nobject order does.\n\nIn this example, we have a MIDX containing two packs, each with a\ndistinct set of objects (pack A corresponds to the tree, blob, and\ncommit from the first patch, and pack B corresponds to the second\npatch).\n\nFirst, a MIDX is written where the 'A' pack is preferred. As expected,\nthe bitmaps generated there are in-tact. But then, we generate an\nidentical MIDX with a different object order: this time preferring pack\n'B'.\n\nDue to a bug which will be explained and fixed in the following commit,\nthe MIDX is updated, but the .rev file is not, causing the .bitmap file\nto be read incorrectly. Specifically, the .bitmap file will contain\ncorrect data, but the auxiliary object order in the .rev file is stale,\ncausing readers to get confused by reading the new bitmaps using the old\nobject order.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex e187f90f29..0ca2868b0b 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,4 +395,35 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n+test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\tpack-$A.idx\n+\t\tpack-$B.idx\n+\t\tEOF\n+\n+\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t--preferred-pack=pack-$A.pack \n---\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 13 +++++-----\n midx.c | 25 ++++++++++++++++----\n t/t5326-multi-pack-bitmaps.sh | 2 +-\n 4 files changed, 30 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex b39c69da8c..f2221d2b44 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -24,6 +24,7 @@ and their offsets into multiple packfiles. It contains:\n ** An offset within the jth packfile for the object.\n * If large offsets are required, we use another list of large\n offsets similar to version 2 pack-indexes.\n+- An optional list of objects in pseudo-pack order (used with MIDX bitmaps).\n \n Thus, we can provide O(log N) lookup time for any number\n of packfiles.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 8d2f42f29e..6d3efb7d16 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -376,6 +376,11 @@ CHUNK DATA:\n \t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n \t 8-byte offsets into large packfiles.\n \n+\t[Optional] Bitmap pack order (ID: {'R', 'I', 'D', 'X'})\n+\t A list of MIDX positions (one per object in the MIDX, num_objects in\n+\t total, each a 4-byte unsigned integer in network byte order), sorted\n+\t according to their relative bitmap/pseudo-pack positions.\n+\n TRAILER:\n \n \tIndex checksum of the above contents.\n@@ -456,9 +461,5 @@ In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n objects in packs stored by the MIDX, laid out in pack order, and the\n packs arranged in MIDX order (with the preferred pack coming first).\n \n-Finally, note that the MIDX's reverse index is not stored as a chunk in\n-the multi-pack-index itself. This is done because the reverse index\n-includes the checksum of the pack or MIDX to which it belongs, which\n-makes it impossible to write in the MIDX. To avoid races when rewriting\n-the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n-filename (e.g., `multi-pack-index-xyz.rev`).\n+The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n+the MIDX itself.\ndiff --git a/midx.c b/midx.c\nindex 837b46b2af..d3179e9c02 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n+#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n@@ -833,6 +834,18 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_midx_revindex(struct hashfile *f,\n+\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\n+\treturn 0;\n+}\n+\n struct midx_pack_order_data {\n \tuint32_t nr;\n \tuint32_t pack;\n@@ -891,7 +904,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n \t\t\t\t\tmidx_hash, WRITE_REV);\n \n-\tif (finalize_object_file(tmp_file, buf.buf))\n+\tif (rename(tmp_file, buf.buf))\n \t\tdie(_(\"cannot store reverse index file\"));\n \n \tstrbuf_release(&buf);\n@@ -1403,15 +1416,19 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n \n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n+\t\t\t ctx.entries_nr * sizeof(uint32_t),\n+\t\t\t write_midx_revindex);\n+\t}\n+\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n-\t\tctx.pack_order = midx_pack_order(&ctx);\n-\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 0ca2868b0b..353282310d 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,7 +395,7 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"445447","messageId":"3279e2eb9b7bbfcc930e4ee146a3bd2476ef91eb.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 3/9] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:15:46Z","receivedAt":"2022-01-04T18:15:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\nsource for the reverse index's data. But it will be useful for tests to\nbe able to determine whether the reverse index was loaded from the\nseparate .rev file, or from a chunk within the MIDX.\n\nTo instrument this, add a trace2 event which the tests can look for in\norder to determine the reverse index's source.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 70d0fbafcb..bd15ebad03 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -301,6 +301,9 @@ int load_midx_revindex(struct multi_pack_index *m)\n \tif (m->revindex_data)\n \t\treturn 0;\n \n+\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n+\t\t\t \"source\", \"rev\");\n+\n \tget_midx_rev_filename(&revindex_name, m);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"445448","messageId":"5818621ea8ae6cb4bb1364621c33016edc208438.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 4/9] t5326: drop unnecessary setup","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:15:49Z","receivedAt":"2022-01-04T18:15:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The core.multiPackIndex config became true by default back in 18e449f86b\n(midx: enable core.multiPackIndex by default, 2020-09-25), so it is no\nlonger necessary to enable it explicitly.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 4 ----\n 1 file changed, 4 deletions(-)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 353282310d..1a9581af30 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -19,10 +19,6 @@ midx_pack_source () {\n \n setup_bitmap_history\n \n-test_expect_success 'enable core.multiPackIndex' '\n-\tgit config core.multiPackIndex true\n-'\n-\n test_expect_success 'create single-pack midx with bitmaps' '\n \tgit repack -ad &&\n \tgit multi-pack-index write --bitmap &&\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"445449","messageId":"33502d6a17b6b927d43a015859e40e6cdaab6667.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 5/9] t5326: extract `test_rev_exists`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:15:51Z","receivedAt":"2022-01-04T18:15:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To determine which source of data is used for the MIDX's reverse index\ncache, introduce a helper which forces loading the reverse index, and\nthen looks for the special trace2 event introduced in a previous commit.\n\nFor now, this helper just looks for when the legacy MIDX .rev file was\nloaded, but in a subsequent commit will become parameterized over the\nthe reverse index's source.\n\nThis function replaces checking for the existence of the .rev file. We\ncould write a similar helper to ensure that the .rev file is cleaned up\nafter repacking, but it will make subsequent tests more difficult to\nwrite, and provides marginal value since we already check that the MIDX\n.bitmap file is removed.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 37 +++++++++++++++++++++--------------\n 1 file changed, 22 insertions(+), 15 deletions(-)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 1a9581af30..999740f8c7 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -17,16 +17,29 @@ midx_pack_source () {\n \ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n }\n \n+test_rev_exists () {\n+\tcommit=\"$1\"\n+\n+\ttest_expect_success 'reverse index exists' '\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n+\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t'\n+}\n+\n setup_bitmap_history\n \n test_expect_success 'create single-pack midx with bitmaps' '\n \tgit repack -ad &&\n \tgit multi-pack-index write --bitmap &&\n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD\n+\n basic_bitmap_tests\n \n test_expect_success 'create new additional packs' '\n@@ -52,10 +65,11 @@ test_expect_success 'create multi-pack midx with bitmaps' '\n \ttest_line_count = 25 packs &&\n \n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD\n+\n basic_bitmap_tests\n \n test_expect_success '--no-bitmap is respected when bitmaps exist' '\n@@ -66,7 +80,6 @@ test_expect_success '--no-bitmap is respected when bitmaps exist' '\n \n \ttest_path_is_file $midx &&\n \ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \tgit multi-pack-index write --no-bitmap &&\n \n@@ -206,10 +219,11 @@ test_expect_success 'setup partial bitmaps' '\n \ttest_commit loose &&\n \tgit multi-pack-index write --bitmap 2>err &&\n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD~\n+\n basic_bitmap_tests HEAD~\n \n test_expect_success 'removing a MIDX clears stale bitmaps' '\n@@ -224,7 +238,6 @@ test_expect_success 'removing a MIDX clears stale bitmaps' '\n \n \t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n \t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\tstale_rev=$midx-$(midx_checksum $objdir).rev &&\n \t\trm $midx &&\n \n \t\t# Then write a new MIDX.\n@@ -234,9 +247,7 @@ test_expect_success 'removing a MIDX clears stale bitmaps' '\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\ttest_path_is_missing $stale_bitmap &&\n-\t\ttest_path_is_missing $stale_rev\n+\t\ttest_path_is_missing $stale_bitmap\n \t)\n '\n \n@@ -257,7 +268,6 @@ test_expect_success 'pack.preferBitmapTips' '\n \t\tgit multi-pack-index write --bitmap &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \t\ttest-tool bitmap list-commits | sort >bitmaps &&\n \t\tcomm -13 bitmaps commits >before &&\n@@ -267,7 +277,6 @@ test_expect_success 'pack.preferBitmapTips' '\n \t\t\tsnapshot &&\n \n \t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n \t\trm -fr $midx &&\n \n \t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"445450","messageId":"76e23cae0f42afef982a4e94ec551335774be6f2.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 6/9] t5326: move tests to t/lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:15:53Z","receivedAt":"2022-01-04T18:15:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In t5326, we have a handful of tests that we would like to run twice:\nonce using the MIDX's new `RIDX` chunk as the source of the\nreverse-index cache, and once using the separate `.rev` file.\n\nBut because these tests mutate the state of the underlying repository,\nand then make assumptions about those mutations occurring in a certain\nsequence, simply running the tests twice in the same repository is\nawkward.\n\nInstead, extract the core of interesting tests into t/lib-bitmap.sh to\nprepare for them to be run twice, each in a separate test script. This\nmeans that they can each operate on a separate repository, removing any\nconcerns about mutating state.\n\nFor now, this patch is a strict cut-and-paste of some tests from t5326.\nThe tests which did not move are not interesting with respect to the\nsource of their reverse index data.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 177 ++++++++++++++++++++++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 173 +--------------------------------\n 2 files changed, 179 insertions(+), 171 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 21d0392dda..48a8730a13 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,6 +1,9 @@\n # Helpers for scripts testing bitmap functionality; see t5310 for\n # example usage.\n \n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -264,3 +267,177 @@ have_delta () {\n midx_checksum () {\n \ttest-tool read-midx --checksum \"$1\"\n }\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+test_rev_exists () {\n+\tcommit=\"$1\"\n+\n+\ttest_expect_success 'reverse index exists' '\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n+\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t'\n+}\n+\n+midx_bitmap_core () {\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'create single-pack midx with bitmaps' '\n+\t\tgit repack -ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'create new additional packs' '\n+\t\tfor i in $(test_seq 1 16)\n+\t\tdo\n+\t\t\ttest_commit \"$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\n+\t\tgit checkout -b other2 HEAD~8 &&\n+\t\tfor i in $(test_seq 1 8)\n+\t\tdo\n+\t\t\ttest_commit \"side-$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\t\tgit checkout second\n+\t'\n+\n+\ttest_expect_success 'create multi-pack midx with bitmaps' '\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\tls $objdir/pack/pack-*.pack >packs &&\n+\t\ttest_line_count = 25 packs &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_commit respect--no-bitmap &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\tgit multi-pack-index write --no-bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n+\t'\n+\n+\ttest_expect_success 'setup midx with base from later pack' '\n+\t\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t\t# prefers to delete contents out of a base rather than add to a shorter\n+\t\t# object.\n+\t\ttest_seq 1 128 >a &&\n+\t\ttest_seq 1 130 >b &&\n+\n+\t\tgit add a b &&\n+\t\tgit commit -m \"initial commit\" &&\n+\n+\t\ta=$(git rev-parse HEAD:a) &&\n+\t\tb=$(git rev-parse HEAD:b) &&\n+\n+\t\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\t\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\t) &&\n+\n+\t\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t\t# any other object.\n+\t\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t\t$b\n+\t\t$(git rev-parse HEAD)\n+\t\t$(git rev-parse HEAD^{tree})\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit prune-packed &&\n+\t\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t\t# reuse.\n+\t\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\t\thave_delta $a $b &&\n+\t\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+\t'\n+\n+\trev_list_tests 'full bitmap with backwards delta'\n+\n+\ttest_expect_success 'clone with bitmaps enabled' '\n+\t\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\t\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\n+\t\t\ttest_commit A &&\n+\t\t\ttest_commit B &&\n+\n+\t\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\t\tpack-$A.idx\n+\t\t\tpack-$B.idx\n+\t\t\tEOF\n+\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\t--preferred-pack=pack-$A.pack err &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD~\n+\n+\tbasic_bitmap_tests HEAD~\n+}\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 999740f8c7..100ac90d15 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -9,134 +9,7 @@ test_description='exercise basic multi-pack bitmap functionality'\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n-objdir=.git/objects\n-midx=$objdir/pack/multi-pack-index\n-\n-# midx_pack_source \n-midx_pack_source () {\n-\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n-}\n-\n-test_rev_exists () {\n-\tcommit=\"$1\"\n-\n-\ttest_expect_success 'reverse index exists' '\n-\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n-\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n-\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n-\t'\n-}\n-\n-setup_bitmap_history\n-\n-test_expect_success 'create single-pack midx with bitmaps' '\n-\tgit repack -ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD\n-\n-basic_bitmap_tests\n-\n-test_expect_success 'create new additional packs' '\n-\tfor i in $(test_seq 1 16)\n-\tdo\n-\t\ttest_commit \"$i\" &&\n-\t\tgit repack -d || return 1\n-\tdone &&\n-\n-\tgit checkout -b other2 HEAD~8 &&\n-\tfor i in $(test_seq 1 8)\n-\tdo\n-\t\ttest_commit \"side-$i\" &&\n-\t\tgit repack -d || return 1\n-\tdone &&\n-\tgit checkout second\n-'\n-\n-test_expect_success 'create multi-pack midx with bitmaps' '\n-\tgit multi-pack-index write --bitmap &&\n-\n-\tls $objdir/pack/pack-*.pack >packs &&\n-\ttest_line_count = 25 packs &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD\n-\n-basic_bitmap_tests\n-\n-test_expect_success '--no-bitmap is respected when bitmaps exist' '\n-\tgit multi-pack-index write --bitmap &&\n-\n-\ttest_commit respect--no-bitmap &&\n-\tgit repack -d &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\n-\tgit multi-pack-index write --no-bitmap &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n-'\n-\n-test_expect_success 'setup midx with base from later pack' '\n-\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n-\t# prefers to delete contents out of a base rather than add to a shorter\n-\t# object.\n-\ttest_seq 1 128 >a &&\n-\ttest_seq 1 130 >b &&\n-\n-\tgit add a b &&\n-\tgit commit -m \"initial commit\" &&\n-\n-\ta=$(git rev-parse HEAD:a) &&\n-\tb=$(git rev-parse HEAD:b) &&\n-\n-\t# In the first pack, \"a\" is stored as a delta to \"b\".\n-\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n-\t$a\n-\t$b\n-\tEOF\n-\t) &&\n-\n-\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n-\t# any other object.\n-\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n-\t$b\n-\t$(git rev-parse HEAD)\n-\t$(git rev-parse HEAD^{tree})\n-\tEOF\n-\t) &&\n-\n-\tgit prune-packed &&\n-\t# Use the second pack as the preferred source, so that \"b\" occurs\n-\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n-\t# reuse.\n-\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n-\n-\thave_delta $a $b &&\n-\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n-'\n-\n-rev_list_tests 'full bitmap with backwards delta'\n-\n-test_expect_success 'clone with bitmaps enabled' '\n-\tgit clone --no-local --bare . clone-reverse-delta.git &&\n-\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n-\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+midx_bitmap_core\n \n bitmap_reuse_tests() {\n \tfrom=$1\n@@ -213,18 +86,7 @@ test_expect_success 'missing object closure fails gracefully' '\n \t)\n '\n \n-test_expect_success 'setup partial bitmaps' '\n-\ttest_commit packed &&\n-\tgit repack &&\n-\ttest_commit loose &&\n-\tgit multi-pack-index write --bitmap 2>err &&\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD~\n-\n-basic_bitmap_tests HEAD~\n+midx_bitmap_partial_tests\n \n test_expect_success 'removing a MIDX clears stale bitmaps' '\n \trm -fr repo &&\n@@ -398,35 +260,4 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit A &&\n-\t\ttest_commit B &&\n-\n-\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n-\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n-\n-\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n-\t\tpack-$A.idx\n-\t\tpack-$B.idx\n-\t\tEOF\n-\n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\t--preferred-pack=pack-$A.pack \n---\n t/lib-bitmap.sh | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 48a8730a13..77b5f46a03 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -275,17 +275,23 @@ midx_pack_source () {\n \n test_rev_exists () {\n \tcommit=\"$1\"\n+\tkind=\"$2\"\n \n \ttest_expect_success 'reverse index exists' '\n \t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n \t\t\tgit rev-list --test-bitmap \"$commit\" &&\n \n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t\tif test \"rev\" = \"$kind\"\n+\t\tthen\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\t\tfi &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"$kind\\\"\" event.trace\n \t'\n }\n \n midx_bitmap_core () {\n+\trev_kind=\"${1:-rev}\"\n+\n \tsetup_bitmap_history\n \n \ttest_expect_success 'create single-pack midx with bitmaps' '\n@@ -295,7 +301,7 @@ midx_bitmap_core () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD\n+\ttest_rev_exists HEAD \"$rev_kind\"\n \n \tbasic_bitmap_tests\n \n@@ -325,7 +331,7 @@ midx_bitmap_core () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD\n+\ttest_rev_exists HEAD \"$rev_kind\"\n \n \tbasic_bitmap_tests\n \n@@ -428,6 +434,8 @@ midx_bitmap_core () {\n }\n \n midx_bitmap_partial_tests () {\n+\trev_kind=\"${1:-rev}\"\n+\n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\n \t\tgit repack &&\n@@ -437,7 +445,7 @@ midx_bitmap_partial_tests () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD~\n+\ttest_rev_exists HEAD~ \"$rev_kind\"\n \n \tbasic_bitmap_tests HEAD~\n }\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"445452","messageId":"55aa69de12c5f82a66836e829f915363cc73b421.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 8/9] midx: read `RIDX` chunk when present","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:16:00Z","receivedAt":"2022-01-04T18:16:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a MIDX contains the new `RIDX` chunk, ensure that the reverse index\nis read from it instead of the on-disk .rev file. Since we need to\nencode the object order in the MIDX itself for correctness reasons,\nthere is no point in storing the same data again outside of the MIDX.\n\nSo, this patch stops writing separate .rev files, and reads it out of\nthe MIDX itself. This is possible to do with relatively little new code,\nsince the format of the RIDX chunk is identical to the data in the .rev\nfile. In other words, we can implement this by pointing the\n`revindex_data` field at the reverse index chunk of the MIDX instead of\nthe .rev file without any other changes.\n\nNote that we have two knobs that are adjusted for the new tests:\nGIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\nwhether the MIDX .rev is written at all, and the latter controls whether\nwe read the MIDX's RIDX chunk.\n\nBoth are necessary to ensure that the test added at the beginning of\nthis series continues to work. This is because we always need to write\nthe RIDX chunk in the MIDX in order to change its checksum, but we want\nto make sure reading the existing .rev file still works (since the RIDX\nchunk takes precedence by default).\n\nArguably this isn't a very interesting mode to test, because the\nprecedence rules mean that we'll always read the RIDX chunk over the\n.rev file. But it makes it impossible for a user to induce corruption in\ntheir repository by adjusting the test knobs (since if we had an\neither/or knob they could stop writing the RIDX chunk, allowing them to\ntweak the MIDX's object order without changing its checksum).\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++++-\n midx.h | 1 +\n pack-revindex.c | 17 +++++++++++++++++\n t/lib-bitmap.sh | 4 ++--\n t/t5326-multi-pack-bitmaps.sh | 6 ++++++\n t/t5327-multi-pack-bitmaps-rev.sh | 23 +++++++++++++++++++++++\n t/t7700-repack.sh | 4 ----\n 7 files changed, 54 insertions(+), 7 deletions(-)\n create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\ndiff --git a/midx.c b/midx.c\nindex d3179e9c02..9aba13b5b1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -162,6 +162,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n \n+\tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n+\t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tCALLOC_ARRAY(m->pack_names, m->num_packs);\n@@ -1429,7 +1432,8 @@ static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\n \t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &ctx,\ndiff --git a/midx.h b/midx.h\nindex b7d79a515c..22e8e53288 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -36,6 +36,7 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_object_offsets;\n \tconst unsigned char *chunk_large_offsets;\n+\tconst unsigned char *chunk_revindex;\n \n \tconst char **pack_names;\n \tstruct packed_git **packs;\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex bd15ebad03..08dc160167 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -298,9 +298,26 @@ int load_midx_revindex(struct multi_pack_index *m)\n {\n \tstruct strbuf revindex_name = STRBUF_INIT;\n \tint ret;\n+\n \tif (m->revindex_data)\n \t\treturn 0;\n \n+\tif (m->chunk_revindex) {\n+\t\t/*\n+\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n+\t\t * the reverse index instead of trying to load a separate `.rev`\n+\t\t * file.\n+\t\t *\n+\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n+\t\t * not want to accidentally call munmap() in the middle of the\n+\t\t * MIDX.\n+\t\t */\n+\t\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n+\t\t\t\t \"source\", \"midx\");\n+\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n+\t\treturn 0;\n+\t}\n+\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t \"source\", \"rev\");\n \ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 77b5f46a03..365d990ce3 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -290,7 +290,7 @@ test_rev_exists () {\n }\n \n midx_bitmap_core () {\n-\trev_kind=\"${1:-rev}\"\n+\trev_kind=\"${1:-midx}\"\n \n \tsetup_bitmap_history\n \n@@ -434,7 +434,7 @@ midx_bitmap_core () {\n }\n \n midx_bitmap_partial_tests () {\n-\trev_kind=\"${1:-rev}\"\n+\trev_kind=\"${1:-midx}\"\n \n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 100ac90d15..c0924074c4 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -9,6 +9,12 @@ test_description='exercise basic multi-pack bitmap functionality'\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n+# This test exercise multi-pack bitmap functionality where the object order is\n+# stored and read from a special chunk within the MIDX, so use the default\n+# behavior here.\n+sane_unset GIT_TEST_MIDX_WRITE_REV\n+sane_unset GIT_TEST_MIDX_READ_RIDX\n+\n midx_bitmap_core\n \n bitmap_reuse_tests() {\ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nnew file mode 100755\nindex 0000000000..d30ba632c8\n--- /dev/null\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality (.rev files)'\n+\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+# Unlike t5326, this test exercise multi-pack bitmap functionality where the\n+# object order is stored in a separate .rev file.\n+GIT_TEST_MIDX_WRITE_REV=1\n+GIT_TEST_MIDX_READ_RIDX=0\n+export GIT_TEST_MIDX_WRITE_REV\n+export GIT_TEST_MIDX_READ_RIDX\n+\n+midx_bitmap_core rev\n+midx_bitmap_partial_tests rev\n+\n+test_done\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 4693f8dc2b..02a6633a16 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -311,16 +311,13 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t\tchecksum=$(midx_checksum $objdir) &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$checksum.bitmap &&\n-\t\ttest_path_is_file $midx-$checksum.rev &&\n \n \t\ttest_commit repack-3 &&\n \t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-$checksum.bitmap &&\n-\t\ttest_path_is_missing $midx-$checksum.rev &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \t\ttest_commit repack-4 &&\n \t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n@@ -353,7 +350,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '\n \t\ttest_line_count = 1 before &&\n \n \t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n \t\trm -fr $midx &&\n \n \t\t# instead of constructing the snapshot ourselves (c.f., the test\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"445453","messageId":"9707d5ea4433d9a5c7f8581dbb2d8a05f410efd3.1641320129.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1641320129.git.me@ttaylorr.com","subject":"[PATCH v3 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-04T18:16:02Z","receivedAt":"2022-01-04T18:16:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When opening a MIDX/pack-bitmap, we call open_midx_bitmap_1() or\nopen_pack_bitmap_1() respectively in a loop over the set of MIDXs/packs.\nBy design, these functions are supposed to be called over every pack and\nMIDX, since only one of them should have a valid bitmap.\n\nOrdinarily we return '0' from these two functions in order to indicate\nthat we successfully loaded a bitmap To signal that we couldn't load a\nbitmap corresponding to the MIDX/pack (either because one doesn't exist,\nor because there was an error with loading it), we can return '-1'. In\neither case, the callers each enumerate all MIDXs/packs to ensure that\nat most one bitmap per-kind is present.\n\nBut when we fail to load a bitmap that does exist (for example, loading\na MIDX bitmap without finding a corresponding reverse index), we'll\nreturn -1 but leave the 'midx' field non-NULL. So when we fallback to\nloading a pack bitmap, we'll complain that the bitmap we're trying to\npopulate already is \"opened\", even though it isn't.\n\nRectify this by setting the '->pack' and '->midx' field back to NULL as\nappropriate. Two tests are added: one to ensure that the MIDX-to-pack\nbitmap fallback works, and another to ensure we still complain when\nthere are multiple pack bitmaps in a repository.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 4 ++++\n t/t5310-pack-bitmaps.sh | 28 ++++++++++++++++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 19 +++++++++++++++++++\n 3 files changed, 51 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f772d3cb7f..9c666cdb8b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -358,7 +358,9 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n cleanup:\n \tmunmap(bitmap_git->map, bitmap_git->map_size);\n \tbitmap_git->map_size = 0;\n+\tbitmap_git->map_pos = 0;\n \tbitmap_git->map = NULL;\n+\tbitmap_git->midx = NULL;\n \treturn -1;\n }\n \n@@ -405,6 +407,8 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\tmunmap(bitmap_git->map, bitmap_git->map_size);\n \t\tbitmap_git->map = NULL;\n \t\tbitmap_git->map_size = 0;\n+\t\tbitmap_git->map_pos = 0;\n+\t\tbitmap_git->pack = NULL;\n \t\treturn -1;\n \t}\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex d05ab716f6..f775fc1ce6 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -397,4 +397,32 @@ test_expect_success 'pack.preferBitmapTips' '\n \t)\n '\n \n+test_expect_success 'complains about multiple pack bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\n+\t\tgit repack -adb &&\n+\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n+\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n+\n+\t\ttest_commit other &&\n+\t\tgit repack -ab &&\n+\n+\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n+\n+\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n+\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n+\t\ttest_line_count = 2 packs &&\n+\t\ttest_line_count = 2 bitmaps &&\n+\n+\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\tgrep \"ignoring extra bitmap file\" err\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex c0924074c4..3c1ecc7e25 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -266,4 +266,23 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n+test_expect_success 'graceful fallback when missing reverse index' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\n+\t\t# write a pack and MIDX bitmap containing base\n+\t\tgit repack -adb &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n+\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t! grep \"ignoring extra bitmap file\" err\n+\t)\n+'\n+\n test_done\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"446253","messageId":"xmqqwnj29f1v.fsf@gitster.g","threadId":"57055","inReplyTo":"7d20c13f8b48d2aef45c2c8c40efb6ecdb865aa8.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-14T21:35:08Z","receivedAt":"2022-01-14T21:35:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> ... It's likely we were using\n> finalize_object_file() instead of a pure rename() because the former\n> also adjusts shared permissions.\n\nI thought the primary reason why we use finalize was because we\nignore EEXIST (and the assumption is that the files with the same\ncontents get the same name computed from their contents).\n\n> \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n> \t\t\t\t\tmidx_hash, WRITE_REV);\n> \n> -\tif (finalize_object_file(tmp_file, buf.buf))\n> +\tif (rename(tmp_file, buf.buf))\n> \t\tdie(_(\"cannot store reverse index file\"));\n\nDoesn't your new code die with it if buf.buf names an existing file?\n"},{"id":"446254","messageId":"xmqqsftq9en8.fsf@gitster.g","threadId":"57055","inReplyTo":"xmqqwnj29f1v.fsf@gitster.g","subject":"Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-14T21:43:55Z","receivedAt":"2022-01-14T21:44:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Taylor Blau writes:\n>\n>> ... It's likely we were using\n>> finalize_object_file() instead of a pure rename() because the former\n>> also adjusts shared permissions.\n>\n> I thought the primary reason why we use finalize was because we\n> ignore EEXIST (and the assumption is that the files with the same\n> contents get the same name computed from their contents).\n>\n>> \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n>> \t\t\t\t\tmidx_hash, WRITE_REV);\n>> \n>> -\tif (finalize_object_file(tmp_file, buf.buf))\n>> +\tif (rename(tmp_file, buf.buf))\n>> \t\tdie(_(\"cannot store reverse index file\"));\n>\n> Doesn't your new code die with it if buf.buf names an existing file?\n\nAh, scratch that. rename() discards the old one atomically, so as\nlong as tmp_file and buf.buf are in the same directory (which I\nthink it is in this case), we wouldn't be affected by the bug that\nis worked around with \"Coda hack\" in finalize_object_file(), either.\n\n"},{"id":"446277","messageId":"YeIcem4bDgmQhs7+@nand.local","threadId":"57055","inReplyTo":"xmqqsftq9en8.fsf@gitster.g","subject":"Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-15T00:59:38Z","receivedAt":"2022-01-15T00:59:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 14, 2022 at 01:43:55PM -0800, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n> > Taylor Blau writes:\n> >\n> >> ... It's likely we were using\n> >> finalize_object_file() instead of a pure rename() because the former\n> >> also adjusts shared permissions.\n> >\n> > I thought the primary reason why we use finalize was because we\n> > ignore EEXIST (and the assumption is that the files with the same\n> > contents get the same name computed from their contents).\n> >\n> >> \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n> >> \t\t\t\t\tmidx_hash, WRITE_REV);\n> >>\n> >> -\tif (finalize_object_file(tmp_file, buf.buf))\n> >> +\tif (rename(tmp_file, buf.buf))\n> >> \t\tdie(_(\"cannot store reverse index file\"));\n> >\n> > Doesn't your new code die with it if buf.buf names an existing file?\n>\n> Ah, scratch that. rename() discards the old one atomically, so as\n> long as tmp_file and buf.buf are in the same directory (which I\n> think it is in this case), we wouldn't be affected by the bug that\n> is worked around with \"Coda hack\" in finalize_object_file(), either.\n\nExactly. In this case, we really did want to overwrite an existing .rev\nfile with the same name. That's because prior to this patch, we didn't\nstore the object order in the MIDX itself. That made it possible for us\nto change the object order, but leave the MIDX checksum alone.\n\nThen we'd keep the old .rev (with the old order, but the same name) in\nplace, if we used link(2) in order to try (and fail) to overwrite the\nexisting .rev with the new one.\n\nIndeed, the temporary file created by write_rev_file_order() is written\nin the pack directory, which is where the .rev file and MIDX will\nultimately live.\n\nThis code is going to be hidden behind a test knob in a few patches\nanyway, but verifying all of this is good nonetheless (especially if you\njust want to apply these first two patches into the v2.35 tree).\n\nThanks,\nTaylor\n"},{"id":"446288","messageId":"xmqqtue54iop.fsf@gitster.g","threadId":"57055","inReplyTo":"YeIcem4bDgmQhs7+@nand.local","subject":"Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-15T06:27:50Z","receivedAt":"2022-01-15T06:27:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Fri, Jan 14, 2022 at 01:43:55PM -0800, Junio C Hamano wrote:\n>> Junio C Hamano writes:\n>>\n>> > Taylor Blau writes:\n>> >\n>> >> ... It's likely we were using\n>> >> finalize_object_file() instead of a pure rename() because the former\n>> >> also adjusts shared permissions.\n>> >\n>> > I thought the primary reason why we use finalize was because we\n>> > ignore EEXIST (and the assumption is that the files with the same\n>> > contents get the same name computed from their contents).\n>> >\n>> >> \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n>> >> \t\t\t\t\tmidx_hash, WRITE_REV);\n>> >>\n>> >> -\tif (finalize_object_file(tmp_file, buf.buf))\n>> >> +\tif (rename(tmp_file, buf.buf))\n>> >> \t\tdie(_(\"cannot store reverse index file\"));\n>> >\n>> > Doesn't your new code die with it if buf.buf names an existing file?\n>>\n>> Ah, scratch that. rename() discards the old one atomically, so as\n>> long as tmp_file and buf.buf are in the same directory (which I\n>> think it is in this case), we wouldn't be affected by the bug that\n>> is worked around with \"Coda hack\" in finalize_object_file(), either.\n>\n> Exactly. In this case, we really did want to overwrite an existing .rev\n> file with the same name. That's because prior to this patch, we didn't\n> store the object order in the MIDX itself. That made it possible for us\n> to change the object order, but leave the MIDX checksum alone.\n\nThe other change in this step is addition of a new chunk type that\nrecords the revindex data. With that, you are guaranteeing that a\nnew file with the same checksum as an old one must have the\nbyte-for-byte identical contents, so at that point, it is OK to use\nfinalize_object_file() and not use rename() to keep the old file, no?\n\nIf that is the case, we may rather want to use f-o-f consistently\nfor stuff inside .git/ directory whose filename is tied with its\ncontents. I do not think we want to chase a bug that comes from\ndifference between link-then-unlink vs rename, which affects loose\nobject files in one way and midx file in another way.\n"},{"id":"446586","messageId":"20220120175541.3099054-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"babce7d29a85df0da54cb651433111bc33097a4e.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-20T17:55:41Z","receivedAt":"2022-01-20T17:55:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> This patch demonstrates a cause of bitmap corruption that can occur when\n> the contents of the multi-pack index does not change, but the underlying\n> object order does.\n> \n> In this example, we have a MIDX containing two packs, each with a\n> distinct set of objects (pack A corresponds to the tree, blob, and\n> commit from the first patch, and pack B corresponds to the second\n> patch).\n> \n> First, a MIDX is written where the 'A' pack is preferred. As expected,\n> the bitmaps generated there are in-tact. But then, we generate an\n> identical MIDX with a different object order: this time preferring pack\n> 'B'.\n> \n> Due to a bug which will be explained and fixed in the following commit,\n> the MIDX is updated, but the .rev file is not, causing the .bitmap file\n> to be read incorrectly. Specifically, the .bitmap file will contain\n> correct data, but the auxiliary object order in the .rev file is stale,\n> causing readers to get confused by reading the new bitmaps using the old\n> object order.\n\nThanks - overall, this looks like a bug that needs to be fixed.\n\nFor the benefit of other reviewers, here's my summary of the problem:\nthe .midx, .rev, and .bitmap files are almost always generated together,\nand it is possible for two different invocations of Git to generate the\nsame .midx but a different .rev and .bitmap. For example, when\ngenerating a .midx+.rev+.bitmap for 2 disjoint packfiles, the 1st time\nwith one packfile as preferred and the 2nd time with the other packfile\nas preferred. In .midx, packfiles are always ordered by lexicographical\norder of their names, and the preferred status only matters when an\nobject is in multiple packfiles (which never happens in this case, since\nthe packfiles are disjoint). But the preferred status affects .rev and\n.bitmap, because they use a concept called \"pseudo-pack order\" (see\npack-format.txt for more details) in which the preferred pack comes\nfirst.\n\nAs an effort to ensure that Git reads coherent .midx, .rev, and .bitmap\nfiles, both the .rev and .bitmap files are keyed on the checksum of the\n.midx file. But the issue here is that a .rev and a .bitmap could both\nrefer to the same .midx checksum when the .rev and .bitmap files are not\ncoherent with respect to each other (e.g. when a Git process has written\nthe .rev, but not the .bitmap yet - but this would appear perfectly\nordinary to another concurrently running Git process, since the .midx\nchecksum in the .rev and .bitmap files match).\n\nThis problem is exacerbated by the fact that the .rev has its .midx\nchecksum in its filename, whereas the .bitmap has its .midx checksum in\nits file contents. When generating .midx+.rev+.bitmap, it would write\nthe .bitmap but not the .rev, since a .rev of the same filename already\nexists.\n\nThe solution is to embed the .rev in the .midx. This means that the\nchecksum stored in .bitmap takes into account the contents of what would\nhave been in .rev, solving the coherency issue. (There are other\nsolutions like storing the name of the preferred pack in .midx, but I\nthink that putting the contents of .rev in the .midx is best.)\n"},{"id":"446587","messageId":"20220120180843.3103044-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"7d20c13f8b48d2aef45c2c8c40efb6ecdb865aa8.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-20T18:08:43Z","receivedAt":"2022-01-20T18:08:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> The previous patch demonstrates a bug where a MIDX's auxiliary object\n> order can become out of sync with a MIDX bitmap.\n> \n> This is because of two confounding factors:\n> \n> - First, the object order is stored in a file which is named according\n> to the multi-pack index's checksum, and the MIDX does not store the\n> object order. This means that the object order can change without\n> altering the checksum.\n> \n> - But the .rev file is moved into place with finalize_object_file(),\n> which link(2)'s the file into place instead of renaming it. For us,\n> that means that a modified .rev file will not be moved into place if\n> MIDX's checksum was unchanged.\n> \n> The fix here is two-fold. First, we need to stop linking the file into\n> place and instead rename it. It's likely we were using\n> finalize_object_file() instead of a pure rename() because the former\n> also adjusts shared permissions. But that is unnecessary, because we\n> already do so in write_rev_file_order(), so rename alone is safe.\n> \n> But we also need to make the MIDX's checksum change in some way when the\n> preferred pack changes without altering the set of packs stored in a\n> MIDX to prevent a race where the new .rev file is moved into place\n> before the MIDX is updated. Here, you'd get the opposite effect: reading\n> old bitmaps with the new object order.\n\nI think the main issue is the first confounding factor you listed above:\neven if we didn't have the other confounding factor, that issue alone is\nenough to motivate the entire patch set. Likewise, as Junio said [1], I\ndon't think we need to switch to rename() if we make the checksum\ndifferent, so the fix is one-fold, not two-fold. For what it's worth, I\nswitched back to finalize_object_file() and ran the tests, and they all\npass.\n\nSo I would simplify the commit message to just talk about the checksum\nissue. (This is definitely not a blocker for merging, though - others\nmight find the additional context helpful.)\n\nThe code up to this patch (apart from the rename()) looks good.\n\n[1] https://lore.kernel.org/git/xmqqtue54iop.fsf@gitster.g/\n"},{"id":"446588","messageId":"20220120181545.3104404-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"3279e2eb9b7bbfcc930e4ee146a3bd2476ef91eb.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 3/9] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-20T18:15:45Z","receivedAt":"2022-01-20T18:15:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\n> source for the reverse index's data. But it will be useful for tests to\n> be able to determine whether the reverse index was loaded from the\n> separate .rev file, or from a chunk within the MIDX.\n\nAs for this patch and all subsequent patches, we (at $DAYJOB) discussed\nduring our Review Club the idea of not supporting .rev files altogether.\nFirstly, because of this bug, we cannot fully trust .rev files anymore.\nAnd even if we decided to trust it (or to trust it after some\nverification step that I can't think of), that file would only be useful\nfor a short time until a regularly scheduled maintenance step\nregenerates the bitmaps anyway.\n\nThat would also simplify the code slightly and eliminate the need to test a\nvariety of cases - so we probably wouldn't need the t/lib-bitmap.sh file\nintroduced in patch 6. I'll hold off on reviewing the rest of the\npatches for now, since they may substantially change if we decide to\nstop supporting .rev files.\n"},{"id":"446601","messageId":"YeneJrzjp160edGG@nand.local","threadId":"57055","inReplyTo":"20220120175541.3099054-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-20T22:11:50Z","receivedAt":"2022-01-20T22:11:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 20, 2022 at 09:55:41AM -0800, Jonathan Tan wrote:\n> As an effort to ensure that Git reads coherent .midx, .rev, and .bitmap\n> files, both the .rev and .bitmap files are keyed on the checksum of the\n> .midx file. But the issue here is that a .rev and a .bitmap could both\n> refer to the same .midx checksum when the .rev and .bitmap files are not\n> coherent with respect to each other (e.g. when a Git process has written\n> the .rev, but not the .bitmap yet - but this would appear perfectly\n> ordinary to another concurrently running Git process, since the .midx\n> checksum in the .rev and .bitmap files match).\n\nKind of, and it's possible that we're saying the same thing here using\ndifferent words.\n\nBut seeing an out-of-sync .bitmap and .rev is really a symptom of the\nunderlying problem, which is that the MIDX checksum could (prior to\nthese patches) be unchanged even when the object order it represents\n_does_ change (e.g., in the case of swapping the preferred pack around\nlike the test here demonstrates).\n\n> This problem is exacerbated by the fact that the .rev has its .midx\n> checksum in its filename, whereas the .bitmap has its .midx checksum in\n> its file contents. When generating .midx+.rev+.bitmap, it would write\n> the .bitmap but not the .rev, since a .rev of the same filename already\n> exists.\n\nThis isn't quite right: both have the MIDX's checksum in their filename.\nFor example after running `git repack --write-midx --write-bitmap-index`\non a random repository, I get these MIDX-related files:\n\n $ find .git/objects/pack -type f -name 'multi-pack-index*'\n .git/objects/pack/multi-pack-index-fd71600b4ceb4caf23a538c7829b9284f2b66d73.rev\n .git/objects/pack/multi-pack-index-fd71600b4ceb4caf23a538c7829b9284f2b66d73.bitmap\n .git/objects/pack/multi-pack-index\n\nBefore these patches, it was possible for the MIDX's object order to\nchange but for its checksum to remain the same. The problem here is that\nwe rename(2)'d the .bitmap into place, but only link(2)'d the .rev into\nplace.\n\nSo one of the two was updated, and the other was left behind. That does\nmake them incoherent with respect to each other; but I find it more\nuseful to think of it as the .rev being out-of-sync with the MIDX.\n\n> The solution is to embed the .rev in the .midx. This means that the\n> checksum stored in .bitmap takes into account the contents of what would\n> have been in .rev, solving the coherency issue. (There are other\n> solutions like storing the name of the preferred pack in .midx, but I\n> think that putting the contents of .rev in the .midx is best.)\n\nRight. The problem being that it's possible to change the MIDX's\nobject order without changing its checksum. Since the .rev file's\ncontents encodes the pseudo-pack order, embedding it into the MIDX is\nsufficient to guarantee that the MIDX's checksum changes whenever the\nobject order does.\n\nApologies if that all exactly matched up with your understanding, and I\nwas just telling you stuff that you already knew. But I figure that this\nbug is subtle enough that a little bit of hair-splitting doesn't hurt\n;).\n\nThanks,\nTaylor\n"},{"id":"446602","messageId":"Yenee7bmEEiR2gkj@nand.local","threadId":"57055","inReplyTo":"20220120180843.3103044-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-20T22:13:15Z","receivedAt":"2022-01-20T22:13:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 20, 2022 at 10:08:43AM -0800, Jonathan Tan wrote:\n> I think the main issue is the first confounding factor you listed above:\n> even if we didn't have the other confounding factor, that issue alone is\n> enough to motivate the entire patch set. Likewise, as Junio said [1], I\n> don't think we need to switch to rename() if we make the checksum\n> different, so the fix is one-fold, not two-fold. For what it's worth, I\n> switched back to finalize_object_file() and ran the tests, and they all\n> pass.\n\nYeah, I agree with both of you and would rather use\nfinalize_object_file() here and be consistent with other callers that\nmodify $GIT_DIR/objects.\n\nIt is safe to do, since the .rev file's name will necessarily be\ndifferent if the MIDX's object order changes.\n\n> [1] https://lore.kernel.org/git/xmqqtue54iop.fsf@gitster.g/\n\nThanks,\nTaylor\n"},{"id":"446603","messageId":"YenfyK8wE5VXuHoo@nand.local","threadId":"57055","inReplyTo":"20220120181545.3104404-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 3/9] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-20T22:18:48Z","receivedAt":"2022-01-20T22:18:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 20, 2022 at 10:15:45AM -0800, Jonathan Tan wrote:\n> Taylor Blau writes:\n> > In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\n> > source for the reverse index's data. But it will be useful for tests to\n> > be able to determine whether the reverse index was loaded from the\n> > separate .rev file, or from a chunk within the MIDX.\n>\n> As for this patch and all subsequent patches, we (at $DAYJOB) discussed\n> during our Review Club the idea of not supporting .rev files altogether.\n> Firstly, because of this bug, we cannot fully trust .rev files anymore.\n> And even if we decided to trust it (or to trust it after some\n> verification step that I can't think of), that file would only be useful\n> for a short time until a regularly scheduled maintenance step\n> regenerates the bitmaps anyway.\n\nI assume that you're talking about MIDX .rev files here, not the usual\npackfile ones which are only related in the sense that they follow the\nsame file format, but are otherwise distinct.\n\nWe could drop support for the MIDX .rev files, but I do not think it is\na very good idea. There are tagged versions of Git which have .rev files\nout in the wild, so suddenly dropping support for them would mean that\nexisting repositories would no longer be able to read their multi-pack\nbitmaps after upgrading to a version of Git which doesn't include\nsupport for separate .rev files.\n\nSo there are some backwards compatibility concerns there, and I think\nthat that alone means we can't do it.\n\nIn order to maintain good test coverage of the soon-to-be deprecated\nexternal MIDX .rev files, this patch series adds some additional testing\nknobs to make sure that we're thoroughly exercising both cases.\n\nIt's somewhat unfortunate that we are stuck with on-disk .rev files for\nMIDXs, since it opened the door for this bug to manifest itself. But I\nam not comfortable getting rid of them at this point.\n\nIn any case, the .rev file's contents are moving to the MIDX, and by the\nend of this series we do not write an external .rev file when creating a\nmulti-pack bitmap, so you are more than free to not trust any existing\n.rev files anymore.\n\nThanks,\nTaylor\n"},{"id":"446608","messageId":"xmqq5yqeghck.fsf@gitster.g","threadId":"57055","inReplyTo":"YeneJrzjp160edGG@nand.local","subject":"Re: [PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-20T22:41:47Z","receivedAt":"2022-01-20T22:41:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Right. The problem being that it's possible to change the MIDX's\n> object order without changing its checksum. Since the .rev file's\n> contents encodes the pseudo-pack order, embedding it into the MIDX is\n> sufficient to guarantee that the MIDX's checksum changes whenever the\n> object order does.\n\nTo slightly rephrase,\n\n The problem being that before the change to embed reverse table in\n MIDX, it was possible to change the object order without changing\n the checksum, but with that change alone, if a rebuilt MIDX has the\n same checksum, it is guaranteed that it records the same objects in\n the same order.\n\nHence, shuffling between rename(2) and link(2), which was the code\nchange that is only necessary to force updating the existing MIDX\nwith newly recreated one because two MIDX with the same checksum\ncould record objects in different orders, is no longer necessary.\n\nAm I on the same page as you two?\n"},{"id":"446609","messageId":"YenmLJ9yvG0sdxoV@nand.local","threadId":"57055","inReplyTo":"xmqq5yqeghck.fsf@gitster.g","subject":"Re: [PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-20T22:46:04Z","receivedAt":"2022-01-20T22:46:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 20, 2022 at 02:41:47PM -0800, Junio C Hamano wrote:\n> Am I on the same page as you two?\n\nYes, exactly.\n\nThanks,\nTaylor\n"},{"id":"446763","messageId":"20220124174008.2431481-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"YeneJrzjp160edGG@nand.local","subject":"Re: [PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-24T17:40:08Z","receivedAt":"2022-01-24T17:40:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> On Thu, Jan 20, 2022 at 09:55:41AM -0800, Jonathan Tan wrote:\n> > As an effort to ensure that Git reads coherent .midx, .rev, and .bitmap\n> > files, both the .rev and .bitmap files are keyed on the checksum of the\n> > .midx file. But the issue here is that a .rev and a .bitmap could both\n> > refer to the same .midx checksum when the .rev and .bitmap files are not\n> > coherent with respect to each other (e.g. when a Git process has written\n> > the .rev, but not the .bitmap yet - but this would appear perfectly\n> > ordinary to another concurrently running Git process, since the .midx\n> > checksum in the .rev and .bitmap files match).\n> \n> Kind of, and it's possible that we're saying the same thing here using\n> different words.\n> \n> But seeing an out-of-sync .bitmap and .rev is really a symptom of the\n> underlying problem, which is that the MIDX checksum could (prior to\n> these patches) be unchanged even when the object order it represents\n> _does_ change (e.g., in the case of swapping the preferred pack around\n> like the test here demonstrates).\n\nYeah - I think the way you're describing it is that the concept of\npreferred packfile is inherent to the .midx, whereas I just think of it\nas an input that helps the algorithm choose between the many .midx\npossible given a set of packfiles. Maybe these are just two different\nways of looking at the same thing.\n\n> > This problem is exacerbated by the fact that the .rev has its .midx\n> > checksum in its filename, whereas the .bitmap has its .midx checksum in\n> > its file contents. When generating .midx+.rev+.bitmap, it would write\n> > the .bitmap but not the .rev, since a .rev of the same filename already\n> > exists.\n> \n> This isn't quite right: both have the MIDX's checksum in their filename.\n> For example after running `git repack --write-midx --write-bitmap-index`\n> on a random repository, I get these MIDX-related files:\n> \n> $ find .git/objects/pack -type f -name 'multi-pack-index*'\n> .git/objects/pack/multi-pack-index-fd71600b4ceb4caf23a538c7829b9284f2b66d73.rev\n> .git/objects/pack/multi-pack-index-fd71600b4ceb4caf23a538c7829b9284f2b66d73.bitmap\n> .git/objects/pack/multi-pack-index\n\nAh, thanks for the clarification. I saw that bitmaps were overridden\ncorrectly and saw BITMAP_OPT_HASH_CACHE in the bitmap-format\ndocumentation, and made an incorrect assumption.\n\n> Apologies if that all exactly matched up with your understanding, and I\n> was just telling you stuff that you already knew. But I figure that this\n> bug is subtle enough that a little bit of hair-splitting doesn't hurt\n> ;).\n\nThanks for all the clarification, especially about the bitmap.\n"},{"id":"446764","messageId":"20220124175322.2433200-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"YenfyK8wE5VXuHoo@nand.local","subject":"Re: [PATCH v3 3/9] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-24T17:53:22Z","receivedAt":"2022-01-24T17:53:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> On Thu, Jan 20, 2022 at 10:15:45AM -0800, Jonathan Tan wrote:\n> > Taylor Blau writes:\n> > > In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\n> > > source for the reverse index's data. But it will be useful for tests to\n> > > be able to determine whether the reverse index was loaded from the\n> > > separate .rev file, or from a chunk within the MIDX.\n> >\n> > As for this patch and all subsequent patches, we (at $DAYJOB) discussed\n> > during our Review Club the idea of not supporting .rev files altogether.\n> > Firstly, because of this bug, we cannot fully trust .rev files anymore.\n> > And even if we decided to trust it (or to trust it after some\n> > verification step that I can't think of), that file would only be useful\n> > for a short time until a regularly scheduled maintenance step\n> > regenerates the bitmaps anyway.\n> \n> I assume that you're talking about MIDX .rev files here, not the usual\n> packfile ones which are only related in the sense that they follow the\n> same file format, but are otherwise distinct.\n\nAh, yes - sorry for the lack of clarity.\n\n> We could drop support for the MIDX .rev files, but I do not think it is\n> a very good idea. There are tagged versions of Git which have .rev files\n> out in the wild, so suddenly dropping support for them would mean that\n> existing repositories would no longer be able to read their multi-pack\n> bitmaps after upgrading to a version of Git which doesn't include\n> support for separate .rev files.\n> \n> So there are some backwards compatibility concerns there, and I think\n> that that alone means we can't do it.\n> \n> In order to maintain good test coverage of the soon-to-be deprecated\n> external MIDX .rev files, this patch series adds some additional testing\n> knobs to make sure that we're thoroughly exercising both cases.\n> \n> It's somewhat unfortunate that we are stuck with on-disk .rev files for\n> MIDXs, since it opened the door for this bug to manifest itself. But I\n> am not comfortable getting rid of them at this point.\n\nThanks for this explanation of your viewpoint. I still think that it's\nfine to drop support for reading something that a tagged and released\nversion of Git has created, since it is a potentially wrong cache (and\ndoes not contain any new information), but what I can do is to review\nthe rest of this patch set assuming that we want to keep support for the\non-disk MIDX .rev files, so I'll do that. This way, if we decide to keep\nsupport for these files, I would be happy to see this patch set merged.\n"},{"id":"446798","messageId":"20220124191503.2447008-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"7ce3dc60f93d5154943a63f1c09345bd47fbc7c9.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 7/9] t/lib-bitmap.sh: parameterize tests over reverse index source","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-24T19:15:03Z","receivedAt":"2022-01-24T19:40:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> To prepare for reading the reverse index data out of the MIDX itself,\n> teach the `test_rev_exists` function to take an expected \"source\" for\n> the reverse index data.\n\nThanks - up to here looks good. Thanks especially for patch 6, which was easy\nto verify using \"--color-moved --color-moved-ws=allow-indentation-change\".\n\n> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> index 48a8730a13..77b5f46a03 100644\n> --- a/t/lib-bitmap.sh\n> +++ b/t/lib-bitmap.sh\n> @@ -275,17 +275,23 @@ midx_pack_source () {\n> \n> test_rev_exists () {\n> \tcommit=\"$1\"\n> +\tkind=\"$2\"\n> \n> \ttest_expect_success 'reverse index exists' '\n\nTo make it easier to understand test failures, we should probably\ninclude \"kind\" in the name of the test case.\n\nIn a separate commit after this one, we should do it for the other\nblocks.\n"},{"id":"446804","messageId":"20220124192702.2452750-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"55aa69de12c5f82a66836e829f915363cc73b421.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 8/9] midx: read `RIDX` chunk when present","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-24T19:27:01Z","receivedAt":"2022-01-24T19:59:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> When a MIDX contains the new `RIDX` chunk, ensure that the reverse index\n> is read from it instead of the on-disk .rev file. Since we need to\n> encode the object order in the MIDX itself for correctness reasons,\n> there is no point in storing the same data again outside of the MIDX.\n> \n> So, this patch stops writing separate .rev files, and reads it out of\n> the MIDX itself. This is possible to do with relatively little new code,\n> since the format of the RIDX chunk is identical to the data in the .rev\n> file. In other words, we can implement this by pointing the\n> `revindex_data` field at the reverse index chunk of the MIDX instead of\n> the .rev file without any other changes.\n\nAh, that's great.\n\n> Note that we have two knobs that are adjusted for the new tests:\n> GIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\n> whether the MIDX .rev is written at all, and the latter controls whether\n> we read the MIDX's RIDX chunk.\n> \n> Both are necessary to ensure that the test added at the beginning of\n> this series continues to work. This is because we always need to write\n> the RIDX chunk in the MIDX in order to change its checksum, but we want\n> to make sure reading the existing .rev file still works (since the RIDX\n> chunk takes precedence by default).\n\nCould we disable the beginning-of-this-series test when the MIDX .rev is\nnot written instead? Then, we can test what the user would actually\nexperience (no reverse index in .midx, reverse index in .rev) instead of\nsimulating it by skipping over the reverse index section in .midx.\n\nIf we can't do that, then I would agree that the approach in this patch\nseems like the best approach.\n"},{"id":"446805","messageId":"20220124192910.2453842-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"9707d5ea4433d9a5c7f8581dbb2d8a05f410efd3.1641320129.git.me@ttaylorr.com","subject":"Re: [PATCH v3 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-24T19:29:10Z","receivedAt":"2022-01-24T20:10:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> When opening a MIDX/pack-bitmap, we call open_midx_bitmap_1() or\n> open_pack_bitmap_1() respectively in a loop over the set of MIDXs/packs.\n> By design, these functions are supposed to be called over every pack and\n> MIDX, since only one of them should have a valid bitmap.\n\nThanks for this series. Apart from my minor comments on patches 7 and 8,\nthis series looks good.\n"},{"id":"446875","messageId":"YfBuUBOjWpvVOE/9@nand.local","threadId":"57055","inReplyTo":"20220124191503.2447008-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 7/9] t/lib-bitmap.sh: parameterize tests over reverse index source","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T21:40:32Z","receivedAt":"2022-01-25T21:40:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 24, 2022 at 11:15:03AM -0800, Jonathan Tan wrote:\n> Taylor Blau writes:\n> > To prepare for reading the reverse index data out of the MIDX itself,\n> > teach the `test_rev_exists` function to take an expected \"source\" for\n> > the reverse index data.\n>\n> Thanks - up to here looks good. Thanks especially for patch 6, which was easy\n> to verify using \"--color-moved --color-moved-ws=allow-indentation-change\".\n\nGood, thanks for verifying.\n\n> > diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\n> > index 48a8730a13..77b5f46a03 100644\n> > --- a/t/lib-bitmap.sh\n> > +++ b/t/lib-bitmap.sh\n> > @@ -275,17 +275,23 @@ midx_pack_source () {\n> >\n> > test_rev_exists () {\n> > \tcommit=\"$1\"\n> > +\tkind=\"$2\"\n> >\n> > \ttest_expect_success 'reverse index exists' '\n>\n> To make it easier to understand test failures, we should probably\n> include \"kind\" in the name of the test case.\n\nSure, that seems good to me.\n\n> In a separate commit after this one, we should do it for the other\n> blocks.\n\nBut I'm not as certain about this one. The tests are already separated\ninto two different test scripts (t5326 for the in-MIDX .rev file, and\nt5327 for the external .rev file). So it should already be clear based\non what test script you're running.\n\n\nThanks,\nTaylor\n"},{"id":"446876","messageId":"YfBvXS841To7ZXcp@nand.local","threadId":"57055","inReplyTo":"20220124192702.2452750-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 8/9] midx: read `RIDX` chunk when present","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T21:45:01Z","receivedAt":"2022-01-25T21:45:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 24, 2022 at 11:27:01AM -0800, Jonathan Tan wrote:\n> > Note that we have two knobs that are adjusted for the new tests:\n> > GIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\n> > whether the MIDX .rev is written at all, and the latter controls whether\n> > we read the MIDX's RIDX chunk.\n> >\n> > Both are necessary to ensure that the test added at the beginning of\n> > this series continues to work. This is because we always need to write\n> > the RIDX chunk in the MIDX in order to change its checksum, but we want\n> > to make sure reading the existing .rev file still works (since the RIDX\n> > chunk takes precedence by default).\n>\n> Could we disable the beginning-of-this-series test when the MIDX .rev is\n> not written instead? Then, we can test what the user would actually\n> experience (no reverse index in .midx, reverse index in .rev) instead of\n> simulating it by skipping over the reverse index section in .midx.\n>\n> If we can't do that, then I would agree that the approach in this patch\n> seems like the best approach.\n\nI considered it, i.e., by having the two knobs be:\n\n - GIT_TEST_MIDX_WRITE_REV\n - GIT_TEST_MIDX_WRITE_RIDX\n\nwhere the pair (true, false) would correspond to what a corrupt\nrepository would look like before this series. But I dislike that it\nallows the caller to alter the MIDX's checksum by controlling whether or\nnot the chunk is written.\n\nSo it's really looking at the same problem from two sides: do you make\nthe RIDX chunk disappear by not reading it, or by never writing it in\nthe first place? And although the latter is more \"accurate\", it did\nallow me to sidestep a lot of gory details like the ones I outlined in\nthe second patch.\n\nI don't remember everything fully, since some time has passed since I\noriginally wrote this, but I remember encountering some of the races\nwhere you'd read the old bitmap in the new object order, and other\nannoyances.\n\nThanks,\nTaylor\n"},{"id":"446877","messageId":"YfBvpKCEFLWHF59p@nand.local","threadId":"57055","inReplyTo":"20220124192910.2453842-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T21:46:12Z","receivedAt":"2022-01-25T21:46:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 24, 2022 at 11:29:10AM -0800, Jonathan Tan wrote:\n> Taylor Blau writes:\n> > When opening a MIDX/pack-bitmap, we call open_midx_bitmap_1() or\n> > open_pack_bitmap_1() respectively in a loop over the set of MIDXs/packs.\n> > By design, these functions are supposed to be called over every pack and\n> > MIDX, since only one of them should have a valid bitmap.\n>\n> Thanks for this series. Apart from my minor comments on patches 7 and 8,\n> this series looks good.\n\nThanks, I think we need a reroll anyway to reinstate the call to\nfinalize_object_file() as well as to have these patches be based on the\ntip of master now that we're on the other side of v2.35.0.\n\nI'll send that shortly. In the meantime, thanks for your review!\n\nThanks,\nTaylor\n"},{"id":"446882","messageId":"cover.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1638991570.git.me@ttaylorr.com","subject":"[PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:40:57Z","receivedAt":"2022-01-25T22:41:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a(n even) small(er) reroll of my series which fixes a serious problem\nwith MIDX bitmaps by which they can become corrupt when permuting their pack\norder.\n\nThe only change is to revert back to using finalize_object_file(), since it\nbehaves consistently with other parts of the code that touch $GIT_DIR/objects.\nThis is safe to do given the other change described in that patch. The\ndescription has been updated slightly to reflect.\n\nA minor tweak to the tests is done towards the end of the series. But otherwise\nthis is unchanged from v3.\n\nIt is prepared on the tip of master (which is 89bece5c8c at the time of\nwriting).\n\nTaylor Blau (9):\n t5326: demonstrate bitmap corruption after permutation\n midx.c: make changing the preferred pack safe\n pack-revindex.c: instrument loading on-disk reverse index\n t5326: drop unnecessary setup\n t5326: extract `test_rev_exists`\n t5326: move tests to t/lib-bitmap.sh\n t/lib-bitmap.sh: parameterize tests over reverse index source\n midx: read `RIDX` chunk when present\n pack-bitmap.c: gracefully fallback after opening pack/MIDX\n\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 13 +-\n midx.c | 29 ++-\n midx.h | 1 +\n pack-bitmap.c | 4 +\n pack-revindex.c | 20 ++\n t/lib-bitmap.sh | 185 +++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 28 +++\n t/t5326-multi-pack-bitmaps.sh | 164 +++-------------\n t/t5327-multi-pack-bitmaps-rev.sh | 23 +++\n t/t7700-repack.sh | 4 -\n 11 files changed, 321 insertions(+), 151 deletions(-)\n create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\nRange-diff against v3:\n 1: babce7d29a = 1: 7ea9cced8e t5326: demonstrate bitmap corruption after permutation\n 2: 7d20c13f8b ! 2: 62d7561482 midx.c: make changing the preferred pack safe\n @@ Commit message\n that means that a modified .rev file will not be moved into place if\n MIDX's checksum was unchanged.\n \n - The fix here is two-fold. First, we need to stop linking the file into\n - place and instead rename it. It's likely we were using\n - finalize_object_file() instead of a pure rename() because the former\n - also adjusts shared permissions. But that is unnecessary, because we\n - already do so in write_rev_file_order(), so rename alone is safe.\n + This fix is to force the MIDX's checksum to change when the preferred\n + pack changes but the set of packs contained in the MIDX does not. In\n + other words, when the object order changes, the MIDX's checksum needs to\n + change with it (regardless of whether the MIDX is tracking the same or\n + different packs).\n \n - But we also need to make the MIDX's checksum change in some way when the\n - preferred pack changes without altering the set of packs stored in a\n - MIDX to prevent a race where the new .rev file is moved into place\n - before the MIDX is updated. Here, you'd get the opposite effect: reading\n - old bitmaps with the new object order.\n + This prevents a race whereby changing the object order (but not the\n + packs themselves) enables a reader to see the new .rev file with the old\n + MIDX, or similarly seeing the new bitmap with the old object order.\n \n - But this race bites us even here: suppose that we didn't change the MIDX\n - checksum, but only renamed the auxiliary object order into place instead\n - of hardlinking it. Then when we go to generate the new bitmap, we'll\n - load the old MIDX bitmap, along with the MIDX that it references. That's\n - fine, since the new MIDX isn't moved into place until after the new\n - bitmap is generated. But the new object order *has* been moved into\n - place. So we'll read the old bitmaps in the new order when generating\n - the new bitmap file, meaning that without this secondary change, bitmap\n - generation itself would become a victim of the race described here.\n + But why can't we just stop hardlinking the .rev into place instead\n + adding additional data to the MIDX? Suppose that's what we did. Then\n + when we go to generate the new bitmap, we'll load the old MIDX bitmap,\n + along with the MIDX that it references. That's fine, since the new MIDX\n + isn't moved into place until after the new bitmap is generated. But the\n + new object order *has* been moved into place. So we'll read the old\n + bitmaps in the new order when generating the new bitmap file, meaning\n + that without this secondary change, bitmap generation itself would\n + become a victim of the race described here.\n \n This can all be prevented by forcing the MIDX's checksum to change when\n - the object order changes. We could include the entire object order in\n - the MIDX, but doing so is somewhat awkward. (For example, the code that\n - writes a .rev file expects to know the checksum of the associated pack\n - or MIDX, but writing that data into the MIDX itself makes that a\n - circular dependency).\n + the object order does. By embedding the entire object order into the\n + MIDX, we do just that. That is, the MIDX's checksum will change in\n + response to any perturbation of the underlying object order. In t5326,\n + this will cause the MIDX's checksum to update (even without changing the\n + set of packs in the MIDX), preventing the stale read problem.\n \n - Instead, make the object order used during bitmap generation part of the\n - MIDX itself. That means that the new test in t5326 will cause the MIDX's\n - checksum to update, preventing the stale read problem.\n + Note that this makes it safe to continue to link(2) the MIDX .rev file\n + into place, since it is now impossible to have a .rev file that is\n + out-of-sync with the MIDX whose checksum it references. (But we will do\n + away with MIDX .rev files later in this series anyway, so this is\n + somewhat of a moot point).\n \n In theory, it is possible to store a \"fingerprint\" of the full object\n order here, so long as that fingerprint changes at least as often as the\n @@ midx.c: static int write_midx_large_offsets(struct hashfile *f,\n struct midx_pack_order_data {\n \tuint32_t nr;\n \tuint32_t pack;\n -@@ midx.c: static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n - \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n - \t\t\t\t\tmidx_hash, WRITE_REV);\n - \n --\tif (finalize_object_file(tmp_file, buf.buf))\n -+\tif (rename(tmp_file, buf.buf))\n - \t\tdie(_(\"cannot store reverse index file\"));\n - \n - \tstrbuf_release(&buf);\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n 3: 3279e2eb9b = 3: abc18613e0 pack-revindex.c: instrument loading on-disk reverse index\n 4: 5818621ea8 = 4: 80589d7ae6 t5326: drop unnecessary setup\n 5: 33502d6a17 = 5: b9c4ff8636 t5326: extract `test_rev_exists`\n 6: 76e23cae0f = 6: 00c6e914b4 t5326: move tests to t/lib-bitmap.sh\n 7: 7ce3dc60f9 ! 7: 3f35ef6499 t/lib-bitmap.sh: parameterize tests over reverse index source\n @@ t/lib-bitmap.sh: midx_pack_source () {\n \tcommit=\"$1\"\n +\tkind=\"$2\"\n \n - \ttest_expect_success 'reverse index exists' '\n +-\ttest_expect_success 'reverse index exists' '\n ++\ttest_expect_success \"reverse index exists ($kind)\" '\n \t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n \t\t\tgit rev-list --test-bitmap \"$commit\" &&\n \n 8: 55aa69de12 = 8: 94563cf038 midx: read `RIDX` chunk when present\n 9: 9707d5ea44 = 9: 581b723792 pack-bitmap.c: gracefully fallback after opening pack/MIDX\n-- \n2.34.1.455.gd6eb6fd089\n"},{"id":"446883","messageId":"7ea9cced8ec79a8e39948a5e4b8dde6e9b54695a.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:01Z","receivedAt":"2022-01-25T22:41:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch demonstrates a cause of bitmap corruption that can occur when\nthe contents of the multi-pack index does not change, but the underlying\nobject order does.\n\nIn this example, we have a MIDX containing two packs, each with a\ndistinct set of objects (pack A corresponds to the tree, blob, and\ncommit from the first patch, and pack B corresponds to the second\npatch).\n\nFirst, a MIDX is written where the 'A' pack is preferred. As expected,\nthe bitmaps generated there are in-tact. But then, we generate an\nidentical MIDX with a different object order: this time preferring pack\n'B'.\n\nDue to a bug which will be explained and fixed in the following commit,\nthe MIDX is updated, but the .rev file is not, causing the .bitmap file\nto be read incorrectly. Specifically, the .bitmap file will contain\ncorrect data, but the auxiliary object order in the .rev file is stale,\ncausing readers to get confused by reading the new bitmaps using the old\nobject order.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex e187f90f29..0ca2868b0b 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,4 +395,35 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n+test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\tpack-$A.idx\n+\t\tpack-$B.idx\n+\t\tEOF\n+\n+\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t--preferred-pack=pack-$A.pack \n---\n Documentation/technical/multi-pack-index.txt | 1 +\n Documentation/technical/pack-format.txt | 13 ++++++-----\n midx.c | 23 +++++++++++++++++---\n t/t5326-multi-pack-bitmaps.sh | 2 +-\n 4 files changed, 29 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex b39c69da8c..f2221d2b44 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -24,6 +24,7 @@ and their offsets into multiple packfiles. It contains:\n ** An offset within the jth packfile for the object.\n * If large offsets are required, we use another list of large\n offsets similar to version 2 pack-indexes.\n+- An optional list of objects in pseudo-pack order (used with MIDX bitmaps).\n \n Thus, we can provide O(log N) lookup time for any number\n of packfiles.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 8d2f42f29e..6d3efb7d16 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -376,6 +376,11 @@ CHUNK DATA:\n \t[Optional] Object Large Offsets (ID: {'L', 'O', 'F', 'F'})\n \t 8-byte offsets into large packfiles.\n \n+\t[Optional] Bitmap pack order (ID: {'R', 'I', 'D', 'X'})\n+\t A list of MIDX positions (one per object in the MIDX, num_objects in\n+\t total, each a 4-byte unsigned integer in network byte order), sorted\n+\t according to their relative bitmap/pseudo-pack positions.\n+\n TRAILER:\n \n \tIndex checksum of the above contents.\n@@ -456,9 +461,5 @@ In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n objects in packs stored by the MIDX, laid out in pack order, and the\n packs arranged in MIDX order (with the preferred pack coming first).\n \n-Finally, note that the MIDX's reverse index is not stored as a chunk in\n-the multi-pack-index itself. This is done because the reverse index\n-includes the checksum of the pack or MIDX to which it belongs, which\n-makes it impossible to write in the MIDX. To avoid races when rewriting\n-the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n-filename (e.g., `multi-pack-index-xyz.rev`).\n+The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n+the MIDX itself.\ndiff --git a/midx.c b/midx.c\nindex 837b46b2af..0248c4577c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n+#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n@@ -833,6 +834,18 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_midx_revindex(struct hashfile *f,\n+\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\n+\treturn 0;\n+}\n+\n struct midx_pack_order_data {\n \tuint32_t nr;\n \tuint32_t pack;\n@@ -1403,15 +1416,19 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n \n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n+\t\t\t ctx.entries_nr * sizeof(uint32_t),\n+\t\t\t write_midx_revindex);\n+\t}\n+\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n-\t\tctx.pack_order = midx_pack_order(&ctx);\n-\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 0ca2868b0b..353282310d 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -395,7 +395,7 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n+test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n \trm -fr repo &&\n \tgit init repo &&\n \ttest_when_finished \"rm -fr repo\" &&\n-- \n2.34.1.455.gd6eb6fd089\n\n"},{"id":"446885","messageId":"abc18613e04a3bb8571926cb653296fe6b49cc60.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 3/9] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:05Z","receivedAt":"2022-01-25T22:41:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\nsource for the reverse index's data. But it will be useful for tests to\nbe able to determine whether the reverse index was loaded from the\nseparate .rev file, or from a chunk within the MIDX.\n\nTo instrument this, add a trace2 event which the tests can look for in\norder to determine the reverse index's source.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 70d0fbafcb..bd15ebad03 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -301,6 +301,9 @@ int load_midx_revindex(struct multi_pack_index *m)\n \tif (m->revindex_data)\n \t\treturn 0;\n \n+\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n+\t\t\t \"source\", \"rev\");\n+\n \tget_midx_rev_filename(&revindex_name, m);\n \n \tret = load_revindex_from_disk(revindex_name.buf,\n-- \n2.34.1.455.gd6eb6fd089\n\n"},{"id":"446886","messageId":"80589d7ae6de8e33211916c7483ed9d9dd9c9ef9.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 4/9] t5326: drop unnecessary setup","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:08Z","receivedAt":"2022-01-25T22:41:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The core.multiPackIndex config became true by default back in 18e449f86b\n(midx: enable core.multiPackIndex by default, 2020-09-25), so it is no\nlonger necessary to enable it explicitly.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 4 ----\n 1 file changed, 4 deletions(-)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 353282310d..1a9581af30 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -19,10 +19,6 @@ midx_pack_source () {\n \n setup_bitmap_history\n \n-test_expect_success 'enable core.multiPackIndex' '\n-\tgit config core.multiPackIndex true\n-'\n-\n test_expect_success 'create single-pack midx with bitmaps' '\n \tgit repack -ad &&\n \tgit multi-pack-index write --bitmap &&\n-- \n2.34.1.455.gd6eb6fd089\n\n"},{"id":"446887","messageId":"b9c4ff863647c3719725b3ab290e055493b7d6af.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 5/9] t5326: extract `test_rev_exists`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:10Z","receivedAt":"2022-01-25T22:41:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To determine which source of data is used for the MIDX's reverse index\ncache, introduce a helper which forces loading the reverse index, and\nthen looks for the special trace2 event introduced in a previous commit.\n\nFor now, this helper just looks for when the legacy MIDX .rev file was\nloaded, but in a subsequent commit will become parameterized over the\nthe reverse index's source.\n\nThis function replaces checking for the existence of the .rev file. We\ncould write a similar helper to ensure that the .rev file is cleaned up\nafter repacking, but it will make subsequent tests more difficult to\nwrite, and provides marginal value since we already check that the MIDX\n.bitmap file is removed.\n\nSigned-off-by: Taylor Blau \n---\n t/t5326-multi-pack-bitmaps.sh | 37 +++++++++++++++++++++--------------\n 1 file changed, 22 insertions(+), 15 deletions(-)\n\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 1a9581af30..999740f8c7 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -17,16 +17,29 @@ midx_pack_source () {\n \ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n }\n \n+test_rev_exists () {\n+\tcommit=\"$1\"\n+\n+\ttest_expect_success 'reverse index exists' '\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n+\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t'\n+}\n+\n setup_bitmap_history\n \n test_expect_success 'create single-pack midx with bitmaps' '\n \tgit repack -ad &&\n \tgit multi-pack-index write --bitmap &&\n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD\n+\n basic_bitmap_tests\n \n test_expect_success 'create new additional packs' '\n@@ -52,10 +65,11 @@ test_expect_success 'create multi-pack midx with bitmaps' '\n \ttest_line_count = 25 packs &&\n \n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD\n+\n basic_bitmap_tests\n \n test_expect_success '--no-bitmap is respected when bitmaps exist' '\n@@ -66,7 +80,6 @@ test_expect_success '--no-bitmap is respected when bitmaps exist' '\n \n \ttest_path_is_file $midx &&\n \ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \tgit multi-pack-index write --no-bitmap &&\n \n@@ -206,10 +219,11 @@ test_expect_success 'setup partial bitmaps' '\n \ttest_commit loose &&\n \tgit multi-pack-index write --bitmap 2>err &&\n \ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n '\n \n+test_rev_exists HEAD~\n+\n basic_bitmap_tests HEAD~\n \n test_expect_success 'removing a MIDX clears stale bitmaps' '\n@@ -224,7 +238,6 @@ test_expect_success 'removing a MIDX clears stale bitmaps' '\n \n \t\t# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.\n \t\tstale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&\n-\t\tstale_rev=$midx-$(midx_checksum $objdir).rev &&\n \t\trm $midx &&\n \n \t\t# Then write a new MIDX.\n@@ -234,9 +247,7 @@ test_expect_success 'removing a MIDX clears stale bitmaps' '\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\ttest_path_is_missing $stale_bitmap &&\n-\t\ttest_path_is_missing $stale_rev\n+\t\ttest_path_is_missing $stale_bitmap\n \t)\n '\n \n@@ -257,7 +268,6 @@ test_expect_success 'pack.preferBitmapTips' '\n \t\tgit multi-pack-index write --bitmap &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \t\ttest-tool bitmap list-commits | sort >bitmaps &&\n \t\tcomm -13 bitmaps commits >before &&\n@@ -267,7 +277,6 @@ test_expect_success 'pack.preferBitmapTips' '\n \t\t\tsnapshot &&\n \n \t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n \t\trm -fr $midx &&\n \n \t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n-- \n2.34.1.455.gd6eb6fd089\n\n"},{"id":"446888","messageId":"00c6e914b45b8a7fd52fd62c7cf020272793f83d.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 6/9] t5326: move tests to t/lib-bitmap.sh","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:13Z","receivedAt":"2022-01-25T22:41:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In t5326, we have a handful of tests that we would like to run twice:\nonce using the MIDX's new `RIDX` chunk as the source of the\nreverse-index cache, and once using the separate `.rev` file.\n\nBut because these tests mutate the state of the underlying repository,\nand then make assumptions about those mutations occurring in a certain\nsequence, simply running the tests twice in the same repository is\nawkward.\n\nInstead, extract the core of interesting tests into t/lib-bitmap.sh to\nprepare for them to be run twice, each in a separate test script. This\nmeans that they can each operate on a separate repository, removing any\nconcerns about mutating state.\n\nFor now, this patch is a strict cut-and-paste of some tests from t5326.\nThe tests which did not move are not interesting with respect to the\nsource of their reverse index data.\n\nSigned-off-by: Taylor Blau \n---\n t/lib-bitmap.sh | 177 ++++++++++++++++++++++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 173 +--------------------------------\n 2 files changed, 179 insertions(+), 171 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 21d0392dda..48a8730a13 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -1,6 +1,9 @@\n # Helpers for scripts testing bitmap functionality; see t5310 for\n # example usage.\n \n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n # Compare a file containing rev-list bitmap traversal output to its non-bitmap\n # counterpart. You can't just use test_cmp for this, because the two produce\n # subtly different output:\n@@ -264,3 +267,177 @@ have_delta () {\n midx_checksum () {\n \ttest-tool read-midx --checksum \"$1\"\n }\n+\n+# midx_pack_source \n+midx_pack_source () {\n+\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n+}\n+\n+test_rev_exists () {\n+\tcommit=\"$1\"\n+\n+\ttest_expect_success 'reverse index exists' '\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n+\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t'\n+}\n+\n+midx_bitmap_core () {\n+\tsetup_bitmap_history\n+\n+\ttest_expect_success 'create single-pack midx with bitmaps' '\n+\t\tgit repack -ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success 'create new additional packs' '\n+\t\tfor i in $(test_seq 1 16)\n+\t\tdo\n+\t\t\ttest_commit \"$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\n+\t\tgit checkout -b other2 HEAD~8 &&\n+\t\tfor i in $(test_seq 1 8)\n+\t\tdo\n+\t\t\ttest_commit \"side-$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\t\tgit checkout second\n+\t'\n+\n+\ttest_expect_success 'create multi-pack midx with bitmaps' '\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\tls $objdir/pack/pack-*.pack >packs &&\n+\t\ttest_line_count = 25 packs &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD\n+\n+\tbasic_bitmap_tests\n+\n+\ttest_expect_success '--no-bitmap is respected when bitmaps exist' '\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_commit respect--no-bitmap &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\tgit multi-pack-index write --no-bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n+\t'\n+\n+\ttest_expect_success 'setup midx with base from later pack' '\n+\t\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n+\t\t# prefers to delete contents out of a base rather than add to a shorter\n+\t\t# object.\n+\t\ttest_seq 1 128 >a &&\n+\t\ttest_seq 1 130 >b &&\n+\n+\t\tgit add a b &&\n+\t\tgit commit -m \"initial commit\" &&\n+\n+\t\ta=$(git rev-parse HEAD:a) &&\n+\t\tb=$(git rev-parse HEAD:b) &&\n+\n+\t\t# In the first pack, \"a\" is stored as a delta to \"b\".\n+\t\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\t) &&\n+\n+\t\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n+\t\t# any other object.\n+\t\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n+\t\t$b\n+\t\t$(git rev-parse HEAD)\n+\t\t$(git rev-parse HEAD^{tree})\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit prune-packed &&\n+\t\t# Use the second pack as the preferred source, so that \"b\" occurs\n+\t\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n+\t\t# reuse.\n+\t\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n+\n+\t\thave_delta $a $b &&\n+\t\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n+\t'\n+\n+\trev_list_tests 'full bitmap with backwards delta'\n+\n+\ttest_expect_success 'clone with bitmaps enabled' '\n+\t\tgit clone --no-local --bare . clone-reverse-delta.git &&\n+\t\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n+\n+\t\tgit rev-parse HEAD >expect &&\n+\t\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n+\t\trm -fr repo &&\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\n+\t\t\ttest_commit A &&\n+\t\t\ttest_commit B &&\n+\n+\t\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n+\t\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n+\n+\t\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n+\t\t\tpack-$A.idx\n+\t\t\tpack-$B.idx\n+\t\t\tEOF\n+\n+\t\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n+\t\t\t\t--preferred-pack=pack-$A.pack err &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n+\t'\n+\n+\ttest_rev_exists HEAD~\n+\n+\tbasic_bitmap_tests HEAD~\n+}\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 999740f8c7..100ac90d15 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -9,134 +9,7 @@ test_description='exercise basic multi-pack bitmap functionality'\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n-objdir=.git/objects\n-midx=$objdir/pack/multi-pack-index\n-\n-# midx_pack_source \n-midx_pack_source () {\n-\ttest-tool read-midx --show-objects .git/objects | grep \"^$1 \" | cut -f2\n-}\n-\n-test_rev_exists () {\n-\tcommit=\"$1\"\n-\n-\ttest_expect_success 'reverse index exists' '\n-\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n-\t\t\tgit rev-list --test-bitmap \"$commit\" &&\n-\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n-\t'\n-}\n-\n-setup_bitmap_history\n-\n-test_expect_success 'create single-pack midx with bitmaps' '\n-\tgit repack -ad &&\n-\tgit multi-pack-index write --bitmap &&\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD\n-\n-basic_bitmap_tests\n-\n-test_expect_success 'create new additional packs' '\n-\tfor i in $(test_seq 1 16)\n-\tdo\n-\t\ttest_commit \"$i\" &&\n-\t\tgit repack -d || return 1\n-\tdone &&\n-\n-\tgit checkout -b other2 HEAD~8 &&\n-\tfor i in $(test_seq 1 8)\n-\tdo\n-\t\ttest_commit \"side-$i\" &&\n-\t\tgit repack -d || return 1\n-\tdone &&\n-\tgit checkout second\n-'\n-\n-test_expect_success 'create multi-pack midx with bitmaps' '\n-\tgit multi-pack-index write --bitmap &&\n-\n-\tls $objdir/pack/pack-*.pack >packs &&\n-\ttest_line_count = 25 packs &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD\n-\n-basic_bitmap_tests\n-\n-test_expect_success '--no-bitmap is respected when bitmaps exist' '\n-\tgit multi-pack-index write --bitmap &&\n-\n-\ttest_commit respect--no-bitmap &&\n-\tgit repack -d &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\n-\tgit multi-pack-index write --no-bitmap &&\n-\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&\n-\ttest_path_is_missing $midx-$(midx_checksum $objdir).rev\n-'\n-\n-test_expect_success 'setup midx with base from later pack' '\n-\t# Write a and b so that \"a\" is a delta on top of base \"b\", since Git\n-\t# prefers to delete contents out of a base rather than add to a shorter\n-\t# object.\n-\ttest_seq 1 128 >a &&\n-\ttest_seq 1 130 >b &&\n-\n-\tgit add a b &&\n-\tgit commit -m \"initial commit\" &&\n-\n-\ta=$(git rev-parse HEAD:a) &&\n-\tb=$(git rev-parse HEAD:b) &&\n-\n-\t# In the first pack, \"a\" is stored as a delta to \"b\".\n-\tp1=$(git pack-objects .git/objects/pack/pack <<-EOF\n-\t$a\n-\t$b\n-\tEOF\n-\t) &&\n-\n-\t# In the second pack, \"a\" is missing, and \"b\" is not a delta nor base to\n-\t# any other object.\n-\tp2=$(git pack-objects .git/objects/pack/pack <<-EOF\n-\t$b\n-\t$(git rev-parse HEAD)\n-\t$(git rev-parse HEAD^{tree})\n-\tEOF\n-\t) &&\n-\n-\tgit prune-packed &&\n-\t# Use the second pack as the preferred source, so that \"b\" occurs\n-\t# earlier in the MIDX object order, rendering \"a\" unusable for pack\n-\t# reuse.\n-\tgit multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&\n-\n-\thave_delta $a $b &&\n-\ttest $(midx_pack_source $a) != $(midx_pack_source $b)\n-'\n-\n-rev_list_tests 'full bitmap with backwards delta'\n-\n-test_expect_success 'clone with bitmaps enabled' '\n-\tgit clone --no-local --bare . clone-reverse-delta.git &&\n-\ttest_when_finished \"rm -fr clone-reverse-delta.git\" &&\n-\n-\tgit rev-parse HEAD >expect &&\n-\tgit --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n+midx_bitmap_core\n \n bitmap_reuse_tests() {\n \tfrom=$1\n@@ -213,18 +86,7 @@ test_expect_success 'missing object closure fails gracefully' '\n \t)\n '\n \n-test_expect_success 'setup partial bitmaps' '\n-\ttest_commit packed &&\n-\tgit repack &&\n-\ttest_commit loose &&\n-\tgit multi-pack-index write --bitmap 2>err &&\n-\ttest_path_is_file $midx &&\n-\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n-'\n-\n-test_rev_exists HEAD~\n-\n-basic_bitmap_tests HEAD~\n+midx_bitmap_partial_tests\n \n test_expect_success 'removing a MIDX clears stale bitmaps' '\n \trm -fr repo &&\n@@ -398,35 +260,4 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n-test_expect_success 'changing the preferred pack does not corrupt bitmaps' '\n-\trm -fr repo &&\n-\tgit init repo &&\n-\ttest_when_finished \"rm -fr repo\" &&\n-\t(\n-\t\tcd repo &&\n-\n-\t\ttest_commit A &&\n-\t\ttest_commit B &&\n-\n-\t\tgit rev-list --objects --no-object-names HEAD^ >A.objects &&\n-\t\tgit rev-list --objects --no-object-names HEAD^.. >B.objects &&\n-\n-\t\tA=$(git pack-objects $objdir/pack/pack indexes <<-EOF &&\n-\t\tpack-$A.idx\n-\t\tpack-$B.idx\n-\t\tEOF\n-\n-\t\tgit multi-pack-index write --bitmap --stdin-packs \\\n-\t\t\t--preferred-pack=pack-$A.pack \n---\n t/lib-bitmap.sh | 20 ++++++++++++++------\n 1 file changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 48a8730a13..253895c04e 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -275,17 +275,23 @@ midx_pack_source () {\n \n test_rev_exists () {\n \tcommit=\"$1\"\n+\tkind=\"$2\"\n \n-\ttest_expect_success 'reverse index exists' '\n+\ttest_expect_success \"reverse index exists ($kind)\" '\n \t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n \t\t\tgit rev-list --test-bitmap \"$commit\" &&\n \n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n-\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"rev\\\"\" event.trace\n+\t\tif test \"rev\" = \"$kind\"\n+\t\tthen\n+\t\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev\n+\t\tfi &&\n+\t\tgrep \"\\\"category\\\":\\\"load_midx_revindex\\\",\\\"key\\\":\\\"source\\\",\\\"value\\\":\\\"$kind\\\"\" event.trace\n \t'\n }\n \n midx_bitmap_core () {\n+\trev_kind=\"${1:-rev}\"\n+\n \tsetup_bitmap_history\n \n \ttest_expect_success 'create single-pack midx with bitmaps' '\n@@ -295,7 +301,7 @@ midx_bitmap_core () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD\n+\ttest_rev_exists HEAD \"$rev_kind\"\n \n \tbasic_bitmap_tests\n \n@@ -325,7 +331,7 @@ midx_bitmap_core () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD\n+\ttest_rev_exists HEAD \"$rev_kind\"\n \n \tbasic_bitmap_tests\n \n@@ -428,6 +434,8 @@ midx_bitmap_core () {\n }\n \n midx_bitmap_partial_tests () {\n+\trev_kind=\"${1:-rev}\"\n+\n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\n \t\tgit repack &&\n@@ -437,7 +445,7 @@ midx_bitmap_partial_tests () {\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap\n \t'\n \n-\ttest_rev_exists HEAD~\n+\ttest_rev_exists HEAD~ \"$rev_kind\"\n \n \tbasic_bitmap_tests HEAD~\n }\n-- \n2.34.1.455.gd6eb6fd089\n\n"},{"id":"446890","messageId":"94563cf038722852d801bd0a8e744a569799f354.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 8/9] midx: read `RIDX` chunk when present","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:17Z","receivedAt":"2022-01-25T22:41:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a MIDX contains the new `RIDX` chunk, ensure that the reverse index\nis read from it instead of the on-disk .rev file. Since we need to\nencode the object order in the MIDX itself for correctness reasons,\nthere is no point in storing the same data again outside of the MIDX.\n\nSo, this patch stops writing separate .rev files, and reads it out of\nthe MIDX itself. This is possible to do with relatively little new code,\nsince the format of the RIDX chunk is identical to the data in the .rev\nfile. In other words, we can implement this by pointing the\n`revindex_data` field at the reverse index chunk of the MIDX instead of\nthe .rev file without any other changes.\n\nNote that we have two knobs that are adjusted for the new tests:\nGIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\nwhether the MIDX .rev is written at all, and the latter controls whether\nwe read the MIDX's RIDX chunk.\n\nBoth are necessary to ensure that the test added at the beginning of\nthis series continues to work. This is because we always need to write\nthe RIDX chunk in the MIDX in order to change its checksum, but we want\nto make sure reading the existing .rev file still works (since the RIDX\nchunk takes precedence by default).\n\nArguably this isn't a very interesting mode to test, because the\nprecedence rules mean that we'll always read the RIDX chunk over the\n.rev file. But it makes it impossible for a user to induce corruption in\ntheir repository by adjusting the test knobs (since if we had an\neither/or knob they could stop writing the RIDX chunk, allowing them to\ntweak the MIDX's object order without changing its checksum).\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++++-\n midx.h | 1 +\n pack-revindex.c | 17 +++++++++++++++++\n t/lib-bitmap.sh | 4 ++--\n t/t5326-multi-pack-bitmaps.sh | 6 ++++++\n t/t5327-multi-pack-bitmaps-rev.sh | 23 +++++++++++++++++++++++\n t/t7700-repack.sh | 4 ----\n 7 files changed, 54 insertions(+), 7 deletions(-)\n create mode 100755 t/t5327-multi-pack-bitmaps-rev.sh\n\ndiff --git a/midx.c b/midx.c\nindex 0248c4577c..6e6cb0eb04 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -162,6 +162,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n \n+\tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n+\t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tCALLOC_ARRAY(m->pack_names, m->num_packs);\n@@ -1429,7 +1432,8 @@ static int write_midx_internal(const char *object_dir,\n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX)\n+\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n \t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\n \t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &ctx,\ndiff --git a/midx.h b/midx.h\nindex b7d79a515c..22e8e53288 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -36,6 +36,7 @@ struct multi_pack_index {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_object_offsets;\n \tconst unsigned char *chunk_large_offsets;\n+\tconst unsigned char *chunk_revindex;\n \n \tconst char **pack_names;\n \tstruct packed_git **packs;\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex bd15ebad03..08dc160167 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -298,9 +298,26 @@ int load_midx_revindex(struct multi_pack_index *m)\n {\n \tstruct strbuf revindex_name = STRBUF_INIT;\n \tint ret;\n+\n \tif (m->revindex_data)\n \t\treturn 0;\n \n+\tif (m->chunk_revindex) {\n+\t\t/*\n+\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n+\t\t * the reverse index instead of trying to load a separate `.rev`\n+\t\t * file.\n+\t\t *\n+\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n+\t\t * not want to accidentally call munmap() in the middle of the\n+\t\t * MIDX.\n+\t\t */\n+\t\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n+\t\t\t\t \"source\", \"midx\");\n+\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n+\t\treturn 0;\n+\t}\n+\n \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n \t\t\t \"source\", \"rev\");\n \ndiff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh\nindex 253895c04e..a95537e759 100644\n--- a/t/lib-bitmap.sh\n+++ b/t/lib-bitmap.sh\n@@ -290,7 +290,7 @@ test_rev_exists () {\n }\n \n midx_bitmap_core () {\n-\trev_kind=\"${1:-rev}\"\n+\trev_kind=\"${1:-midx}\"\n \n \tsetup_bitmap_history\n \n@@ -434,7 +434,7 @@ midx_bitmap_core () {\n }\n \n midx_bitmap_partial_tests () {\n-\trev_kind=\"${1:-rev}\"\n+\trev_kind=\"${1:-midx}\"\n \n \ttest_expect_success 'setup partial bitmaps' '\n \t\ttest_commit packed &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 100ac90d15..c0924074c4 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -9,6 +9,12 @@ test_description='exercise basic multi-pack bitmap functionality'\n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n \n+# This test exercise multi-pack bitmap functionality where the object order is\n+# stored and read from a special chunk within the MIDX, so use the default\n+# behavior here.\n+sane_unset GIT_TEST_MIDX_WRITE_REV\n+sane_unset GIT_TEST_MIDX_READ_RIDX\n+\n midx_bitmap_core\n \n bitmap_reuse_tests() {\ndiff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh\nnew file mode 100755\nindex 0000000000..d30ba632c8\n--- /dev/null\n+++ b/t/t5327-multi-pack-bitmaps-rev.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='exercise basic multi-pack bitmap functionality (.rev files)'\n+\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+\n+# We'll be writing our own midx and bitmaps, so avoid getting confused by the\n+# automatic ones.\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+# Unlike t5326, this test exercise multi-pack bitmap functionality where the\n+# object order is stored in a separate .rev file.\n+GIT_TEST_MIDX_WRITE_REV=1\n+GIT_TEST_MIDX_READ_RIDX=0\n+export GIT_TEST_MIDX_WRITE_REV\n+export GIT_TEST_MIDX_READ_RIDX\n+\n+midx_bitmap_core rev\n+midx_bitmap_partial_tests rev\n+\n+test_done\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex e489869dd9..5922fb5bdd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -312,16 +312,13 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t\tchecksum=$(midx_checksum $objdir) &&\n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_file $midx-$checksum.bitmap &&\n-\t\ttest_path_is_file $midx-$checksum.rev &&\n \n \t\ttest_commit repack-3 &&\n \t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n \n \t\ttest_path_is_file $midx &&\n \t\ttest_path_is_missing $midx-$checksum.bitmap &&\n-\t\ttest_path_is_missing $midx-$checksum.rev &&\n \t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n-\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n \n \t\ttest_commit repack-4 &&\n \t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n@@ -354,7 +351,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '\n \t\ttest_line_count = 1 before &&\n \n \t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n-\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n \t\trm -fr $midx &&\n \n \t\t# instead of constructing the snapshot ourselves (c.f., the test\n-- \n2.34.1.455.gd6eb6fd089\n\n"},{"id":"446891","messageId":"581b723792c2bec1ba66025c7ebb0344d5541865.1643150456.git.me@ttaylorr.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"[PATCH v4 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-25T22:41:20Z","receivedAt":"2022-01-25T22:41:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When opening a MIDX/pack-bitmap, we call open_midx_bitmap_1() or\nopen_pack_bitmap_1() respectively in a loop over the set of MIDXs/packs.\nBy design, these functions are supposed to be called over every pack and\nMIDX, since only one of them should have a valid bitmap.\n\nOrdinarily we return '0' from these two functions in order to indicate\nthat we successfully loaded a bitmap To signal that we couldn't load a\nbitmap corresponding to the MIDX/pack (either because one doesn't exist,\nor because there was an error with loading it), we can return '-1'. In\neither case, the callers each enumerate all MIDXs/packs to ensure that\nat most one bitmap per-kind is present.\n\nBut when we fail to load a bitmap that does exist (for example, loading\na MIDX bitmap without finding a corresponding reverse index), we'll\nreturn -1 but leave the 'midx' field non-NULL. So when we fallback to\nloading a pack bitmap, we'll complain that the bitmap we're trying to\npopulate already is \"opened\", even though it isn't.\n\nRectify this by setting the '->pack' and '->midx' field back to NULL as\nappropriate. Two tests are added: one to ensure that the MIDX-to-pack\nbitmap fallback works, and another to ensure we still complain when\nthere are multiple pack bitmaps in a repository.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 4 ++++\n t/t5310-pack-bitmaps.sh | 28 ++++++++++++++++++++++++++++\n t/t5326-multi-pack-bitmaps.sh | 19 +++++++++++++++++++\n 3 files changed, 51 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex f772d3cb7f..9c666cdb8b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -358,7 +358,9 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n cleanup:\n \tmunmap(bitmap_git->map, bitmap_git->map_size);\n \tbitmap_git->map_size = 0;\n+\tbitmap_git->map_pos = 0;\n \tbitmap_git->map = NULL;\n+\tbitmap_git->midx = NULL;\n \treturn -1;\n }\n \n@@ -405,6 +407,8 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git\n \t\tmunmap(bitmap_git->map, bitmap_git->map_size);\n \t\tbitmap_git->map = NULL;\n \t\tbitmap_git->map_size = 0;\n+\t\tbitmap_git->map_pos = 0;\n+\t\tbitmap_git->pack = NULL;\n \t\treturn -1;\n \t}\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex d05ab716f6..f775fc1ce6 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -397,4 +397,32 @@ test_expect_success 'pack.preferBitmapTips' '\n \t)\n '\n \n+test_expect_success 'complains about multiple pack bitmaps' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\n+\t\tgit repack -adb &&\n+\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n+\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n+\n+\t\ttest_commit other &&\n+\t\tgit repack -ab &&\n+\n+\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n+\n+\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n+\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n+\t\ttest_line_count = 2 packs &&\n+\t\ttest_line_count = 2 bitmaps &&\n+\n+\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\tgrep \"ignoring extra bitmap file\" err\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex c0924074c4..3c1ecc7e25 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -266,4 +266,23 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n \t)\n '\n \n+test_expect_success 'graceful fallback when missing reverse index' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\n+\t\t# write a pack and MIDX bitmap containing base\n+\t\tgit repack -adb &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\tGIT_TEST_MIDX_READ_RIDX=0 \\\n+\t\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n+\t\t! grep \"ignoring extra bitmap file\" err\n+\t)\n+'\n+\n test_done\n-- \n2.34.1.455.gd6eb6fd089\n"},{"id":"446966","messageId":"220126.8635lafskv.gmgdl@evledraar.gmail.com","threadId":"57055","inReplyTo":"7ea9cced8ec79a8e39948a5e4b8dde6e9b54695a.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-26T15:01:20Z","receivedAt":"2022-01-26T15:02:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 25 2022, Taylor Blau wrote:\n\n> +test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n> +\trm -fr repo &&\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n\nNit: The initial \"rm -fr\" isn't needed here, and we should aim to have\ntests clean up after themselves, not needing to clean up after other\ntests.\n\nThis appears to have been copy/pasted from the test you added in\n54156af0d66 (t5326: test propagating hashcache values, 2021-09-17),\nwhich needlessly used that pattern, the tests you added preceding it\nfollow the \"clean up your own mess\" pattern.\n"},{"id":"446967","messageId":"220126.86y232edxn.gmgdl@evledraar.gmail.com","threadId":"57055","inReplyTo":"abc18613e04a3bb8571926cb653296fe6b49cc60.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 3/9] pack-revindex.c: instrument loading on-disk reverse index","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-26T15:03:50Z","receivedAt":"2022-01-26T15:04:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 25 2022, Taylor Blau wrote:\n\n> In a subsequent commit, we'll use the MIDX's new 'RIDX' chunk as a\n> source for the reverse index's data. But it will be useful for tests to\n> be able to determine whether the reverse index was loaded from the\n> separate .rev file, or from a chunk within the MIDX.\n>\n> To instrument this, add a trace2 event which the tests can look for in\n> order to determine the reverse index's source.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> pack-revindex.c | 3 +++\n> 1 file changed, 3 insertions(+)\n>\n> diff --git a/pack-revindex.c b/pack-revindex.c\n> index 70d0fbafcb..bd15ebad03 100644\n> --- a/pack-revindex.c\n> +++ b/pack-revindex.c\n> @@ -301,6 +301,9 @@ int load_midx_revindex(struct multi_pack_index *m)\n> \tif (m->revindex_data)\n> \t\treturn 0;\n> \n> +\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> +\t\t\t \"source\", \"rev\");\n> +\n> \tget_midx_rev_filename(&revindex_name, m);\n> \n> \tret = load_revindex_from_disk(revindex_name.buf,\n\nIMO easier to read/reason about if squashed into 5/9, where it's first\nused, or at least re-arrange it so that 4/9 isn't between the two...\n"},{"id":"446968","messageId":"220126.86tudqeduk.gmgdl@evledraar.gmail.com","threadId":"57055","inReplyTo":"b9c4ff863647c3719725b3ab290e055493b7d6af.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 5/9] t5326: extract `test_rev_exists`","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-26T15:04:37Z","receivedAt":"2022-01-26T15:06:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 25 2022, Taylor Blau wrote:\n\nI left a comment on 3/9, but re $subject: It really suggests we're just\nrefactoring some code to a function (\"extract\"...), but really we're\nusing the new trace2 event added earlier here to test things in a new\nway.\n\nPerhaps (if squashed): \"midx + 5326: add trace2 evet to test revision\nexistence\" ?\n"},{"id":"446969","messageId":"220126.86pmoeedpw.gmgdl@evledraar.gmail.com","threadId":"57055","inReplyTo":"581b723792c2bec1ba66025c7ebb0344d5541865.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-26T15:08:07Z","receivedAt":"2022-01-26T15:09:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 25 2022, Taylor Blau wrote:\n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index d05ab716f6..f775fc1ce6 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -397,4 +397,32 @@ test_expect_success 'pack.preferBitmapTips' '\n> \t)\n> '\n> \n> +test_expect_success 'complains about multiple pack bitmaps' '\n> +\trm -fr repo &&\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n\nNit: the \"rm -rf\" isn't needed per the comment on 1/9...\n\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit base &&\n> +\n> +\t\tgit repack -adb &&\n> +\t\tbitmap=\"$(ls .git/objects/pack/pack-*.bitmap)\" &&\n> +\t\tmv \"$bitmap\" \"$bitmap.bak\" &&\n> +\n> +\t\ttest_commit other &&\n> +\t\tgit repack -ab &&\n> +\n> +\t\tmv \"$bitmap.bak\" \"$bitmap\" &&\n> +\n> +\t\tfind .git/objects/pack -type f -name \"*.pack\" >packs &&\n> +\t\tfind .git/objects/pack -type f -name \"*.bitmap\" >bitmaps &&\n> +\t\ttest_line_count = 2 packs &&\n> +\t\ttest_line_count = 2 bitmaps &&\n> +\n> +\t\tgit rev-list --use-bitmap-index HEAD 2>err &&\n> +\t\tgrep \"ignoring extra bitmap file\" err\n> +\t)\n> +'\n> +\n> test_done\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> index c0924074c4..3c1ecc7e25 100755\n> --- a/t/t5326-multi-pack-bitmaps.sh\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -266,4 +266,23 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '\n> \t)\n> '\n> \n> +test_expect_success 'graceful fallback when missing reverse index' '\n> +\trm -fr repo &&\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\t(\n\n...and here either, where we're adding a new test to the same file.\n"},{"id":"446971","messageId":"220126.86lez2edae.gmgdl@evledraar.gmail.com","threadId":"57055","inReplyTo":"94563cf038722852d801bd0a8e744a569799f354.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 8/9] midx: read `RIDX` chunk when present","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-26T15:10:08Z","receivedAt":"2022-01-26T15:18:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 25 2022, Taylor Blau wrote:\n\n> @@ -298,9 +298,26 @@ int load_midx_revindex(struct multi_pack_index *m)\n> {\n> \tstruct strbuf revindex_name = STRBUF_INIT;\n> \tint ret;\n> +\n\nnit: good addition for style in general, but a stay whitespace change in\nan otherwise narrowly focused patch.\n\n> \tif (m->revindex_data)\n> \t\treturn 0;\n> \n> +\tif (m->chunk_revindex) {\n> +\t\t/*\n> +\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n> +\t\t * the reverse index instead of trying to load a separate `.rev`\n> +\t\t * file.\n> +\t\t *\n> +\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n> +\t\t * not want to accidentally call munmap() in the middle of the\n> +\t\t * MIDX.\n> +\t\t */\n> +\t\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> +\t\t\t\t \"source\", \"midx\");\n> +\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n> +\t\treturn 0;\n> +\t}\n> +\n> \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> \t\t\t \"source\", \"rev\");\n\nThis trace2_data_string() is repeated with just \"midx\" v.s. \"rev\"\nparameters.\n\nI was going to suggest that maybe we should add a \"goto\" here, since\nyou're trying to juggle the early return and logging this.\n\nBut wouldn't this be both simpler and give you better logging if it was\nthe usual region start/end pattern, with just the \"data string\"\nin-between?\n\nThen you'd get both performance data on the index loading, and the\nsource.\n\nOr maybe it's overkill in this case, I don't know...\n\n"},{"id":"446980","messageId":"220126.86czkee675.gmgdl@evledraar.gmail.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-26T17:50:27Z","receivedAt":"2022-01-26T17:51:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 25 2022, Taylor Blau wrote:\n\n> Here is a(n even) small(er) reroll of my series which fixes a serious problem\n> with MIDX bitmaps by which they can become corrupt when permuting their pack\n> order.\n>\n> The only change is to revert back to using finalize_object_file(), since it\n> behaves consistently with other parts of the code that touch $GIT_DIR/objects.\n> This is safe to do given the other change described in that patch. The\n> description has been updated slightly to reflect.\n>\n> A minor tweak to the tests is done towards the end of the series. But otherwise\n> this is unchanged from v3.\n>\n> It is prepared on the tip of master (which is 89bece5c8c at the time of\n> writing).\n\nI gave this some light reading earlier, and left a few nit-y comments\nalong the way. This isn't my area of expertise, but without giving it\nmuch of a deep look this all looked good to me.\n\nI don't see a reason not to carry this forward, with or without\naddressing any of the nits I brought up.\n\nThanks for working on this.\n"},{"id":"446991","messageId":"YfGsfdHjYMBb2JWH@nand.local","threadId":"57055","inReplyTo":"220126.8635lafskv.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 1/9] t5326: demonstrate bitmap corruption after permutation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-26T20:18:05Z","receivedAt":"2022-01-26T20:18:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 26, 2022 at 04:01:20PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Tue, Jan 25 2022, Taylor Blau wrote:\n>\n> > +test_expect_failure 'changing the preferred pack does not corrupt bitmaps' '\n> > +\trm -fr repo &&\n> > +\tgit init repo &&\n> > +\ttest_when_finished \"rm -fr repo\" &&\n>\n> Nit: The initial \"rm -fr\" isn't needed here, and we should aim to have\n> tests clean up after themselves, not needing to clean up after other\n> tests.\n>\n> This appears to have been copy/pasted from the test you added in\n> 54156af0d66 (t5326: test propagating hashcache values, 2021-09-17),\n> which needlessly used that pattern, the tests you added preceding it\n> follow the \"clean up your own mess\" pattern.\n\nI don't think I copy/pasted it, but just have it embedded deep into my\nmuscle-memory from every time that I've written:\n\n rm -fr repo\n git init repo\n cd repo\n\ninto $HOME/a.sh, which I use as a dumping ground for little tests.\n\nYes, I agree it's useless here, but it is consistent with the style (for\nbetter or worse).\n\nIt's fine if Junio wants to remove it when queueing, but I don't think\nit's a big enough deal to matter here (probably not in general, either,\nbut I wouldn't be opposed to a concerted effort to clean these up across\nall of `t` if that's something you're interested in).\n\nThanks,\nTaylor\n"},{"id":"446992","messageId":"YfGs4WG+S9f4/eqW@nand.local","threadId":"57055","inReplyTo":"220126.86tudqeduk.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 5/9] t5326: extract `test_rev_exists`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-26T20:19:45Z","receivedAt":"2022-01-26T20:19:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 26, 2022 at 04:04:37PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Tue, Jan 25 2022, Taylor Blau wrote:\n>\n> I left a comment on 3/9, but re $subject: It really suggests we're just\n> refactoring some code to a function (\"extract\"...), but really we're\n> using the new trace2 event added earlier here to test things in a new\n> way.\n\nI can see what you're saying. I'd rather not squash the two patches\ntogether (because it makes sense to me as-is, and I'd much rather get\nthis bug fixed than split hairs on the patch structure), but if you feel\nstrongly I would do it.\n\nThanks,\nTaylor\n"},{"id":"446993","messageId":"YfGtruutEDPZlvTE@nand.local","threadId":"57055","inReplyTo":"220126.86lez2edae.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 8/9] midx: read `RIDX` chunk when present","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-26T20:23:10Z","receivedAt":"2022-01-26T20:23:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 26, 2022 at 04:10:08PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Tue, Jan 25 2022, Taylor Blau wrote:\n>\n> > @@ -298,9 +298,26 @@ int load_midx_revindex(struct multi_pack_index *m)\n> > {\n> > \tstruct strbuf revindex_name = STRBUF_INIT;\n> > \tint ret;\n> > +\n>\n> nit: good addition for style in general, but a stay whitespace change in\n> an otherwise narrowly focused patch.\n>\n> > \tif (m->revindex_data)\n> > \t\treturn 0;\n> >\n> > +\tif (m->chunk_revindex) {\n> > +\t\t/*\n> > +\t\t * If the MIDX `m` has a `RIDX` chunk, then use its contents for\n> > +\t\t * the reverse index instead of trying to load a separate `.rev`\n> > +\t\t * file.\n> > +\t\t *\n> > +\t\t * Note that we do *not* set `m->revindex_map` here, since we do\n> > +\t\t * not want to accidentally call munmap() in the middle of the\n> > +\t\t * MIDX.\n> > +\t\t */\n> > +\t\ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> > +\t\t\t\t \"source\", \"midx\");\n> > +\t\tm->revindex_data = (const uint32_t *)m->chunk_revindex;\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > \ttrace2_data_string(\"load_midx_revindex\", the_repository,\n> > \t\t\t \"source\", \"rev\");\n>\n> This trace2_data_string() is repeated with just \"midx\" v.s. \"rev\"\n> parameters.\n>\n> I was going to suggest that maybe we should add a \"goto\" here, since\n> you're trying to juggle the early return and logging this.\n\nRight; that matches how we handle the two cases separately here. I'm not\nsure exactly what you're suggesting with by adding a \"goto\" label; I\nthink it would make things more awkward rather than more readable, but\nit's certainly possible that I'm missing something.\n\n> But wouldn't this be both simpler and give you better logging if it was\n> the usual region start/end pattern, with just the \"data string\"\n> in-between?\n>\n> Then you'd get both performance data on the index loading, and the\n> source.\n>\n> Or maybe it's overkill in this case, I don't know...\n\nI think it's overkill; in the \"embedded\" case (which we expect to be\npretty common in the future) we're not really measuring anything except\nfor the pointer assignment.\n\nWe *could* instrument pack-revindex.c::load_revindex_from_disk(), but I\nthink that's a separate issue from this one.\n\nThanks,\nTaylor\n"},{"id":"446995","messageId":"YfGuC0wowQnYi/Eq@nand.local","threadId":"57055","inReplyTo":"220126.86czkee675.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-26T20:24:43Z","receivedAt":"2022-01-26T20:24:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 26, 2022 at 06:50:27PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> I gave this some light reading earlier, and left a few nit-y comments\n> along the way. This isn't my area of expertise, but without giving it\n> much of a deep look this all looked good to me.\n\nThanks so much!\n\n> I don't see a reason not to carry this forward, with or without\n> addressing any of the nits I brought up.\n\nI responded to your comments throughout this thread, but the \"executive\nsummary\" is that I think we're OK to continue with this version as-is.\n\nCould I get an ACK from either Stolee or Jonathan Tan that this version\nlooks OK?\n\nThanks,\nTaylor\n"},{"id":"446997","messageId":"20220126210031.2855730-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"YfBuUBOjWpvVOE/9@nand.local","subject":"Re: [PATCH v3 7/9] t/lib-bitmap.sh: parameterize tests over reverse index source","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-26T21:00:31Z","receivedAt":"2022-01-26T21:00:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> > In a separate commit after this one, we should do it for the other\n> > blocks.\n> \n> But I'm not as certain about this one. The tests are already separated\n> into two different test scripts (t5326 for the in-MIDX .rev file, and\n> t5327 for the external .rev file). So it should already be clear based\n> on what test script you're running.\n\nAh, yes, you're right.\n"},{"id":"446999","messageId":"20220126212847.2860166-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"YfBvXS841To7ZXcp@nand.local","subject":"Re: [PATCH v3 8/9] midx: read `RIDX` chunk when present","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-26T21:28:47Z","receivedAt":"2022-01-26T21:28:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> On Mon, Jan 24, 2022 at 11:27:01AM -0800, Jonathan Tan wrote:\n> > > Note that we have two knobs that are adjusted for the new tests:\n> > > GIT_TEST_MIDX_WRITE_REV and GIT_TEST_MIDX_READ_RIDX. The former controls\n> > > whether the MIDX .rev is written at all, and the latter controls whether\n> > > we read the MIDX's RIDX chunk.\n> > >\n> > > Both are necessary to ensure that the test added at the beginning of\n> > > this series continues to work. This is because we always need to write\n> > > the RIDX chunk in the MIDX in order to change its checksum, but we want\n> > > to make sure reading the existing .rev file still works (since the RIDX\n> > > chunk takes precedence by default).\n> >\n> > Could we disable the beginning-of-this-series test when the MIDX .rev is\n> > not written instead? Then, we can test what the user would actually\n> > experience (no reverse index in .midx, reverse index in .rev) instead of\n> > simulating it by skipping over the reverse index section in .midx.\n> >\n> > If we can't do that, then I would agree that the approach in this patch\n> > seems like the best approach.\n> \n> I considered it, i.e., by having the two knobs be:\n> \n> - GIT_TEST_MIDX_WRITE_REV\n> - GIT_TEST_MIDX_WRITE_RIDX\n> \n> where the pair (true, false) would correspond to what a corrupt\n> repository would look like before this series. But I dislike that it\n> allows the caller to alter the MIDX's checksum by controlling whether or\n> not the chunk is written.\n\nI think that if the user really wanted to manipulate the checksum by\nusing an environment variable for Git tests, they could just as easily\ndo so by using an old version of Git.\n\n> So it's really looking at the same problem from two sides: do you make\n> the RIDX chunk disappear by not reading it, or by never writing it in\n> the first place? And although the latter is more \"accurate\", it did\n> allow me to sidestep a lot of gory details like the ones I outlined in\n> the second patch.\n> \n> I don't remember everything fully, since some time has passed since I\n> originally wrote this, but I remember encountering some of the races\n> where you'd read the old bitmap in the new object order, and other\n> annoyances.\n\nI think that feeling the need to avoid the gory details and races in\ntest is an argument for also avoiding it in production (by not\nsupporting external .rev files any more), but I've already explained my\nopinion on this and will leave it to others to decide.\n\nIn any case, I still think that this patch set in its current form is\nworth merging - it is a strict improvement over what we had before (the\nrace condition I'm worried about has been there before this patch set).\n"},{"id":"447075","messageId":"8842fdd9-438d-43dc-b7b5-0d6bb456e313@gmail.com","threadId":"57055","inReplyTo":"cover.1643150456.git.me@ttaylorr.com","subject":"Re: [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2022-01-27T14:13:55Z","receivedAt":"2022-01-27T14:13:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/25/2022 5:40 PM, Taylor Blau wrote:\n> Here is a(n even) small(er) reroll of my series which fixes a serious problem\n> with MIDX bitmaps by which they can become corrupt when permuting their pack\n> order.\n> \n> The only change is to revert back to using finalize_object_file(), since it\n> behaves consistently with other parts of the code that touch $GIT_DIR/objects.\n> This is safe to do given the other change described in that patch. The\n> description has been updated slightly to reflect.\n> \n> A minor tweak to the tests is done towards the end of the series. But otherwise\n> this is unchanged from v3.\n\nIt's been a while since I've looked at this topic, so I re-read the\npatches and the discussion on this version. I found the patches to\nbe ready to go and the comments that exist to be interesting style\npoints, but not valuable enough to hold up the series.\n\nIf you _do_ decide to re-roll, then please add\n\nReviewed-by: Derrick Stolee \n\nThanks,\n-Stolee\n"},{"id":"447100","messageId":"20220127171509.3073226-1-jonathantanmy@google.com","threadId":"57055","inReplyTo":"YfGuC0wowQnYi/Eq@nand.local","subject":"Re: [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-01-27T17:15:09Z","receivedAt":"2022-01-27T17:15:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> Could I get an ACK from either Stolee or Jonathan Tan that this version\n> looks OK?\n\nI think Stolee has already given his ACK [1]. Here's my ACK:\n\nReviewed-by: Jonathan Tan \n\n[1] https://lore.kernel.org/git/8842fdd9-438d-43dc-b7b5-0d6bb456e313@gmail.com/\n"},{"id":"449526","messageId":"YhgLz5tAgLYLMPks@nand.local","threadId":"57055","inReplyTo":"20220127171509.3073226-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-02-24T22:50:55Z","receivedAt":"2022-02-24T22:50:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 27, 2022 at 09:15:09AM -0800, Jonathan Tan wrote:\n> Taylor Blau writes:\n> > Could I get an ACK from either Stolee or Jonathan Tan that this version\n> > looks OK?\n>\n> I think Stolee has already given his ACK [1]. Here's my ACK:\n>\n> Reviewed-by: Jonathan Tan \n\nThanks, both. Junio picked this up via f2cb46a6b3 (Merge branch\n'tb/midx-bitmap-corruption-fix', 2022-02-16), which is on master.\n\nHopefully this is the last that any of us will have to think about this\nbug ;-).\n\n> [1] https://lore.kernel.org/git/8842fdd9-438d-43dc-b7b5-0d6bb456e313@gmail.com/\n\nThanks,\nTaylor\n"}]}