# [PATCH] rev-list: skip bitmap traversal for --left-right 6 messages from 2024-11-01 to 2025-02-02. Participants: Jeff King, Taylor Blau, Ayush Chandekar. Thread: https://gitlist.dev/t/62435 ## Jeff King, 2024-11-01 12:16 Subject: [PATCH] rev-list: skip bitmap traversal for --left-right Message-ID: <20241101121606.GA2327410@coredump.intra.peff.net> URL: https://gitlist.dev/e/20241101121606.GA2327410%40coredump.intra.peff.net ``` Running: git rev-list --left-right --use-bitmap-index one...two will produce output without any left-right markers, since the bitmap traversal returns only a single set of reachable commits. Instead we should refuse to use bitmaps here and produce the correct output using a traditional traversal. This is probably not the only remaining option that misbehaves with bitmaps, but it's particularly egregious in that it feels like it _could_ work. Doing two separate traversals for the left/right sides and then taking the symmetric set differences should yield the correct answer, but our traversal code doesn't know how to do that. It's not clear if naively doing two separate traversals would always be a performance win. A traditional traversal only needs to walk down to the merge base, but bitmaps always fill out the full reachability set. So depending on your bitmap coverage, we could end up walking old bits of history twice to fill out the same uninteresting bits on both sides. We'd also of course end up with a very large --boundary set, if the user asked for that. So this might or might not be something worth implementing later. But for now, let's make sure we don't produce the wrong answer if somebody tries it. The test covers this, but also the same thing with "--count" (which is what I originally tried in a real-world case). Ironically the try_bitmap_count() code already realizes that "--left-right" won't work there. But that just causes us to fall back to the regular bitmap traversal code, which itself doesn't handle counting (we produce a list of objects rather than a count). Signed-off-by: Jeff King --- builtin/rev-list.c | 7 +++++++ t/t5310-pack-bitmaps.sh | 12 ++++++++++++ 2 files changed, 19 insertions(+) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index f62bcbf2b1..3078787115 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -485,6 +485,13 @@ static int try_bitmap_traversal(struct rev_info *revs, if (revs->max_count >= 0) return -1; + /* + * We can't know which commits were left/right in a single traversal, + * and we don't yet know how to traverse them separately. + */ + if (revs->left_right) + return -1; + bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects); if (!bitmap_git) return -1; diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index 7044c7d7c6..6bcbea64cc 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -503,6 +503,18 @@ test_expect_success 'boundary-based traversal is used when requested' ' done ' +test_expect_success 'left-right not confused by bitmap index' ' + git rev-list --left-right other...HEAD >expect && + git rev-list --use-bitmap-index --left-right other...HEAD >actual && + test_cmp expect actual +' + +test_expect_success 'left-right count not confused by bitmap-index' ' + git rev-list --left-right --count other...HEAD >expect && + git rev-list --use-bitmap-index --left-right --count other...HEAD >actual && + test_cmp expect actual +' + test_bitmap_cases "pack.writeBitmapLookupTable" test_expect_success 'verify writing bitmap lookup table when enabled' ' -- 2.47.0.441.g1a09955689 ``` ## Taylor Blau, 2024-11-01 14:40 Subject: Re: [PATCH] rev-list: skip bitmap traversal for --left-right Message-ID: URL: https://gitlist.dev/e/ZyToaRxR7upLg0IE%40nand.local In-Reply-To: <20241101121606.GA2327410@coredump.intra.peff.net> ``` On Fri, Nov 01, 2024 at 08:16:06AM -0400, Jeff King wrote: > builtin/rev-list.c | 7 +++++++ > t/t5310-pack-bitmaps.sh | 12 ++++++++++++ > 2 files changed, 19 insertions(+) Nice find, and well explained (not just why it doesn't work today but could in the future, but why making it work in the future with bitmaps does not necessarily a clear performance improvement). Probably you and I should think more about other rev-list options that *don't* work with --use-bitmap-index. I share your feeling there that there probably exist such options which silently do nothing (or the wrong thing) in the presence of '--use-bitmap-index'. Let's merge this one down. Thanks, Taylor ``` ## Jeff King, 2024-11-04 17:41 Subject: Re: [PATCH] rev-list: skip bitmap traversal for --left-right Message-ID: <20241104174126.GB2985568@coredump.intra.peff.net> URL: https://gitlist.dev/e/20241104174126.GB2985568%40coredump.intra.peff.net In-Reply-To: ``` On Fri, Nov 01, 2024 at 10:40:41AM -0400, Taylor Blau wrote: > Nice find, and well explained (not just why it doesn't work today but > could in the future, but why making it work in the future with bitmaps > does not necessarily a clear performance improvement). > > Probably you and I should think more about other rev-list options that > *don't* work with --use-bitmap-index. I share your feeling there that > there probably exist such options which silently do nothing (or the > wrong thing) in the presence of '--use-bitmap-index'. I'm pretty sure --cherry-pick and --cherry-mark are examples, but I suspect it's the tip of the iceberg. I don't know if it's worth spending much effort on this. It's certainly a wart, but there's a certain amount of "if it hurts, don't do it". The --left-right one bugged me so much just because I thought it _would_ work. ;) -Peff ``` ## Jeff King, 2024-11-04 17:46 Subject: Re: [PATCH] rev-list: skip bitmap traversal for --left-right Message-ID: <20241104174628.GA2990236@coredump.intra.peff.net> URL: https://gitlist.dev/e/20241104174628.GA2990236%40coredump.intra.peff.net In-Reply-To: <20241104174126.GB2985568@coredump.intra.peff.net> ``` On Mon, Nov 04, 2024 at 12:41:27PM -0500, Jeff King wrote: > On Fri, Nov 01, 2024 at 10:40:41AM -0400, Taylor Blau wrote: > > > Nice find, and well explained (not just why it doesn't work today but > > could in the future, but why making it work in the future with bitmaps > > does not necessarily a clear performance improvement). > > > > Probably you and I should think more about other rev-list options that > > *don't* work with --use-bitmap-index. I share your feeling there that > > there probably exist such options which silently do nothing (or the > > wrong thing) in the presence of '--use-bitmap-index'. > > I'm pretty sure --cherry-pick and --cherry-mark are examples, but I > suspect it's the tip of the iceberg. I don't know if it's worth spending > much effort on this. It's certainly a wart, but there's a certain amount > of "if it hurts, don't do it". The --left-right one bugged me so much > just because I thought it _would_ work. ;) Actually, I wonder if this could be an interesting #leftoverbits or intern mini-project: 1. Look at the set of rev-list options and pick one. 2. Repack a repo (say, git.git) with reachability bitmaps using "git repack -adb". 3. Compare output of "rev-list --your-option" with and without --use-bitmap-index. If the outputs do not reasonably match, add code to try_bitmap_traversal() to forbid it and bail to the non-bitmap path. Maybe also other try_bitmap_*() functions, as well. It's a little more involved than other miniprojects just because there may be some judgement needed for "reasonably match". Being byte-for-byte identical is good, but in some cases the output will be different (e.g., bitmap output is in a different order, lacks path annotations, etc). So it would help in step (1) if you understand what the option is supposed to do and how you'd expect it to work with bitmaps. ;) -Peff ``` ## Taylor Blau, 2024-11-04 18:34 Subject: Re: [PATCH] rev-list: skip bitmap traversal for --left-right Message-ID: URL: https://gitlist.dev/e/ZykTwaWaveGGvapw%40nand.local In-Reply-To: <20241104174628.GA2990236@coredump.intra.peff.net> ``` On Mon, Nov 04, 2024 at 12:46:28PM -0500, Jeff King wrote: > On Mon, Nov 04, 2024 at 12:41:27PM -0500, Jeff King wrote: > > > On Fri, Nov 01, 2024 at 10:40:41AM -0400, Taylor Blau wrote: > > > > > Nice find, and well explained (not just why it doesn't work today but > > > could in the future, but why making it work in the future with bitmaps > > > does not necessarily a clear performance improvement). > > > > > > Probably you and I should think more about other rev-list options that > > > *don't* work with --use-bitmap-index. I share your feeling there that > > > there probably exist such options which silently do nothing (or the > > > wrong thing) in the presence of '--use-bitmap-index'. > > > > I'm pretty sure --cherry-pick and --cherry-mark are examples, but I > > suspect it's the tip of the iceberg. I don't know if it's worth spending > > much effort on this. It's certainly a wart, but there's a certain amount > > of "if it hurts, don't do it". The --left-right one bugged me so much > > just because I thought it _would_ work. ;) > > Actually, I wonder if this could be an interesting #leftoverbits or > intern mini-project: I agree; that would be a good project for interns. Great idea! Thanks, Taylor ``` ## Ayush Chandekar, 2025-02-02 12:28 Subject: Re: [PATCH] rev-list: skip bitmap traversal for --left-right Message-ID: <20250202122837.335101-1-ayu.chandekar@gmail.com> URL: https://gitlist.dev/e/20250202122837.335101-1-ayu.chandekar%40gmail.com In-Reply-To: ``` Hey, I found this issue quite interesting. Would you like to discuss a bit more, maybe about how to implement it? I am looking forward to solving this. Regards, Ayush ```