Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests

From: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Date: Aug 10, 2022, 10:04 UTC
Message-ID: <CAPOJW5w2NYbRkFOaqrNYVFkp5ud=aAxhGGV6gpdDPwnyx5TAVw@mail.gmail.com>
In-Reply-To: <4rs1s351-73np-4sq8-p6o8-r7178rp0p0n0@tzk.qr>

On Wed, Aug 10, 2022 at 2:50 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 5 quoted lines

>
> Hi Abhradeep,
> I instrumented this, and saw that the `multi-pack-index` and
> `multi-pack-index*.bitmap` files were unchanged by the `git repack`
> invocation.

Yeah, those two files remain unchanged here.

Show 24 quoted lines

> Re-generating the MIDX bitmap forcefully after the repack seems to fix
> things over here:
>
> -- snip --
> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh
> index a95537e759b..564124bda27 100644
> --- a/t/lib-bitmap.sh
> +++ b/t/lib-bitmap.sh
> @@ -438,7 +438,10 @@ midx_bitmap_partial_tests () {
>
>         test_expect_success 'setup partial bitmaps' '
>                 test_commit packed &&
> +ls -l .git/objects/pack/ &&
>                 git repack &&
> +git multi-pack-index write --bitmap &&
> +ls -l .git/objects/pack/ &&
>                 test_commit loose &&
>                 git multi-pack-index write --bitmap 2>err &&
>                 test_path_is_file $midx &&
> -- snap --
>
> This suggests to me that the `multi-pack-index write --bitmap 2>err` call
> in this hunk might reuse a stale MIDX bitmap, and that _that_  might be
> the root cause of this breakage.

Yeah, the `multi-pack-index write --bitmap 2>err` is creating the problem. More specifically the `multi-pack-index write` part. As you can see in my previous comment (if you get the comment), I shared a screenshot there which pointed out that the multi-pack-index files in both cases are different. The portion from which it started to differ belongs to the `RIDX` chunk.

So, I used some debug lines in `midx_pack_order()` function[1] and found that the objects are sorted differently in those cases (i.e. passing case and failing case). For passing case, the RIDX chunk contents are like below -

pack_order = [ 1, 36, 11, 6, 18, 3, 19, 12, 5, 31, 27, 23, 29, 8, 38, 22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17, 32, 0, 21, 16, 25, 13, 40, 20,]

And in the failing case, this is -

pack_order = [ 12, 18, 3, 19, 1, 36, 11, 6, 5, 31, 27, 23, 29, 8, 38, 22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17, 32, 0, 21, 16, 25, 13, 40, 20,]

I went further and realized that this is due to the line[2] -

    if (!e->preferred)
        data[i].pack |= (1U << 31);

I.e. 4- 5 `pack_midx_entry` objects have different `preferred` values in those cases. For example, "46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b" (with pack order 12) is `preferred` in failing case (that's why it is in the first position) and the same is `not preferred` in the passing case.

It may be because of reusing a stale midx bitmap (as you said). But I am not sure. Just to ensure myself, I compared all the other packfiles, idx files and a pack `.bitmap` file (which you can see using ls command) of failing and passing cases and found that they are the same.

Thanks :)

[1] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/midx.c#L861 [2] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/midx.c#L872