Re: [External Mail]Re: why git is so slow for a tiny git push?

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Oct 12, 2021, 10:06 UTC
Message-ID: <87pmsak0hl.fsf@evledraar.gmail.com>
In-Reply-To: <YWVJyRJhRTdg39tX@coredump.intra.peff.net>

On Tue, Oct 12 2021, Jeff King wrote:

Show 37 quoted lines

> On Tue, Oct 12, 2021 at 08:04:44AM +0000, 程洋 wrote:
>
>> I have bitmap indeed because my master server also serves as download server.
>> However I'm using git 2.17.0, and I didn't set repack.writeBitmaps
>
> On that version and without the config, then perhaps you (or somebody)
> passed "-b" to git-repack.
>
>> But why bitmaps can cause push to be slow? Do you mean that if
>> writeBitmaps is true, every push will regenerate bitmap file? If
>> that's what you mean, what I see is the only bitmap file in my repo
>> didn't change across time (the modify time is one month ago, long
>> before I run the experiment)
>
> No, it is not regenerating the on-disk bitmaps. But when deciding the
> set of objects to send, pack-objects will generate an internal bitmap
> which is the set difference of objects reachable from the pushed refs,
> minus objects reachable from the refs the other the other side told us
> they had.
>
> It uses the on-disk bitmaps as much as possible, but there may be
> commits not covered by bitmaps (either because they were pushed since
> the last repack which built bitmaps, or simply because it's too
> expensive to put a bitmap on every commit, so we sprinkle them
> throughout the commit history). In those cases we have to traverse parts
> of the object graph by walking commits and opening up trees. This can be
> expensive, and is where your time is going.
>
> Reachability bitmaps _usually_ make things faster, but they have some
> cases where they make things worse (especially if you have a ton of
> refs, or haven't repacked recently).
>
> If bitmaps are causing a problem for your push, they are likely to be
> causing problems for fetches, too. But if you want to keep them to serve
> fetches, but not use them for push, you should be able to do:
>
>   git -c pack.usebitmaps=false push

For the last on-list discussion to (probably the same) problem, which in turn references an even earlier one: https://lore.kernel.org/git/878s6nfq54.fsf@evledraar.gmail.com/

I don't remember if my own report from mid-2019 said so or contradicts this (and I didn't re-read the thread), but FWIW I *vaguely* recall that the case I ran into *might* have had to do with a user running into this on a shared "staging" server.

I.e. one where users logged in as their own user, cd'd to a shared git repo they got a lock on, and ran fetch/push/deploy commands. One user had a pack.useBitmaps=true or equivalent in their config (or had manually run such a repack), so there were some very old stale bitmaps around.

This was also a setup with a gc.bigPackThreshold configured, which now that I think about it might have made it much worse, i.e. we'd keep that *.bitmap on the "big pack" around forever.

But more generally with these side-indexes it seems to me that the code involved might not be considering these sorts of edge cases, i.e. my understanding from you above is that if we have bitmaps anywhere we'll try to in-memory use them for all the objects in play? Or that otherwise having "partial" bitmaps leads to pathological behavior.

tl;dr: We can't assume that a config of "I like to write side-index A/B/C/... when I repack" means that the repo is in that state *now*, but it seems that we do.