Re: [Internet]Re: reachability-bitmap makes push performance worse ?

From: Derrick Stolee <derrickstolee@github.com>
Date: Jun 14, 2022, 14:22 UTC
Message-ID: <78ce313d-a93c-ca2a-12f9-a6afbd6f15a2@github.com>
In-Reply-To: <d92e5a26e7664d7ab8ab1215ecaea6bf@tencent.com>

On 6/14/2022 7:00 AM, kylezhao(赵柯宇) wrote:

Show 25 quoted lines

>> This is a known issue, I think you've found the same problem discussed in these past threads:
>>
>> https://lore.kernel.org/git/38b99459158a45b1bea09037f3dd092d@exmbdft7.ad.twosigma.com/
>> https://lore.kernel.org/git/87zhoz8b9o.fsf@evledraar.gmail.com/
> 
> Thanks.
> 
>> The latter one in particular has a lot of extra details. The former also
>> has the suggestion of a per-push bitmap configuration as a workaround.
>>
>> As your numbers show it's still an issue today, but those threads should
>> help you if you're looking to dig further into the root cause.
>>
>> Aside from the underlying root causes it would be very nice to fix the
>> progress code in that area, i.e. we "stall" on "Enumerating objects",
>> which is just a matter of us not having a separate progress bar for the
>> very expensive bitmap work we're doing.
> 
> It looks like optimizing the bitmap to solve the problem will be a long
> process. This requires developers to have a deep understanding of the
> algorithm.
> 
> A per-push bitmap configuration as a workaround can't completely solve the
> problem, but it works for me. After all, bitmap was not designed to optimize
> git push. Most of time, git push is not been called as frequently as git fetch.

I think the issue is that bitmaps are designed to support servers, which don't exactly use "git push" but instead use "git upload-pack" with a very different type of data (a lot of branches simultaneously from a large variety of bases).

In general, clients that use "git push" don't generally have bitmaps, so this has not been a priority. For clients, it is faster to do a more focused object walk. See these commits and blog post for more details:

* d5d2e93577 (revision: implement sparse algorithm, 2019-01-16)
* 3d036eb0d2 (pack-objects: create pack.useSparse setting, 2019-01-16)
* https://devblogs.microsoft.com/devops/exploring-new-frontiers-for-git-push-performance/

Hopefully that gives enough context as to why one would want to disable bitmaps for most "git push" operations.

> The problem has been around for 3 years, has the community considered providing
> a config like "push.useBitmap" to prevent git push using bitmap? It would be
> appreciated if there is such a config, which can quickly solve my problem and
> doesn't seem like a lot of work.

I think this config would be a good idea, and I would even argue that we might want to set it to "false" by default.

> If no one is interested in it, I can also try to submit a patch (although it
> may be a bit slow since all I am new to the git community).

We would welcome the contribution! I look forward to seeing it when you're ready.

Thanks, -Stolee