RE: [External Mail]Re: why git is so slow for a tiny git push?

From: 程洋 <chengyang@xiaomi.com>
Date: Oct 12, 2021, 08:04 UTC
Message-ID: <ef2aa0d3ea8a4d98b910abdfd55191d0@xiaomi.com>
In-Reply-To: <YWRr9g32cMlIc37V@coredump.intra.peff.net>

I have bitmap indeed because my master server also serves as download server. However I'm using git 2.17.0, and I didn't set repack.writeBitmaps

Also I tried "GIT_TRACE2_PERF" and the it is "enumerating objects" cost most of the time. But why bitmaps can cause push to be slow? Do you mean that if writeBitmaps is true, every push will regenerate bitmap file? If that's what you mean, what I see is the only bitmap file in my repo didn't change across time (the modify time is one month ago, long before I run the experiment)

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I came up with an idea, since I found if I decrease the refs number, push goes much faster than before. So I use receive.hiderefs to hide most of refs. And I comment "reject_updates_to_hidden" in receive-pack.c (because I need to update those hide refs, or add new refs)

-----Original Message-----
From: Jeff King <peff@peff.net>
Sent: Tuesday, October 12, 2021 12:53 AM
To: 程洋 <chengyang@xiaomi.com>
Cc: git@vger.kernel.org
Subject: [External Mail]Re: why git is so slow for a tiny git push?

*This message originated from outside of XIAOMI. Please treat this email with caution*

On Sat, Oct 09, 2021 at 06:05:56PM +0000, 程洋 wrote:

Show 7 quoted lines

> I have a really big repository which has 9m objects and maybe 300k refs.
> I noticed that git push is really slow for a tiny change. An example
> shows below
>
> 3 objects which is only 7 kb takes 36 seconds to pack-objects (it's
> the time after i enable pack.usesparse) However if I manually call
> “pack-objects” with the exactly same objects SHA1. It only take less than 0.005 second What is really pass to “pack-objects” when I call “git push”?

Do you have an objects/pack/pack-*.bitmap file on the sending side?

The bitmap code is eager to produce an exact set difference between what is being sent and what the other side has. If you have incomplete bitmap coverage (which is almost a certainty if you have 300k refs), it may do a lot of traversal filling in the "what the other side has" part of the bitmap, even though it does not end up helping the final result in this case.

Bitmaps are enabled by default on bare repos since Git v2.22.0. You can override this with:

  git config repack.writeBitmaps false
  git gc

(or if you don't want to do the gc, you can safely remove the '.bitmap' file).

I notice you used GIT_TRACE_PERFORMANCE below. Try GIT_TRACE2_PERF instead, which goes into detail within particular processes. If this is related to bitmaps, I'd expect the time to go to the "enumerate-objects" region.

-Peff #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#