Re: Slow git pack-refs --all

From: Patrick Steinhardt <ps@pks.im>
Date: Jan 8, 2026, 06:33 UTC
Message-ID: <aV9PwnouO8652XQR@pks.im>
In-Reply-To: <CH3PR12MB9026C8C940270F02CEF83C4FC284A@CH3PR12MB9026.namprd12.prod.outlook.com>

On Wed, Jan 07, 2026 at 10:58:36PM +0000, Martin Fick wrote:

Show 34 quoted lines

> > From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM
> On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
> > > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > > > Did you try using perf(1) to profile the process and generate a flame
> > > > graph from it? That should likely make it immediately obvious where Git
> > > > is spending all of its time.
> > >
> > > I will pursue this. Unfortunately this might be difficult on this
> > > particular server.
> > 
> > True, on the server side this can be a bit tricky.
> 
> I ran perf, and got a flame graph, I am not sure what the best way to share that
> is, but I will try to summarize what looked important:
> 
> About one third of the time is in this section:
> 
> libc-2.17.so 32.5%
>  _memcmp_sse4_1 29.8%
>  page_fault 7.23%
>  ...
> 
> I am not really sure what that is doing?
> 
> 
> Another third is doing:
> 
> unpack_object_header_buffer 30%
>  page_fault 26.9%
>  ...
>  nfs_read_page 10%
> 
> Which could very well be looking at the headers of objects to see if they are 
> tags needing to be peeled?

Both of these are lacking some information to be able to tell. Are you by any chance able to share the whole flame graph? That'd make this a bit easier to figure out.

Show 7 quoted lines

> And the remaining third was a bit all over the place with small sections,
> the largest two of those sections being:
> 
> packed_refs_store_create ~8.7%
>  unknown 4.4%
>  memchr 4.4%
>  page_fault 4.4%

We spend ~9% of time in `packed_refs_store_create()`? That looks seriously broken to me, the function shouldn't even do much.

Show 7 quoted lines

> nth_packed_object_offset 7%
>  page_fault 3.2%
> 
> This was way less informative (to me) then I would have hoped. :( Maybe
> this means more to you? 
> 
> It does look like a lot of page_faults, likely due to the use of mmap?

Certainly looks like the page faults are to blame here overall. It's still surprising to me it's _that_ slow. Quoting the other mail you sent:

On Wed, Jan 07, 2026 at 05:05:53PM +0000, Martin Fick wrote:

> rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev

I know that back when we still supported NFS we recommended to use an rsize and wsize of 1MB to reduce the round trip times.

Patrick