Re: Slow git pack-refs --all

From: Martin Fick <mfick@nvidia.com>
Date: Jan 5, 2026, 23:45 UTC
Message-ID: <CH3PR12MB90260C4887067C88629BBE52C286A@CH3PR12MB9026.namprd12.prod.outlook.com>
In-Reply-To: <20260102074901.GD2581074@coredump.intra.peff.net>

> From: Jeff King <peff@peff.net>
>  Sent: Friday, January 2, 2026 12:49 AM
> On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:
> > Except for the fact that repacking objects made it faster, my

I have now confirmed that repacking does NOT actually make things faster. I believe filesystem caching interfered with much of my testing. Using echo 3 > /proc/sys/vm/drop_caches has helped to get more consistent results.

By repacking to get one used, and one cruft pack only, and no loose objects, I have confirmed that pack-refs it is still slow. This rules out the idea that the loose object, or pack file counts were making things slow.

Show 8 quoted lines

> > observations make it look like it's the writing that is actually slow,
> > not the reads. Could there be too many small unbuffered writes, could
> > this write path have missed being optimized (it likely isn't used
> > elsewhere)?
>
> All of the packed-refs writes are through fprintf(), which should be
> fully buffered. You should be able to confirm with strace (I get
> 4096-byte writes on my system).

I can confirm that my system is actually printing 8192 bytes at a time.

> If writing were slow, I'd also expect that to scale with the total
> number of refs, not the number of changed refs (since we have to rewrite
> the whole file, but only new entries need to be peeled).

OK, after discovering the strace -r and -T options, I have determined that the 29K writes were all very fast in themselves. However, most of the writes seem to follow each other with no other system calls in between. This explains why it looks like the writes are slow, even though they aren't.

If I tally up the time between the previous system call, and each write(), it adds up to the bulk of the time (4mins out of 4m15s) that it takes to pack refs. This tells me that no visible I/O or system calls are the problem, but rather that the program itself is taking a long time between writes. I very much doubt that this is heavy CPU time, but rather I am going to guess that this is hidden system time spent accessing mmaped memory. Could it be really slow reading the packed-refs file? I can see the packed-refs file is mmaped() before the writes start, and then munmapped after the writes are completed. If I had to guess, that likely means that the packed-refs file is being read in small increments by the kernel via mmap, and that is what is making things very slow over NFS. My alternative theory, is that each ref is being looked up via a binary search, but I don't think git does this?

-Martin