Re: Slow git pack-refs --all

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Dec 25, 2025, 23:38 UTC
Message-ID: <aU3K9lGbHw68Vv5U@fruit.crustytoothpaste.net>
In-Reply-To: <CH3PR12MB9026B5872FD42F031970074BC2B3A@CH3PR12MB9026.namprd12.prod.outlook.com>

On 2025-12-25 at 22:13:54, Martin Fick wrote:

Show 11 quoted lines

> Although the packed-refs file is large, copying it takes less than 1s,
> so there isn't a writing throughput issue with the filesystem.
> Additionally, jgit can pack-refs --all in under 20s on the same repo,
> so I don't believe there is an issue locking the 200 loose refs
> either. When observing the filesystem, I do see the packed-refs.new
> growing at a rate that seems slower than expected as if much more is
> happening while writing this file, than just writing the file.
> 
> An strace shows about 200+ open("./objects..") calls interspersed
> between around ~26K write() calls. I am surprised to see pack-refs
> reading objects at all.

I think this is from `should_pack_ref`:

    /* Do not pack broken refs: */
    if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
    	return 0;

So Git is going to need to verify that the object at least exists. I don't know why we would need to _open_ them, however. Perhaps someone else has ideas.

Show 5 quoted lines

> Although the repository is not in terrible shape before packing refs
> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo
> first does speed it up so that packing refs then takes under 20s.
> 
> This repository is on NFS.

That's almost certainly part of your performance problem, too. Loading a single pack file and index is going to be way, way faster than making lots of network calls to open 37 pack file and 37 index files, plus at least stat some loose objects.

I will note that at least some forges always have Git write pack files and try to avoid loose objects altogether since that almost always improves performance. You may want to set `receive.unpackLimit` to 1 to see if that helps in the general case.

-- 
brian m. carlson (they/them)
Toronto, Ontario, CA