From: Patrick Steinhardt <ps@pks.im>
Date: Tue, 06 Jan 2026 06:53:34 GMT
Subject: Re: Slow git pack-refs --all
Message-ID: <aVyxbqk-2QQIgDXK@pks.im>
In-Reply-To: <CH3PR12MB90260C4887067C88629BBE52C286A@CH3PR12MB9026.namprd12.prod.outlook.com>

Hi Martin,

On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> OK, after discovering the strace -r and -T options, I have determined that
> the 29K writes were all very fast in themselves. However, most of the
> writes seem to follow each other with no other system calls in between.
> This explains why it looks like the writes are slow, even though they aren't.
> 
> If I tally up the time between the previous system call, and each write(),
> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> pack refs. This tells me that no visible I/O or system calls are the problem,
> but rather that the program itself is taking a long time between writes.
> I very much doubt that this is heavy CPU time, but rather I am going to 
> guess that this is hidden system time spent accessing mmaped memory. 
> Could it be really slow reading the packed-refs file? I can see the 
> packed-refs file is mmaped() before the writes start, and then 
> munmapped after the writes are completed. If I had to guess, that likely
> means that the packed-refs file is being read in small increments by the 
> kernel via mmap, and that is what is making things very slow over NFS. 

I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
be a constant source of issues, which is why we eventually sunsetted the
use of it completely. Do you use any special flags for mounting the NFS
filesystem?

> My alternative theory, is that each ref is being looked up via a binary 
> search, but I don't think git does this?

Did you try using perf(1) to profile the process and generate a flame
graph from it? That should likely make it immediately obvious where Git
is spending all of its time.

Patrick