Re: Slow git pack-refs --all

> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > > OK, after discovering the strace -r and -T options, I have determined that
> > > the 29K writes were all very fast in themselves. However, most of the
> > > writes seem to follow each other with no other system calls in between.
> > > This explains why it looks like the writes are slow, even though they aren't.
> > >
> > > If I tally up the time between the previous system call, and each write(),
> > > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > > pack refs. This tells me that no visible I/O or system calls are the problem,
> > > but rather that the program itself is taking a long time between writes.
> > > I very much doubt that this is heavy CPU time, but rather I am going to
> > > guess that this is hidden system time spent accessing mmaped memory.
> > > Could it be really slow reading the packed-refs file? I can see the
> > > packed-refs file is mmaped() before the writes start, and then
> > > munmapped after the writes are completed. If I had to guess, that likely
> > > means that the packed-refs file is being read in small increments by the
> > > kernel via mmap, and that is what is making things very slow over NFS.
> > 
> > I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
> > be a constant source of issues, which is why we eventually sunsetted the
> > use of it completely. Do you use any special flags for mounting the NFS
> > filesystem?
> 
> I am open to alternatives to NFS. Do you know of any NFS alternatives that 
> provides instantaneous replication to potentially hundreds of mirrors? I 
> have used Gerrit and git-daemon for many years on NFS, and it generally 
> has performed very well for us, and it solves many real performance issues 
> which I have yet to find a viable alternative able to even come close to
> matching. NFS with all it warts it is for us (and likely will be for many) until 
> there is a viable enterprise ready alternative with low (zero) replication 
> latency and high throughput.

> That being said, NFS can cause many issues. In this case, I would say that
> something is particularly "broken" here with git, and I believe that it
> would be helpful to the git community to be aware of this fairly specific 
> broken case which clearly has a lot of room for improvement (as seen
> by the fact that jgit, in java, can do essentially the same thing more 
> than 10Xs faster). While I have been mostly assuming that this is a 
> particularly specific bad case since git daemon generally is fast for most
> users, this might actually be something that if improved would greatly 
> improve many parts of git (not just this use case).

> It would be nice to improve git to not hold the packed-refs.lock so long 
> to avoid this blocking behavior on servers. Of course, to be fair, this 
> likely only blocks Gerrit servers since Gerrit uses the packed-refs file to 
> perform atomic updates for many things, and most other servers use 
> loose refs instead. It would be great if git were optimized to avoid any 
> unnecessary reads while the lock is held.  In theory, almost all of the 
> data that git needs to read here (including tags for peeling) could be 
> read before acquiring the lock, and it would only need to double 
> check certain reads after it acquires the lock in case things changed. 
> That wouldn't make git pack-refs faster, but it would drastically 
> reduce the impact of any problematic I/O by not holding the lock for 
> almost the entire operation.

> > Did you try using perf(1) to profile the process and generate a flame
> > graph from it? That should likely make it immediately obvious where Git
> > is spending all of its time.
> 
> I will pursue this. Unfortunately this might be difficult on this 
> particular server.