Slow git pack-refs --all

I was hoping to get some help debugging a busy large repository where git pack-refs --all tends to regularly take over 5mins to run in production. As you can imagine this is particularly problematic on a busy Gerrit server since it tends to hold the packed-refs.lock file for most of this duration. Any help is greatly appreciated, see the details below.

-Martin

What did you do before the bug happened? (Steps to reproduce your issue)

I have a large repository (~90M objects, ~50GB, ~3M refs) which is regularly (every ~2hours) repacked and maintained, but generally gets at least 300+ updates per maintenance cycle.

What did you expect to happen? (Expected behavior)

git pack-refs --all to complete in under 20s when there are only 200 loose refs

What happened instead? (Actual behavior)

git pack-refs --all takes more than 3 minutes

What's different between what you expected and what actually happened?

This is much slower than expected

Anything else you want to add:

Although the packed-refs file is large, copying it takes less than 1s, so there isn't a writing throughput issue with the filesystem. Additionally, jgit can pack-refs --all in under 20s on the same repo, so I don't believe there is an issue locking the 200 loose refs either. When observing the filesystem, I do see the packed-refs.new growing at a rate that seems slower than expected as if much more is happening while writing this file, than just writing the file.

An strace shows about 200+ open("./objects..") calls interspersed between around ~26K write() calls. I am surprised to see pack-refs reading objects at all.

Although the repository is not in terrible shape before packing refs (~1500 loose objects, 37pack files). Surprisingly, repacking the repo first does speed it up so that packing refs then takes under 20s.

This repository is on NFS.

[System Info] git version: git version 2.45.2 cpu: x86_64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh uname: Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 compiler info: gnuc: 14.1 libc info: glibc: 2.17 $SHELL (typically, interactive shell): /bin/bash

[Enabled Hooks] not run from a git repository - no hooks to show