threads / discuss / 64681

Slow git pack-refs --all

Subject: Slow git pack-refs --all

## tl;dr

19 messages between Dec 25, 2025 and Jan 16, 2026.

replies: 18people: 4as markdown or json

Martin Fick· Dec 25, 2025, 22:13 UTC · lore
I was hoping to get some help debugging a busy large repository where git pack-refs --all tends to regularly take over 5mins to run in production. As you can imagine this is particularly problematic on a busy Gerrit server since it tends to hold the packed-refs.lock file for most of this duration. Any help is greatly appreciated, see the details below.
-Martin
What did you do before the bug happened? (Steps to reproduce your issue)
I have a large repository (~90M objects, ~50GB, ~3M refs) which is regularly (every ~2hours) repacked and maintained, but generally gets at least 300+ updates per maintenance cycle.
What did you expect to happen? (Expected behavior)
git pack-refs --all to complete in under 20s when there are only 200 loose refs
What happened instead? (Actual behavior)
git pack-refs --all takes more than 3 minutes
What's different between what you expected and what actually happened?
This is much slower than expected
Anything else you want to add:
Although the packed-refs file is large, copying it takes less than 1s, so there isn't a writing throughput issue with the filesystem. Additionally, jgit can pack-refs --all in under 20s on the same repo, so I don't believe there is an issue locking the 200 loose refs either. When observing the filesystem, I do see the packed-refs.new growing at a rate that seems slower than expected as if much more is happening while writing this file, than just writing the file.
An strace shows about 200+ open("./objects..") calls interspersed between around ~26K write() calls. I am surprised to see pack-refs reading objects at all.
Although the repository is not in terrible shape before packing refs (~1500 loose objects, 37pack files). Surprisingly, repacking the repo first does speed it up so that packing refs then takes under 20s.
This repository is on NFS.

[System Info] git version: git version 2.45.2 cpu: x86_64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh uname: Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 compiler info: gnuc: 14.1 libc info: glibc: 2.17 $SHELL (typically, interactive shell): /bin/bash

[Enabled Hooks] not run from a git repository - no hooks to show

brian m. carlson· Dec 25, 2025, 23:38 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

On 2025-12-25 at 22:13:54, Martin Fick wrote:
Show 11 quoted lines
> Although the packed-refs file is large, copying it takes less than 1s,
> so there isn't a writing throughput issue with the filesystem.
> Additionally, jgit can pack-refs --all in under 20s on the same repo,
> so I don't believe there is an issue locking the 200 loose refs
> either. When observing the filesystem, I do see the packed-refs.new
> growing at a rate that seems slower than expected as if much more is
> happening while writing this file, than just writing the file.
> 
> An strace shows about 200+ open("./objects..") calls interspersed
> between around ~26K write() calls. I am surprised to see pack-refs
> reading objects at all.
I think this is from `should_pack_ref`:
    /* Do not pack broken refs: */
    if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
    	return 0;

So Git is going to need to verify that the object at least exists. I don't know why we would need to _open_ them, however. Perhaps someone else has ideas.

Show 5 quoted lines
> Although the repository is not in terrible shape before packing refs
> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo
> first does speed it up so that packing refs then takes under 20s.
> 
> This repository is on NFS.

That's almost certainly part of your performance problem, too. Loading a single pack file and index is going to be way, way faster than making lots of network calls to open 37 pack file and 37 index files, plus at least stat some loose objects.

I will note that at least some forges always have Git write pack files and try to avoid loose objects altogether since that almost always improves performance. You may want to set `receive.unpackLimit` to 1 to see if that helps in the general case.

-- 
brian m. carlson (they/them)
Toronto, Ontario, CA
Jeff King· Dec 26, 2025, 04:45 UTC · re: brian m. carlson · lore

Re: Slow git pack-refs --all

On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:
Show 9 quoted lines
> I think this is from `should_pack_ref`:
> 
>     /* Do not pack broken refs: */
>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
>     	return 0;
> 
> So Git is going to need to verify that the object at least exists.  I
> don't know why we would need to _open_ them, however.  Perhaps someone
> else has ideas.

The packed-refs file stores tag-peeling information. So pack-refs opens the object for any newly written ref via peel_object(), which has to at least read the header to get the type. That call happens via write_with_updates() in packed-backend.c.

  If we wanted to be really pedantic, anything in refs/heads/ should not
  point to a non-commit and thus should never need to be peeled. I'm not
  sure if we want to embed that assumption in this code path, though
  (nor would it necessarily help Martin's case if the refs are not in
  refs/heads anyway).
-Peff
brian m. carlson· Dec 26, 2025, 17:15 UTC · re: Jeff King · lore

Re: Slow git pack-refs --all

On 2025-12-26 at 04:45:07, Jeff King wrote:
Show 22 quoted lines
> On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:
> 
> > I think this is from `should_pack_ref`:
> > 
> >     /* Do not pack broken refs: */
> >     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
> >     	return 0;
> > 
> > So Git is going to need to verify that the object at least exists.  I
> > don't know why we would need to _open_ them, however.  Perhaps someone
> > else has ideas.
> 
> The packed-refs file stores tag-peeling information. So pack-refs opens
> the object for any newly written ref via peel_object(), which has to at
> least read the header to get the type. That call happens via
> write_with_updates() in packed-backend.c.
> 
>   If we wanted to be really pedantic, anything in refs/heads/ should not
>   point to a non-commit and thus should never need to be peeled. I'm not
>   sure if we want to embed that assumption in this code path, though
>   (nor would it necessarily help Martin's case if the refs are not in
>   refs/heads anyway).

I don't think that would be a good idea. I know that people definitely do updates of the loose refs by hand (although they should not) and so it's entirely possible for them to contain invalid values, such as having branches contain non-commit objects.

I wonder if reftable would avoid the need for this kind of expensive check since it would already have the data peeled if need be and wouldn't need to recompute the values.

-- 
brian m. carlson (they/them)
Toronto, Ontario, CA
Jeff King· Dec 27, 2025, 07:36 UTC · re: brian m. carlson · lore

Re: Slow git pack-refs --all

On Fri, Dec 26, 2025 at 05:15:31PM +0000, brian m. carlson wrote:
Show 10 quoted lines
> >   If we wanted to be really pedantic, anything in refs/heads/ should not
> >   point to a non-commit and thus should never need to be peeled. I'm not
> >   sure if we want to embed that assumption in this code path, though
> >   (nor would it necessarily help Martin's case if the refs are not in
> >   refs/heads anyway).
> 
> I don't think that would be a good idea.  I know that people definitely
> do updates of the loose refs by hand (although they should not) and so
> it's entirely possible for them to contain invalid values, such as
> having branches contain non-commit objects.
Yeah, that matches my inclination.
> I wonder if reftable would avoid the need for this kind of expensive
> check since it would already have the data peeled if need be and
> wouldn't need to recompute the values.

It does the same amount of peeling, but it's amortized across more operations (i.e., whatever did those ref updates in the first place) rather than during the pack operation. And of course there really is no pack operation per se with reftables, but I believe it avoids re-peeling when rewriting entries during compaction.

It might actually do fewer object accesses overall if the ref-writing operations have already loaded the objects in question (and thus it knows whether they're tags or not, and may even have parsed tags in memory). It can also do more in some cases (e.g., two loose writes will peel for each write, whereas the files backend only bothers to peel during packing).

-Peff
Martin Fick· Dec 31, 2025, 05:48 UTC · re: Jeff King · lore

Re: Slow git pack-refs --all

Show 18 quoted lines
> From: Jeff King <peff@peff.net>
> Sent: Thursday, December 25, 2025 9:45 PM
>
> On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:
>> I think this is from `should_pack_ref`:
>>
>>     /* Do not pack broken refs: */
>>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
>>       return 0;
>>
>> So Git is going to need to verify that the object at least exists.  I
>> don't know why we would need to _open_ them, however.  Perhaps someone
>> else has ideas.
>
>The packed-refs file stores tag-peeling information. So pack-refs opens
>the object for any newly written ref via peel_object(), which has to at
>least read the header to get the type. That call happens via
>write_with_updates() in packed-backend.c.
Thanks, this makes sense. However, since jgit needs to peel these objects also, it doesn't make sense to me that this would be the bottleneck unless git is doing something terribly inefficient here. :(
Except for the fact that repacking objects made it faster, my observations make it look like it's the writing that is actually slow, not the reads. Could there be too many small unbuffered writes, could this write path have missed being optimized (it likely isn't used elsewhere)?
-Martin
Jeff King· Jan 2, 2026, 07:49 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:
Show 18 quoted lines
> >> I think this is from `should_pack_ref`:
> >>
> >>     /* Do not pack broken refs: */
> >>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
> >>       return 0;
> >>
> >> So Git is going to need to verify that the object at least exists.  I
> >> don't know why we would need to _open_ them, however.  Perhaps someone
> >> else has ideas.
> >
> >The packed-refs file stores tag-peeling information. So pack-refs opens
> >the object for any newly written ref via peel_object(), which has to at
> >least read the header to get the type. That call happens via
> >write_with_updates() in packed-backend.c.
> 
> Thanks, this makes sense. However, since jgit needs to peel these
> objects also, it doesn't make sense to me that this would be the
> bottleneck unless git is doing something terribly inefficient here. :(

I'd expect both git and jgit to open each loose object once. I tried running "git pack-refs --all --prune" under strace on a test repo. It does seem to open the object once. Then I tried the same with jgit (though it does not understand --prune), and got the same results.

So...I dunno.
Show 5 quoted lines
> Except for the fact that repacking objects made it faster, my
> observations make it look like it's the writing that is actually slow,
> not the reads. Could there be too many small unbuffered writes, could
> this write path have missed being optimized (it likely isn't used
> elsewhere)?

All of the packed-refs writes are through fprintf(), which should be fully buffered. You should be able to confirm with strace (I get 4096-byte writes on my system).

If writing were slow, I'd also expect that to scale with the total number of refs, not the number of changed refs (since we have to rewrite the whole file, but only new entries need to be peeled).

-Peff
Martin Fick· Jan 5, 2026, 23:45 UTC · re: Jeff King · lore

Re: Slow git pack-refs --all

> From: Jeff King <peff@peff.net>
>  Sent: Friday, January 2, 2026 12:49 AM
> On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:
> > Except for the fact that repacking objects made it faster, my

I have now confirmed that repacking does NOT actually make things faster. I believe filesystem caching interfered with much of my testing. Using echo 3 > /proc/sys/vm/drop_caches has helped to get more consistent results.

By repacking to get one used, and one cruft pack only, and no loose objects, I have confirmed that pack-refs it is still slow. This rules out the idea that the loose object, or pack file counts were making things slow.

Show 8 quoted lines
> > observations make it look like it's the writing that is actually slow,
> > not the reads. Could there be too many small unbuffered writes, could
> > this write path have missed being optimized (it likely isn't used
> > elsewhere)?
>
> All of the packed-refs writes are through fprintf(), which should be
> fully buffered. You should be able to confirm with strace (I get
> 4096-byte writes on my system).
I can confirm that my system is actually printing 8192 bytes at a time.
> If writing were slow, I'd also expect that to scale with the total
> number of refs, not the number of changed refs (since we have to rewrite
> the whole file, but only new entries need to be peeled).

OK, after discovering the strace -r and -T options, I have determined that the 29K writes were all very fast in themselves. However, most of the writes seem to follow each other with no other system calls in between. This explains why it looks like the writes are slow, even though they aren't.

If I tally up the time between the previous system call, and each write(), it adds up to the bulk of the time (4mins out of 4m15s) that it takes to pack refs. This tells me that no visible I/O or system calls are the problem, but rather that the program itself is taking a long time between writes. I very much doubt that this is heavy CPU time, but rather I am going to guess that this is hidden system time spent accessing mmaped memory. Could it be really slow reading the packed-refs file? I can see the packed-refs file is mmaped() before the writes start, and then munmapped after the writes are completed. If I had to guess, that likely means that the packed-refs file is being read in small increments by the kernel via mmap, and that is what is making things very slow over NFS. My alternative theory, is that each ref is being looked up via a binary search, but I don't think git does this?

-Martin
Patrick Steinhardt· Jan 6, 2026, 06:53 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

Hi Martin,
On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
Show 16 quoted lines
> OK, after discovering the strace -r and -T options, I have determined that
> the 29K writes were all very fast in themselves. However, most of the
> writes seem to follow each other with no other system calls in between.
> This explains why it looks like the writes are slow, even though they aren't.
> 
> If I tally up the time between the previous system call, and each write(),
> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> pack refs. This tells me that no visible I/O or system calls are the problem,
> but rather that the program itself is taking a long time between writes.
> I very much doubt that this is heavy CPU time, but rather I am going to 
> guess that this is hidden system time spent accessing mmaped memory. 
> Could it be really slow reading the packed-refs file? I can see the 
> packed-refs file is mmaped() before the writes start, and then 
> munmapped after the writes are completed. If I had to guess, that likely
> means that the packed-refs file is being read in small increments by the 
> kernel via mmap, and that is what is making things very slow over NFS.

I wouldn't be surprised if NFS was the culprit. At GitLab we found it to be a constant source of issues, which is why we eventually sunsetted the use of it completely. Do you use any special flags for mounting the NFS filesystem?

> My alternative theory, is that each ref is being looked up via a binary 
> search, but I don't think git does this?

Did you try using perf(1) to profile the process and generate a flame graph from it? That should likely make it immediately obvious where Git is spending all of its time.

Patrick
Martin Fick· Jan 6, 2026, 23:02 UTC · re: Patrick Steinhardt · lore

Re: Slow git pack-refs --all

Show 23 quoted lines
> From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > OK, after discovering the strace -r and -T options, I have determined that
> > the 29K writes were all very fast in themselves. However, most of the
> > writes seem to follow each other with no other system calls in between.
> > This explains why it looks like the writes are slow, even though they aren't.
> >
> > If I tally up the time between the previous system call, and each write(),
> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > pack refs. This tells me that no visible I/O or system calls are the problem,
> > but rather that the program itself is taking a long time between writes.
> > I very much doubt that this is heavy CPU time, but rather I am going to
> > guess that this is hidden system time spent accessing mmaped memory.
> > Could it be really slow reading the packed-refs file? I can see the
> > packed-refs file is mmaped() before the writes start, and then
> > munmapped after the writes are completed. If I had to guess, that likely
> > means that the packed-refs file is being read in small increments by the
> > kernel via mmap, and that is what is making things very slow over NFS.
> 
> I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
> be a constant source of issues, which is why we eventually sunsetted the
> use of it completely. Do you use any special flags for mounting the NFS
> filesystem?

I am open to alternatives to NFS. Do you know of any NFS alternatives that provides instantaneous replication to potentially hundreds of mirrors? I have used Gerrit and git-daemon for many years on NFS, and it generally has performed very well for us, and it solves many real performance issues which I have yet to find a viable alternative able to even come close to matching. NFS with all it warts it is for us (and likely will be for many) until there is a viable enterprise ready alternative with low (zero) replication latency and high throughput.

That being said, NFS can cause many issues. In this case, I would say that something is particularly "broken" here with git, and I believe that it would be helpful to the git community to be aware of this fairly specific broken case which clearly has a lot of room for improvement (as seen by the fact that jgit, in java, can do essentially the same thing more than 10Xs faster). While I have been mostly assuming that this is a particularly specific bad case since git daemon generally is fast for most users, this might actually be something that if improved would greatly improve many parts of git (not just this use case).

It would be nice to improve git to not hold the packed-refs.lock so long to avoid this blocking behavior on servers. Of course, to be fair, this likely only blocks Gerrit servers since Gerrit uses the packed-refs file to perform atomic updates for many things, and most other servers use loose refs instead. It would be great if git were optimized to avoid any unnecessary reads while the lock is held. In theory, almost all of the data that git needs to read here (including tags for peeling) could be read before acquiring the lock, and it would only need to double check certain reads after it acquires the lock in case things changed. That wouldn't make git pack-refs faster, but it would drastically reduce the impact of any problematic I/O by not holding the lock for almost the entire operation.

> Did you try using perf(1) to profile the process and generate a flame
> graph from it? That should likely make it immediately obvious where Git
> is spending all of its time.

I will pursue this. Unfortunately this might be difficult on this particular server.

Thanks for the feedback, and suggestions,
-Martin
Patrick Steinhardt· Jan 7, 2026, 11:42 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
Show 32 quoted lines
> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > > OK, after discovering the strace -r and -T options, I have determined that
> > > the 29K writes were all very fast in themselves. However, most of the
> > > writes seem to follow each other with no other system calls in between.
> > > This explains why it looks like the writes are slow, even though they aren't.
> > >
> > > If I tally up the time between the previous system call, and each write(),
> > > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > > pack refs. This tells me that no visible I/O or system calls are the problem,
> > > but rather that the program itself is taking a long time between writes.
> > > I very much doubt that this is heavy CPU time, but rather I am going to
> > > guess that this is hidden system time spent accessing mmaped memory.
> > > Could it be really slow reading the packed-refs file? I can see the
> > > packed-refs file is mmaped() before the writes start, and then
> > > munmapped after the writes are completed. If I had to guess, that likely
> > > means that the packed-refs file is being read in small increments by the
> > > kernel via mmap, and that is what is making things very slow over NFS.
> > 
> > I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
> > be a constant source of issues, which is why we eventually sunsetted the
> > use of it completely. Do you use any special flags for mounting the NFS
> > filesystem?
> 
> I am open to alternatives to NFS. Do you know of any NFS alternatives that 
> provides instantaneous replication to potentially hundreds of mirrors? I 
> have used Gerrit and git-daemon for many years on NFS, and it generally 
> has performed very well for us, and it solves many real performance issues 
> which I have yet to find a viable alternative able to even come close to
> matching. NFS with all it warts it is for us (and likely will be for many) until 
> there is a viable enterprise ready alternative with low (zero) replication 
> latency and high throughput.

Yeah, agreed, NFS can get you a long way, until you eventually start to hit some road blocks once you reach a certain scale. Unfortunately though, there isn't really a ready-made alternative solution that serves your needs, or at least none that I know of. That's why GitLab eventually settled on Gitaly Cluster with Praefect handling replication, and why GitHub has its Spokes architecture that does basically the same thing.

Show 9 quoted lines
> That being said, NFS can cause many issues. In this case, I would say that
> something is particularly "broken" here with git, and I believe that it
> would be helpful to the git community to be aware of this fairly specific 
> broken case which clearly has a lot of room for improvement (as seen
> by the fact that jgit, in java, can do essentially the same thing more 
> than 10Xs faster). While I have been mostly assuming that this is a 
> particularly specific bad case since git daemon generally is fast for most
> users, this might actually be something that if improved would greatly 
> improve many parts of git (not just this use case).

Chances are that if we can improve the case for NFS, other filesystems might benefit, as well. So if this is something that we can improve I agree that we should. It's too early to tell though, as we don't really know what the actual root cause is just yet.

Show 12 quoted lines
> It would be nice to improve git to not hold the packed-refs.lock so long 
> to avoid this blocking behavior on servers. Of course, to be fair, this 
> likely only blocks Gerrit servers since Gerrit uses the packed-refs file to 
> perform atomic updates for many things, and most other servers use 
> loose refs instead. It would be great if git were optimized to avoid any 
> unnecessary reads while the lock is held.  In theory, almost all of the 
> data that git needs to read here (including tags for peeling) could be 
> read before acquiring the lock, and it would only need to double 
> check certain reads after it acquires the lock in case things changed. 
> That wouldn't make git pack-refs faster, but it would drastically 
> reduce the impact of any problematic I/O by not holding the lock for 
> almost the entire operation.

It can probably be improved, true. I think that it's a bit of a wasted effort, as I'd rather invest the time into improving reftables as a more future-proof solution. But as you are well aware I'm quite biased here, and I'd welcome any efforts to also improve the files backend. I am just unlikely to work on it myself :)

Show 6 quoted lines
> > Did you try using perf(1) to profile the process and generate a flame
> > graph from it? That should likely make it immediately obvious where Git
> > is spending all of its time.
> 
> I will pursue this. Unfortunately this might be difficult on this 
> particular server.
True, on the server side this can be a bit tricky.
Patrick
Martin Fick· Jan 7, 2026, 22:58 UTC · re: Patrick Steinhardt · lore

Re: Slow git pack-refs --all

> From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM
On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
Show 9 quoted lines
> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > > Did you try using perf(1) to profile the process and generate a flame
> > > graph from it? That should likely make it immediately obvious where Git
> > > is spending all of its time.
> >
> > I will pursue this. Unfortunately this might be difficult on this
> > particular server.
> 
> True, on the server side this can be a bit tricky.

I ran perf, and got a flame graph, I am not sure what the best way to share that is, but I will try to summarize what looked important:

About one third of the time is in this section:
libc-2.17.so 32.5%
 _memcmp_sse4_1 29.8%
 page_fault 7.23%
 ...
I am not really sure what that is doing?
Another third is doing:
unpack_object_header_buffer 30%
 page_fault 26.9%
 ...
 nfs_read_page 10%

Which could very well be looking at the headers of objects to see if they are tags needing to be peeled?

And the remaining third was a bit all over the place with small sections, the largest two of those sections being:

packed_refs_store_create ~8.7%
 unknown 4.4%
 memchr 4.4%
 page_fault 4.4%
nth_packed_object_offset 7%
 page_fault 3.2%

This was way less informative (to me) then I would have hoped. :( Maybe this means more to you? 

It does look like a lot of page_faults, likely due to the use of mmap?
-Martin
Patrick Steinhardt· Jan 8, 2026, 06:33 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

On Wed, Jan 07, 2026 at 10:58:36PM +0000, Martin Fick wrote:
Show 34 quoted lines
> > From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM
> On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
> > > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > > > Did you try using perf(1) to profile the process and generate a flame
> > > > graph from it? That should likely make it immediately obvious where Git
> > > > is spending all of its time.
> > >
> > > I will pursue this. Unfortunately this might be difficult on this
> > > particular server.
> > 
> > True, on the server side this can be a bit tricky.
> 
> I ran perf, and got a flame graph, I am not sure what the best way to share that
> is, but I will try to summarize what looked important:
> 
> About one third of the time is in this section:
> 
> libc-2.17.so 32.5%
>  _memcmp_sse4_1 29.8%
>  page_fault 7.23%
>  ...
> 
> I am not really sure what that is doing?
> 
> 
> Another third is doing:
> 
> unpack_object_header_buffer 30%
>  page_fault 26.9%
>  ...
>  nfs_read_page 10%
> 
> Which could very well be looking at the headers of objects to see if they are 
> tags needing to be peeled?

Both of these are lacking some information to be able to tell. Are you by any chance able to share the whole flame graph? That'd make this a bit easier to figure out.

Show 7 quoted lines
> And the remaining third was a bit all over the place with small sections,
> the largest two of those sections being:
> 
> packed_refs_store_create ~8.7%
>  unknown 4.4%
>  memchr 4.4%
>  page_fault 4.4%

We spend ~9% of time in `packed_refs_store_create()`? That looks seriously broken to me, the function shouldn't even do much.

Show 7 quoted lines
> nth_packed_object_offset 7%
>  page_fault 3.2%
> 
> This was way less informative (to me) then I would have hoped. :( Maybe
> this means more to you? 
> 
> It does look like a lot of page_faults, likely due to the use of mmap?

Certainly looks like the page faults are to blame here overall. It's still surprising to me it's _that_ slow. Quoting the other mail you sent:

On Wed, Jan 07, 2026 at 05:05:53PM +0000, Martin Fick wrote:
> rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev

I know that back when we still supported NFS we recommended to use an rsize and wsize of 1MB to reduce the round trip times.

Patrick
Jeff King· Jan 15, 2026, 21:09 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

On Wed, Jan 07, 2026 at 10:58:36PM +0000, Martin Fick wrote:
Show 11 quoted lines
> I ran perf, and got a flame graph, I am not sure what the best way to share that
> is, but I will try to summarize what looked important:
> 
> About one third of the time is in this section:
> 
> libc-2.17.so 32.5%
>  _memcmp_sse4_1 29.8%
>  page_fault 7.23%
>  ...
> 
> I am not really sure what that is doing?

Probably this is the call to strcmp(iter->ref.name, update->refname) in packed-backend.c:write_with_updates().

We have to write out the new packed-refs file with our updates in sorted order. So it's a big O(n) merge between the existing ones (from the "iter" side) and the new ones (from the "update" side).

It could also be caused by sorting of the packed-refs entries. We generally shouldn't need to do that, but I think I may have found something useful. See below.

Show 7 quoted lines
> unpack_object_header_buffer 30%
>  page_fault 26.9%
>  ...
>  nfs_read_page 10%
> 
> Which could very well be looking at the headers of objects to see if they are 
> tags needing to be peeled?
Yeah, that's what I'd expect here.
Show 7 quoted lines
> And the remaining third was a bit all over the place with small sections,
> the largest two of those sections being:
> 
> packed_refs_store_create ~8.7%
>  unknown 4.4%
>  memchr 4.4%
>  page_fault 4.4%

Hmm, I don't think we have a function "packed_refs_store_create". Did you typo while transferring the name over?

At any rate, we can assume this is poking through the packed-refs file itself, looking for trailing newlines via memchr.

But why would we do that immediately when creating the packed-refs store in memory? In modern versions of Git, we try to avoid reading the packed-refs file as much as possible, binary-searching when we can. Of course that means it has to be sorted, which was not something promised by the original format. So we have a "sorted" tag that we write. E.g., this is from my clone of git, packed with git itself:

  $ head -n 1 .git/packed-refs
  # pack-refs with: peeled fully-peeled sorted
Now let's try something with jgit:
  git init
  git commit --allow-empty -m foo
  git branch foo
  git branch bar
  jgit pack-refs --all
  cat .git/packed-refs
That gives me this:
  # pack-refs with: peeled
  86054aaedc64c24aec8aaad988f6979a3cb82ee0 refs/heads/bar
  86054aaedc64c24aec8aaad988f6979a3cb82ee0 refs/heads/foo
  86054aaedc64c24aec8aaad988f6979a3cb82ee0 refs/heads/main

Aha! So jgit is not writing out the "sorted" tag. As a result, when git reads the file, its logic is:

  1. Check for the sorted tag. It's not here, so...
  2. Check if the file is sorted by reading each entry linearly. If it's
     not, then...
  3. Read it all into memory and sort the result. We can then
     binary-search that (and iterate it in sorted order, which is
     important for pack-refs).

So when git reads the packed-refs file, we are ending up at least with step 2, an extra pass through the whole file, and maybe to step 3 (depending on whether jgit actually sorts the file).

You mentioned that Gerrit writes the packed-refs file directly itself, presumably using jgit. So it sounds like it is constantly undoing Git's "sorted" marker, which causes git-pack-refs to spend extra effort checking the sortedness, and rewrite the marker, which then gets hosed again by jgit, and so on.

And that may explain why jgit is faster, if it is not doing the extra sort check. If it is not even trying to maintain the sorted property that it would be faster still (it takes one linear pass while writing out the file, omitting entries that match our updates, and then appends our updates at the end).

If jgit _is_ sorting the file but not writing out the sorted marker, then it should start doing so. ;)

If it's not sorting the file, then probably it should start doing so (and writing the marker). This will make subsequent reads much faster (mmap + binary-search). It shouldn't even be slower to write (assuming jgit's writes are doing the usual "rewrite the whole thing to a tempfile and atomic-rename into place", and not taking some shortcut by appending to the file).

Unrelated to your problem, but also jgit should support the fully-peeled tag, another thing that makes readers faster. ;)

The jgit version I'm using is:
  $ jgit version
  jgit version 7.5.0.202512021534-r

One way you could test this theory is to sort and mark the file yourself, before running "git pack-refs". One easy way to do that is to convince git to rewrite it by removing an entry. I.e., find some ref mentioned in the pack-refs file, and then "git update-ref -d $ref". And check out the first line of .git/packed-refs before and after. If it goes faster (and similarly fast to jgit) only when the "sorted" tag appears, then that would be our culprit.

-Peff
Martin Fick· Jan 16, 2026, 20:35 UTC · re: Jeff King · lore

Re: Slow git pack-refs --all

Show 12 quoted lines
> From: Jeff King <peff@peff.net> Sent: Thursday, January 15, 2026 2:09 PM
> > ...
> > And the remaining third was a bit all over the place with small sections,
> > the largest two of those sections being:
> >
> > packed_refs_store_create ~8.7%
> >  unknown 4.4%
> >  memchr 4.4%
> >  page_fault 4.4%
> 
> Hmm, I don't think we have a function "packed_refs_store_create". Did
> you typo while transferring the name over?
Yes, this should have been packed_ref_store_create (singular), sorry.
Show 37 quoted lines
> At any rate, we can assume this is poking through the packed-refs file
> itself, looking for trailing newlines via memchr.
> 
> But why would we do that immediately when creating the packed-refs store
> in memory? In modern versions of Git, we try to avoid reading the
> packed-refs file as much as possible, binary-searching when we can. Of
> course that means it has to be sorted, which was not something promised
> by the original format. So we have a "sorted" tag that we write. E.g.,
> this is from my clone of git, packed with git itself:
> ... 
>   # pack-refs with: peeled fully-peeled sorted
> ...
>   jgit pack-refs --all
> 
> That gives me this:
> 
>   # pack-refs with: peeled
> ...
>  Aha! So jgit is not writing out the "sorted" tag. As a result, when git
> reads the file, its logic is:
> 
>   1. Check for the sorted tag. It's not here, so...
>   2. Check if the file is sorted by reading each entry linearly. If it's
>     not, then...
>   3. Read it all into memory and sort the result. We can then
>      binary-search that (and iterate it in sorted order, which is
>      important for pack-refs).
> 
> So when git reads the packed-refs file, we are ending up at least with
> step 2, an extra pass through the whole file, and maybe to step 3
> (depending on whether jgit actually sorts the file).
> 
> You mentioned that Gerrit writes the packed-refs file directly itself,
> presumably using jgit. So it sounds like it is constantly undoing Git's
> "sorted" marker, which causes git-pack-refs to spend extra effort
> checking the sortedness, and rewrite the marker, which then gets hosed
> again by jgit, and so on.
Agreed, this is likely the case, but not for the sorted marker, see below...
Show 5 quoted lines
> And that may explain why jgit is faster, if it is not doing the extra
> sort check. If it is not even trying to maintain the sorted property
> that it would be faster still (it takes one linear pass while writing
> out the file, omitting entries that match our updates, and then appends
> our updates at the end).
Unfortunately, this does not actually seem to be the reason.
> If jgit _is_ sorting the file but not writing out the sorted marker,
> then it should start doing so. ;)

Agreed, I will see to it that this gets fixed. Unfortunately, adding the sorted tag does not seem to speed things up. :(

Show 6 quoted lines
> If it's not sorting the file, then probably it should start doing so
> (and writing the marker). This will make subsequent reads much faster
> (mmap + binary-search). It shouldn't even be slower to write (assuming
> jgit's writes are doing the usual "rewrite the whole thing to a tempfile
> and atomic-rename into place", and not taking some shortcut by appending
> to the file).

FYI, jgit does seem to order things, it does not append. The resulting output from jgit after a repack with new refs add matches that from git for all but the header.

> Unrelated to your problem, but also jgit should support the fully-peeled
> tag, another thing that makes readers faster. ;)

Ironically, this is not just related, it appears to be the trigger!!! When I add this tag (and not the sorted tag), the cache flushed time drops down to under 4s (from over 5mins)!

I will see to it that jgit fixes this too. That should help solve my problem.

That being said, it seems like something is still broken in git here despite this tag being missing?

Thanks so much Peff for helping get to this point!
-Martin
Martin Fick· Jan 7, 2026, 17:05 UTC · re: Patrick Steinhardt · lore

Re: Slow git pack-refs --all

Show 20 quoted lines
> From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > OK, after discovering the strace -r and -T options, I have determined that
> > the 29K writes were all very fast in themselves. However, most of the
> > writes seem to follow each other with no other system calls in between.
> > This explains why it looks like the writes are slow, even though they aren't.
> 
> > If I tally up the time between the previous system call, and each write(),
> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > pack refs. This tells me that no visible I/O or system calls are the problem,
> > but rather that the program itself is taking a long time between writes.
> > I very much doubt that this is heavy CPU time, but rather I am going to
> > guess that this is hidden system time spent accessing mmaped memory.
> > Could it be really slow reading the packed-refs file? I can see the
> > packed-refs file is mmaped() before the writes start, and then
> > munmapped after the writes are completed. If I had to guess, that likely
> > means that the packed-refs file is being read in small increments by the
> > kernel via mmap, and that is what is making things very slow over NFS.
> 
> ... Do you use any special flags for mounting the NFS filesystem?
Oh sorry, I forgot to reply to this last time. We use the following mount flags:
rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev
-Martin
Jeff King· Jan 6, 2026, 10:38 UTC · re: Martin Fick · lore

Re: Slow git pack-refs --all

On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> By repacking to get one used, and one cruft pack only, and no loose 
> objects, I have confirmed that pack-refs it is still slow. This rules out the 
> idea that the loose object, or pack file counts were making things slow.

OK, that is interesting. I'd still expect opening the objects to be the dominating factor, but now the load would be on jumping around the mmap'd packfile rather than open/read/close calls.

Show 11 quoted lines
> OK, after discovering the strace -r and -T options, I have determined that
> the 29K writes were all very fast in themselves. However, most of the
> writes seem to follow each other with no other system calls in between.
> This explains why it looks like the writes are slow, even though they aren't.
> 
> If I tally up the time between the previous system call, and each write(),
> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> pack refs. This tells me that no visible I/O or system calls are the problem,
> but rather that the program itself is taking a long time between writes.
> I very much doubt that this is heavy CPU time, but rather I am going to 
> guess that this is hidden system time spent accessing mmaped memory.

That would be consistent with reading object data from the packfile. We'll jump around within the packfile to get that data.

Show 5 quoted lines
> Could it be really slow reading the packed-refs file? I can see the 
> packed-refs file is mmaped() before the writes start, and then 
> munmapped after the writes are completed. If I had to guess, that likely
> means that the packed-refs file is being read in small increments by the 
> kernel via mmap, and that is what is making things very slow over NFS.

The packed-refs file is mmap'd, but we'll be reading it sequentially. I guess whether or not there is good read-ahead there may depend on the NFS implementation.

> My alternative theory, is that each ref is being looked up via a binary 
> search, but I don't think git does this?

Git does binary search within the packed-refs file, but it shouldn't be doing so here. The write-out phase of packing refs is a straight merge between two lists: the existing packed-refs entries and the new entries we are adding.

I'd second Patrick's suggestion to use perf or similar to try to see where the time is going.

You might also try building Git with NO_MMAP. That might make the I/O costs more apparent via strace, because they'll be coming via pread().

-Peff
Martin Fick· Jan 6, 2026, 23:03 UTC · re: Jeff King · lore

Re: Slow git pack-refs --all

From: Jeff King <peff@peff.net> Sent: Tuesday, January 6, 2026 3:38 AM
Show 8 quoted lines
> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > By repacking to get one used, and one cruft pack only, and no loose
> > objects, I have confirmed that pack-refs it is still slow. This rules out the
> > idea that the loose object, or pack file counts were making things slow.
> 
> OK, that is interesting. I'd still expect opening the objects to be the
> dominating factor, but now the load would be on jumping around the
> mmap'd packfile rather than open/read/close calls.
I believe I have confirmed this now with more testing...

By first dropping the system caches, and then catting the pack file to /dev/null, it sped things up to under 20s!

Note that neither catting the idx, nor the packed-refs file helped to noticeably speed things up.

Show 14 quoted lines
> > OK, after discovering the strace -r and -T options, I have determined that
> > the 29K writes were all very fast in themselves. However, most of the
> > writes seem to follow each other with no other system calls in between.
> > This explains why it looks like the writes are slow, even though they aren't.
>
> > If I tally up the time between the previous system call, and each write(),
> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > pack refs. This tells me that no visible I/O or system calls are the problem,
> > but rather that the program itself is taking a long time between writes.
> > I very much doubt that this is heavy CPU time, but rather I am going to
> > guess that this is hidden system time spent accessing mmaped memory.
> 
> That would be consistent with reading object data from the packfile.
> We'll jump around within the packfile to get that data.
Agreed, but boy is that really bad performance!
Show 9 quoted lines
> > Could it be really slow reading the packed-refs file? I can see the
> > packed-refs file is mmaped() before the writes start, and then
> > munmapped after the writes are completed. If I had to guess, that likely
> > means that the packed-refs file is being read in small increments by the
> > kernel via mmap, and that is what is making things very slow over NFS.
> 
> The packed-refs file is mmap'd, but we'll be reading it sequentially. I
> guess whether or not there is good read-ahead there may depend on the
> NFS implementation.

Yeah, ruled out now by dropping the system caches, and then catting the packed-refs file before running git pack-refs, which did NOT help speed things up.

Show 7 quoted lines
> > My alternative theory, is that each ref is being looked up via a binary
> > search, but I don't think git does this?
> 
> Git does binary search within the packed-refs file, but it shouldn't be
> doing so here. The write-out phase of packing refs is a straight merge
> between two lists: the existing packed-refs entries and the new entries
> we are adding.

Agreed, and I should have ruled this out by realizing that this would likely not have been affected by the system caches in my earlier tests.

> I'd second Patrick's suggestion to use perf or similar to try to see
> where the time is going.
Noted, thanks.
> You might also try building Git with NO_MMAP. That might make the I/O
> costs more apparent via strace, because they'll be coming via pread().

Agreed, I will try to do this. I think that the jgit results hint that this this might even eliminate most of the I/O costs (jgit is not using MMAP in my tests). It would be nice if this were a runtime config instead of requiring a rebuild, as some use cases might be better with, and some without MMAP.

Thanks for all the input,
-Martin
Martin Fick· Dec 31, 2025, 05:39 UTC · re: brian m. carlson · lore

Re: Slow git pack-refs --all

Show 8 quoted lines
>From: brian m. carlson
> Sent: Thursday, December 25, 2025 4:38 PM
> On 2025-12-25 at 22:13:54, Martin Fick wrote:
>> Although the repository is not in terrible shape before packing refs
>> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo
>> first does speed it up so that packing refs then takes under 20s.
>>
>> This repository is on NFS.
Show 9 quoted lines
> That's almost certainly part of your performance problem, too.  Loading
> a single pack file and index is going to be way, way faster than making
> lots of network calls to open 37 pack file and 37 index files, plus at
> least stat some loose objects.
>
> I will note that at least some forges always have Git write pack files
> and try to avoid loose objects altogether since that almost always
> improves performance.  You may want to set `receive.unpackLimit` to 1 to
> see if that helps in the general case.
This would not explain why jgit can pack-refs much faster, since it has to deal with these NFS latencies also. In my experience with jgit, we generally don't see performance issues unless packfile counts exceed 300 or so on NFS, definitely not with only 37 of them. It could be that git is doing some things less efficiently here, but I would be pretty surprised if git could not also perform well typically on NFS with only 37 packfiles. I don't think that 200 something object lookups, should ever take 3+mins, even on NFS, and even with way more packfiles than this. To be that slow, it would have to take about 1s per lookup! Something really seems fishy here to me. I have to think that somehow it isn't these reads that are slow, but I can't explain what else it would be?
-Martin

← back to recent threads