{"thread":{"id":"64681","subject":"Slow git pack-refs --all","startedAt":"2025-12-25T22:13:59Z","lastAt":"2026-01-16T20:35:43Z","messageCount":19,"participants":["Martin Fick","brian m. carlson","Jeff King","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"532742","messageId":"CH3PR12MB9026B5872FD42F031970074BC2B3A@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":null,"subject":"Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2025-12-25T22:13:54Z","receivedAt":"2025-12-25T22:13:59Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"I was hoping to get some help debugging a busy large repository where git pack-refs --all tends to regularly take over 5mins to run in production. As you can imagine this is particularly problematic on a busy Gerrit server since it tends to hold the packed-refs.lock file for most of this duration. Any help is greatly appreciated, see the details below.\n\n-Martin\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nI have a large repository (~90M objects, ~50GB, ~3M refs) which is regularly (every ~2hours) repacked and maintained, but generally gets at least 300+ updates per maintenance cycle.\n\nWhat did you expect to happen? (Expected behavior)\n\ngit pack-refs --all to complete in under 20s when there are only 200 loose refs\n\nWhat happened instead? (Actual behavior)\n\ngit pack-refs --all takes more than 3 minutes\n\nWhat's different between what you expected and what actually happened?\n\nThis is much slower than expected\n\nAnything else you want to add:\n\nAlthough the packed-refs file is large, copying it takes less than 1s, so there isn't a writing throughput issue with the filesystem. Additionally, jgit can pack-refs --all in under 20s on the same repo, so I don't believe there is an issue locking the 200 loose refs either. When observing the filesystem, I do see the packed-refs.new growing at a rate that seems slower than expected as if much more is happening while writing this file, than just writing the file.\n\nAn strace shows about 200+ open(\"./objects..\") calls interspersed between around ~26K write() calls. I am surprised to see pack-refs reading objects at all.\n\nAlthough the repository is not in terrible shape before packing refs (~1500 loose objects, 37pack files). Surprisingly, repacking the repo first does speed it up so that packing refs then takes under 20s.\n\nThis repository is on NFS.\n\n[System Info]\ngit version:\ngit version 2.45.2\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64\ncompiler info: gnuc: 14.1\nlibc info: glibc: 2.17\n$SHELL (typically, interactive shell): /bin/bash\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n"},{"id":"532743","messageId":"aU3K9lGbHw68Vv5U@fruit.crustytoothpaste.net","threadId":"64681","inReplyTo":"CH3PR12MB9026B5872FD42F031970074BC2B3A@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-12-25T23:38:30Z","receivedAt":"2025-12-25T23:38:37Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-12-25 at 22:13:54, Martin Fick wrote:\n> Although the packed-refs file is large, copying it takes less than 1s,\n> so there isn't a writing throughput issue with the filesystem.\n> Additionally, jgit can pack-refs --all in under 20s on the same repo,\n> so I don't believe there is an issue locking the 200 loose refs\n> either. When observing the filesystem, I do see the packed-refs.new\n> growing at a rate that seems slower than expected as if much more is\n> happening while writing this file, than just writing the file.\n> \n> An strace shows about 200+ open(\"./objects..\") calls interspersed\n> between around ~26K write() calls. I am surprised to see pack-refs\n> reading objects at all.\n\nI think this is from `should_pack_ref`:\n\n    /* Do not pack broken refs: */\n    if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))\n    \treturn 0;\n\nSo Git is going to need to verify that the object at least exists.  I\ndon't know why we would need to _open_ them, however.  Perhaps someone\nelse has ideas.\n\n> Although the repository is not in terrible shape before packing refs\n> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo\n> first does speed it up so that packing refs then takes under 20s.\n> \n> This repository is on NFS.\n\nThat's almost certainly part of your performance problem, too.  Loading\na single pack file and index is going to be way, way faster than making\nlots of network calls to open 37 pack file and 37 index files, plus at\nleast stat some loose objects.\n\nI will note that at least some forges always have Git write pack files\nand try to avoid loose objects altogether since that almost always\nimproves performance.  You may want to set `receive.unpackLimit` to 1 to\nsee if that helps in the general case.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"532749","messageId":"20251226044507.GA1971832@coredump.intra.peff.net","threadId":"64681","inReplyTo":"aU3K9lGbHw68Vv5U@fruit.crustytoothpaste.net","subject":"Re: Slow git pack-refs --all","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-12-26T04:45:07Z","receivedAt":"2025-12-26T04:45:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:\n\n> I think this is from `should_pack_ref`:\n> \n>     /* Do not pack broken refs: */\n>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))\n>     \treturn 0;\n> \n> So Git is going to need to verify that the object at least exists.  I\n> don't know why we would need to _open_ them, however.  Perhaps someone\n> else has ideas.\n\nThe packed-refs file stores tag-peeling information. So pack-refs opens\nthe object for any newly written ref via peel_object(), which has to at\nleast read the header to get the type. That call happens via\nwrite_with_updates() in packed-backend.c.\n\n  If we wanted to be really pedantic, anything in refs/heads/ should not\n  point to a non-commit and thus should never need to be peeled. I'm not\n  sure if we want to embed that assumption in this code path, though\n  (nor would it necessarily help Martin's case if the refs are not in\n  refs/heads anyway).\n\n-Peff\n"},{"id":"532767","messageId":"aU7Cs2pXiXInfBh4@fruit.crustytoothpaste.net","threadId":"64681","inReplyTo":"20251226044507.GA1971832@coredump.intra.peff.net","subject":"Re: Slow git pack-refs --all","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-12-26T17:15:31Z","receivedAt":"2025-12-26T17:15:34Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-12-26 at 04:45:07, Jeff King wrote:\n> On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:\n> \n> > I think this is from `should_pack_ref`:\n> > \n> >     /* Do not pack broken refs: */\n> >     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))\n> >     \treturn 0;\n> > \n> > So Git is going to need to verify that the object at least exists.  I\n> > don't know why we would need to _open_ them, however.  Perhaps someone\n> > else has ideas.\n> \n> The packed-refs file stores tag-peeling information. So pack-refs opens\n> the object for any newly written ref via peel_object(), which has to at\n> least read the header to get the type. That call happens via\n> write_with_updates() in packed-backend.c.\n> \n>   If we wanted to be really pedantic, anything in refs/heads/ should not\n>   point to a non-commit and thus should never need to be peeled. I'm not\n>   sure if we want to embed that assumption in this code path, though\n>   (nor would it necessarily help Martin's case if the refs are not in\n>   refs/heads anyway).\n\nI don't think that would be a good idea.  I know that people definitely\ndo updates of the loose refs by hand (although they should not) and so\nit's entirely possible for them to contain invalid values, such as\nhaving branches contain non-commit objects.\n\nI wonder if reftable would avoid the need for this kind of expensive\ncheck since it would already have the data peeled if need be and\nwouldn't need to recompute the values.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"532771","messageId":"20251227073634.GA2071715@coredump.intra.peff.net","threadId":"64681","inReplyTo":"aU7Cs2pXiXInfBh4@fruit.crustytoothpaste.net","subject":"Re: Slow git pack-refs --all","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-12-27T07:36:34Z","receivedAt":"2025-12-27T07:36:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 26, 2025 at 05:15:31PM +0000, brian m. carlson wrote:\n\n> >   If we wanted to be really pedantic, anything in refs/heads/ should not\n> >   point to a non-commit and thus should never need to be peeled. I'm not\n> >   sure if we want to embed that assumption in this code path, though\n> >   (nor would it necessarily help Martin's case if the refs are not in\n> >   refs/heads anyway).\n> \n> I don't think that would be a good idea.  I know that people definitely\n> do updates of the loose refs by hand (although they should not) and so\n> it's entirely possible for them to contain invalid values, such as\n> having branches contain non-commit objects.\n\nYeah, that matches my inclination.\n\n> I wonder if reftable would avoid the need for this kind of expensive\n> check since it would already have the data peeled if need be and\n> wouldn't need to recompute the values.\n\nIt does the same amount of peeling, but it's amortized across more\noperations (i.e., whatever did those ref updates in the first place)\nrather than during the pack operation. And of course there really is no\npack operation per se with reftables, but I believe it avoids re-peeling\nwhen rewriting entries during compaction.\n\nIt might actually do fewer object accesses overall if the ref-writing\noperations have already loaded the objects in question (and thus it\nknows whether they're tags or not, and may even have parsed tags in\nmemory). It can also do more in some cases (e.g., two loose writes will\npeel for each write, whereas the files backend only bothers to peel\nduring packing).\n\n-Peff\n"},{"id":"532862","messageId":"CH3PR12MB90260871D7B0D3516FCF3DCBC2BDA@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"aU3K9lGbHw68Vv5U@fruit.crustytoothpaste.net","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2025-12-31T05:39:55Z","receivedAt":"2025-12-31T05:39:58Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":">From: brian m. carlson\n> Sent: Thursday, December 25, 2025 4:38 PM\n> On 2025-12-25 at 22:13:54, Martin Fick wrote:\n>> Although the repository is not in terrible shape before packing refs\n>> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo\n>> first does speed it up so that packing refs then takes under 20s.\n>>\n>> This repository is on NFS.\n\n> That's almost certainly part of your performance problem, too.  Loading\n> a single pack file and index is going to be way, way faster than making\n> lots of network calls to open 37 pack file and 37 index files, plus at\n> least stat some loose objects.\n>\n> I will note that at least some forges always have Git write pack files\n> and try to avoid loose objects altogether since that almost always\n> improves performance.  You may want to set `receive.unpackLimit` to 1 to\n> see if that helps in the general case.\n\nThis would not explain why jgit can pack-refs much faster, since it has to deal with these NFS latencies also. In my experience with jgit, we generally don't see performance issues unless packfile counts exceed 300 or so on NFS, definitely not with only 37 of them. It could be that git is doing some things less efficiently here, but I would be pretty surprised if git could not also perform well typically on NFS with only 37 packfiles. I don't think that 200 something object lookups, should ever take 3+mins, even on NFS, and even with way more packfiles than this. To be that slow, it would have to take about 1s per lookup! Something really seems fishy here to me. I have to think that somehow it isn't these reads that are slow, but I can't explain what else it would be?\n\n-Martin\n\n"},{"id":"532863","messageId":"CH3PR12MB9026DFCF7AF4ED1A249B16A5C2BDA@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"20251226044507.GA1971832@coredump.intra.peff.net","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2025-12-31T05:48:11Z","receivedAt":"2025-12-31T05:48:14Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"> From: Jeff King <peff@peff.net>\n> Sent: Thursday, December 25, 2025 9:45 PM\n>\n> On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:\n>> I think this is from `should_pack_ref`:\n>>\n>>     /* Do not pack broken refs: */\n>>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))\n>>       return 0;\n>>\n>> So Git is going to need to verify that the object at least exists.  I\n>> don't know why we would need to _open_ them, however.  Perhaps someone\n>> else has ideas.\n>\n>The packed-refs file stores tag-peeling information. So pack-refs opens\n>the object for any newly written ref via peel_object(), which has to at\n>least read the header to get the type. That call happens via\n>write_with_updates() in packed-backend.c.\n\nThanks, this makes sense. However, since jgit needs to peel these objects also, it doesn't make sense to me that this would be the bottleneck unless git is doing something terribly inefficient here. :(\n\nExcept for the fact that repacking objects made it faster, my observations make it look like it's the writing that is actually slow, not the reads. Could there be too many small unbuffered writes, could this write path have missed being optimized (it likely isn't used elsewhere)?\n\n-Martin\n"},{"id":"532904","messageId":"20260102074901.GD2581074@coredump.intra.peff.net","threadId":"64681","inReplyTo":"CH3PR12MB9026DFCF7AF4ED1A249B16A5C2BDA@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-01-02T07:49:01Z","receivedAt":"2026-01-02T07:49:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:\n\n> >> I think this is from `should_pack_ref`:\n> >>\n> >>     /* Do not pack broken refs: */\n> >>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))\n> >>       return 0;\n> >>\n> >> So Git is going to need to verify that the object at least exists.  I\n> >> don't know why we would need to _open_ them, however.  Perhaps someone\n> >> else has ideas.\n> >\n> >The packed-refs file stores tag-peeling information. So pack-refs opens\n> >the object for any newly written ref via peel_object(), which has to at\n> >least read the header to get the type. That call happens via\n> >write_with_updates() in packed-backend.c.\n> \n> Thanks, this makes sense. However, since jgit needs to peel these\n> objects also, it doesn't make sense to me that this would be the\n> bottleneck unless git is doing something terribly inefficient here. :(\n\nI'd expect both git and jgit to open each loose object once. I tried\nrunning \"git pack-refs --all --prune\" under strace on a test repo. It\ndoes seem to open the object once. Then I tried the same with jgit\n(though it does not understand --prune), and got the same results.\n\nSo...I dunno.\n\n> Except for the fact that repacking objects made it faster, my\n> observations make it look like it's the writing that is actually slow,\n> not the reads. Could there be too many small unbuffered writes, could\n> this write path have missed being optimized (it likely isn't used\n> elsewhere)?\n\nAll of the packed-refs writes are through fprintf(), which should be\nfully buffered. You should be able to confirm with strace (I get\n4096-byte writes on my system).\n\nIf writing were slow, I'd also expect that to scale with the total\nnumber of refs, not the number of changed refs (since we have to rewrite\nthe whole file, but only new entries need to be peeled).\n\n-Peff\n"},{"id":"533092","messageId":"CH3PR12MB90260C4887067C88629BBE52C286A@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"20260102074901.GD2581074@coredump.intra.peff.net","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2026-01-05T23:45:41Z","receivedAt":"2026-01-05T23:45:45Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"> From: Jeff King <peff@peff.net>\n>  Sent: Friday, January 2, 2026 12:49 AM\n> On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:\n> > Except for the fact that repacking objects made it faster, my\n\nI have now confirmed that repacking does NOT actually make things \nfaster. I believe filesystem caching interfered with much of my \ntesting. Using echo 3 > /proc/sys/vm/drop_caches has helped to\nget more consistent results.\n\nBy repacking to get one used, and one cruft pack only, and no loose \nobjects, I have confirmed that pack-refs it is still slow. This rules out the \nidea that the loose object, or pack file counts were making things slow.\n\n> > observations make it look like it's the writing that is actually slow,\n> > not the reads. Could there be too many small unbuffered writes, could\n> > this write path have missed being optimized (it likely isn't used\n> > elsewhere)?\n>\n> All of the packed-refs writes are through fprintf(), which should be\n> fully buffered. You should be able to confirm with strace (I get\n> 4096-byte writes on my system).\n\nI can confirm that my system is actually printing 8192 bytes at a time.\n\n> If writing were slow, I'd also expect that to scale with the total\n> number of refs, not the number of changed refs (since we have to rewrite\n> the whole file, but only new entries need to be peeled).\n\nOK, after discovering the strace -r and -T options, I have determined that\nthe 29K writes were all very fast in themselves. However, most of the\nwrites seem to follow each other with no other system calls in between.\nThis explains why it looks like the writes are slow, even though they aren't.\n\nIf I tally up the time between the previous system call, and each write(),\nit adds up to the bulk of the time (4mins out of 4m15s) that it takes to\npack refs. This tells me that no visible I/O or system calls are the problem,\nbut rather that the program itself is taking a long time between writes.\nI very much doubt that this is heavy CPU time, but rather I am going to \nguess that this is hidden system time spent accessing mmaped memory. \nCould it be really slow reading the packed-refs file? I can see the \npacked-refs file is mmaped() before the writes start, and then \nmunmapped after the writes are completed. If I had to guess, that likely\nmeans that the packed-refs file is being read in small increments by the \nkernel via mmap, and that is what is making things very slow over NFS. \nMy alternative theory, is that each ref is being looked up via a binary \nsearch, but I don't think git does this?\n\n-Martin\n"},{"id":"533096","messageId":"aVyxbqk-2QQIgDXK@pks.im","threadId":"64681","inReplyTo":"CH3PR12MB90260C4887067C88629BBE52C286A@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-06T06:53:34Z","receivedAt":"2026-01-06T06:53:40Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi Martin,\n\nOn Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:\n> OK, after discovering the strace -r and -T options, I have determined that\n> the 29K writes were all very fast in themselves. However, most of the\n> writes seem to follow each other with no other system calls in between.\n> This explains why it looks like the writes are slow, even though they aren't.\n> \n> If I tally up the time between the previous system call, and each write(),\n> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to\n> pack refs. This tells me that no visible I/O or system calls are the problem,\n> but rather that the program itself is taking a long time between writes.\n> I very much doubt that this is heavy CPU time, but rather I am going to \n> guess that this is hidden system time spent accessing mmaped memory. \n> Could it be really slow reading the packed-refs file? I can see the \n> packed-refs file is mmaped() before the writes start, and then \n> munmapped after the writes are completed. If I had to guess, that likely\n> means that the packed-refs file is being read in small increments by the \n> kernel via mmap, and that is what is making things very slow over NFS. \n\nI wouldn't be surprised if NFS was the culprit. At GitLab we found it to\nbe a constant source of issues, which is why we eventually sunsetted the\nuse of it completely. Do you use any special flags for mounting the NFS\nfilesystem?\n\n> My alternative theory, is that each ref is being looked up via a binary \n> search, but I don't think git does this?\n\nDid you try using perf(1) to profile the process and generate a flame\ngraph from it? That should likely make it immediately obvious where Git\nis spending all of its time.\n\nPatrick\n"},{"id":"533125","messageId":"20260106103803.GA69061@coredump.intra.peff.net","threadId":"64681","inReplyTo":"CH3PR12MB90260C4887067C88629BBE52C286A@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-01-06T10:38:03Z","receivedAt":"2026-01-06T10:38:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:\n\n> By repacking to get one used, and one cruft pack only, and no loose \n> objects, I have confirmed that pack-refs it is still slow. This rules out the \n> idea that the loose object, or pack file counts were making things slow.\n\nOK, that is interesting. I'd still expect opening the objects to be the\ndominating factor, but now the load would be on jumping around the\nmmap'd packfile rather than open/read/close calls.\n\n> OK, after discovering the strace -r and -T options, I have determined that\n> the 29K writes were all very fast in themselves. However, most of the\n> writes seem to follow each other with no other system calls in between.\n> This explains why it looks like the writes are slow, even though they aren't.\n> \n> If I tally up the time between the previous system call, and each write(),\n> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to\n> pack refs. This tells me that no visible I/O or system calls are the problem,\n> but rather that the program itself is taking a long time between writes.\n> I very much doubt that this is heavy CPU time, but rather I am going to \n> guess that this is hidden system time spent accessing mmaped memory.\n\nThat would be consistent with reading object data from the packfile.\nWe'll jump around within the packfile to get that data.\n\n> Could it be really slow reading the packed-refs file? I can see the \n> packed-refs file is mmaped() before the writes start, and then \n> munmapped after the writes are completed. If I had to guess, that likely\n> means that the packed-refs file is being read in small increments by the \n> kernel via mmap, and that is what is making things very slow over NFS.\n\nThe packed-refs file is mmap'd, but we'll be reading it sequentially. I\nguess whether or not there is good read-ahead there may depend on the\nNFS implementation.\n\n> My alternative theory, is that each ref is being looked up via a binary \n> search, but I don't think git does this?\n\nGit does binary search within the packed-refs file, but it shouldn't be\ndoing so here. The write-out phase of packing refs is a straight merge\nbetween two lists: the existing packed-refs entries and the new entries\nwe are adding.\n\nI'd second Patrick's suggestion to use perf or similar to try to see\nwhere the time is going.\n\nYou might also try building Git with NO_MMAP. That might make the I/O\ncosts more apparent via strace, because they'll be coming via pread().\n\n-Peff\n"},{"id":"533165","messageId":"CH3PR12MB9026F1E4B99D32E138800EEBC287A@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"aVyxbqk-2QQIgDXK@pks.im","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2026-01-06T23:02:19Z","receivedAt":"2026-01-06T23:02:25Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"> From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM\n> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:\n> > OK, after discovering the strace -r and -T options, I have determined that\n> > the 29K writes were all very fast in themselves. However, most of the\n> > writes seem to follow each other with no other system calls in between.\n> > This explains why it looks like the writes are slow, even though they aren't.\n> >\n> > If I tally up the time between the previous system call, and each write(),\n> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to\n> > pack refs. This tells me that no visible I/O or system calls are the problem,\n> > but rather that the program itself is taking a long time between writes.\n> > I very much doubt that this is heavy CPU time, but rather I am going to\n> > guess that this is hidden system time spent accessing mmaped memory.\n> > Could it be really slow reading the packed-refs file? I can see the\n> > packed-refs file is mmaped() before the writes start, and then\n> > munmapped after the writes are completed. If I had to guess, that likely\n> > means that the packed-refs file is being read in small increments by the\n> > kernel via mmap, and that is what is making things very slow over NFS.\n> \n> I wouldn't be surprised if NFS was the culprit. At GitLab we found it to\n> be a constant source of issues, which is why we eventually sunsetted the\n> use of it completely. Do you use any special flags for mounting the NFS\n> filesystem?\n\nI am open to alternatives to NFS. Do you know of any NFS alternatives that \nprovides instantaneous replication to potentially hundreds of mirrors? I \nhave used Gerrit and git-daemon for many years on NFS, and it generally \nhas performed very well for us, and it solves many real performance issues \nwhich I have yet to find a viable alternative able to even come close to\nmatching. NFS with all it warts it is for us (and likely will be for many) until \nthere is a viable enterprise ready alternative with low (zero) replication \nlatency and high throughput.\n\nThat being said, NFS can cause many issues. In this case, I would say that\nsomething is particularly \"broken\" here with git, and I believe that it\nwould be helpful to the git community to be aware of this fairly specific \nbroken case which clearly has a lot of room for improvement (as seen\nby the fact that jgit, in java, can do essentially the same thing more \nthan 10Xs faster). While I have been mostly assuming that this is a \nparticularly specific bad case since git daemon generally is fast for most\nusers, this might actually be something that if improved would greatly \nimprove many parts of git (not just this use case).\n\nIt would be nice to improve git to not hold the packed-refs.lock so long \nto avoid this blocking behavior on servers. Of course, to be fair, this \nlikely only blocks Gerrit servers since Gerrit uses the packed-refs file to \nperform atomic updates for many things, and most other servers use \nloose refs instead. It would be great if git were optimized to avoid any \nunnecessary reads while the lock is held.  In theory, almost all of the \ndata that git needs to read here (including tags for peeling) could be \nread before acquiring the lock, and it would only need to double \ncheck certain reads after it acquires the lock in case things changed. \nThat wouldn't make git pack-refs faster, but it would drastically \nreduce the impact of any problematic I/O by not holding the lock for \nalmost the entire operation.\n\n> Did you try using perf(1) to profile the process and generate a flame\n> graph from it? That should likely make it immediately obvious where Git\n> is spending all of its time.\n\nI will pursue this. Unfortunately this might be difficult on this \nparticular server.\n\nThanks for the feedback, and suggestions,\n\n-Martin\n"},{"id":"533166","messageId":"CH3PR12MB902640F983E7FB0FAB92D404C287A@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"20260106103803.GA69061@coredump.intra.peff.net","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2026-01-06T23:03:35Z","receivedAt":"2026-01-06T23:03:37Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"From: Jeff King <peff@peff.net> Sent: Tuesday, January 6, 2026 3:38 AM\n\n> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:\n> > By repacking to get one used, and one cruft pack only, and no loose\n> > objects, I have confirmed that pack-refs it is still slow. This rules out the\n> > idea that the loose object, or pack file counts were making things slow.\n> \n> OK, that is interesting. I'd still expect opening the objects to be the\n> dominating factor, but now the load would be on jumping around the\n> mmap'd packfile rather than open/read/close calls.\n\nI believe I have confirmed this now with more testing...\n\nBy first dropping the system caches, and then catting the pack file to\n/dev/null, it sped things up to under 20s!\n\nNote that neither catting the idx, nor the packed-refs file helped to \nnoticeably speed things up.\n\n> > OK, after discovering the strace -r and -T options, I have determined that\n> > the 29K writes were all very fast in themselves. However, most of the\n> > writes seem to follow each other with no other system calls in between.\n> > This explains why it looks like the writes are slow, even though they aren't.\n>\n> > If I tally up the time between the previous system call, and each write(),\n> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to\n> > pack refs. This tells me that no visible I/O or system calls are the problem,\n> > but rather that the program itself is taking a long time between writes.\n> > I very much doubt that this is heavy CPU time, but rather I am going to\n> > guess that this is hidden system time spent accessing mmaped memory.\n> \n> That would be consistent with reading object data from the packfile.\n> We'll jump around within the packfile to get that data.\n\nAgreed, but boy is that really bad performance!\n\n\n> > Could it be really slow reading the packed-refs file? I can see the\n> > packed-refs file is mmaped() before the writes start, and then\n> > munmapped after the writes are completed. If I had to guess, that likely\n> > means that the packed-refs file is being read in small increments by the\n> > kernel via mmap, and that is what is making things very slow over NFS.\n> \n> The packed-refs file is mmap'd, but we'll be reading it sequentially. I\n> guess whether or not there is good read-ahead there may depend on the\n> NFS implementation.\n\nYeah, ruled out now by dropping the system caches, and then catting the \npacked-refs file before running git pack-refs, which did NOT help speed \nthings up.\n\n\n> > My alternative theory, is that each ref is being looked up via a binary\n> > search, but I don't think git does this?\n> \n> Git does binary search within the packed-refs file, but it shouldn't be\n> doing so here. The write-out phase of packing refs is a straight merge\n> between two lists: the existing packed-refs entries and the new entries\n> we are adding.\n\nAgreed, and I should have ruled this out by realizing that this would likely\nnot have been affected by the system caches in my earlier tests.\n\n\n> I'd second Patrick's suggestion to use perf or similar to try to see\n> where the time is going.\n\nNoted, thanks.\n\n\n> You might also try building Git with NO_MMAP. That might make the I/O\n> costs more apparent via strace, because they'll be coming via pread().\n\nAgreed, I will try to do this. I think that the jgit results hint that this \nthis might even eliminate most of the I/O costs (jgit is not using\nMMAP in my tests). It would be nice if this were a runtime config\ninstead of requiring a rebuild, as some use cases might be better\nwith, and some without MMAP.\n\nThanks for all the input,\n\n-Martin\n"},{"id":"533210","messageId":"aV5GwOS_N2jyIFaz@pks.im","threadId":"64681","inReplyTo":"CH3PR12MB9026F1E4B99D32E138800EEBC287A@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-07T11:42:56Z","receivedAt":"2026-01-07T11:43:02Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:\n> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM\n> > On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:\n> > > OK, after discovering the strace -r and -T options, I have determined that\n> > > the 29K writes were all very fast in themselves. However, most of the\n> > > writes seem to follow each other with no other system calls in between.\n> > > This explains why it looks like the writes are slow, even though they aren't.\n> > >\n> > > If I tally up the time between the previous system call, and each write(),\n> > > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to\n> > > pack refs. This tells me that no visible I/O or system calls are the problem,\n> > > but rather that the program itself is taking a long time between writes.\n> > > I very much doubt that this is heavy CPU time, but rather I am going to\n> > > guess that this is hidden system time spent accessing mmaped memory.\n> > > Could it be really slow reading the packed-refs file? I can see the\n> > > packed-refs file is mmaped() before the writes start, and then\n> > > munmapped after the writes are completed. If I had to guess, that likely\n> > > means that the packed-refs file is being read in small increments by the\n> > > kernel via mmap, and that is what is making things very slow over NFS.\n> > \n> > I wouldn't be surprised if NFS was the culprit. At GitLab we found it to\n> > be a constant source of issues, which is why we eventually sunsetted the\n> > use of it completely. Do you use any special flags for mounting the NFS\n> > filesystem?\n> \n> I am open to alternatives to NFS. Do you know of any NFS alternatives that \n> provides instantaneous replication to potentially hundreds of mirrors? I \n> have used Gerrit and git-daemon for many years on NFS, and it generally \n> has performed very well for us, and it solves many real performance issues \n> which I have yet to find a viable alternative able to even come close to\n> matching. NFS with all it warts it is for us (and likely will be for many) until \n> there is a viable enterprise ready alternative with low (zero) replication \n> latency and high throughput.\n\nYeah, agreed, NFS can get you a long way, until you eventually start to\nhit some road blocks once you reach a certain scale. Unfortunately\nthough, there isn't really a ready-made alternative solution that serves\nyour needs, or at least none that I know of. That's why GitLab\neventually settled on Gitaly Cluster with Praefect handling replication,\nand why GitHub has its Spokes architecture that does basically the same\nthing.\n\n> That being said, NFS can cause many issues. In this case, I would say that\n> something is particularly \"broken\" here with git, and I believe that it\n> would be helpful to the git community to be aware of this fairly specific \n> broken case which clearly has a lot of room for improvement (as seen\n> by the fact that jgit, in java, can do essentially the same thing more \n> than 10Xs faster). While I have been mostly assuming that this is a \n> particularly specific bad case since git daemon generally is fast for most\n> users, this might actually be something that if improved would greatly \n> improve many parts of git (not just this use case).\n\nChances are that if we can improve the case for NFS, other filesystems\nmight benefit, as well. So if this is something that we can improve I\nagree that we should. It's too early to tell though, as we don't really\nknow what the actual root cause is just yet.\n\n> It would be nice to improve git to not hold the packed-refs.lock so long \n> to avoid this blocking behavior on servers. Of course, to be fair, this \n> likely only blocks Gerrit servers since Gerrit uses the packed-refs file to \n> perform atomic updates for many things, and most other servers use \n> loose refs instead. It would be great if git were optimized to avoid any \n> unnecessary reads while the lock is held.  In theory, almost all of the \n> data that git needs to read here (including tags for peeling) could be \n> read before acquiring the lock, and it would only need to double \n> check certain reads after it acquires the lock in case things changed. \n> That wouldn't make git pack-refs faster, but it would drastically \n> reduce the impact of any problematic I/O by not holding the lock for \n> almost the entire operation.\n\nIt can probably be improved, true. I think that it's a bit of a wasted\neffort, as I'd rather invest the time into improving reftables as a more\nfuture-proof solution. But as you are well aware I'm quite biased here,\nand I'd welcome any efforts to also improve the files backend. I am just\nunlikely to work on it myself :)\n\n> > Did you try using perf(1) to profile the process and generate a flame\n> > graph from it? That should likely make it immediately obvious where Git\n> > is spending all of its time.\n> \n> I will pursue this. Unfortunately this might be difficult on this \n> particular server.\n\nTrue, on the server side this can be a bit tricky.\n\nPatrick\n"},{"id":"533232","messageId":"CH3PR12MB90266A2B11493D5E02E90D02C284A@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"aVyxbqk-2QQIgDXK@pks.im","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2026-01-07T17:05:53Z","receivedAt":"2026-01-07T17:05:56Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"> From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM\n> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:\n> > OK, after discovering the strace -r and -T options, I have determined that\n> > the 29K writes were all very fast in themselves. However, most of the\n> > writes seem to follow each other with no other system calls in between.\n> > This explains why it looks like the writes are slow, even though they aren't.\n> \n> > If I tally up the time between the previous system call, and each write(),\n> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to\n> > pack refs. This tells me that no visible I/O or system calls are the problem,\n> > but rather that the program itself is taking a long time between writes.\n> > I very much doubt that this is heavy CPU time, but rather I am going to\n> > guess that this is hidden system time spent accessing mmaped memory.\n> > Could it be really slow reading the packed-refs file? I can see the\n> > packed-refs file is mmaped() before the writes start, and then\n> > munmapped after the writes are completed. If I had to guess, that likely\n> > means that the packed-refs file is being read in small increments by the\n> > kernel via mmap, and that is what is making things very slow over NFS.\n> \n> ... Do you use any special flags for mounting the NFS filesystem?\n\nOh sorry, I forgot to reply to this last time. We use the following mount flags:\n\nrw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev\n\n-Martin"},{"id":"533243","messageId":"CH3PR12MB9026C8C940270F02CEF83C4FC284A@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"aV5GwOS_N2jyIFaz@pks.im","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2026-01-07T22:58:36Z","receivedAt":"2026-01-07T22:58:39Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"> From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM\nOn Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:\n> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM\n> > > Did you try using perf(1) to profile the process and generate a flame\n> > > graph from it? That should likely make it immediately obvious where Git\n> > > is spending all of its time.\n> >\n> > I will pursue this. Unfortunately this might be difficult on this\n> > particular server.\n> \n> True, on the server side this can be a bit tricky.\n\nI ran perf, and got a flame graph, I am not sure what the best way to share that\nis, but I will try to summarize what looked important:\n\nAbout one third of the time is in this section:\n\nlibc-2.17.so 32.5%\n _memcmp_sse4_1 29.8%\n page_fault 7.23%\n ...\n\nI am not really sure what that is doing?\n\n\nAnother third is doing:\n\nunpack_object_header_buffer 30%\n page_fault 26.9%\n ...\n nfs_read_page 10%\n\nWhich could very well be looking at the headers of objects to see if they are \ntags needing to be peeled?\n\n\nAnd the remaining third was a bit all over the place with small sections,\nthe largest two of those sections being:\n\npacked_refs_store_create ~8.7%\n unknown 4.4%\n memchr 4.4%\n page_fault 4.4%\n\nnth_packed_object_offset 7%\n page_fault 3.2%\n\nThis was way less informative (to me) then I would have hoped. :( Maybe\nthis means more to you? \n\nIt does look like a lot of page_faults, likely due to the use of mmap?\n\n-Martin\n"},{"id":"533269","messageId":"aV9PwnouO8652XQR@pks.im","threadId":"64681","inReplyTo":"CH3PR12MB9026C8C940270F02CEF83C4FC284A@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-08T06:33:38Z","receivedAt":"2026-01-08T06:33:45Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 07, 2026 at 10:58:36PM +0000, Martin Fick wrote:\n> > From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM\n> On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:\n> > > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM\n> > > > Did you try using perf(1) to profile the process and generate a flame\n> > > > graph from it? That should likely make it immediately obvious where Git\n> > > > is spending all of its time.\n> > >\n> > > I will pursue this. Unfortunately this might be difficult on this\n> > > particular server.\n> > \n> > True, on the server side this can be a bit tricky.\n> \n> I ran perf, and got a flame graph, I am not sure what the best way to share that\n> is, but I will try to summarize what looked important:\n> \n> About one third of the time is in this section:\n> \n> libc-2.17.so 32.5%\n>  _memcmp_sse4_1 29.8%\n>  page_fault 7.23%\n>  ...\n> \n> I am not really sure what that is doing?\n> \n> \n> Another third is doing:\n> \n> unpack_object_header_buffer 30%\n>  page_fault 26.9%\n>  ...\n>  nfs_read_page 10%\n> \n> Which could very well be looking at the headers of objects to see if they are \n> tags needing to be peeled?\n\nBoth of these are lacking some information to be able to tell. Are you\nby any chance able to share the whole flame graph? That'd make this a\nbit easier to figure out.\n\n> And the remaining third was a bit all over the place with small sections,\n> the largest two of those sections being:\n> \n> packed_refs_store_create ~8.7%\n>  unknown 4.4%\n>  memchr 4.4%\n>  page_fault 4.4%\n\nWe spend ~9% of time in `packed_refs_store_create()`? That looks\nseriously broken to me, the function shouldn't even do much.\n\n> nth_packed_object_offset 7%\n>  page_fault 3.2%\n> \n> This was way less informative (to me) then I would have hoped. :( Maybe\n> this means more to you? \n> \n> It does look like a lot of page_faults, likely due to the use of mmap?\n\nCertainly looks like the page faults are to blame here overall. It's\nstill surprising to me it's _that_ slow. Quoting the other mail you sent:\n\nOn Wed, Jan 07, 2026 at 05:05:53PM +0000, Martin Fick wrote:\n> rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev\n\nI know that back when we still supported NFS we recommended to use an\nrsize and wsize of 1MB to reduce the round trip times.\n\nPatrick\n"},{"id":"533987","messageId":"20260115210908.GE1053259@coredump.intra.peff.net","threadId":"64681","inReplyTo":"CH3PR12MB9026C8C940270F02CEF83C4FC284A@CH3PR12MB9026.namprd12.prod.outlook.com","subject":"Re: Slow git pack-refs --all","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-01-15T21:09:08Z","receivedAt":"2026-01-15T21:09:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 07, 2026 at 10:58:36PM +0000, Martin Fick wrote:\n\n> I ran perf, and got a flame graph, I am not sure what the best way to share that\n> is, but I will try to summarize what looked important:\n> \n> About one third of the time is in this section:\n> \n> libc-2.17.so 32.5%\n>  _memcmp_sse4_1 29.8%\n>  page_fault 7.23%\n>  ...\n> \n> I am not really sure what that is doing?\n\nProbably this is the call to strcmp(iter->ref.name, update->refname) in\npacked-backend.c:write_with_updates().\n\nWe have to write out the new packed-refs file with our updates in sorted\norder. So it's a big O(n) merge between the existing ones (from the\n\"iter\" side) and the new ones (from the \"update\" side).\n\nIt could also be caused by sorting of the packed-refs entries. We\ngenerally shouldn't need to do that, but I think I may have found\nsomething useful. See below.\n\n> unpack_object_header_buffer 30%\n>  page_fault 26.9%\n>  ...\n>  nfs_read_page 10%\n> \n> Which could very well be looking at the headers of objects to see if they are \n> tags needing to be peeled?\n\nYeah, that's what I'd expect here.\n\n> And the remaining third was a bit all over the place with small sections,\n> the largest two of those sections being:\n> \n> packed_refs_store_create ~8.7%\n>  unknown 4.4%\n>  memchr 4.4%\n>  page_fault 4.4%\n\nHmm, I don't think we have a function \"packed_refs_store_create\". Did\nyou typo while transferring the name over?\n\nAt any rate, we can assume this is poking through the packed-refs file\nitself, looking for trailing newlines via memchr.\n\nBut why would we do that immediately when creating the packed-refs store\nin memory? In modern versions of Git, we try to avoid reading the\npacked-refs file as much as possible, binary-searching when we can. Of\ncourse that means it has to be sorted, which was not something promised\nby the original format. So we have a \"sorted\" tag that we write. E.g.,\nthis is from my clone of git, packed with git itself:\n\n  $ head -n 1 .git/packed-refs\n  # pack-refs with: peeled fully-peeled sorted\n\nNow let's try something with jgit:\n\n  git init\n  git commit --allow-empty -m foo\n  git branch foo\n  git branch bar\n\n  jgit pack-refs --all\n  cat .git/packed-refs\n\nThat gives me this:\n\n  # pack-refs with: peeled\n  86054aaedc64c24aec8aaad988f6979a3cb82ee0 refs/heads/bar\n  86054aaedc64c24aec8aaad988f6979a3cb82ee0 refs/heads/foo\n  86054aaedc64c24aec8aaad988f6979a3cb82ee0 refs/heads/main\n\nAha! So jgit is not writing out the \"sorted\" tag. As a result, when git\nreads the file, its logic is:\n\n  1. Check for the sorted tag. It's not here, so...\n\n  2. Check if the file is sorted by reading each entry linearly. If it's\n     not, then...\n\n  3. Read it all into memory and sort the result. We can then\n     binary-search that (and iterate it in sorted order, which is\n     important for pack-refs).\n\nSo when git reads the packed-refs file, we are ending up at least with\nstep 2, an extra pass through the whole file, and maybe to step 3\n(depending on whether jgit actually sorts the file).\n\nYou mentioned that Gerrit writes the packed-refs file directly itself,\npresumably using jgit. So it sounds like it is constantly undoing Git's\n\"sorted\" marker, which causes git-pack-refs to spend extra effort\nchecking the sortedness, and rewrite the marker, which then gets hosed\nagain by jgit, and so on.\n\nAnd that may explain why jgit is faster, if it is not doing the extra\nsort check. If it is not even trying to maintain the sorted property\nthat it would be faster still (it takes one linear pass while writing\nout the file, omitting entries that match our updates, and then appends\nour updates at the end).\n\nIf jgit _is_ sorting the file but not writing out the sorted marker,\nthen it should start doing so. ;)\n\nIf it's not sorting the file, then probably it should start doing so\n(and writing the marker). This will make subsequent reads much faster\n(mmap + binary-search). It shouldn't even be slower to write (assuming\njgit's writes are doing the usual \"rewrite the whole thing to a tempfile\nand atomic-rename into place\", and not taking some shortcut by appending\nto the file).\n\nUnrelated to your problem, but also jgit should support the fully-peeled\ntag, another thing that makes readers faster. ;)\n\nThe jgit version I'm using is:\n\n  $ jgit version\n  jgit version 7.5.0.202512021534-r\n\nOne way you could test this theory is to sort and mark the file\nyourself, before running \"git pack-refs\". One easy way to do that is to\nconvince git to rewrite it by removing an entry. I.e., find some ref\nmentioned in the pack-refs file, and then \"git update-ref -d $ref\". And\ncheck out the first line of .git/packed-refs before and after. If it\ngoes faster (and similarly fast to jgit) only when the \"sorted\" tag\nappears, then that would be our culprit.\n\n-Peff\n"},{"id":"534085","messageId":"CH3PR12MB902665032350D502E3D31ACBC28DA@CH3PR12MB9026.namprd12.prod.outlook.com","threadId":"64681","inReplyTo":"20260115210908.GE1053259@coredump.intra.peff.net","subject":"Re: Slow git pack-refs --all","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2026-01-16T20:35:33Z","receivedAt":"2026-01-16T20:35:43Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"> From: Jeff King <peff@peff.net> Sent: Thursday, January 15, 2026 2:09 PM\n> > ...\n> > And the remaining third was a bit all over the place with small sections,\n> > the largest two of those sections being:\n> >\n> > packed_refs_store_create ~8.7%\n> >  unknown 4.4%\n> >  memchr 4.4%\n> >  page_fault 4.4%\n> \n> Hmm, I don't think we have a function \"packed_refs_store_create\". Did\n> you typo while transferring the name over?\n\nYes, this should have been packed_ref_store_create (singular), sorry.\n\n\n> At any rate, we can assume this is poking through the packed-refs file\n> itself, looking for trailing newlines via memchr.\n> \n> But why would we do that immediately when creating the packed-refs store\n> in memory? In modern versions of Git, we try to avoid reading the\n> packed-refs file as much as possible, binary-searching when we can. Of\n> course that means it has to be sorted, which was not something promised\n> by the original format. So we have a \"sorted\" tag that we write. E.g.,\n> this is from my clone of git, packed with git itself:\n> ... \n>   # pack-refs with: peeled fully-peeled sorted\n> ...\n>   jgit pack-refs --all\n> \n> That gives me this:\n> \n>   # pack-refs with: peeled\n> ...\n>  Aha! So jgit is not writing out the \"sorted\" tag. As a result, when git\n> reads the file, its logic is:\n> \n>   1. Check for the sorted tag. It's not here, so...\n>   2. Check if the file is sorted by reading each entry linearly. If it's\n>     not, then...\n>   3. Read it all into memory and sort the result. We can then\n>      binary-search that (and iterate it in sorted order, which is\n>      important for pack-refs).\n> \n> So when git reads the packed-refs file, we are ending up at least with\n> step 2, an extra pass through the whole file, and maybe to step 3\n> (depending on whether jgit actually sorts the file).\n> \n> You mentioned that Gerrit writes the packed-refs file directly itself,\n> presumably using jgit. So it sounds like it is constantly undoing Git's\n> \"sorted\" marker, which causes git-pack-refs to spend extra effort\n> checking the sortedness, and rewrite the marker, which then gets hosed\n> again by jgit, and so on.\n\nAgreed, this is likely the case, but not for the sorted marker, see below...\n\n> And that may explain why jgit is faster, if it is not doing the extra\n> sort check. If it is not even trying to maintain the sorted property\n> that it would be faster still (it takes one linear pass while writing\n> out the file, omitting entries that match our updates, and then appends\n> our updates at the end).\n\nUnfortunately, this does not actually seem to be the reason.\n\n> If jgit _is_ sorting the file but not writing out the sorted marker,\n> then it should start doing so. ;)\n\nAgreed, I will see to it that this gets fixed. Unfortunately, adding the \nsorted tag does not seem to speed things up. :(\n\n> If it's not sorting the file, then probably it should start doing so\n> (and writing the marker). This will make subsequent reads much faster\n> (mmap + binary-search). It shouldn't even be slower to write (assuming\n> jgit's writes are doing the usual \"rewrite the whole thing to a tempfile\n> and atomic-rename into place\", and not taking some shortcut by appending\n> to the file).\n\nFYI, jgit does seem to order things, it does not append. The resulting output\nfrom jgit after a repack with new refs add matches that from git for all but\nthe header.\n\n> Unrelated to your problem, but also jgit should support the fully-peeled\n> tag, another thing that makes readers faster. ;)\n\nIronically, this is not just related, it appears to be the trigger!!! When I add \nthis tag (and not the sorted tag), the cache flushed time drops down to \nunder 4s (from over 5mins)!\n\nI will see to it that jgit fixes this too. That should help solve my problem.\n\nThat being said, it seems like something is still broken in git here \ndespite this tag being missing?\n\nThanks so much Peff for helping get to this point!\n\n-Martin"}]}