{"thread":{"id":"63917","subject":"cat-file --batch-command info performance problem","startedAt":"2025-08-05T21:26:43Z","lastAt":"2025-08-05T22:10:04Z","messageCount":2,"participants":["Rob Browning"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"523607","messageId":"871ppp1n33.fsf@trouble.defaultvalue.org","threadId":"63917","inReplyTo":null,"subject":"cat-file --batch-command info performance problem","fromName":"Rob Browning","fromEmail":"rlb@defaultvalue.org","sentAt":"2025-08-05T21:21:36Z","receivedAt":"2025-08-05T21:26:43Z","isPatch":false,"sender":{"key":"rlb@defaultvalue.org","avatar":null},"body":"\nWhile doing some testing before a bup release, I ran in to a performance\nproblem that I've narrowed down to git (2.47.2) cat-file --batch-command\n(currently, bup often relies on git cat-file).\n\nI can reproduce the problem with a ~125GB (~3M object) repository on\nan external SSD and a system with 16GB RAM via \"git --batch-command <\nfetch-oids\" where fetch-oids contains 8k \"info HASH\" commands.\n\nThat process runs at 32 hashes/sec (overall average), with a cold\ncache (echo 3 > /proc/sys/vm/drop_caches), and does not improve over\nrepeated runs.  While that's running, every time I check, git's\nreading about 300-400+ MB/s.\n\nThat didn't seem right, so I wrote a test command that produces the same\ninformation via direct index access and packfile reads/seeks. With a\ncold cache, that runs at 1.5k hashes/sec (even from python), and on the\nsecond or third run, 9 hashes/seck.  Interestingly, if I run cat-file\nafter the test command has warmed up the cache, cat-file then reaches\n53k hashes/sec.\n\nMy current guess is that cat-file's approach (all mmap?) is causing\nsome kind of kernel derived read amplification that's vastly\nincreasing the working set.\n\nThis is easily repeatable here, so I'd be happy to help test if that's\ndesirable, and perhaps even to pursue it if I can, and if it seems like\nsomething that could/should be addressed in git.\n\nIf not, that's also helpful to know, and then we'll just handle the\nlookups ourselves.\n\nThanks\n-- \nRob Browning\nrlb @defaultvalue.org and @debian.org\nGPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A\nGPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4\n"},{"id":"523612","messageId":"87v7n1zah0.fsf@trouble.defaultvalue.org","threadId":"63917","inReplyTo":"871ppp1n33.fsf@trouble.defaultvalue.org","subject":"Re: cat-file --batch-command info performance problem","fromName":"Rob Browning","fromEmail":"rlb@defaultvalue.org","sentAt":"2025-08-05T22:10:03Z","receivedAt":"2025-08-05T22:10:04Z","isPatch":false,"sender":{"key":"rlb@defaultvalue.org","avatar":null},"body":"Rob Browning <rlb@defaultvalue.org> writes:\n\n> While doing some testing before a bup release, I ran in to a performance\n> problem that I've narrowed down to git (2.47.2) cat-file --batch-command\n> (currently, bup often relies on git cat-file).\n>\n> I can reproduce the problem with a ~125GB (~3M object) repository on\n> an external SSD and a system with 16GB RAM via \"git --batch-command <\n> fetch-oids\" where fetch-oids contains 8k \"info HASH\" commands.\n\nJust after sending this, I thought to wonder whether the issue might\nsomehow be with the hardware/driver/etc. (that particular drive is an\nnvme ssd in an external usb-3 case), and while I'm not testing the exact\nsame repository (it's a slightly larger, related one), preliminary\nresults suggest cat-file behaves much more reasonably with an internal\nnvme drive, i.e. it starts slow, gets faster, and ends up repeatably at\n37k hashes/sec after the second run.\n\nSo I'll probably test a bit more, but while I'd be quite interested in\nthe cause, it seems likely my solution should just be to replace that\nhardware.\n\n-- \nRob Browning\nrlb @defaultvalue.org and @debian.org\nGPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A\nGPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4\n"}]}