Re: cat-file --batch-command info performance problem
- From
- Rob Browning <rlb@defaultvalue.org>
- Date
- Aug 5, 2025, 22:10 UTC
- Message-ID
- <87v7n1zah0.fsf@trouble.defaultvalue.org>
- In-Reply-To
- <871ppp1n33.fsf@trouble.defaultvalue.org>
Rob Browning <rlb@defaultvalue.org> writes:
Show 7 quoted lines
> While doing some testing before a bup release, I ran in to a performance > problem that I've narrowed down to git (2.47.2) cat-file --batch-command > (currently, bup often relies on git cat-file). > > I can reproduce the problem with a ~125GB (~3M object) repository on > an external SSD and a system with 16GB RAM via "git --batch-command < > fetch-oids" where fetch-oids contains 8k "info HASH" commands.
Just after sending this, I thought to wonder whether the issue might somehow be with the hardware/driver/etc. (that particular drive is an nvme ssd in an external usb-3 case), and while I'm not testing the exact same repository (it's a slightly larger, related one), preliminary results suggest cat-file behaves much more reasonably with an internal nvme drive, i.e. it starts slow, gets faster, and ends up repeatably at 37k hashes/sec after the second run.
So I'll probably test a bit more, but while I'd be quite interested in the cause, it seems likely my solution should just be to replace that hardware.
-- Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4