From: Patrick Steinhardt Date: Fri, 13 Mar 2026 06:45:11 GMT Subject: [PATCH v4 00/10] upload-pack: reduce lock contention when writing packfile data Message-ID: <20260313-pks-upload-pack-write-contention-v4-0-7a9668061f7f@pks.im> In-Reply-To: <20260227-pks-upload-pack-write-contention-v1-0-7166fe255704@pks.im> Hi, this small patch series fixes some heavy lock contention when writing data from git-upload-pack(1) into pipes. This lock contention can be observed when having hundreds of git-upload-pack(1) processes active at the same time that write data into pipes at dozens of gigabits per second. I have uploaded the flame graph that clearly shows the lock contention at [1]. Changes in v4: - Drop a stale half-sentence that I wanted to remove. - Link to v3: https://lore.kernel.org/r/20260310-pks-upload-pack-write-contention-v3-0-8bc97aa3e267@pks.im Changes in v3: - Fix handling of `iov_len` overflows in writev(3p) wrapper. - Add another patch that causes us to flush out data instead of sending a 0005 keepalive packet. - Link to v2: https://lore.kernel.org/r/20260303-pks-upload-pack-write-contention-v2-0-7321830f08fe@pks.im Changes in v2: - Change the buffer size in git-pack-objects(1) to also reduce the number of write syscalls over there. - Introduce writev to half the number of syscalls when writing pktlines. - Use `sizeof(os->buffer)` instead of open-coding its size. - Improve keepalive logic in git-upload-pack(1) to account for buffering. - Link to v1: https://lore.kernel.org/r/20260227-pks-upload-pack-write-contention-v1-0-7166fe255704@pks.im Thanks! Patrick [1]: https://gitlab.com/gitlab-org/git/-/work_items/675 --- Patrick Steinhardt (10): upload-pack: fix debug statement when flushing packfile data upload-pack: adapt keepalives based on buffering upload-pack: prefer flushing data over sending keepalive upload-pack: reduce lock contention when writing packfile data compat/posix: introduce writev(3p) wrapper wrapper: introduce writev(3p) wrappers sideband: use writev(3p) to send pktlines csum-file: introduce `hashfd_ext()` csum-file: drop `hashfd_throughput()` builtin/pack-objects: reduce lock contention when writing packfile data Makefile | 4 +++ builtin/pack-objects.c | 23 +++++++++++--- compat/posix.h | 14 +++++++++ compat/writev.c | 44 +++++++++++++++++++++++++++ config.mak.uname | 2 ++ csum-file.c | 28 +++++------------ csum-file.h | 16 ++++++++-- meson.build | 1 + sideband.c | 14 +++++++-- upload-pack.c | 81 +++++++++++++++++++++++++++++++++++++++----------- wrapper.c | 41 +++++++++++++++++++++++++ wrapper.h | 9 ++++++ write-or-die.c | 8 +++++ write-or-die.h | 1 + 14 files changed, 239 insertions(+), 47 deletions(-) Range-diff versus v3: 1: 65471f969b = 1: f9896a8451 upload-pack: fix debug statement when flushing packfile data 2: c5705c1cb1 = 2: b1bc6f5749 upload-pack: adapt keepalives based on buffering 3: f34fa584f4 ! 3: fcf5f06375 upload-pack: prefer flushing data over sending keepalive @@ Commit message the early bit waiting for packfile URIs. But the optimization is easy enough to realize. - Do so and flush out data instead of sending an empty pktline. While at - it, drop the useless + Do so and flush out data instead of sending an empty pktline. Suggested-by: Jeff King Signed-off-by: Patrick Steinhardt 4: 8d08e93929 = 4: f2bb7a38aa upload-pack: reduce lock contention when writing packfile data 5: aa42217e43 = 5: 78a1bbb810 compat/posix: introduce writev(3p) wrapper 6: c5d194ca2b = 6: c199dd398a wrapper: introduce writev(3p) wrappers 7: e4df805840 = 7: b7fe2b818f sideband: use writev(3p) to send pktlines 8: 16f9968bcd = 8: b9d33b93cd csum-file: introduce `hashfd_ext()` 9: 47ae58440b = 9: b2af62c4ff csum-file: drop `hashfd_throughput()` 10: c92ccf9df2 = 10: 47f5090b66 builtin/pack-objects: reduce lock contention when writing packfile data --- base-commit: fb1b83bcddff60463f6e86bb021784c88d0b748c change-id: 20260227-pks-upload-pack-write-contention-435ce01f5fe9