Re: [PATCH 2/2] upload-pack: reduce lock contention when writing packfile data

From

Jeff King <peff@peff.net>

Date

Feb 27, 2026, 19:37 UTC

Message-ID

<20260227193758.GA2931515@coredump.intra.peff.net>

In-Reply-To

<20260227-pks-upload-pack-write-contention-v1-2-7166fe255704@pks.im>

On Fri, Feb 27, 2026 at 12:23:01PM +0100, Patrick Steinhardt wrote:

> Extend our use of the buffering infrastructure so that we soak up bytes
> until the buffer is filled up at least 2/3rds of its capacity. The
> change is relatively simple to implement as we already know to flush the
> buffer in `create_pack_file()` after git-pack-objects(1) has finished.

Something like this:

diff --git a/csum-file.c b/csum-file.c
index 6e21e3cac8..94798fa429 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -206,7 +206,7 @@ struct hashfile *hashfd_throughput(const struct git_hash_algo *algop,
 	 * size so the progress indicators arrive at a more
 	 * frequent rate.
 	 */
-	return hashfd_internal(algop, fd, name, tp, 8 * 1024);
+	return hashfd_internal(algop, fd, name, tp, 32 * 1024);
 }
 
 void hashfile_checkpoint_init(struct hashfile *f,

reduces the number of write calls reported by:

  git clone \
    --upload-pack='perf stat -e syscalls:sys_enter_write git-upload-pack' \
    --bare --no-local linux.git foo.git

from ~420k to ~160k. In theory we expect ~8x reduction in our target
area, 4x for each of pack-objects and upload-pack, but of course there
are other writes going on, too, including the extra sideband ones. And
obviously we could push it further towards LARGE_PACKET_MAX to save even
more.

> Now git-upload-pack(1) already has the infrastructure in place to buffer
> some of the data it reads from git-pack-objects(1) before actually
> sending it out. We only use this infrastructure in very limited ways
> though, so we generally end up matching one read(3p) call with one
> write(3p) call. Even worse, when the sideband is enabled we end up
> matching one read with _two_ writes: one for the pkt-line length, and
> one for the packfile data.

Using writev() would be an easy-ish fix here, modulo portability
concerns (though of course it is easy to implement a fallback writev()
in terms of write()). Doing this:

diff --git a/sideband.c b/sideband.c
index ea7c25211e..b5509fbaa2 100644
--- a/sideband.c
+++ b/sideband.c
@@ -266,19 +266,25 @@ void send_sideband(int fd, int band, const char *data, ssize_t sz, int packet_ma
 	while (sz) {
 		unsigned n;
 		char hdr[5];
+		struct iovec iov[2];
 
 		n = sz;
 		if (packet_max - 5 < n)
 			n = packet_max - 5;
 		if (0 <= band) {
 			xsnprintf(hdr, sizeof(hdr), "%04x", n + 5);
 			hdr[4] = band;
-			write_or_die(fd, hdr, 5);
+			iov[0].iov_base = hdr;
+			iov[0].iov_len = 5;
 		} else {
 			xsnprintf(hdr, sizeof(hdr), "%04x", n + 4);
-			write_or_die(fd, hdr, 4);
+			iov[0].iov_base = hdr;
+			iov[0].iov_len = 4;
 		}
-		write_or_die(fd, p, n);
+		iov[1].iov_base = p;
+		iov[1].iov_len = n;
+		/* obviously needs looping and error detection */
+		writev(fd, iov, 2);
 		p += n;
 		sz -= n;
 	}

drops my 160k write calls down to 82k.

Another option here is teaching the packet-forming code to reserve a few
bytes at the front of the packet. There's a little discussion here:

  https://lore.kernel.org/git/YBkeYSA5UfQP1m%2Fx@coredump.intra.peff.net/

In theory it's easy and elegant to do, but I'm not sure what the
refactoring fallout would be like.

> This significantly reduces the number of write(3p) syscalls we need to
> do. Before this change, cloning the Linux repository resulted in around
> 400,000 write(3p) syscalls. With the buffering in place we only do
> around 130,000 syscalls.

Out of curiosity, how did you end up measuring? I first tried with
strace (without "-f") on the upload-pack process, but strace slowed it
enough that it ended up collecting multiple of pack-object's 8k write()
calls in a single read() call. ;) The "perf stat" above seemed to work
OK, though of course it's counting child processes, too.

-Peff