git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB

From
Jeff King <peff@peff.net>
Date
Apr 28, 2026, 02:09 UTC
Message-ID
<20260428020933.GA660154@coredump.intra.peff.net>
In-Reply-To
<xmqqecjz26wr.fsf@gitster.g>
On Tue, Apr 28, 2026 at 10:46:44AM +0900, Junio C Hamano wrote:
Show 13 quoted lines
> The application may have produced only 2kB before it issues a
> "flush".  Whether the buffer size is 4kB or 128kB, such a flush will
> only write out 2kB, and the larger buffer size does not help at all.
> But if the application has produced 90kB before it issues a "flush",
> the larger buffer size would give us a great improvement.  With 4kB
> buffer, before such an application level "flush", we would have seen
> 22 = floor(90/4) calls of write(2) to flush the buffer, plus a 2kB
> write(2).  With 128kB buffer, we would see a single 90kB write(2).
> 
> So the apparently lower improvement than I naively have expected may
> be attributable to the fact that many application level "flush" was
> not large enough to benefit from 128kB buffer?  How much of the
> total number of bytes written came in large batches, vs tiny ones?

The input to index-pack in a fetch is going to be the demuxing of the sideband via git-fetch. So it's probably flushing 64k or less each time (because that's the max size of a packet), and unless index-pack is going much slower than the input, that maximizes how much it will read.

Depending on the source, though, it may be possible to go faster than index-pack (which has to at least update the pack checksum for every byte, and may even zlib inflate and hash the object itself if it's a non-delta). In which case the sideband demuxer would start filling the pipe and index-pack may get larger reads.

We could actually reduce the number of syscalls further if index-pack did the demuxing itself, and we just handed it the descriptor. That probably doesn't help all that much in this case, though, if the problem is not raw reads/writes on pipes, but rather ones that go to the slow FUSE filesystem. And as long as those pipe reads/writes are "wide" (allowing the eventual filesystem writes to also be wide), then the exact number may not be as important.

But the demuxing may also explain why the total number of writes did not decrease as much as you expected, since those ones will probably not be reduced by the patch in question. So the improvement is a percentage of only a smaller portion of the total (but not necessarily half, because they may have been larger writes in the first place).

-Peff
Previous: Junio C HamanoNext: Scott Bauersfeld via GitGitGadget
Message 5 of 12 in “index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB”
  1. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 24, 2026
  2. Junio C HamanoApr 25, 2026
  3. Derrick StoleeApr 27, 2026
  4. Junio C HamanoApr 28, 2026
  5. Jeff KingApr 28, 2026
  6. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  7. Derrick StoleeApr 27, 2026
  8. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  9. Derrick StoleeApr 27, 2026
  10. Junio C HamanoApr 28, 2026
  11. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 28, 2026
  12. Junio C HamanoMay 12, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.