git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4] index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB

From
Junio C Hamano <gitster@pobox.com>
Date
May 12, 2026, 05:51 UTC
Message-ID
<xmqqy0hpnpkb.fsf@gitster.g>
In-Reply-To
<pull.2282.v4.git.git.1777387660841.gitgitgadget@gmail.com>
"Scott Bauersfeld via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 49 quoted lines
> From: Scott Bauersfeld <sbauersfeld@g.ucla.edu>
>
> index-pack and unpack-objects both read pack data from stdin through
> a 4 KiB static buffer. In index-pack, each fill() flushes consumed
> bytes to the pack file via write_or_die(), capping every write(2)
> at 4 KiB. unpack-objects uses the same buffer pattern for reads.
>
> On FUSE-backed filesystems every write(2) is a synchronous round
> trip through the FUSE protocol (userspace -> kernel -> userspace ->
> back), so the 4 KiB buffer turns a clone into many unnecessary tiny
> writes with noticeable latency overhead.
>
> Increase the buffer from 4 KiB to 128 KiB. Introduce a shared
> DEFAULT_IO_BUFFER_SIZE constant in git-compat-util.h (next to
> MAX_IO_SIZE) and use it in index-pack, unpack-objects, and the
> hashfile layer in csum-file (which already used 128 KiB but
> hardcoded the value).
>
> Pack file writes to a FUSE filesystem with writeback caching
> disabled during HTTPS clones of git/git (~293 MB pack):
>
>   74,958 -> 4,687 (94% fewer)
>
> Wall-clock time of git clone over HTTPS onto a FUSE passthrough
> filesystem with writeback caching disabled, 3 runs per variant:
>
>   vscode (~1.26 GB pack): 84.5s -> 75.7s avg (10% faster)
>   git/git (~306 MB pack):  22.6s -> 20.0s avg (11% faster)
>
> Signed-off-by: Scott Bauersfeld <sbauersfeld@g.ucla.edu>
> ---
>...
>     
>     Changes since v3
>     ================
>     
>      * Replaced strace-based syscall measurements with FUSE daemon write
>        logging. The earlier strace numbers (72,465 → 24,943, 65% reduction)
>        were distorted: strace -f ptrace intercepts every syscall in all
>        traced processes and added enough overhead to distort the
>        measurements. The FUSE daemon logging captures write sizes without
>        perturbing the traced processes, showing the true reduction is 94%
>        (74,958 → 4,687).
>      * Note: Why 4,687 writes instead of ~2k writes as would be expected
>        with a 128 KiB buffer size? It appears that fill() is calling xread()
>        on a pipe and the linux default buffer size for pipes is 64KiB. I
>        also tested using fcntl(F_SETPIPE_SZ) to increase the pipe's buffer
>        size to 128KiB, which does indeed reduce total pack file writes to
>        ~2.4K.

It seems that everybody was happy with v3 already, so let's merge it down to 'next'.

Thanks.
Previous: Scott Bauersfeld via GitGitGadget
Message 12 of 12 in “index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB”
  1. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 24, 2026
  2. Junio C HamanoApr 25, 2026
  3. Derrick StoleeApr 27, 2026
  4. Junio C HamanoApr 28, 2026
  5. Jeff KingApr 28, 2026
  6. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  7. Derrick StoleeApr 27, 2026
  8. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  9. Derrick StoleeApr 27, 2026
  10. Junio C HamanoApr 28, 2026
  11. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 28, 2026
  12. Junio C HamanoMay 12, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.