git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB

From
Derrick Stolee <stolee@gmail.com>
Date
Apr 27, 2026, 12:36 UTC
Message-ID
<c19a0e29-1218-4239-a362-df514153b5ff@gmail.com>
In-Reply-To
<xmqqldeb9w8e.fsf@gitster.g>
On 4/25/2026 6:21 AM, Junio C Hamano wrote:
Show 22 quoted lines
> "Scott Bauersfeld via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Scott Bauersfeld <sbauersfeld@g.ucla.edu>
>>
>> On FUSE-backed filesystems every write(2) is a synchronous round
>> trip through the FUSE protocol (userspace -> kernel -> userspace ->
>> back), so the 4 KiB buffer turns a clone into many unnecessary tiny
>> writes with noticeable latency overhead.
>>
>> Increase the buffer from 4 KiB to 128 KiB, matching the default
>> already used by the hashfile layer in csum-file.c.
> 
> Quite sensible reasoning presented very nicely.
> 
> It may probably be a #leftoverbit but these three instances of (128
> * 1024) may want to have a common symbolic constant, like
> 
>     #define DEFAULT_IOBUFFER_SIZE_IN_BYTES (128 * 1024)
> 
> in a bit more central header file.  Especially for the one in
> csum-file.c where there is no symbolic constant used for that
> purpose.
I also had this thought. Would environment.h be the best place? 
Show 12 quoted lines
>> Testing with strace on HTTPS clones of git/git (~296 MB pack, 5 runs
>> per variant, isolated builds from the same v2.54.0 source) shows:
>>
>>   index-pack pack file writes: 72,465 -> 24,943 avg (66% reduction)
>>   total write() syscalls:     310,192 -> 259,530 avg (17% reduction)
>>   writes of exactly 4096 bytes: ~40,077 -> 0 (eliminated)
> 
> Hmph, I would have expected more like (1 - 4/128) ~ 97% reduction.
> The difference between that and 66% is coming from where?  There are
> inherently short writes that do not utilize the new larger buffer
> beyond 4kB?  If so, another number of interest might be the number
> of writes smaller than 4096 bytes, perhaps?
 
One way to reword what you're asking is to measure "number of writes
not using the whole buffer" which is basically going to be "the
number of flush events from the application layer". Every time the
application intends to flush, the current buffer is likely to not
be exactly full. I would expect this number to not change between
implementations in real experiments.

The improvement here comes from the reduced number of flushes due to buffer limits. I see that this can be measured in the number of system-level events, but what impact does this have on the end-to- end time of 'git index-pack' or 'git unpack-objects'? Is there a t/perf/ test that can demonstrate this improvement for a variety of real repos using GIT_PERF_REPO?

Thanks, -Stolee

Previous: Junio C HamanoNext: Junio C Hamano
Message 3 of 12 in “index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB”
  1. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 24, 2026
  2. Junio C HamanoApr 25, 2026
  3. Derrick StoleeApr 27, 2026
  4. Junio C HamanoApr 28, 2026
  5. Jeff KingApr 28, 2026
  6. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  7. Derrick StoleeApr 27, 2026
  8. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  9. Derrick StoleeApr 27, 2026
  10. Junio C HamanoApr 28, 2026
  11. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 28, 2026
  12. Junio C HamanoMay 12, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.