git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB

From
Junio C Hamano <gitster@pobox.com>
Date
Apr 28, 2026, 01:46 UTC
Message-ID
<xmqqecjz26wr.fsf@gitster.g>
In-Reply-To
<c19a0e29-1218-4239-a362-df514153b5ff@gmail.com>
Derrick Stolee <stolee@gmail.com> writes:
Show 8 quoted lines
>> The difference between that and 66% is coming from where?  There are
>> inherently short writes that do not utilize the new larger buffer
>> beyond 4kB?  If so, another number of interest might be the number
>> of writes smaller than 4096 bytes, perhaps?
>  
> One way to reword what you're asking is to measure "number of writes
> not using the whole buffer" which is basically going to be "the
> number of flush events from the application layer".

I do not think it would differ between the old and the new implementation.

> Every time the
> application intends to flush, the current buffer is likely to not
> be exactly full. I would expect this number to not change between
> implementations in real experiments.
Yes, I agree.
But what I was trying to get at was a bit different.  

The application may have produced only 2kB before it issues a "flush". Whether the buffer size is 4kB or 128kB, such a flush will only write out 2kB, and the larger buffer size does not help at all. But if the application has produced 90kB before it issues a "flush", the larger buffer size would give us a great improvement. With 4kB buffer, before such an application level "flush", we would have seen 22 = floor(90/4) calls of write(2) to flush the buffer, plus a 2kB write(2). With 128kB buffer, we would see a single 90kB write(2).

So the apparently lower improvement than I naively have expected may be attributable to the fact that many application level "flush" was not large enough to benefit from 128kB buffer? How much of the total number of bytes written came in large batches, vs tiny ones?

> The improvement here comes from the reduced number of flushes due
> to buffer limits.
Yes.
Show 5 quoted lines
> I see that this can be measured in the number of
> system-level events, but what impact does this have on the end-to-
> end time of 'git index-pack' or 'git unpack-objects'? Is there a
> t/perf/ test that can demonstrate this improvement for a variety
> of real repos using GIT_PERF_REPO?

Interesting thought, but the number of system-level events (or the number of write(2) system calls) is not reduced by 97% because we apparently are issuing too many of them, and the reason is? I suspect the reason why we still issue too many write(2) is because we often do not send enough data between application-level flushes.

Previous: Derrick StoleeNext: Jeff King
Message 4 of 12 in “index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB”
  1. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 24, 2026
  2. Junio C HamanoApr 25, 2026
  3. Derrick StoleeApr 27, 2026
  4. Junio C HamanoApr 28, 2026
  5. Jeff KingApr 28, 2026
  6. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  7. Derrick StoleeApr 27, 2026
  8. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  9. Derrick StoleeApr 27, 2026
  10. Junio C HamanoApr 28, 2026
  11. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 28, 2026
  12. Junio C HamanoMay 12, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.