git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2] index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB

From
Derrick Stolee <stolee@gmail.com>
Date
Apr 27, 2026, 17:23 UTC
Message-ID
<5498637e-178f-48aa-8cdc-adc38b100627@gmail.com>
In-Reply-To
<pull.2282.v2.git.git.1777306114914.gitgitgadget@gmail.com>
On 4/27/2026 12:08 PM, Scott Bauersfeld via GitGitGadget wrote:
> From: Scott Bauersfeld <sbauersfeld@g.ucla.edu>
...
Show 5 quoted lines
> Wall-clock time of git clone over HTTPS onto a FUSE passthrough
> filesystem with writeback caching disabled, 3 runs per variant:
> 
>   vscode (~1.26 GB pack): 84.5s -> 75.7s avg (10% faster)
>   git/git (~306 MB pack):  22.6s -> 20.0s avg (11% faster)
Wow! This is much higher than I expected. Great find.

I imagine that other platforms or non-FUSE setups will not have the same benefits. As long as they aren't _regressions_ then this is a great find.

> -/* We always read in 4kB chunks. */
> -static unsigned char input_buffer[4096];
> +static unsigned char input_buffer[DEFAULT_PACKFILE_BUFFER_SIZE];
> -/* We always read in 4kB chunks. */
> -static unsigned char buffer[4096];
> +static unsigned char buffer[DEFAULT_PACKFILE_BUFFER_SIZE];
These changes are what I expected in v2.
Show 12 quoted lines
> diff --git a/csum-file.c b/csum-file.c
> index 9558177a11..c1aeaf587a 100644
> --- a/csum-file.c
> +++ b/csum-file.c
> @@ -178,7 +178,7 @@ struct hashfile *hashfd_ext(const struct git_hash_algo *algop,
>  	f->algop = unsafe_hash_algo(algop);
>  	f->algop->init_fn(&f->ctx);
>  
> -	f->buffer_len = opts->buffer_len ? opts->buffer_len : 128 * 1024;
> +	f->buffer_len = opts->buffer_len ? opts->buffer_len : DEFAULT_PACKFILE_BUFFER_SIZE;
>  	f->buffer = xmalloc(f->buffer_len);
>  	f->check_buffer = NULL;
This one surprised me, as this hunk wasn't in your v1 patch.

I think using this replacement makes sense, since it _is_ an exact value. It did make me think as to how we landed on 128K for this example.

The previous line is due to a1118c0a446 (csum-file: introduce `hashfd_ext()`, 2026-03-13), but it only moved the 128K default from hashfd(). Notably, hashfd_throughput() still uses an 8K setting in opt->buffer_len.

Hilariously, I went spelunking for the original reason for the 128K and it was 2ca245f8be5 (csum-file.h: increase hashfile buffer size, 2021-05-18) written by...me. The motivation was due to using the hashfile logic for the .git/index file which also used 128K buffers in f279894 (read-cache: make the index write buffer size 128K, 2021-02-18).

All this is to say that we now have two constants of identical value, where WRITE_BUFFER_SIZE in read-cache.c could be replaced with your new DEFAULT_PACKFILE_BUFFER_SIZE.

This does make me think that maybe DEFAULT_PACKFILE_BUFFER_SIZE is misnamed? Should it be DEFAULT_HASHFILE_BUFFER_SIZE or DEFAULT_FILESYSTEM_BUFFER_SIZE to better fit this size value being used in both packfiles and index files?

Show 14 quoted lines
> diff --git a/git-compat-util.h b/git-compat-util.h
> index ae1bdc90a4..a2f037811c 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -712,6 +712,12 @@ static inline uint64_t u64_add(uint64_t a, uint64_t b)
>  # endif
>  #endif
>  
> +/*
> + * Default buffer size for buffered I/O in pack file operations (index-pack,
> + * unpack-objects) and the hashfile layer in csum-file.
> + */
> +#define DEFAULT_PACKFILE_BUFFER_SIZE (128 * 1024)
> +

I see. Putting this in git-compat-util.h makes the rest of the changes good without any need to add a new include.

Thanks, -Stolee

Previous: Scott Bauersfeld via GitGitGadgetNext: Scott Bauersfeld via GitGitGadget
Message 7 of 12 in “index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB”
  1. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 24, 2026
  2. Junio C HamanoApr 25, 2026
  3. Derrick StoleeApr 27, 2026
  4. Junio C HamanoApr 28, 2026
  5. Jeff KingApr 28, 2026
  6. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  7. Derrick StoleeApr 27, 2026
  8. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 27, 2026
  9. Derrick StoleeApr 27, 2026
  10. Junio C HamanoApr 28, 2026
  11. index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiBScott Bauersfeld via GitGitGadget, Apr 28, 2026
  12. Junio C HamanoMay 12, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.