Re: [PATCH v2] index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB
- From
Derrick Stolee <stolee@gmail.com>
- Date
- Apr 27, 2026, 17:23 UTC
- Message-ID
- <5498637e-178f-48aa-8cdc-adc38b100627@gmail.com>
- In-Reply-To
- <pull.2282.v2.git.git.1777306114914.gitgitgadget@gmail.com>
On 4/27/2026 12:08 PM, Scott Bauersfeld via GitGitGadget wrote:
> From: Scott Bauersfeld <sbauersfeld@g.ucla.edu>
...
Show 5 quoted lines
> Wall-clock time of git clone over HTTPS onto a FUSE passthrough > filesystem with writeback caching disabled, 3 runs per variant: > > vscode (~1.26 GB pack): 84.5s -> 75.7s avg (10% faster) > git/git (~306 MB pack): 22.6s -> 20.0s avg (11% faster)
Wow! This is much higher than I expected. Great find.
I imagine that other platforms or non-FUSE setups will not have the same benefits. As long as they aren't _regressions_ then this is a great find.
> -/* We always read in 4kB chunks. */ > -static unsigned char input_buffer[4096]; > +static unsigned char input_buffer[DEFAULT_PACKFILE_BUFFER_SIZE];
> -/* We always read in 4kB chunks. */ > -static unsigned char buffer[4096]; > +static unsigned char buffer[DEFAULT_PACKFILE_BUFFER_SIZE];
These changes are what I expected in v2.
Show 12 quoted lines
> diff --git a/csum-file.c b/csum-file.c > index 9558177a11..c1aeaf587a 100644 > --- a/csum-file.c > +++ b/csum-file.c > @@ -178,7 +178,7 @@ struct hashfile *hashfd_ext(const struct git_hash_algo *algop, > f->algop = unsafe_hash_algo(algop); > f->algop->init_fn(&f->ctx); > > - f->buffer_len = opts->buffer_len ? opts->buffer_len : 128 * 1024; > + f->buffer_len = opts->buffer_len ? opts->buffer_len : DEFAULT_PACKFILE_BUFFER_SIZE; > f->buffer = xmalloc(f->buffer_len); > f->check_buffer = NULL;
This one surprised me, as this hunk wasn't in your v1 patch.
I think using this replacement makes sense, since it _is_ an exact value. It did make me think as to how we landed on 128K for this example.
The previous line is due to a1118c0a446 (csum-file: introduce `hashfd_ext()`, 2026-03-13), but it only moved the 128K default from hashfd(). Notably, hashfd_throughput() still uses an 8K setting in opt->buffer_len.
Hilariously, I went spelunking for the original reason for the 128K and it was 2ca245f8be5 (csum-file.h: increase hashfile buffer size, 2021-05-18) written by...me. The motivation was due to using the hashfile logic for the .git/index file which also used 128K buffers in f279894 (read-cache: make the index write buffer size 128K, 2021-02-18).
All this is to say that we now have two constants of identical value, where WRITE_BUFFER_SIZE in read-cache.c could be replaced with your new DEFAULT_PACKFILE_BUFFER_SIZE.
This does make me think that maybe DEFAULT_PACKFILE_BUFFER_SIZE is misnamed? Should it be DEFAULT_HASHFILE_BUFFER_SIZE or DEFAULT_FILESYSTEM_BUFFER_SIZE to better fit this size value being used in both packfiles and index files?
Show 14 quoted lines
> diff --git a/git-compat-util.h b/git-compat-util.h > index ae1bdc90a4..a2f037811c 100644 > --- a/git-compat-util.h > +++ b/git-compat-util.h > @@ -712,6 +712,12 @@ static inline uint64_t u64_add(uint64_t a, uint64_t b) > # endif > #endif > > +/* > + * Default buffer size for buffered I/O in pack file operations (index-pack, > + * unpack-objects) and the hashfile layer in csum-file. > + */ > +#define DEFAULT_PACKFILE_BUFFER_SIZE (128 * 1024) > +
I see. Putting this in git-compat-util.h makes the rest of the changes good without any need to add a new include.
Thanks, -Stolee