Re: [PATCH] read-cache: make the index write buffer size 128K
- From
Junio C Hamano <gitster@pobox.com>
- Date
- Feb 20, 2021, 03:28 UTC
- Message-ID
- <xmqqv9ana05b.fsf@gitster.g>
- In-Reply-To
- <f52df30b-4ab0-fd6f-17f8-70daed81df39@jeffhostetler.com>
Jeff Hostetler <git@jeffhostetler.com> writes:
Show 22 quoted lines
> On 2/17/21 9:48 PM, Neeraj K. Singh via GitGitGadget wrote: >> From: Neeraj Singh <neerajsi@ntdev.microsoft.com> >> Writing an index 8K at a time invokes the OS filesystem and caching >> code >> very frequently, introducing noticeable overhead while writing large >> indexes. When experimenting with different write buffer sizes on Windows >> writing the Windows OS repo index (260MB), most of the benefit came by >> bumping the index write buffer size to 64K. I picked 128K to ensure that >> we're past the knee of the curve. >> With this change, the time under do_write_index for an index with 3M >> files goes from ~1.02s to ~0.72s. > > [...] > >> -#define WRITE_BUFFER_SIZE 8192 >> +#define WRITE_BUFFER_SIZE (128 * 1024) >> static unsigned char write_buffer[WRITE_BUFFER_SIZE]; >> static unsigned long write_buffer_len; > > [...] > > Very nice.
I wonder if we gain more by going say 4M buffer size or even larger?
Is this something we can make the system auto-tune itself? This is not about reading but writing, so we already have enough information to estimate how much we would need to write out.
Thanks.