git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] read-cache: make the index write buffer size 128K

From
Junio C Hamano <gitster@pobox.com>
Date
Feb 20, 2021, 03:28 UTC
Message-ID
<xmqqv9ana05b.fsf@gitster.g>
In-Reply-To
<f52df30b-4ab0-fd6f-17f8-70daed81df39@jeffhostetler.com>
Jeff Hostetler <git@jeffhostetler.com> writes:
Show 22 quoted lines
> On 2/17/21 9:48 PM, Neeraj K. Singh via GitGitGadget wrote:
>> From: Neeraj Singh <neerajsi@ntdev.microsoft.com>
>> Writing an index 8K at a time invokes the OS filesystem and caching
>> code
>> very frequently, introducing noticeable overhead while writing large
>> indexes. When experimenting with different write buffer sizes on Windows
>> writing the Windows OS repo index (260MB), most of the benefit came by
>> bumping the index write buffer size to 64K. I picked 128K to ensure that
>> we're past the knee of the curve.
>> With this change, the time under do_write_index for an index with 3M
>> files goes from ~1.02s to ~0.72s.
>
> [...]
>
>>   -#define WRITE_BUFFER_SIZE 8192
>> +#define WRITE_BUFFER_SIZE (128 * 1024)
>>   static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>>   static unsigned long write_buffer_len;
>
> [...]
>
> Very nice.
I wonder if we gain more by going say 4M buffer size or even larger?

Is this something we can make the system auto-tune itself? This is not about reading but writing, so we already have enough information to estimate how much we would need to write out.

Thanks.
Previous: Jeff HostetlerNext: Neeraj Singh
Message 3 of 11 in “read-cache: make the index write buffer size 128K”
  1. read-cache: make the index write buffer size 128KNeeraj K. Singh via GitGitGadget, Feb 18, 2021
  2. Jeff HostetlerFeb 19, 2021
  3. Junio C HamanoFeb 20, 2021
  4. Neeraj SinghFeb 20, 2021
  5. Junio C HamanoFeb 21, 2021
  6. Neeraj SinghFeb 24, 2021
  7. Junio C HamanoFeb 25, 2021
  8. Chris TorekFeb 25, 2021
  9. Junio C HamanoFeb 25, 2021
  10. Neeraj SinghFeb 25, 2021
  11. Chris TorekFeb 25, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.