Re: [PATCH] read-cache: make the index write buffer size 128K

From: Jeff Hostetler <git@jeffhostetler.com>
Date: Feb 19, 2021, 19:12 UTC
Message-ID: <f52df30b-4ab0-fd6f-17f8-70daed81df39@jeffhostetler.com>
In-Reply-To: <pull.877.git.1613616506949.gitgitgadget@gmail.com>

On 2/17/21 9:48 PM, Neeraj K. Singh via GitGitGadget wrote:

Show 11 quoted lines

> From: Neeraj Singh <neerajsi@ntdev.microsoft.com>
> 
> Writing an index 8K at a time invokes the OS filesystem and caching code
> very frequently, introducing noticeable overhead while writing large
> indexes. When experimenting with different write buffer sizes on Windows
> writing the Windows OS repo index (260MB), most of the benefit came by
> bumping the index write buffer size to 64K. I picked 128K to ensure that
> we're past the knee of the curve.
> 
> With this change, the time under do_write_index for an index with 3M
> files goes from ~1.02s to ~0.72s.

[...]

Show 5 quoted lines

>   
> -#define WRITE_BUFFER_SIZE 8192
> +#define WRITE_BUFFER_SIZE (128 * 1024)
>   static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>   static unsigned long write_buffer_len;

[...]

Very nice.

I can confirm that this gives nice gains on Windows. (I'm using the Office repo which has a 188MB index file (2.1M files at HEAD). Running "git status" shows a gain of about 200ms.

We get a smaller gain on Mac of about 50ms (again, using the Office repo).

So, you may add my sign-off or ACK to this.
     Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>

FWIW, You might take a look at `t/perf/p0007-write-cache.sh` Update it as follows:

```

diff --git a/t/perf/p0007-write-cache.sh b/t/perf/p0007-write-cache.sh
index 09595264f0..337280ff1c 100755
--- a/t/perf/p0007-write-cache.sh
+++ b/t/perf/p0007-write-cache.sh
@@ -4,7 +4,8 @@ test_description="Tests performance of writing the index"

  . ./perf-lib.sh

-test_perf_default_repo
+test_perf_large_repo

  test_expect_success "setup repo" '
         if git rev-parse --verify refs/heads/p0006-ballast^{commit}
```


Then you can run it like this:

     $ cd t/perf
     $ GIT_PERF_LARGE_REPO=/path/to/your/enlistment ./p0007-write-cache

Then you can run it with the small and then with the large buffer and
get times for essentially just the index write in isolation.

Hope this helps,
Jeff