threads / patch / 41746

patchread-cache: increase write buffer size

Subject: [PATCH] read-cache: increase write buffer size

## tl;dr

5 messages between Mar 19, 2016 and Mar 21, 2016. Diffs are folded; open one to read it.

replies: 4people: 4as markdown or json

David Turner· Mar 19, 2016, 01:19 UTC · lore

Each write() has syscall overhead, and writing a large index entails many such calls. A larger write buffer reduces the overhead, leading to increased performance.

On my repo, which has an index size of 30m, this saves about 10ms of time writing the index.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 read-cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to read-cache.c +1 −1
diff --git a/read-cache.c b/read-cache.c
index d9fb78b..09ebe08 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)
 	return 0;
 }
 
-#define WRITE_BUFFER_SIZE 8192
+#define WRITE_BUFFER_SIZE 131072
 static unsigned char write_buffer[WRITE_BUFFER_SIZE];
 static unsigned long write_buffer_len;
 

-- 
2.4.2.767.g62658d5-twtrsrc
Torsten Bögershausen· Mar 19, 2016, 08:05 UTC · re: David Turner · lore

Re: [PATCH] read-cache: increase write buffer size

On 2016-03-19 02.19, David Turner wrote:
Show 26 quoted lines
> Each write() has syscall overhead, and writing a large index entails
> many such calls.  A larger write buffer reduces the overhead,
> leading to increased performance.
> 
> On my repo, which has an index size of 30m, this saves about 10ms of
> time writing the index.
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  read-cache.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/read-cache.c b/read-cache.c
> index d9fb78b..09ebe08 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)
>  	return 0;
>  }
>  
> -#define WRITE_BUFFER_SIZE 8192
> +#define WRITE_BUFFER_SIZE 131072
>  static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>  static unsigned long write_buffer_len;
>  
>

Do I read that right, it saves 10 milliseconds ? What happens to small system (like Raspberry PI), when you want 128K write buffer ? Could the buffer size be turned into a makefile variable, defaulting to 8192 ?

Pranit Bauva· Mar 19, 2016, 10:25 UTC · re: Torsten Bögershausen · lore

Re: [PATCH] read-cache: increase write buffer size

On Sat, Mar 19, 2016 at 1:35 PM, Torsten Bögershausen <tboegi@web.de> wrote:
Show 31 quoted lines
> On 2016-03-19 02.19, David Turner wrote:
>> Each write() has syscall overhead, and writing a large index entails
>> many such calls.  A larger write buffer reduces the overhead,
>> leading to increased performance.
>>
>> On my repo, which has an index size of 30m, this saves about 10ms of
>> time writing the index.
>>
>> Signed-off-by: David Turner <dturner@twopensource.com>
>> ---
>>  read-cache.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/read-cache.c b/read-cache.c
>> index d9fb78b..09ebe08 100644
>> --- a/read-cache.c
>> +++ b/read-cache.c
>> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)
>>       return 0;
>>  }
>>
>> -#define WRITE_BUFFER_SIZE 8192
>> +#define WRITE_BUFFER_SIZE 131072
>>  static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>>  static unsigned long write_buffer_len;
>>
>>
> Do I read that right, it saves 10 milliseconds ?
> What happens to small system (like Raspberry PI), when you
> want 128K write buffer ?
> Could the buffer size be turned into a makefile variable, defaulting to 8192 ?
Maybe for all ARM devices not just Raspberry Pi.

There is one way, though its not that good. This is just a sample code. I am unsure about if using bash in makefile is a good idea

in Makefile,
uname -a | grep "arm" >out
if [ -s out ]; then
    export ARM=arm
fi
rm out
in read-cache.c,
#define ARCH (getenv("ARCH"))
#ifdef ARCH,
    #define WRITE_BUFFER_SIZE 128000
#endif
Duy Nguyen· Mar 19, 2016, 11:18 UTC · re: David Turner · lore

Re: [PATCH] read-cache: increase write buffer size

On Sat, Mar 19, 2016 at 8:19 AM, David Turner <dturner@twopensource.com> wrote:
Show 6 quoted lines
> Each write() has syscall overhead, and writing a large index entails
> many such calls.  A larger write buffer reduces the overhead,
> leading to increased performance.
>
> On my repo, which has an index size of 30m, this saves about 10ms of
> time writing the index.

I wonder if split-index does not work as intended. Because if it does, you should rarely need to write that such big index files (or is this 30mb the small part and the base index is even bigger?).

But I agree with Torsten that we should make this configurable, preferably via config file, if not we can still move this define back in Makefile, overridable using config.mak

-- 
Duy
David Turner· Mar 21, 2016, 16:22 UTC · re: Duy Nguyen · lore

Re: [PATCH] read-cache: increase write buffer size

On Sat, 2016-03-19 at 18:18 +0700, Duy Nguyen wrote:
Show 19 quoted lines
> On Sat, Mar 19, 2016 at 8:19 AM, David Turner <
> dturner@twopensource.com> wrote:
> > Each write() has syscall overhead, and writing a large index
> > entails
> > many such calls.  A larger write buffer reduces the overhead,
> > leading to increased performance.
> > 
> > On my repo, which has an index size of 30m, this saves about 10ms
> > of
> > time writing the index.
> 
> I wonder if split-index does not work as intended. Because if it
> does,
> you should rarely need to write that such big index files (or is this
> 30mb the small part and the base index is even bigger?).
> 
> But I agree with Torsten that we should make this configurable,
> preferably via config file, if not we can still move this define back
> in Makefile, overridable using config.mak
Oh, right, split index.  Yeah, I should just turn that on.  Nevermind.

← back to recent threads