# [PATCH] read-cache: increase write buffer size 5 messages from 2016-03-19 to 2016-03-21. Participants: David Turner, Torsten Bögershausen, Pranit Bauva, Duy Nguyen. Thread: https://gitlist.dev/t/41746 ## David Turner, 2016-03-19 01:19 Subject: [PATCH] read-cache: increase write buffer size Message-ID: <1458350341-12276-1-git-send-email-dturner@twopensource.com> URL: https://gitlist.dev/e/1458350341-12276-1-git-send-email-dturner%40twopensource.com ``` Each write() has syscall overhead, and writing a large index entails many such calls. A larger write buffer reduces the overhead, leading to increased performance. On my repo, which has an index size of 30m, this saves about 10ms of time writing the index. Signed-off-by: David Turner --- read-cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/read-cache.c b/read-cache.c index d9fb78b..09ebe08 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate) return 0; } -#define WRITE_BUFFER_SIZE 8192 +#define WRITE_BUFFER_SIZE 131072 static unsigned char write_buffer[WRITE_BUFFER_SIZE]; static unsigned long write_buffer_len; -- 2.4.2.767.g62658d5-twtrsrc ``` ## Torsten Bögershausen, 2016-03-19 08:05 Subject: Re: [PATCH] read-cache: increase write buffer size Message-ID: <56ED0863.3020709@web.de> URL: https://gitlist.dev/e/56ED0863.3020709%40web.de In-Reply-To: <1458350341-12276-1-git-send-email-dturner@twopensource.com> ``` On 2016-03-19 02.19, David Turner wrote: > Each write() has syscall overhead, and writing a large index entails > many such calls. A larger write buffer reduces the overhead, > leading to increased performance. > > On my repo, which has an index size of 30m, this saves about 10ms of > time writing the index. > > Signed-off-by: David Turner > --- > read-cache.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/read-cache.c b/read-cache.c > index d9fb78b..09ebe08 100644 > --- a/read-cache.c > +++ b/read-cache.c > @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate) > return 0; > } > > -#define WRITE_BUFFER_SIZE 8192 > +#define WRITE_BUFFER_SIZE 131072 > static unsigned char write_buffer[WRITE_BUFFER_SIZE]; > static unsigned long write_buffer_len; > > Do I read that right, it saves 10 milliseconds ? What happens to small system (like Raspberry PI), when you want 128K write buffer ? Could the buffer size be turned into a makefile variable, defaulting to 8192 ? ``` ## Pranit Bauva, 2016-03-19 10:25 Subject: Re: [PATCH] read-cache: increase write buffer size Message-ID: URL: https://gitlist.dev/e/CAFZEwPMtv0JF%2BQ_1P53NB2oxH-XJXGzmiUsOxk0KBaa%2BAJE%2BvA%40mail.gmail.com In-Reply-To: <56ED0863.3020709@web.de> ``` On Sat, Mar 19, 2016 at 1:35 PM, Torsten Bögershausen wrote: > On 2016-03-19 02.19, David Turner wrote: >> Each write() has syscall overhead, and writing a large index entails >> many such calls. A larger write buffer reduces the overhead, >> leading to increased performance. >> >> On my repo, which has an index size of 30m, this saves about 10ms of >> time writing the index. >> >> Signed-off-by: David Turner >> --- >> read-cache.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/read-cache.c b/read-cache.c >> index d9fb78b..09ebe08 100644 >> --- a/read-cache.c >> +++ b/read-cache.c >> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate) >> return 0; >> } >> >> -#define WRITE_BUFFER_SIZE 8192 >> +#define WRITE_BUFFER_SIZE 131072 >> static unsigned char write_buffer[WRITE_BUFFER_SIZE]; >> static unsigned long write_buffer_len; >> >> > Do I read that right, it saves 10 milliseconds ? > What happens to small system (like Raspberry PI), when you > want 128K write buffer ? > Could the buffer size be turned into a makefile variable, defaulting to 8192 ? Maybe for all ARM devices not just Raspberry Pi. There is one way, though its not that good. This is just a sample code. I am unsure about if using bash in makefile is a good idea in Makefile, uname -a | grep "arm" >out if [ -s out ]; then export ARM=arm fi rm out in read-cache.c, #define ARCH (getenv("ARCH")) #ifdef ARCH, #define WRITE_BUFFER_SIZE 128000 #endif ``` ## Duy Nguyen, 2016-03-19 11:18 Subject: Re: [PATCH] read-cache: increase write buffer size Message-ID: URL: https://gitlist.dev/e/CACsJy8ABvs822-vtELCm2wvvpxRV6wtEEr5Ge1qxH_zOcqdwSA%40mail.gmail.com In-Reply-To: <1458350341-12276-1-git-send-email-dturner@twopensource.com> ``` On Sat, Mar 19, 2016 at 8:19 AM, David Turner wrote: > Each write() has syscall overhead, and writing a large index entails > many such calls. A larger write buffer reduces the overhead, > leading to increased performance. > > On my repo, which has an index size of 30m, this saves about 10ms of > time writing the index. I wonder if split-index does not work as intended. Because if it does, you should rarely need to write that such big index files (or is this 30mb the small part and the base index is even bigger?). But I agree with Torsten that we should make this configurable, preferably via config file, if not we can still move this define back in Makefile, overridable using config.mak -- Duy ``` ## David Turner, 2016-03-21 16:22 Subject: Re: [PATCH] read-cache: increase write buffer size Message-ID: <1458577365.9385.31.camel@twopensource.com> URL: https://gitlist.dev/e/1458577365.9385.31.camel%40twopensource.com In-Reply-To: ``` On Sat, 2016-03-19 at 18:18 +0700, Duy Nguyen wrote: > On Sat, Mar 19, 2016 at 8:19 AM, David Turner < > dturner@twopensource.com> wrote: > > Each write() has syscall overhead, and writing a large index > > entails > > many such calls. A larger write buffer reduces the overhead, > > leading to increased performance. > > > > On my repo, which has an index size of 30m, this saves about 10ms > > of > > time writing the index. > > I wonder if split-index does not work as intended. Because if it > does, > you should rarely need to write that such big index files (or is this > 30mb the small part and the base index is even bigger?). > > But I agree with Torsten that we should make this configurable, > preferably via config file, if not we can still move this define back > in Makefile, overridable using config.mak Oh, right, split index. Yeah, I should just turn that on. Nevermind. ```