{"thread":{"id":"41746","subject":"[PATCH] read-cache: increase write buffer size","startedAt":"2016-03-19T01:19:01Z","lastAt":"2016-03-21T16:22:45Z","messageCount":5,"participants":["David Turner","Torsten Bögershausen","Pranit Bauva","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"281209","messageId":"1458350341-12276-1-git-send-email-dturner@twopensource.com","threadId":"41746","inReplyTo":null,"subject":"[PATCH] read-cache: increase write buffer size","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-19T01:19:01Z","receivedAt":"2016-03-19T01:19:01Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Each write() has syscall overhead, and writing a large index entails\nmany such calls.  A larger write buffer reduces the overhead,\nleading to increased performance.\n\nOn my repo, which has an index size of 30m, this saves about 10ms of\ntime writing the index.\n\nSigned-off-by: David Turner <dturner@twopensource.com>\n---\n read-cache.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex d9fb78b..09ebe08 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)\n \treturn 0;\n }\n \n-#define WRITE_BUFFER_SIZE 8192\n+#define WRITE_BUFFER_SIZE 131072\n static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n static unsigned long write_buffer_len;\n \n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"281220","messageId":"56ED0863.3020709@web.de","threadId":"41746","inReplyTo":"1458350341-12276-1-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH] read-cache: increase write buffer size","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-03-19T08:05:55Z","receivedAt":"2016-03-19T08:05:55Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2016-03-19 02.19, David Turner wrote:\n> Each write() has syscall overhead, and writing a large index entails\n> many such calls.  A larger write buffer reduces the overhead,\n> leading to increased performance.\n> \n> On my repo, which has an index size of 30m, this saves about 10ms of\n> time writing the index.\n> \n> Signed-off-by: David Turner <dturner@twopensource.com>\n> ---\n>  read-cache.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/read-cache.c b/read-cache.c\n> index d9fb78b..09ebe08 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)\n>  \treturn 0;\n>  }\n>  \n> -#define WRITE_BUFFER_SIZE 8192\n> +#define WRITE_BUFFER_SIZE 131072\n>  static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n>  static unsigned long write_buffer_len;\n>  \n> \nDo I read that right, it saves 10 milliseconds ?\nWhat happens to small system (like Raspberry PI), when you\nwant 128K write buffer ?\nCould the buffer size be turned into a makefile variable, defaulting to 8192 ?\n"},{"id":"281224","messageId":"CAFZEwPMtv0JF+Q_1P53NB2oxH-XJXGzmiUsOxk0KBaa+AJE+vA@mail.gmail.com","threadId":"41746","inReplyTo":"56ED0863.3020709@web.de","subject":"Re: [PATCH] read-cache: increase write buffer size","fromName":"Pranit Bauva","fromEmail":"pranit.bauva@gmail.com","sentAt":"2016-03-19T10:25:46Z","receivedAt":"2016-03-19T10:25:46Z","isPatch":true,"sender":{"key":"pranit.bauva@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2959938?v=4"},"body":"On Sat, Mar 19, 2016 at 1:35 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n> On 2016-03-19 02.19, David Turner wrote:\n>> Each write() has syscall overhead, and writing a large index entails\n>> many such calls.  A larger write buffer reduces the overhead,\n>> leading to increased performance.\n>>\n>> On my repo, which has an index size of 30m, this saves about 10ms of\n>> time writing the index.\n>>\n>> Signed-off-by: David Turner <dturner@twopensource.com>\n>> ---\n>>  read-cache.c | 2 +-\n>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/read-cache.c b/read-cache.c\n>> index d9fb78b..09ebe08 100644\n>> --- a/read-cache.c\n>> +++ b/read-cache.c\n>> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)\n>>       return 0;\n>>  }\n>>\n>> -#define WRITE_BUFFER_SIZE 8192\n>> +#define WRITE_BUFFER_SIZE 131072\n>>  static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n>>  static unsigned long write_buffer_len;\n>>\n>>\n> Do I read that right, it saves 10 milliseconds ?\n> What happens to small system (like Raspberry PI), when you\n> want 128K write buffer ?\n> Could the buffer size be turned into a makefile variable, defaulting to 8192 ?\n\nMaybe for all ARM devices not just Raspberry Pi.\n\nThere is one way, though its not that good. This is just a sample\ncode. I am unsure about if using bash in makefile is a good idea\n\nin Makefile,\nuname -a | grep \"arm\" >out\nif [ -s out ]; then\n    export ARM=arm\nfi\nrm out\n\nin read-cache.c,\n#define ARCH (getenv(\"ARCH\"))\n#ifdef ARCH,\n    #define WRITE_BUFFER_SIZE 128000\n#endif\n"},{"id":"281226","messageId":"CACsJy8ABvs822-vtELCm2wvvpxRV6wtEEr5Ge1qxH_zOcqdwSA@mail.gmail.com","threadId":"41746","inReplyTo":"1458350341-12276-1-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH] read-cache: increase write buffer size","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-19T11:18:56Z","receivedAt":"2016-03-19T11:18:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 19, 2016 at 8:19 AM, David Turner <dturner@twopensource.com> wrote:\n> Each write() has syscall overhead, and writing a large index entails\n> many such calls.  A larger write buffer reduces the overhead,\n> leading to increased performance.\n>\n> On my repo, which has an index size of 30m, this saves about 10ms of\n> time writing the index.\n\nI wonder if split-index does not work as intended. Because if it does,\nyou should rarely need to write that such big index files (or is this\n30mb the small part and the base index is even bigger?).\n\nBut I agree with Torsten that we should make this configurable,\npreferably via config file, if not we can still move this define back\nin Makefile, overridable using config.mak\n-- \nDuy\n"},{"id":"281348","messageId":"1458577365.9385.31.camel@twopensource.com","threadId":"41746","inReplyTo":"CACsJy8ABvs822-vtELCm2wvvpxRV6wtEEr5Ge1qxH_zOcqdwSA@mail.gmail.com","subject":"Re: [PATCH] read-cache: increase write buffer size","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-21T16:22:45Z","receivedAt":"2016-03-21T16:22:45Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2016-03-19 at 18:18 +0700, Duy Nguyen wrote:\n> On Sat, Mar 19, 2016 at 8:19 AM, David Turner <\n> dturner@twopensource.com> wrote:\n> > Each write() has syscall overhead, and writing a large index\n> > entails\n> > many such calls.  A larger write buffer reduces the overhead,\n> > leading to increased performance.\n> > \n> > On my repo, which has an index size of 30m, this saves about 10ms\n> > of\n> > time writing the index.\n> \n> I wonder if split-index does not work as intended. Because if it\n> does,\n> you should rarely need to write that such big index files (or is this\n> 30mb the small part and the base index is even bigger?).\n> \n> But I agree with Torsten that we should make this configurable,\n> preferably via config file, if not we can still move this define back\n> in Makefile, overridable using config.mak\n\nOh, right, split index.  Yeah, I should just turn that on.  Nevermind.\n"}]}