{"thread":{"id":"36580","subject":"[PATCH] Bump core.deltaBaseCacheLimit to 96m","startedAt":"2014-05-04T17:13:57Z","lastAt":"2014-05-05T20:19:28Z","messageCount":7,"participants":["David Kastrup","Matthieu Moy","Duy Nguyen","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"240688","messageId":"1399223637-29964-1-git-send-email-dak@gnu.org","threadId":"36580","inReplyTo":null,"subject":"[PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-05-04T17:13:57Z","receivedAt":"2014-05-04T17:13:57Z","isPatch":true,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"The default of 16m causes serious thrashing for large delta chains\ncombined with large files.\n\nHere are some benchmarks (pu variant of git blame):\n\ntime git blame -C src/xdisp.c >/dev/null\n\nfor a repository of Emacs repacked with git gc --aggressive (v1.9,\nresulting in a window size of 250) located on an SSD drive. The file in\nquestion has about 30000 lines, 1Mb of size, and a history with about\n2500 commits.\n\n16m (previous default):\nreal\t3m33.936s\nuser\t2m15.396s\nsys\t1m17.352s\n\n32m:\nreal\t3m1.319s\nuser\t2m8.660s\nsys\t0m51.904s\n\n64m:\nreal\t2m20.636s\nuser\t1m55.780s\nsys\t0m23.964s\n\n96m:\nreal\t2m5.668s\nuser\t1m50.784s\nsys\t0m14.288s\n\n128m:\nreal\t2m4.337s\nuser\t1m50.764s\nsys\t0m12.832s\n\n192m:\nreal\t2m3.567s\nuser\t1m49.508s\nsys\t0m13.312s\n\nSigned-off-by: David Kastrup \n---\n Documentation/config.txt | 2 +-\n environment.c | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 1932e9b..21a3c86 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -489,7 +489,7 @@ core.deltaBaseCacheLimit::\n \tto avoid unpacking and decompressing frequently used base\n \tobjects multiple times.\n +\n-Default is 16 MiB on all platforms. This should be reasonable\n+Default is 96 MiB on all platforms. This should be reasonable\n for all users/operating systems, except on the largest projects.\n You probably do not need to adjust this value.\n +\ndiff --git a/environment.c b/environment.c\nindex 5c4815d..37354c8 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -37,7 +37,7 @@ int core_compression_seen;\n int fsync_object_files;\n size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;\n size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;\n-size_t delta_base_cache_limit = 16 * 1024 * 1024;\n+size_t delta_base_cache_limit = 96 * 1024 * 1024;\n unsigned long big_file_threshold = 512 * 1024 * 1024;\n const char *pager_program;\n int pager_use_color = 1;\n-- \n1.9.1\n"},{"id":"240745","messageId":"CACsJy8BG8fRPk74R_-YABCGMn-YwbDcLHtjUNX7KE66jX1mR4A@mail.gmail.com","threadId":"36580","inReplyTo":"1399223637-29964-1-git-send-email-dak@gnu.org","subject":"Re: [PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-05T10:26:56Z","receivedAt":"2014-05-05T10:26:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 5, 2014 at 12:13 AM, David Kastrup wrote:\n> The default of 16m causes serious thrashing for large delta chains\n> combined with large files.\n>\n> Here are some benchmarks (pu variant of git blame):\n>\n> time git blame -C src/xdisp.c >/dev/null\n\n...\n\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 1932e9b..21a3c86 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -489,7 +489,7 @@ core.deltaBaseCacheLimit::\n> to avoid unpacking and decompressing frequently used base\n> objects multiple times.\n> +\n> -Default is 16 MiB on all platforms. This should be reasonable\n> +Default is 96 MiB on all platforms. This should be reasonable\n> for all users/operating systems, except on the largest projects.\n> You probably do not need to adjust this value.\n\nSo emacs.git falls exactly into the \"except on the largest projects\"\npart. Would it make more sense to advise git devs to set this per repo\ninstead? The majority of (open source) repositories out there are\nsmall if I'm not mistaken. Of those few big repos, we could have a\nsection listing all the tips and tricks to tune git. This is one of\nthem. Index v4 and sparse checkout are some other. In future, maybe\nwatchman support, split index and untracked cache as well.\n-- \nDuy\n"},{"id":"240740","messageId":"CACsJy8D3xXM3ht3JeoowiFQfoL28WDxyijRhyRKGDn4rfn4aSw@mail.gmail.com","threadId":"36580","inReplyTo":"CACsJy8BG8fRPk74R_-YABCGMn-YwbDcLHtjUNX7KE66jX1mR4A@mail.gmail.com","subject":"Re: [PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-05T10:27:49Z","receivedAt":"2014-05-05T10:27:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 5, 2014 at 5:26 PM, Duy Nguyen wrote:\n> part. Would it make more sense to advise git devs to set this per repo\n\ns/advise git devs/advise emacs devs/\n-- \nDuy\n"},{"id":"240731","messageId":"vpqwqe0a3ne.fsf@anie.imag.fr","threadId":"36580","inReplyTo":"CACsJy8BG8fRPk74R_-YABCGMn-YwbDcLHtjUNX7KE66jX1mR4A@mail.gmail.com","subject":"Re: [PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-05-05T11:03:33Z","receivedAt":"2014-05-05T11:03:33Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Duy Nguyen writes:\n\n> On Mon, May 5, 2014 at 12:13 AM, David Kastrup wrote:\n>> The default of 16m causes serious thrashing for large delta chains\n>> combined with large files.\n>>\n>> Here are some benchmarks (pu variant of git blame):\n>>\n>> time git blame -C src/xdisp.c >/dev/null\n>\n> ...\n>\n>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>> index 1932e9b..21a3c86 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -489,7 +489,7 @@ core.deltaBaseCacheLimit::\n>> to avoid unpacking and decompressing frequently used base\n>> objects multiple times.\n>> +\n>> -Default is 16 MiB on all platforms. This should be reasonable\n>> +Default is 96 MiB on all platforms. This should be reasonable\n>> for all users/operating systems, except on the largest projects.\n>> You probably do not need to adjust this value.\n>\n> So emacs.git falls exactly into the \"except on the largest projects\"\n> part. Would it make more sense to advise git devs to set this per repo\n> instead?\n\nWhat's the impact of changing the default for small projects?\n\nMy guess is that changing from 16 to 96Mb is just following Moore's law.\nMachines average RAM has increased a lot since the time 16Mb has been\nchosen, and few people would actually notice the difference in RAM usage\nnowadays.\n\nIf increasing the default does not harm small projects and benefits to\nbig projects, then we should obviously go this way.\n\n(perhaps adding advices for people using Git on machines with low RAM)\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"240773","messageId":"874n14tqty.fsf@fencepost.gnu.org","threadId":"36580","inReplyTo":"CACsJy8BG8fRPk74R_-YABCGMn-YwbDcLHtjUNX7KE66jX1mR4A@mail.gmail.com","subject":"Re: [PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-05-05T11:20:09Z","receivedAt":"2014-05-05T11:20:09Z","isPatch":true,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Duy Nguyen writes:\n\n> On Mon, May 5, 2014 at 12:13 AM, David Kastrup wrote:\n>> The default of 16m causes serious thrashing for large delta chains\n>> combined with large files.\n>>\n>> Here are some benchmarks (pu variant of git blame):\n>>\n>> time git blame -C src/xdisp.c >/dev/null\n>\n> ...\n>\n>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>> index 1932e9b..21a3c86 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -489,7 +489,7 @@ core.deltaBaseCacheLimit::\n>> to avoid unpacking and decompressing frequently used base\n>> objects multiple times.\n>> +\n>> -Default is 16 MiB on all platforms. This should be reasonable\n>> +Default is 96 MiB on all platforms. This should be reasonable\n>> for all users/operating systems, except on the largest projects.\n>> You probably do not need to adjust this value.\n>\n> So emacs.git falls exactly into the \"except on the largest projects\"\n> part.\n\ngit gc --aggressive has been used/recommended for _all_ projects\nregularly, leading to delta chains with a length of 250. So this delta\nchain size is not exceptional but will eventually occur in any archive\nthat has been created and maintained according to the recommendations of\nGit's documentation (which recommends gc --aggressive every few hundreds\nof revisions). I was illustrating the effect on a file of size 1MB.\nThat's not an egregiously large file either.\n\n96MB is the point of diminuishing returns for this case which is _6_\ntimes larger than the current default and _small_ in comparison with the\nmemory installed on developer machines nowadays. Similar slowdowns\noccur with other examples. Git will with the current defaults accept\nfiles of 512Mb size into its compression scheme (and thus its core\nmemory) before punting.\n\nThe current delteBaseCacheLimit of 16Mb is rather ridiculous in\nparticular with the pre-2.0 settings for gc --aggressive and causes\nserious performance degration. It was actually ridiculous even 10 years\nago.\n\n> Would it make more sense to advise git devs to set this per repo\n> instead? The majority of (open source) repositories out there are\n> small if I'm not mistaken. Of those few big repos, we could have a\n> section listing all the tips and tricks to tune git. This is one of\n> them. Index v4 and sparse checkout are some other. In future, maybe\n> watchman support, split index and untracked cache as well.\n\nShrug. The last version of the patch was refused because of wanting\nmore evidence. I added the evidence.\n\nAnd I have it on record in the mailing list and can point to it when\npeople ask me why Git is so slow for \"git blame\" in comparison to other\nversion control systems in spite of my purporting to having improved it.\n\nI'm definitely not going to jump through any more hoops here. I don't\nsee a point in this kind of spectacle.\n\n-- \nDavid Kastrup\n"},{"id":"240763","messageId":"CACsJy8CtcbfUHL+yntyU9zLW7Qc7EobbvFTwVzVsiwCJTsVtOg@mail.gmail.com","threadId":"36580","inReplyTo":"vpqwqe0a3ne.fsf@anie.imag.fr","subject":"Re: [PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-05T11:35:40Z","receivedAt":"2014-05-05T11:35:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 5, 2014 at 6:03 PM, Matthieu Moy\n wrote:\n>>> -Default is 16 MiB on all platforms. This should be reasonable\n>>> +Default is 96 MiB on all platforms. This should be reasonable\n>>> for all users/operating systems, except on the largest projects.\n>>> You probably do not need to adjust this value.\n>>\n>> So emacs.git falls exactly into the \"except on the largest projects\"\n>> part. Would it make more sense to advise git devs to set this per repo\n>> instead?\n>\n> What's the impact of changing the default for small projects?\n\nGood question. With \"git log --patch\" or something like that, we could\nuse up to the limit, which is now 96MB. On modern machines that's\nprobably nothing.\n\n> My guess is that changing from 16 to 96Mb is just following Moore's law.\n> Machines average RAM has increased a lot since the time 16Mb has been\n> chosen, and few people would actually notice the difference in RAM usage\n> nowadays.\n>\n> If increasing the default does not harm small projects and benefits to\n> big projects, then we should obviously go this way.\n\nI wrote without thinking it through. I agree with you.\n\n> (perhaps adding advices for people using Git on machines with low RAM)\n-- \nDuy\n"},{"id":"240775","messageId":"20140505201928.GA24266@sigill.intra.peff.net","threadId":"36580","inReplyTo":"874n14tqty.fsf@fencepost.gnu.org","subject":"Re: [PATCH] Bump core.deltaBaseCacheLimit to 96m","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-05-05T20:19:28Z","receivedAt":"2014-05-05T20:19:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 05, 2014 at 01:20:09PM +0200, David Kastrup wrote:\n\n> > Would it make more sense to advise git devs to set this per repo\n> > instead? The majority of (open source) repositories out there are\n> > small if I'm not mistaken. Of those few big repos, we could have a\n> > section listing all the tips and tricks to tune git. This is one of\n> > them. Index v4 and sparse checkout are some other. In future, maybe\n> > watchman support, split index and untracked cache as well.\n> \n> Shrug. The last version of the patch was refused because of wanting\n> more evidence. I added the evidence.\n\nFWIW, I was the one who asked for the evidence, and this patch looks\npretty straightforward and good. We may also want to revisit the data\nstructure for the delta cache, but that can come separately. My earlier\ntests had not shown improvement with just bumping the cache size, but\nthese ones obviously do. So I think it's worth bumping the default.\n\n-Peff\n"}]}