{"thread":{"id":"7310","subject":"[PATCH] Limit the size of the new delta_base_cache","startedAt":"2007-03-19T04:48:41Z","lastAt":"2007-03-19T04:48:41Z","messageCount":1,"participants":["Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"37465","messageId":"20070319044841.GA22357@spearce.org","threadId":"7310","inReplyTo":null,"subject":"[PATCH] Limit the size of the new delta_base_cache","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-19T04:48:41Z","receivedAt":"2007-03-19T04:48:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The new configuration variable core.deltaBaseCacheLimit allows the\nuser to control how much memory they are willing to give to Git for\ncaching base objects of deltas.  This is not normally meant to be\na user tweakable knob; the \"out of the box\" settings are meant to\nbe suitable for almost all workloads.\n\nWe default to 16 MiB under the assumption that the cache is not\nmeant to consume all of the user's available memory, and that the\ncache's main purpose was to cache trees, for faster path limiters\nduring revision traversal.  Since trees tend to be relatively small\nobjects, this relatively small limit should still allow a large\nnumber of objects.\n\nOn the other hand we don't want the cache to start storing 200\ndifferent versions of a 200 MiB blob, as this could easily blow\nthe entire address space of a 32 bit process.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n Documentation/config.txt |   13 +++++++++++++\n cache.h                  |    1 +\n config.c                 |    5 +++++\n environment.c            |    1 +\n sha1_file.c              |   23 ++++++++++++++++++++---\n 5 files changed, 40 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 953acae..8796929 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -240,6 +240,19 @@ the largest projects.  You probably do not need to adjust this value.\n +\n Common unit suffixes of 'k', 'm', or 'g' are supported.\n \n+core.deltaBaseCacheLimit::\n+\tMaximum number of bytes to reserve for caching base objects\n+\tthat multiple deltafied objects reference.  By storing the\n+\tentire decompressed base objects in a cache Git is able\n+\tto avoid unpacking and decompressing frequently used base\n+\tobjects multiple times.\n++\n+Default is 16 MiB on all platforms.  This should be reasonable\n+for all users/operating systems, except on the largest projects.\n+You probably do not need to adjust this value.\n++\n+Common unit suffixes of 'k', 'm', or 'g' are supported.\n+\n alias.*::\n \tCommand aliases for the gitlink:git[1] command wrapper - e.g.\n \tafter defining \"alias.last = cat-file commit HEAD\", the invocation\ndiff --git a/cache.h b/cache.h\nindex 3818e10..9a3c8c8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -228,6 +228,7 @@ extern const char *apply_default_whitespace;\n extern int zlib_compression_level;\n extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n+extern size_t delta_base_cache_limit;\n extern int auto_crlf;\n \n #define GIT_REPO_VERSION 0\ndiff --git a/config.c b/config.c\nindex d537311..6479855 100644\n--- a/config.c\n+++ b/config.c\n@@ -331,6 +331,11 @@ int git_default_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.deltabasecachelimit\")) {\n+\t\tdelta_base_cache_limit = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.autocrlf\")) {\n \t\tif (value && !strcasecmp(value, \"input\")) {\n \t\t\tauto_crlf = -1;\ndiff --git a/environment.c b/environment.c\nindex 0151ad0..ef2f490 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -27,6 +27,7 @@ const char *apply_default_whitespace;\n int zlib_compression_level = Z_DEFAULT_COMPRESSION;\n size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;\n size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;\n+size_t delta_base_cache_limit = 16 * 1024 * 1024;\n int pager_in_use;\n int pager_use_color = 1;\n int auto_crlf = 0;\t/* 1: both ways, -1: only when adding git objects */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ee64865..6d8b86a 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1354,6 +1354,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \n #define MAX_DELTA_CACHE (256)\n \n+static size_t delta_base_cached;\n static struct delta_base_cache_entry {\n \tstruct packed_git *p;\n \toff_t base_offset;\n@@ -1384,9 +1385,10 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \treturn unpack_entry(p, base_offset, type, base_size);\n \n found_cache_entry:\n-\tif (!keep_cache)\n+\tif (!keep_cache) {\n \t\tent->data = NULL;\n-\telse {\n+\t\tdelta_base_cached -= ent->size;\n+\t} else {\n \t\tret = xmalloc(ent->size + 1);\n \t\tmemcpy(ret, ent->data, ent->size);\n \t\t((char *)ret)[ent->size] = 0;\n@@ -1401,9 +1403,24 @@ static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n {\n \tunsigned long hash = pack_entry_hash(p, base_offset);\n \tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\tstruct delta_base_cache_entry *f = delta_base_cache;\n+\tstruct delta_base_cache_entry *e = f + ARRAY_SIZE(delta_base_cache);\n \n-\tif (ent->data)\n+\tif (ent->data) {\n \t\tfree(ent->data);\n+\t\tent->data = NULL;\n+\t\tdelta_base_cached -= ent->size;\n+\t}\n+\n+\tdelta_base_cached += base_size;\n+\tfor (; delta_base_cached > delta_base_cache_limit && f < e; f++) {\n+\t\tif (f->data) {\n+\t\t\tfree(f->data);\n+\t\t\tf->data = NULL;\n+\t\t\tdelta_base_cached -= f->size;\n+\t\t}\n+\t}\n+\n \tent->p = p;\n \tent->base_offset = base_offset;\n \tent->type = type;\n-- \n1.5.0.5.1030.gc69a\n"}]}