{"thread":{"id":"9001","subject":"[PATCH 2/6] Support fetching the memory usage of a delta index","startedAt":"2007-07-12T12:55:46Z","lastAt":"2007-07-13T18:23:02Z","messageCount":13,"participants":["Brian Downing","Johannes Schindelin","Nicolas Pitre","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"47146","messageId":"1184244952554-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":null,"subject":"[PATCH 0/6] Pack window memory limit, take 2","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:46Z","receivedAt":"2007-07-12T12:55:46Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"This series has my circular buffer errors (hopefully) corrected, and\nthe code is a lot cleaner as a bonus.  Also, the options are now named\n--window-memory and pack.windowMemory, and both can take {k,m,g} suffixes.\n\nI split out the {k,m,g} parsing code from git_config_int into\ngit_parse_long and git_parse_ulong, so it can be used for command-line\narguments as well.  Hopefully these will be useful elsewhere.\n\nFinally the documentation has been cleaned up a bit and information on\nthe defaults has been added.\n\nPatches 1 and 2 are unmodified from last time.\n\n [PATCH 1/6] Don't try to delta if target is much smaller than source\n [PATCH 2/6] Support fetching the memory usage of a delta index\n [PATCH 3/6] Add functions for parsing integers with size suffixes\n [PATCH 4/6] Add pack-objects window memory usage limit\n [PATCH 5/6] Add --window-memory option to git-repack\n [PATCH 6/6] Add documentation for --window-memory, pack.windowMemory\n\n Documentation/config.txt           |    6 +++\n Documentation/git-pack-objects.txt |   11 ++++++\n Documentation/git-repack.txt       |   11 ++++++\n builtin-pack-objects.c             |   52 ++++++++++++++++++++++++++----\n cache.h                            |    3 ++\n config.c                           |   61 +++++++++++++++++++++++++++++------\n delta.h                            |    7 ++++\n diff-delta.c                       |   10 ++++++\n git-repack.sh                      |    3 +-\n 9 files changed, 145 insertions(+), 19 deletions(-)\n\n-bcd\n"},{"id":"47148","messageId":"11842449523154-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"[PATCH 1/6] Don't try to delta if target is much smaller than source","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:47Z","receivedAt":"2007-07-12T12:55:47Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Add a new try_delta heuristic:  Don't bother trying to make a delta if\nthe target object size is much smaller (currently 1/32) than the source,\nas it's very likely not going to get a match.  Even if it does, you will\nhave to read at least 32x the size of the new file to reassemble it,\nwhich isn't such a good deal.  This leads to a considerable performance\nimprovement when deltifying a mix of small and large files with a very\nlarge window, because you don't have to wait for the large files to\npercolate out of the window before things start going fast again.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n builtin-pack-objects.c |    2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 54b9d26..132ce96 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -1342,6 +1342,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n+\tif (trg_size < src_size / 32)\n+\t\treturn 0;\n \n \t/* Load data if not already done */\n \tif (!trg->data) {\n-- \n1.5.2.GIT\n"},{"id":"47145","messageId":"11842449522040-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"[PATCH 2/6] Support fetching the memory usage of a delta index","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:48Z","receivedAt":"2007-07-12T12:55:48Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Delta indexes, at least on 64-bit platforms, tend to be larger than\nthe actual uncompressed data.  As such, keeping track of this storage\nis important if you want to successfully limit the memory size of your\npack window.\n\nSquirrel away the total allocation size inside the delta_index struct,\nand add an accessor \"sizeof_delta_index\" to access it.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n delta.h      |    7 +++++++\n diff-delta.c |   10 ++++++++++\n 2 files changed, 17 insertions(+), 0 deletions(-)\n\ndiff --git a/delta.h b/delta.h\nindex 7b3f86d..40ccf5a 100644\n--- a/delta.h\n+++ b/delta.h\n@@ -24,6 +24,13 @@ create_delta_index(const void *buf, unsigned long bufsize);\n extern void free_delta_index(struct delta_index *index);\n \n /*\n+ * sizeof_delta_index: returns memory usage of delta index\n+ *\n+ * Given pointer must be what create_delta_index() returned, or NULL.\n+ */\n+extern unsigned long sizeof_delta_index(struct delta_index *index);\n+\n+/*\n  * create_delta: create a delta from given index for the given buffer\n  *\n  * This function may be called multiple times with different buffers using\ndiff --git a/diff-delta.c b/diff-delta.c\nindex faf96e4..3af5835 100644\n--- a/diff-delta.c\n+++ b/diff-delta.c\n@@ -119,6 +119,7 @@ struct index_entry {\n };\n \n struct delta_index {\n+\tunsigned long memsize;\n \tconst void *src_buf;\n \tunsigned long src_size;\n \tunsigned int hash_mask;\n@@ -159,6 +160,7 @@ struct delta_index * create_delta_index(const void *buf, unsigned long bufsize)\n \tmem = hash + hsize;\n \tentry = mem;\n \n+\tindex->memsize = memsize;\n \tindex->src_buf = buf;\n \tindex->src_size = bufsize;\n \tindex->hash_mask = hmask;\n@@ -228,6 +230,14 @@ void free_delta_index(struct delta_index *index)\n \tfree(index);\n }\n \n+unsigned long sizeof_delta_index(struct delta_index *index)\n+{\n+\tif (index)\n+\t\treturn index->memsize;\n+\telse\n+\t\treturn 0;\n+}\n+\n /*\n  * The maximum size for any opcode sequence, including the initial header\n  * plus Rabin window plus biggest copy.\n-- \n1.5.2.GIT\n"},{"id":"47151","messageId":"1184244952173-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"[PATCH 3/6] Add functions for parsing integers with size suffixes","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:49Z","receivedAt":"2007-07-12T12:55:49Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Split out the nnn{k,m,g} parsing code from git_config_int into\ngit_parse_long, so command-line parameters can enjoy the same\nfunctionality.  Also add get_parse_ulong for unsigned values.\n\nMake git_config_int use git_parse_long, and add get_config_ulong\nas well.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n cache.h  |    3 +++\n config.c |   61 ++++++++++++++++++++++++++++++++++++++++++++++++++-----------\n 2 files changed, 53 insertions(+), 11 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e64071e..917a7e3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -521,7 +521,10 @@ typedef int (*config_fn_t)(const char *, const char *);\n extern int git_default_config(const char *, const char *);\n extern int git_config_from_file(config_fn_t fn, const char *);\n extern int git_config(config_fn_t fn);\n+extern int git_parse_long(const char *, long *);\n+extern int git_parse_ulong(const char *, unsigned long *);\n extern int git_config_int(const char *, const char *);\n+extern unsigned long git_config_ulong(const char *, const char *);\n extern int git_config_bool(const char *, const char *);\n extern int git_config_set(const char *, const char *);\n extern int git_config_set_multivar(const char *, const char *, const char *, int);\ndiff --git a/config.c b/config.c\nindex 561ee3b..ee338d1 100644\n--- a/config.c\n+++ b/config.c\n@@ -233,21 +233,60 @@ static int git_parse_file(config_fn_t fn)\n \tdie(\"bad config file line %d in %s\", config_linenr, config_file_name);\n }\n \n-int git_config_int(const char *name, const char *value)\n+int git_parse_long(const char *value, long *ret)\n+{\n+\tif (value && *value) {\n+\t\tchar *end;\n+\t\tlong val = strtol(value, &end, 0);\n+\t\tif (!*end)\n+\t\t\t*ret = val;\n+\t\telse if (!strcasecmp(end, \"k\"))\n+\t\t\t*ret = val * 1024;\n+\t\telse if (!strcasecmp(end, \"m\"))\n+\t\t\t*ret = val * 1024 * 1024;\n+\t\telse if (!strcasecmp(end, \"g\"))\n+\t\t\t*ret = val * 1024 * 1024 * 1024;\n+\t\telse\n+\t\t\treturn 0;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int git_parse_ulong(const char *value, unsigned long *ret)\n {\n \tif (value && *value) {\n \t\tchar *end;\n-\t\tint val = strtol(value, &end, 0);\n+\t\tunsigned long val = strtoul(value, &end, 0);\n \t\tif (!*end)\n-\t\t\treturn val;\n-\t\tif (!strcasecmp(end, \"k\"))\n-\t\t\treturn val * 1024;\n-\t\tif (!strcasecmp(end, \"m\"))\n-\t\t\treturn val * 1024 * 1024;\n-\t\tif (!strcasecmp(end, \"g\"))\n-\t\t\treturn val * 1024 * 1024 * 1024;\n-\t}\n-\tdie(\"bad config value for '%s' in %s\", name, config_file_name);\n+\t\t\t*ret = val;\n+\t\telse if (!strcasecmp(end, \"k\"))\n+\t\t\t*ret = val * 1024;\n+\t\telse if (!strcasecmp(end, \"m\"))\n+\t\t\t*ret = val * 1024 * 1024;\n+\t\telse if (!strcasecmp(end, \"g\"))\n+\t\t\t*ret = val * 1024 * 1024 * 1024;\n+\t\telse\n+\t\t\treturn 0;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int git_config_int(const char *name, const char *value)\n+{\n+\tlong ret;\n+\tif (!git_parse_long(value, &ret))\n+\t\tdie(\"bad config value for '%s' in %s\", name, config_file_name);\n+\treturn ret;\n+}\n+\n+unsigned long git_config_ulong(const char *name, const char *value)\n+{\n+\tunsigned long ret;\n+\tif (!git_parse_ulong(value, &ret))\n+\t\tdie(\"bad config value for '%s' in %s\", name, config_file_name);\n+\treturn ret;\n }\n \n int git_config_bool(const char *name, const char *value)\n-- \n1.5.2.GIT\n"},{"id":"47147","messageId":"11842449521798-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"[PATCH 4/6] Add pack-objects window memory usage limit","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:50Z","receivedAt":"2007-07-12T12:55:50Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"This adds an option (--window-memory=N) and configuration variable\n(pack.windowMemory = N) to limit the memory size of the pack-objects\ndelta search window.  This works by removing the oldest unpacked objects\nwhenever the total size goes above the limit.  It will always leave\nat least one object, though, so as not to completely eliminate the\npossibility of computing deltas.\n\nThis is an extra limit on top of the normal window size (--window=N);\nthe window will not dynamically grow above the fixed number of entries\nspecified to fill the memory limit.\n\nWith this, repacking a repository with a mix of large and small objects\nis possible even with a very large window.\n\nCleaner and correct circular buffer handling courtesy of Nicolas Pitre.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n builtin-pack-objects.c |   50 +++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 43 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 132ce96..dc6a5f4 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -16,8 +16,9 @@\n #include \"progress.h\"\n \n static const char pack_usage[] = \"\\\n-git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n-\t[--local] [--incremental] [--window=N] [--depth=N] \\n\\\n+git-pack-objects [{ -q | --progress | --all-progress }] \\n\\\n+\t[--max-pack-size=N] [--local] [--incremental] \\n\\\n+\t[--window=N] [--window-memory=N] [--depth=N] \\n\\\n \t[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \\n\\\n \t[--non-empty] [--revs [--unpacked | --all]*] [--reflog] \\n\\\n \t[--stdout | base-name] [<ref-list | <object-list]\";\n@@ -79,6 +80,9 @@ static unsigned long delta_cache_size = 0;\n static unsigned long max_delta_cache_size = 0;\n static unsigned long cache_max_small_delta_size = 1000;\n \n+static unsigned long window_memory_usage = 0;\n+static unsigned long window_memory_limit = 0;\n+\n /*\n  * The object names in objects array are hashed with this hashtable,\n  * to help looking up the entry by object name.\n@@ -1351,12 +1355,14 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tif (sz != trg_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(trg_entry->idx.sha1), sz, trg_size);\n+\t\twindow_memory_usage += sz;\n \t}\n \tif (!src->data) {\n \t\tsrc->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);\n \t\tif (sz != src_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(src_entry->idx.sha1), sz, src_size);\n+\t\twindow_memory_usage += sz;\n \t}\n \tif (!src->index) {\n \t\tsrc->index = create_delta_index(src->data, src_size);\n@@ -1366,6 +1372,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\twarning(\"suboptimal pack - out of memory\");\n \t\t\treturn 0;\n \t\t}\n+\t\twindow_memory_usage += sizeof_delta_index(src->index);\n \t}\n \n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n@@ -1408,9 +1415,22 @@ static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n \treturn m;\n }\n \n+static void free_unpacked(struct unpacked *n)\n+{\n+\twindow_memory_usage -= sizeof_delta_index(n->index);\n+\tfree_delta_index(n->index);\n+\tn->index = NULL;\n+\tif (n->data) {\n+\t\tfree(n->data);\n+\t\tn->data = NULL;\n+\t\twindow_memory_usage -= n->entry->size;\n+\t}\n+\tn->entry = NULL;\n+}\n+\n static void find_deltas(struct object_entry **list, int window, int depth)\n {\n-\tuint32_t i = nr_objects, idx = 0, processed = 0;\n+\tuint32_t i = nr_objects, idx = 0, count = 0, processed = 0;\n \tunsigned int array_size = window * sizeof(struct unpacked);\n \tstruct unpacked *array;\n \tint max_depth;\n@@ -1445,12 +1465,17 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \t\tif (entry->no_try_delta)\n \t\t\tcontinue;\n \n-\t\tfree_delta_index(n->index);\n-\t\tn->index = NULL;\n-\t\tfree(n->data);\n-\t\tn->data = NULL;\n+\t\tfree_unpacked(n);\n \t\tn->entry = entry;\n \n+\t\twhile (window_memory_limit &&\n+\t\t       window_memory_usage > window_memory_limit &&\n+\t\t       count > 1) {\n+\t\t\tuint32_t tail = (idx + window - count) % window;\n+\t\t\tfree_unpacked(array + tail);\n+\t\t\tcount--;\n+\t\t}\n+\n \t\t/*\n \t\t * If the current object is at pack edge, take the depth the\n \t\t * objects that depend on the current object into account\n@@ -1485,6 +1510,8 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \n \t\tnext:\n \t\tidx++;\n+\t\tif (count + 1 < window)\n+\t\t\tcount++;\n \t\tif (idx >= window)\n \t\t\tidx = 0;\n \t} while (i > 0);\n@@ -1523,6 +1550,10 @@ static int git_pack_config(const char *k, const char *v)\n \t\twindow = git_config_int(k, v);\n \t\treturn 0;\n \t}\n+\tif(!strcmp(k, \"pack.windowmemory\")) {\n+\t\twindow_memory_limit = git_config_ulong(k, v);\n+\t\treturn 0;\n+\t}\n \tif(!strcmp(k, \"pack.depth\")) {\n \t\tdepth = git_config_int(k, v);\n \t\treturn 0;\n@@ -1699,6 +1730,11 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t\tusage(pack_usage);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--window-memory=\")) {\n+\t\t\tif (!git_parse_ulong(arg+15, &window_memory_limit))\n+\t\t\t\tusage(pack_usage);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--depth=\")) {\n \t\t\tchar *end;\n \t\t\tdepth = strtoul(arg+8, &end, 0);\n-- \n1.5.2.GIT\n"},{"id":"47149","messageId":"11842449523248-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"[PATCH 5/6] Add --window-memory option to git-repack","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:51Z","receivedAt":"2007-07-12T12:55:51Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Signed-off-by: Brian Downing <bdowning@lavos.net>\n---\n git-repack.sh |    3 ++-\n 1 files changed, 2 insertions(+), 1 deletions(-)\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex b5c6671..156c5e8 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -3,7 +3,7 @@\n # Copyright (c) 2005 Linus Torvalds\n #\n \n-USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--max-pack-size=N] [--window=N] [--depth=N]'\n+USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--max-pack-size=N] [--window=N] [--window-memory=N] [--depth=N]'\n SUBDIRECTORY_OK='Yes'\n . git-sh-setup\n \n@@ -20,6 +20,7 @@ do\n \t-l)\tlocal=--local ;;\n \t--max-pack-size=*) extra=\"$extra $1\" ;;\n \t--window=*) extra=\"$extra $1\" ;;\n+\t--window-memory=*) extra=\"$extra $1\" ;;\n \t--depth=*) extra=\"$extra $1\" ;;\n \t*)\tusage ;;\n \tesac\n-- \n1.5.2.GIT\n"},{"id":"47150","messageId":"11842449521427-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"[PATCH 6/6] Add documentation for --window-memory, pack.windowMemory","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T12:55:52Z","receivedAt":"2007-07-12T12:55:52Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Signed-off-by: Brian Downing <bdowning@lavos.net>\n---\n Documentation/config.txt           |    6 ++++++\n Documentation/git-pack-objects.txt |   11 +++++++++++\n Documentation/git-repack.txt       |   11 +++++++++++\n 3 files changed, 28 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 4b67f0a..11b3321 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -589,6 +589,12 @@ pack.depth::\n \tThe maximum delta depth used by gitlink:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n \n+pack.windowMemory::\n+\tThe window memory size limit used by gitlink:git-pack-objects[1]\n+\twhen no limit is given on the command line.  The value can be\n+\tsuffixed with \"k\", \"m\", or \"g\".  Defaults to 0, meaning no\n+\tlimit.\n+\n pack.compression::\n \tAn integer -1..9, indicating the compression level for objects\n \tin a pack file. -1 is the zlib default. 0 means no\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e3549b5..6f17cff 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -85,6 +85,17 @@ base-name::\n \ttimes to get to the necessary object.\n \tThe default value for --window is 10 and --depth is 50.\n \n+--window-memory=[N]::\n+\tThis option provides an additional limit on top of `--window`;\n+\tthe window size will dynamically scale down so as to not take\n+\tup more than N bytes in memory.  This is useful in\n+\trepositories with a mix of large and small objects to not run\n+\tout of memory with a large window, but still be able to take\n+\tadvantage of the large window for the smaller objects.  The\n+\tsize can be suffixed with \"k\", \"m\", or \"g\".\n+\t`--window-memory=0` makes memory usage unlimited, which is the\n+\tdefault.\n+\n --max-pack-size=<n>::\n \tMaximum size of each output packfile, expressed in MiB.\n \tIf specified,  multiple packfiles may be created.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 2894939..5283ef8 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -68,6 +68,17 @@ OPTIONS\n \tto be applied that many times to get to the necessary object.\n \tThe default value for --window is 10 and --depth is 50.\n \n+--window-memory=[N]::\n+\tThis option provides an additional limit on top of `--window`;\n+\tthe window size will dynamically scale down so as to not take\n+\tup more than N bytes in memory.  This is useful in\n+\trepositories with a mix of large and small objects to not run\n+\tout of memory with a large window, but still be able to take\n+\tadvantage of the large window for the smaller objects.  The\n+\tsize can be suffixed with \"k\", \"m\", or \"g\".\n+\t`--window-memory=0` makes memory usage unlimited, which is the\n+\tdefault.\n+\n --max-pack-size=<n>::\n \tMaximum size of each output packfile, expressed in MiB.\n \tIf specified,  multiple packfiles may be created.\n-- \n1.5.2.GIT\n"},{"id":"47152","messageId":"20070712130400.GU4087@lavos.net","threadId":"9001","inReplyTo":"11842449521798-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 4/6] Add pack-objects window memory usage limit","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T13:04:00Z","receivedAt":"2007-07-12T13:04:00Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"On Thu, Jul 12, 2007 at 07:55:50AM -0500, Brian Downing wrote:\n> +\t\tif (!prefixcmp(arg, \"--window-memory=\")) {\n> +\t\t\tif (!git_parse_ulong(arg+15, &window_memory_limit))\n> +\t\t\t\tusage(pack_usage);\n> +\t\t\tcontinue;\n> +\t\t}\n\nThis is incorrect.  I had this fixed to +16, but somewhere in my remaking\nthe series it got lost.  I will resend the correct patch.\n\n-bcd\n"},{"id":"47156","messageId":"Pine.LNX.4.64.0707121404450.4516@racer.site","threadId":"9001","inReplyTo":"1184244952173-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 3/6] Add functions for parsing integers with size suffixes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-07-12T13:07:39Z","receivedAt":"2007-07-12T13:07:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 12 Jul 2007, Brian Downing wrote:\n\n> Split out the nnn{k,m,g} parsing code from git_config_int into \n> git_parse_long, so command-line parameters can enjoy the same \n> functionality.  Also add get_parse_ulong for unsigned values.\n\nNice!\n\n> +\t\tif (!*end)\n> +\t\t\t*ret = val;\n> +\t\telse if (!strcasecmp(end, \"k\"))\n> +\t\t\t*ret = val * 1024;\n> +\t\telse if (!strcasecmp(end, \"m\"))\n> +\t\t\t*ret = val * 1024 * 1024;\n> +\t\telse if (!strcasecmp(end, \"g\"))\n> +\t\t\t*ret = val * 1024 * 1024 * 1024;\n> +\t\telse\n> +\t\t\treturn 0;\n\nThis could be an own static function, like this:\n\nunsigned long get_unit_factor(const char *end)\n{\n\tif (!*end)\n\t\treturn 1;\n\tif (!strcasecmp(end, \"k\"))\n\t\treturn 1024;\n\t...\n\terror(\"Unknown unit: %s\", end);\n\treturn 1;\n}\n\nto avoid duplicated code.\n\nCiao,\nDscho\n"},{"id":"47155","messageId":"11842456662346-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"20070712130400.GU4087@lavos.net","subject":"[PATCH] Add pack-objects window memory usage limit","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T13:07:46Z","receivedAt":"2007-07-12T13:07:46Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"This adds an option (--window-memory=N) and configuration variable\n(pack.windowMemory = N) to limit the memory size of the pack-objects\ndelta search window.  This works by removing the oldest unpacked objects\nwhenever the total size goes above the limit.  It will always leave\nat least one object, though, so as not to completely eliminate the\npossibility of computing deltas.\n\nThis is an extra limit on top of the normal window size (--window=N);\nthe window will not dynamically grow above the fixed number of entries\nspecified to fill the memory limit.\n\nWith this, repacking a repository with a mix of large and small objects\nis possible even with a very large window.\n\nCleaner and correct circular buffer handling courtesy of Nicolas Pitre.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n builtin-pack-objects.c |   50 +++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 43 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 132ce96..5cc2148 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -16,8 +16,9 @@\n #include \"progress.h\"\n \n static const char pack_usage[] = \"\\\n-git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n-\t[--local] [--incremental] [--window=N] [--depth=N] \\n\\\n+git-pack-objects [{ -q | --progress | --all-progress }] \\n\\\n+\t[--max-pack-size=N] [--local] [--incremental] \\n\\\n+\t[--window=N] [--window-memory=N] [--depth=N] \\n\\\n \t[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \\n\\\n \t[--non-empty] [--revs [--unpacked | --all]*] [--reflog] \\n\\\n \t[--stdout | base-name] [<ref-list | <object-list]\";\n@@ -79,6 +80,9 @@ static unsigned long delta_cache_size = 0;\n static unsigned long max_delta_cache_size = 0;\n static unsigned long cache_max_small_delta_size = 1000;\n \n+static unsigned long window_memory_usage = 0;\n+static unsigned long window_memory_limit = 0;\n+\n /*\n  * The object names in objects array are hashed with this hashtable,\n  * to help looking up the entry by object name.\n@@ -1351,12 +1355,14 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tif (sz != trg_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(trg_entry->idx.sha1), sz, trg_size);\n+\t\twindow_memory_usage += sz;\n \t}\n \tif (!src->data) {\n \t\tsrc->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);\n \t\tif (sz != src_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(src_entry->idx.sha1), sz, src_size);\n+\t\twindow_memory_usage += sz;\n \t}\n \tif (!src->index) {\n \t\tsrc->index = create_delta_index(src->data, src_size);\n@@ -1366,6 +1372,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\twarning(\"suboptimal pack - out of memory\");\n \t\t\treturn 0;\n \t\t}\n+\t\twindow_memory_usage += sizeof_delta_index(src->index);\n \t}\n \n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n@@ -1408,9 +1415,22 @@ static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n \treturn m;\n }\n \n+static void free_unpacked(struct unpacked *n)\n+{\n+\twindow_memory_usage -= sizeof_delta_index(n->index);\n+\tfree_delta_index(n->index);\n+\tn->index = NULL;\n+\tif (n->data) {\n+\t\tfree(n->data);\n+\t\tn->data = NULL;\n+\t\twindow_memory_usage -= n->entry->size;\n+\t}\n+\tn->entry = NULL;\n+}\n+\n static void find_deltas(struct object_entry **list, int window, int depth)\n {\n-\tuint32_t i = nr_objects, idx = 0, processed = 0;\n+\tuint32_t i = nr_objects, idx = 0, count = 0, processed = 0;\n \tunsigned int array_size = window * sizeof(struct unpacked);\n \tstruct unpacked *array;\n \tint max_depth;\n@@ -1445,12 +1465,17 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \t\tif (entry->no_try_delta)\n \t\t\tcontinue;\n \n-\t\tfree_delta_index(n->index);\n-\t\tn->index = NULL;\n-\t\tfree(n->data);\n-\t\tn->data = NULL;\n+\t\tfree_unpacked(n);\n \t\tn->entry = entry;\n \n+\t\twhile (window_memory_limit &&\n+\t\t       window_memory_usage > window_memory_limit &&\n+\t\t       count > 1) {\n+\t\t\tuint32_t tail = (idx + window - count) % window;\n+\t\t\tfree_unpacked(array + tail);\n+\t\t\tcount--;\n+\t\t}\n+\n \t\t/*\n \t\t * If the current object is at pack edge, take the depth the\n \t\t * objects that depend on the current object into account\n@@ -1485,6 +1510,8 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \n \t\tnext:\n \t\tidx++;\n+\t\tif (count + 1 < window)\n+\t\t\tcount++;\n \t\tif (idx >= window)\n \t\t\tidx = 0;\n \t} while (i > 0);\n@@ -1523,6 +1550,10 @@ static int git_pack_config(const char *k, const char *v)\n \t\twindow = git_config_int(k, v);\n \t\treturn 0;\n \t}\n+\tif(!strcmp(k, \"pack.windowmemory\")) {\n+\t\twindow_memory_limit = git_config_ulong(k, v);\n+\t\treturn 0;\n+\t}\n \tif(!strcmp(k, \"pack.depth\")) {\n \t\tdepth = git_config_int(k, v);\n \t\treturn 0;\n@@ -1699,6 +1730,11 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t\tusage(pack_usage);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--window-memory=\")) {\n+\t\t\tif (!git_parse_ulong(arg+16, &window_memory_limit))\n+\t\t\t\tusage(pack_usage);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--depth=\")) {\n \t\t\tchar *end;\n \t\t\tdepth = strtoul(arg+8, &end, 0);\n-- \n1.5.2.GIT\n"},{"id":"47160","messageId":"11842471462637-git-send-email-bdowning@lavos.net","threadId":"9001","inReplyTo":"Pine.LNX.4.64.0707121404450.4516@racer.site","subject":"[PATCH] Add functions for parsing integers with size suffixes","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T13:32:26Z","receivedAt":"2007-07-12T13:32:26Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Split out the nnn{k,m,g} parsing code from git_config_int into\ngit_parse_long, so command-line parameters can enjoy the same\nfunctionality.  Also add get_parse_ulong for unsigned values.\n\nMake git_config_int use git_parse_long, and add get_config_ulong\nas well.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n    Good idea!\n\n cache.h  |    3 +++\n config.c |   58 ++++++++++++++++++++++++++++++++++++++++++++++------------\n 2 files changed, 49 insertions(+), 12 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e64071e..917a7e3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -521,7 +521,10 @@ typedef int (*config_fn_t)(const char *, const char *);\n extern int git_default_config(const char *, const char *);\n extern int git_config_from_file(config_fn_t fn, const char *);\n extern int git_config(config_fn_t fn);\n+extern int git_parse_long(const char *, long *);\n+extern int git_parse_ulong(const char *, unsigned long *);\n extern int git_config_int(const char *, const char *);\n+extern unsigned long git_config_ulong(const char *, const char *);\n extern int git_config_bool(const char *, const char *);\n extern int git_config_set(const char *, const char *);\n extern int git_config_set_multivar(const char *, const char *, const char *, int);\ndiff --git a/config.c b/config.c\nindex 561ee3b..f89a611 100644\n--- a/config.c\n+++ b/config.c\n@@ -233,21 +233,55 @@ static int git_parse_file(config_fn_t fn)\n \tdie(\"bad config file line %d in %s\", config_linenr, config_file_name);\n }\n \n-int git_config_int(const char *name, const char *value)\n+static unsigned long get_unit_factor(const char *end)\n+{\n+\tif (!*end)\n+\t\treturn 1;\n+\telse if (!strcasecmp(end, \"k\"))\n+\t\treturn 1024;\n+\telse if (!strcasecmp(end, \"m\"))\n+\t\treturn 1024 * 1024;\n+\telse if (!strcasecmp(end, \"g\"))\n+\t\treturn 1024 * 1024 * 1024;\n+\tdie(\"unknown unit: '%s'\", end);\n+}\n+\n+int git_parse_long(const char *value, long *ret)\n+{\n+\tif (value && *value) {\n+\t\tchar *end;\n+\t\tlong val = strtol(value, &end, 0);\n+\t\t*ret = val * get_unit_factor(end);\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int git_parse_ulong(const char *value, unsigned long *ret)\n {\n \tif (value && *value) {\n \t\tchar *end;\n-\t\tint val = strtol(value, &end, 0);\n-\t\tif (!*end)\n-\t\t\treturn val;\n-\t\tif (!strcasecmp(end, \"k\"))\n-\t\t\treturn val * 1024;\n-\t\tif (!strcasecmp(end, \"m\"))\n-\t\t\treturn val * 1024 * 1024;\n-\t\tif (!strcasecmp(end, \"g\"))\n-\t\t\treturn val * 1024 * 1024 * 1024;\n-\t}\n-\tdie(\"bad config value for '%s' in %s\", name, config_file_name);\n+\t\tunsigned long val = strtoul(value, &end, 0);\n+\t\t*ret = val * get_unit_factor(end);\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int git_config_int(const char *name, const char *value)\n+{\n+\tlong ret;\n+\tif (!git_parse_long(value, &ret))\n+\t\tdie(\"bad config value for '%s' in %s\", name, config_file_name);\n+\treturn ret;\n+}\n+\n+unsigned long git_config_ulong(const char *name, const char *value)\n+{\n+\tunsigned long ret;\n+\tif (!git_parse_ulong(value, &ret))\n+\t\tdie(\"bad config value for '%s' in %s\", name, config_file_name);\n+\treturn ret;\n }\n \n int git_config_bool(const char *name, const char *value)\n-- \n1.5.2.GIT\n"},{"id":"47171","messageId":"alpine.LFD.0.999.0707121144110.32552@xanadu.home","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 0/6] Pack window memory limit, take 2","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-07-12T15:46:28Z","receivedAt":"2007-07-12T15:46:28Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 12 Jul 2007, Brian Downing wrote:\n\n> This series has my circular buffer errors (hopefully) corrected, and\n> the code is a lot cleaner as a bonus.  Also, the options are now named\n> --window-memory and pack.windowMemory, and both can take {k,m,g} suffixes.\n\nAcked-by: Nicolas Pitre <nico@cam.org>\n(including the amended patches).\n\n\nNicolas\n"},{"id":"47281","messageId":"7vlkdknoxl.fsf@assigned-by-dhcp.cox.net","threadId":"9001","inReplyTo":"1184244952554-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 0/6] Pack window memory limit, take 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-07-13T18:23:02Z","receivedAt":"2007-07-13T18:23:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Very nicely done.  Thanks, everybody involved.\n"}]}