{"thread":{"id":"8997","subject":"[PATCH 2/5] Support fetching the memory usage of a delta index","startedAt":"2007-07-12T03:14:13Z","lastAt":"2007-07-12T10:02:51Z","messageCount":10,"participants":["Brian Downing","Nicolas Pitre"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"47110","messageId":"11842100581060-git-send-email-bdowning@lavos.net","threadId":"8997","inReplyTo":null,"subject":"[PATCH 0/5] Memory-limited pack-object window support","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T03:14:13Z","receivedAt":"2007-07-12T03:14:13Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"This patch series implements a memory limit on the window size for\npack-objects and repack.  Basically, the window size will temporarily\ngrow smaller than the --window option specifies if the total memory\nusage of the window is over the specified limit.\n\nWith this series I can run even with --window=1000 on my troublesome\nrepository (described in an email a couple of days ago; basically, it\nhas a 20MB file with around 200 revisions, plus a bunch of normal sized\nfiles) with memory usage limited to 512MB and it will happily scale down\nthe window to accommodate the file without blowing out my machine.\n\nThe --window option still specifies the size of the window and is still\nrequired; the window will not grow to reach the memory limit, it will\nonly shrink to fit.  I think this may be a feature, as running with a\nvery large window depth and a memory limit basically means that you will\npack at an approximately constant slow speed, rather than rushing ahead\nas it does now for very small objects.\n\nI took the easy way out and expire objects out of the window after\nallocation has occurred, rather than figuring out how much needed to\nbe cleared before allocating.  This made the logic much more feasible,\nthough.\n\nI chose --window-bytes=N and pack.windowBytes=N as my option and\nconfiguration names.  I'm not in love with them, though; no other\nconfiguration in Git mentions bytes in the name as far as I can see,\nbut I wanted to clearly differentiate it from the window \"size\" that\n--window gets you.\n\nThe first patch in this series is optional, but I recommend it.  It makes\nit so that files that are much smaller than a potential delta source\ndon't even try to delta with it.  This is handy for running in a mixed\nfile size repository with a large window, as it means that when you get\nto small files again you start moving fast without having to wait for\nthe large objects to trickle out of the window.  The cut-off is currently\nif it is 1/32 the size, but that number was completely arbitrary.\n\n [PATCH 1/5] Don't try to delta if target is much smaller than source\n [PATCH 2/5] Support fetching the memory usage of a delta index\n [PATCH 3/5] Add pack-objects window memory usage limit\n [PATCH 4/5] Add --window-bytes option to git-repack\n [PATCH 5/5] Add documentation for --window-bytes, pack.windowBytes\n\n Documentation/config.txt           |    5 +++\n Documentation/git-pack-objects.txt |    8 +++++\n Documentation/git-repack.txt       |    8 +++++\n builtin-pack-objects.c             |   58 +++++++++++++++++++++++++++++++----\n delta.h                            |    7 ++++\n diff-delta.c                       |   10 ++++++\n git-repack.sh                      |    3 +-\n 7 files changed, 91 insertions(+), 8 deletions(-)\n\n-bcd\n"},{"id":"47109","messageId":"11842100581148-git-send-email-bdowning@lavos.net","threadId":"8997","inReplyTo":"11842100581060-git-send-email-bdowning@lavos.net","subject":"[PATCH 1/5] Don't try to delta if target is much smaller than source","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T03:14:14Z","receivedAt":"2007-07-12T03:14:14Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Add a new try_delta heuristic:  Don't bother trying to make a delta if\nthe target object size is much smaller (currently 1/32) than the source,\nas it's very likely not going to get a match.  Even if it does, you will\nhave to read at least 32x the size of the new file to reassemble it,\nwhich isn't such a good deal.  This leads to a considerable performance\nimprovement when deltifying a mix of small and large files with a very\nlarge window, because you don't have to wait for the large files to\npercolate out of the window before things start going fast again.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n builtin-pack-objects.c |    2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 54b9d26..132ce96 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -1342,6 +1342,8 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n+\tif (trg_size < src_size / 32)\n+\t\treturn 0;\n \n \t/* Load data if not already done */\n \tif (!trg->data) {\n-- \n1.5.2.GIT\n"},{"id":"47108","messageId":"1184210058269-git-send-email-bdowning@lavos.net","threadId":"8997","inReplyTo":"11842100581060-git-send-email-bdowning@lavos.net","subject":"[PATCH 2/5] Support fetching the memory usage of a delta index","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T03:14:15Z","receivedAt":"2007-07-12T03:14:15Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Delta indexes, at least on 64-bit platforms, tend to be larger than\nthe actual uncompressed data.  As such, keeping track of this storage\nis important if you want to successfully limit the memory size of your\npack window.\n\nSquirrel away the total allocation size inside the delta_index struct,\nand add an accessor \"sizeof_delta_index\" to access it.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n delta.h      |    7 +++++++\n diff-delta.c |   10 ++++++++++\n 2 files changed, 17 insertions(+), 0 deletions(-)\n\ndiff --git a/delta.h b/delta.h\nindex 7b3f86d..40ccf5a 100644\n--- a/delta.h\n+++ b/delta.h\n@@ -24,6 +24,13 @@ create_delta_index(const void *buf, unsigned long bufsize);\n extern void free_delta_index(struct delta_index *index);\n \n /*\n+ * sizeof_delta_index: returns memory usage of delta index\n+ *\n+ * Given pointer must be what create_delta_index() returned, or NULL.\n+ */\n+extern unsigned long sizeof_delta_index(struct delta_index *index);\n+\n+/*\n  * create_delta: create a delta from given index for the given buffer\n  *\n  * This function may be called multiple times with different buffers using\ndiff --git a/diff-delta.c b/diff-delta.c\nindex faf96e4..3af5835 100644\n--- a/diff-delta.c\n+++ b/diff-delta.c\n@@ -119,6 +119,7 @@ struct index_entry {\n };\n \n struct delta_index {\n+\tunsigned long memsize;\n \tconst void *src_buf;\n \tunsigned long src_size;\n \tunsigned int hash_mask;\n@@ -159,6 +160,7 @@ struct delta_index * create_delta_index(const void *buf, unsigned long bufsize)\n \tmem = hash + hsize;\n \tentry = mem;\n \n+\tindex->memsize = memsize;\n \tindex->src_buf = buf;\n \tindex->src_size = bufsize;\n \tindex->hash_mask = hmask;\n@@ -228,6 +230,14 @@ void free_delta_index(struct delta_index *index)\n \tfree(index);\n }\n \n+unsigned long sizeof_delta_index(struct delta_index *index)\n+{\n+\tif (index)\n+\t\treturn index->memsize;\n+\telse\n+\t\treturn 0;\n+}\n+\n /*\n  * The maximum size for any opcode sequence, including the initial header\n  * plus Rabin window plus biggest copy.\n-- \n1.5.2.GIT\n"},{"id":"47111","messageId":"11842100582887-git-send-email-bdowning@lavos.net","threadId":"8997","inReplyTo":"11842100581060-git-send-email-bdowning@lavos.net","subject":"[PATCH 3/5] Add pack-objects window memory usage limit","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T03:14:16Z","receivedAt":"2007-07-12T03:14:16Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"This adds an option (--window-bytes=N) and configuration variable\n(pack.windowBytes = N) to limit the memory size of the pack-objects\ndelta search window.  This works by removing the oldest unpacked objects\nwhenever the total size goes above the limit.  It will always leave\nat least one object, though, so as not to completely eliminate the\npossibility of computing deltas.\n\nThis is an extra limit on top of the normal window size (--window=N);\nthe window will not dynamically grow above the fixed number of entries\nspecified to fill the memory limit.\n\nWith this, repacking a repository with a mix of large and small objects\nis possible even with a very large window.\n\nSigned-off-by: Brian Downing <bdowning@lavos.net>\n---\n builtin-pack-objects.c |   56 ++++++++++++++++++++++++++++++++++++++++++------\n 1 files changed, 49 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 132ce96..6e441b7 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -16,8 +16,9 @@\n #include \"progress.h\"\n \n static const char pack_usage[] = \"\\\n-git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n-\t[--local] [--incremental] [--window=N] [--depth=N] \\n\\\n+git-pack-objects [{ -q | --progress | --all-progress }] \\n\\\n+\t[--max-pack-size=N] [--local] [--incremental] \\n\\\n+\t[--window=N] [--window-bytes=N] [--depth=N] \\n\\\n \t[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \\n\\\n \t[--non-empty] [--revs [--unpacked | --all]*] [--reflog] \\n\\\n \t[--stdout | base-name] [<ref-list | <object-list]\";\n@@ -79,6 +80,9 @@ static unsigned long delta_cache_size = 0;\n static unsigned long max_delta_cache_size = 0;\n static unsigned long cache_max_small_delta_size = 1000;\n \n+static unsigned long window_memory_usage = 0;\n+static unsigned long window_memory_limit = 0;\n+\n /*\n  * The object names in objects array are hashed with this hashtable,\n  * to help looking up the entry by object name.\n@@ -1351,12 +1355,14 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tif (sz != trg_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(trg_entry->idx.sha1), sz, trg_size);\n+\t\twindow_memory_usage += sz;\n \t}\n \tif (!src->data) {\n \t\tsrc->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);\n \t\tif (sz != src_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(src_entry->idx.sha1), sz, src_size);\n+\t\twindow_memory_usage += sz;\n \t}\n \tif (!src->index) {\n \t\tsrc->index = create_delta_index(src->data, src_size);\n@@ -1366,6 +1372,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\t\t\twarning(\"suboptimal pack - out of memory\");\n \t\t\treturn 0;\n \t\t}\n+\t\twindow_memory_usage += sizeof_delta_index(src->index);\n \t}\n \n \tdelta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);\n@@ -1408,9 +1415,22 @@ static unsigned int check_delta_limit(struct object_entry *me, unsigned int n)\n \treturn m;\n }\n \n+static void free_unpacked(struct unpacked *n)\n+{\n+\twindow_memory_usage -= sizeof_delta_index(n->index);\n+\tfree_delta_index(n->index);\n+\tn->index = NULL;\n+\tif (n->data) {\n+\t\tfree(n->data);\n+\t\tn->data = NULL;\n+\t\twindow_memory_usage -= n->entry->size;\n+\t}\n+\tn->entry = NULL;\n+}\n+\n static void find_deltas(struct object_entry **list, int window, int depth)\n {\n-\tuint32_t i = nr_objects, idx = 0, processed = 0;\n+\tuint32_t i = nr_objects, idx = 0, count = 0, processed = 0;\n \tunsigned int array_size = window * sizeof(struct unpacked);\n \tstruct unpacked *array;\n \tint max_depth;\n@@ -1445,12 +1465,21 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \t\tif (entry->no_try_delta)\n \t\t\tcontinue;\n \n-\t\tfree_delta_index(n->index);\n-\t\tn->index = NULL;\n-\t\tfree(n->data);\n-\t\tn->data = NULL;\n+\t\tfree_unpacked(n);\n \t\tn->entry = entry;\n \n+\t\twhile (window_memory_limit &&\n+\t\t       window_memory_usage > window_memory_limit &&\n+\t\t       count > 1) {\n+\t\t\tuint32_t tail = idx - count;\n+\t\t\tif (tail > idx) {\n+\t\t\t\ttail += window + 1;\n+\t\t\t\ttail %= window;\n+\t\t\t}\n+\t\t\tfree_unpacked(array + tail);\n+\t\t\tcount--;\n+\t\t}\n+\n \t\t/*\n \t\t * If the current object is at pack edge, take the depth the\n \t\t * objects that depend on the current object into account\n@@ -1485,6 +1514,8 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \n \t\tnext:\n \t\tidx++;\n+\t\tif (count < window)\n+\t\t\tcount++;\n \t\tif (idx >= window)\n \t\t\tidx = 0;\n \t} while (i > 0);\n@@ -1523,6 +1554,10 @@ static int git_pack_config(const char *k, const char *v)\n \t\twindow = git_config_int(k, v);\n \t\treturn 0;\n \t}\n+\tif(!strcmp(k, \"pack.windowbytes\")) {\n+\t\twindow_memory_limit = git_config_int(k, v);\n+\t\treturn 0;\n+\t}\n \tif(!strcmp(k, \"pack.depth\")) {\n \t\tdepth = git_config_int(k, v);\n \t\treturn 0;\n@@ -1699,6 +1734,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t\tusage(pack_usage);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--window-bytes=\")) {\n+\t\t\tchar *end;\n+\t\t\twindow_memory_limit = strtoul(arg+15, &end, 0);\n+\t\t\tif (!arg[15] || *end)\n+\t\t\t\tusage(pack_usage);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--depth=\")) {\n \t\t\tchar *end;\n \t\t\tdepth = strtoul(arg+8, &end, 0);\n-- \n1.5.2.GIT\n"},{"id":"47112","messageId":"11842100582743-git-send-email-bdowning@lavos.net","threadId":"8997","inReplyTo":"11842100581060-git-send-email-bdowning@lavos.net","subject":"[PATCH 4/5] Add --window-bytes option to git-repack","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T03:14:17Z","receivedAt":"2007-07-12T03:14:17Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Signed-off-by: Brian Downing <bdowning@lavos.net>\n---\n git-repack.sh |    3 ++-\n 1 files changed, 2 insertions(+), 1 deletions(-)\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex b5c6671..4cff812 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -3,7 +3,7 @@\n # Copyright (c) 2005 Linus Torvalds\n #\n \n-USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--max-pack-size=N] [--window=N] [--depth=N]'\n+USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--max-pack-size=N] [--window=N] [--window-bytes=N] [--depth=N]'\n SUBDIRECTORY_OK='Yes'\n . git-sh-setup\n \n@@ -20,6 +20,7 @@ do\n \t-l)\tlocal=--local ;;\n \t--max-pack-size=*) extra=\"$extra $1\" ;;\n \t--window=*) extra=\"$extra $1\" ;;\n+\t--window-bytes=*) extra=\"$extra $1\" ;;\n \t--depth=*) extra=\"$extra $1\" ;;\n \t*)\tusage ;;\n \tesac\n-- \n1.5.2.GIT\n"},{"id":"47113","messageId":"1184210058514-git-send-email-bdowning@lavos.net","threadId":"8997","inReplyTo":"11842100581060-git-send-email-bdowning@lavos.net","subject":"[PATCH 5/5] Add documentation for --window-bytes, pack.windowBytes","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T03:14:18Z","receivedAt":"2007-07-12T03:14:18Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"Signed-off-by: Brian Downing <bdowning@lavos.net>\n---\n Documentation/config.txt           |    5 +++++\n Documentation/git-pack-objects.txt |    8 ++++++++\n Documentation/git-repack.txt       |    8 ++++++++\n 3 files changed, 21 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex aeece84..83c7dc1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -592,6 +592,11 @@ pack.depth::\n \tThe maximum delta depth used by gitlink:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n \n+pack.windowBytes::\n+\tThis option provides an additional limit on top of `pack.window`;\n+\tthe window size will dynamically scale down so as to not take\n+\tup more than N bytes in memory.\n+\n pack.compression::\n \tAn integer -1..9, indicating the compression level for objects\n \tin a pack file. -1 is the zlib default. 0 means no\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e3549b5..21ed198 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -85,6 +85,14 @@ base-name::\n \ttimes to get to the necessary object.\n \tThe default value for --window is 10 and --depth is 50.\n \n+--window-bytes=[N]::\n+\tThis option provides an additional limit on top of `--window`;\n+\tthe window size will dynamically scale down so as to not take\n+\tup more than N bytes in memory.  This is useful in\n+\trepositories with a mix of large and small objects to not run\n+\tout of memory with a large window, but still be able to take\n+\tadvantage of the large window for the smaller objects.\n+\n --max-pack-size=<n>::\n \tMaximum size of each output packfile, expressed in MiB.\n \tIf specified,  multiple packfiles may be created.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 2894939..805d930 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -68,6 +68,14 @@ OPTIONS\n \tto be applied that many times to get to the necessary object.\n \tThe default value for --window is 10 and --depth is 50.\n \n+--window-bytes=[N]::\n+\tThis option provides an additional limit on top of `--window`;\n+\tthe window size will dynamically scale down so as to not take\n+\tup more than N bytes in memory.  This is useful in\n+\trepositories with a mix of large and small objects to not run\n+\tout of memory with a large window, but still be able to take\n+\tadvantage of the large window for the smaller objects.\n+\n --max-pack-size=<n>::\n \tMaximum size of each output packfile, expressed in MiB.\n \tIf specified,  multiple packfiles may be created.\n-- \n1.5.2.GIT\n"},{"id":"47114","messageId":"alpine.LFD.0.999.0707112356330.32552@xanadu.home","threadId":"8997","inReplyTo":"11842100582887-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 3/5] Add pack-objects window memory usage limit","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-07-12T04:25:54Z","receivedAt":"2007-07-12T04:25:54Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Jul 2007, Brian Downing wrote:\n\n> +\t\twhile (window_memory_limit &&\n> +\t\t       window_memory_usage > window_memory_limit &&\n> +\t\t       count > 1) {\n> +\t\t\tuint32_t tail = idx - count;\n> +\t\t\tif (tail > idx) {\n> +\t\t\t\ttail += window + 1;\n> +\t\t\t\ttail %= window;\n> +\t\t\t}\n> +\t\t\tfree_unpacked(array + tail);\n> +\t\t\tcount--;\n> +\t\t}\n\nThis is bogus.  Suppose window = 10 and only array entries 8, 9, 0, 1 \nand 2 are populated.  In that case idx = 2 and count should be 4 (not \ncounting the current entry yet).  You want to evict entry 8.\n\n\t-- tail = 2 - 4 = -2 (or a big uint32_t value)\n\n\t-- tail > idx is true\n\n\t-- tail += window + 1 -> -2 + 10 + 1 = 9\n\n\t-- tail %= window is useless\n\n\t-- you free entry 9 instead of entry 8.\n\nInstead, you should do:\n\n\ttail = idx - count;\n\tif (tail > idx)\n\t\ttail += window;\n\nor even:\n\n\ttail = (idx + window - count) % window;\n\n>  \t\tnext:\n>  \t\tidx++;\n> +\t\tif (count < window)\n> +\t\t\tcount++;\n\nAnd of course you want:\n\n\tif (count + 1 < window)\n\t\tcount++;\n\nSo not to count the new entry when the window gets full.\n\n\nNicolas\n"},{"id":"47115","messageId":"alpine.LFD.0.999.0707120028020.32552@xanadu.home","threadId":"8997","inReplyTo":"1184210058514-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 5/5] Add documentation for --window-bytes, pack.windowBytes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-07-12T04:35:22Z","receivedAt":"2007-07-12T04:35:22Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Jul 2007, Brian Downing wrote:\n\n> Signed-off-by: Brian Downing <bdowning@lavos.net>\n> ---\n>  Documentation/config.txt           |    5 +++++\n>  Documentation/git-pack-objects.txt |    8 ++++++++\n>  Documentation/git-repack.txt       |    8 ++++++++\n>  3 files changed, 21 insertions(+), 0 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index aeece84..83c7dc1 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -592,6 +592,11 @@ pack.depth::\n>  \tThe maximum delta depth used by gitlink:git-pack-objects[1] when no\n>  \tmaximum depth is given on the command line. Defaults to 50.\n>  \n> +pack.windowBytes::\n> +\tThis option provides an additional limit on top of `pack.window`;\n> +\tthe window size will dynamically scale down so as to not take\n> +\tup more than N bytes in memory.\n> +\n\nThis doesn't say what the default (unlimited) is.\n\n>  pack.compression::\n>  \tAn integer -1..9, indicating the compression level for objects\n>  \tin a pack file. -1 is the zlib default. 0 means no\n> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n> index e3549b5..21ed198 100644\n> --- a/Documentation/git-pack-objects.txt\n> +++ b/Documentation/git-pack-objects.txt\n> @@ -85,6 +85,14 @@ base-name::\n>  \ttimes to get to the necessary object.\n>  \tThe default value for --window is 10 and --depth is 50.\n>  \n> +--window-bytes=[N]::\n> +\tThis option provides an additional limit on top of `--window`;\n> +\tthe window size will dynamically scale down so as to not take\n> +\tup more than N bytes in memory.  This is useful in\n> +\trepositories with a mix of large and small objects to not run\n> +\tout of memory with a large window, but still be able to take\n> +\tadvantage of the large window for the smaller objects.\n\nDitto here.\n\nAlso it is a bit akward to specify a size in bytes when you probably \nwant to specify a limit which is in the megabyte range.  I'd call them \n--window_mem and pack.windowmemory, and allow for unit suffixes of 'k', \n'm', or 'g' to be supported if not already.\n\n\nNicolas\n"},{"id":"47116","messageId":"alpine.LFD.0.999.0707120036200.32552@xanadu.home","threadId":"8997","inReplyTo":"11842100581060-git-send-email-bdowning@lavos.net","subject":"Re: [PATCH 0/5] Memory-limited pack-object window support","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-07-12T04:38:27Z","receivedAt":"2007-07-12T04:38:27Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Jul 2007, Brian Downing wrote:\n\n> This patch series implements a memory limit on the window size for\n> pack-objects and repack.  Basically, the window size will temporarily\n> grow smaller than the --window option specifies if the total memory\n> usage of the window is over the specified limit.\n\nBesides the small issues I've pointed out already, I think this is a \nvery good thing.\n\n\nNicolas\n"},{"id":"47135","messageId":"20070712100251.GT4087@lavos.net","threadId":"8997","inReplyTo":"alpine.LFD.0.999.0707112356330.32552@xanadu.home","subject":"Re: [PATCH 3/5] Add pack-objects window memory usage limit","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-12T10:02:51Z","receivedAt":"2007-07-12T10:02:51Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"On Thu, Jul 12, 2007 at 12:25:54AM -0400, Nicolas Pitre wrote:\n> On Wed, 11 Jul 2007, Brian Downing wrote:\n> \n> > +\t\twhile (window_memory_limit &&\n> > +\t\t       window_memory_usage > window_memory_limit &&\n> > +\t\t       count > 1) {\n> > +\t\t\tuint32_t tail = idx - count;\n> > +\t\t\tif (tail > idx) {\n> > +\t\t\t\ttail += window + 1;\n> > +\t\t\t\ttail %= window;\n> > +\t\t\t}\n> > +\t\t\tfree_unpacked(array + tail);\n> > +\t\t\tcount--;\n> > +\t\t}\n> \n> This is bogus.  Suppose window = 10 and only array entries 8, 9, 0, 1 \n> and 2 are populated.  In that case idx = 2 and count should be 4 (not \n> counting the current entry yet).  You want to evict entry 8.\n\nThe current idx has already been depopulated by the time that code is\nrun, and count is probably one higher than you are expecting, so this\ndoes actually work.\n\nHowever, looking at it again, I think if the window hasn't been saturated\nyet in my current code count will be what you expect in this situation\nand it will screw up as you describe.\n\nBesides, it is admittedly clumsy as hell (a common affliction when\ndealing with circular buffers for me it seems).  I'll see if I can get\nsomething better that works.\n\nThanks,\n-bcd\n"}]}