{"thread":{"id":"8277","subject":"[PATCH v2] Prevent megablobs from gunking up git packs","startedAt":"2007-05-23T02:41:15Z","lastAt":"2007-05-23T02:41:15Z","messageCount":1,"participants":["Dana How"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"43013","messageId":"4653A9CB.8070309@gmail.com","threadId":"8277","inReplyTo":null,"subject":"[PATCH v2] Prevent megablobs from gunking up git packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-23T02:41:15Z","receivedAt":"2007-05-23T02:41:15Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"\ngit stores data in loose blobs or in packfiles.  The former has\nessentially now become an exception mechanism,  to store unusually\n*young* blobs.  Here we re-use all this \"exception\" machinery to\nstore unusually *large* blobs as well.\n\nThis patch implements the following:\n1. git pack-objects takes a new --max-blob-size=N flag,  with the\n   effect that only blobs less than N KB are written to the\n   packfiles(s).  If an already packed blob violates this limit\n   (perhaps these are fast-import packs or max-blob-size was\n   reduced),  a new loose object is made as needed so data is not lost.\n2. git repack inspects repack.maxblobsize .  If set,  its\n   value is passed to git pack-objects on the command line.\n   --max-blob-size=N is also accepted by git repack.\n3. No other git pack-objects caller uses this feature or sees any change.\n\nDuring pack *creation* this avoids copying & deltifying megablobs.\nTherefore,  for 100GB repositories,  git repack can run in hours\nfor the first time after git fast-import,  and in minutes thereafter.\nYou can still include megablobs in your packs thus:\n % git-repack -a [-d]\t{with repack.maxblobsize = 256}\n % git-repack --max-blob-size=0 [--max-pack-size=2047]\nThe first step creates a tight packfile and index with all metadata\nand reasonable blob data,  and the second collects the megablobs,\nwhich could be enhanced with .keep file manipulation,  etc.\n\nDuring pack *use* this feature helps tighten access to heavily-used\nmetadata since it can't be separated by large blobs,  and allows all\nsuch metadata to be accessed through the smallest number of index files.\nMegablobs are accessible with some overhead as loose objects,\nwhich can be corrected with a second packing step as shown above\nat the cost of more index files and packing time.\n\nDocumentation has been updated and operation with pack-object's\n--stdout is prevented.  This patch is based on \"next\".\n\nSigned-off-by: Dana L. How <danahow@gmail.com>\n---\n Documentation/config.txt           |    5 ++++\n Documentation/git-pack-objects.txt |    6 +++++\n Documentation/git-repack.txt       |   10 ++++++++\n builtin-pack-objects.c             |   41 +++++++++++++++++++++++++++++++----\n cache.h                            |    2 +\n git-repack.sh                      |    9 +++++++-\n sha1_file.c                        |   17 +++++++++++++-\n 7 files changed, 82 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 179cb17..70fd3b1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -599,6 +599,11 @@ remotes.<group>::\n \tThe list of remotes which are fetched by \"git remote update\n \t<group>\".  See gitlink:git-remote[1].\n \n+repack.maxblobsize::\n+\tPrevent gitlink:git-repack[1] from including blobs larger than\n+\tthe specified number in kB,  unless overridden by --max-blob-size=N switch.\n+\tDefaults to zero which means no maximum size is in effect.\n+\n repack.usedeltabaseoffset::\n \tAllow gitlink:git-repack[1] to create packs that uses\n \tdelta-base offset.  Defaults to false.\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex cfe127a..ebd36b9 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -85,6 +85,12 @@ base-name::\n \ttimes to get to the necessary object.\n \tThe default value for --window is 10 and --depth is 50.\n \n+--max-blob-size=<n>::\n+\tMaximum size of included blobs, expressed in kB.\n+\tIf specified,  affected blobs only existing in packfiles\n+\tmay be written out as new loose objects to prevent their loss.\n+\tThe default is unlimited.\n+\n --max-pack-size=<n>::\n \tMaximum size of each output packfile, expressed in MiB.\n \tIf specified,  multiple packfiles may be created.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 2847c9b..7daa697 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -65,6 +65,12 @@ OPTIONS\n \tto be applied that many times to get to the necessary object.\n \tThe default value for --window is 10 and --depth is 50.\n \n+--max-blob-size=<n>::\n+\tMaximum size of included blobs, expressed in kB.\n+\tIf specified,  affected blobs only existing in packfiles\n+\tmay be written out as new loose objects to prevent their loss.\n+\tThe default is unlimited.\n+\n --max-pack-size=<n>::\n \tMaximum size of each output packfile, expressed in MiB.\n \tIf specified,  multiple packfiles may be created.\n@@ -84,6 +90,10 @@ be able to read (this includes repositories from which packs can\n be copied out over http or rsync, and people who obtained packs\n that way can try to use older git with it).\n \n+The configuration variable `repack.MaxBlobSize` provides the\n+default for the --max-blob-size option if set.  The latter\n+takes precedence.\n+\n \n Author\n ------\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 19b0aa1..38e2a2b 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -17,7 +17,7 @@\n \n static const char pack_usage[] = \"\\\n git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n-\t[--local] [--incremental] [--window=N] [--depth=N] \\n\\\n+\t[--local] [--incremental] [--window=N] [--depth=N] [--max-blob-size=N]\\n\\\n \t[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \\n\\\n \t[--non-empty] [--revs [--unpacked | --all]*] [--reflog] \\n\\\n \t[--stdout | base-name] [<ref-list | <object-list]\";\n@@ -75,6 +75,7 @@ static int num_preferred_base;\n static struct progress progress_state;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n+static uint32_t max_blob_size;\n \n /*\n  * The object names in objects array are hashed with this hashtable,\n@@ -371,8 +372,6 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t\tpack_size_limit - write_offset : 0;\n \t\t\t\t/* no if no delta */\n \tint usable_delta =\t!entry->delta ? 0 :\n-\t\t\t\t/* yes if unlimited packfile */\n-\t\t\t\t!pack_size_limit ? 1 :\n \t\t\t\t/* no if base written to previous pack */\n \t\t\t\tentry->delta->offset == (off_t)-1 ? 0 :\n \t\t\t\t/* otherwise double-check written to this\n@@ -408,7 +407,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\tbuf = read_sha1_file(entry->sha1, &type, &size);\n \t\tif (!buf)\n \t\t\tdie(\"unable to read %s\", sha1_to_hex(entry->sha1));\n-\t\tif (size != entry->size)\n+\t\tif (size != entry->size && type == obj_type)\n \t\t\tdie(\"object %s size inconsistency (%lu vs %lu)\",\n \t\t\t    sha1_to_hex(entry->sha1), size, entry->size);\n \t\tif (usable_delta) {\n@@ -564,6 +563,25 @@ static off_t write_one(struct sha1file *f,\n \t\t\treturn 0;\n \t}\n \n+\t/* refuse to include megablobs */\n+\tif (max_blob_size && e->size >= max_blob_size) {\n+\t\tif (e->in_pack) {\n+\t\t\t/* rewrite as loose object so git-repack doesn't lose data */\n+\t\t\tvoid *buf;\n+\t\t\tenum object_type type;\n+\t\t\tunsigned long size;\n+\t\t\tbuf = read_sha1_file(e->sha1, &type, &size);\n+\t\t\tif (!buf)\n+\t\t\t\tdie(\"unable to read %s\", sha1_to_hex(e->sha1));\n+\t\t\tif (write_sha1_file_ignore_packs(buf, size, typename(type), NULL) < 0)\n+\t\t\t\tdie(\"failed to write object\");\n+\t\t\tfree(buf);\n+\t\t}\n+\t\te->offset = (off_t)-1;\t/* might eject a reused delta base if mbs decreases */\n+\t\twritten++;\n+\t\treturn offset;\n+\t}\n+\n \te->offset = offset;\n \tsize = write_object(f, e, offset);\n \tif (!size) {\n@@ -1422,13 +1440,16 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Now some size filtering heuristics. */\n \ttrg_size = trg_entry->size;\n+\tsrc_size = src_entry->size;\n+\t/* prevent use if later dropped from packfile */\n+\tif (max_blob_size && (trg_size >= max_blob_size || src_size >= max_blob_size))\n+\t\treturn 0;\n \tmax_size = trg_size/2 - 20;\n \tmax_size = max_size * (max_depth - src_entry->depth) / max_depth;\n \tif (max_size == 0)\n \t\treturn 0;\n \tif (trg_entry->delta && trg_entry->delta_size <= max_size)\n \t\tmax_size = trg_entry->delta_size-1;\n-\tsrc_size = src_entry->size;\n \tsizediff = src_size < trg_size ? trg_size - src_size : 0;\n \tif (sizediff >= max_size)\n \t\treturn 0;\n@@ -1735,6 +1756,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\tincremental = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--max-blob-size=\")) {\n+\t\t\tchar *end;\n+\t\t\tmax_blob_size = strtoul(arg+16, &end, 0) * 1024;\n+\t\t\tif (!arg[16] || *end)\n+\t\t\t\tusage(pack_usage);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--compression=\")) {\n \t\t\tchar *end;\n \t\t\tint level = strtoul(arg+14, &end, 0);\n@@ -1855,6 +1883,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!pack_to_stdout && thin)\n \t\tdie(\"--thin cannot be used to build an indexable pack.\");\n \n+\tif (pack_to_stdout && max_blob_size)\n+\t\tdie(\"--max-blob-size cannot be used to build a pack for transfer.\");\n+\n \tprepare_packed_git();\n \n \tif (progress)\ndiff --git a/cache.h b/cache.h\nindex ec85d93..a415924 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -343,6 +343,8 @@ extern int sha1_object_info(const unsigned char *, unsigned long *);\n extern void * read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size);\n extern int hash_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1);\n extern int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n+extern int write_sha1_file_ignore_packs(void *buf, unsigned long len, const char *type,\n+\t\t\t\t\tunsigned char *return_sha1);\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 4ea6e5b..6b4e1af 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -8,7 +8,7 @@ SUBDIRECTORY_OK='Yes'\n . git-sh-setup\n \n no_update_info= all_into_one= remove_redundant=\n-local= quiet= no_reuse= extra=\n+local= quiet= no_reuse= extra= max_blob_size=\n while case \"$#\" in 0) break ;; esac\n do\n \tcase \"$1\" in\n@@ -18,6 +18,7 @@ do\n \t-q)\tquiet=-q ;;\n \t-f)\tno_reuse=--no-reuse-object ;;\n \t-l)\tlocal=--local ;;\n+\t--max-blob-size=*) extra=\"$extra $1\" max_blob_size=t ;;\n \t--max-pack-size=*) extra=\"$extra $1\" ;;\n \t--window=*) extra=\"$extra $1\" ;;\n \t--depth=*) extra=\"$extra $1\" ;;\n@@ -35,6 +36,12 @@ true)\n \textra=\"$extra --delta-base-offset\" ;;\n esac\n \n+# handle blob limiting\n+if [ -z \"$max_blob_size\" ]; then\n+\tmbs=\"`git config --int repack.maxblobsize`\"\n+\t[ -n \"$mbs\" ] && extra=\"$extra --max-blob-size=$mbs\"\n+fi\n+\n PACKDIR=\"$GIT_OBJECT_DIRECTORY/pack\"\n PACKTMP=\"$GIT_OBJECT_DIRECTORY/.tmp-$$-pack\"\n rm -f \"$PACKTMP\"-*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 12d2ef2..1424756 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1979,7 +1979,8 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n \treturn 0;\n }\n \n-int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n+static int write_sha1_file_core(void *buf, unsigned long len, const char *type,\n+\t\t\t\tint checkpacks, unsigned char *returnsha1)\n {\n \tint size, ret;\n \tunsigned char *compressed;\n@@ -1997,7 +1998,7 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n \tfilename = sha1_file_name(sha1);\n \tif (returnsha1)\n \t\thashcpy(returnsha1, sha1);\n-\tif (has_sha1_file(sha1))\n+\tif (checkpacks && has_sha1_file(sha1))\n \t\treturn 0;\n \tfd = open(filename, O_RDONLY);\n \tif (fd >= 0) {\n@@ -2062,6 +2063,18 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n \treturn move_temp_to_file(tmpfile, filename);\n }\n \n+int write_sha1_file(void *buf, unsigned long len, const char *type,\n+\t\t    unsigned char *returnsha1)\n+{\n+\treturn write_sha1_file_core(buf, len, type, 1, returnsha1);\n+}\n+\n+int write_sha1_file_ignore_packs(void *buf, unsigned long len, const char *type,\n+\t\t\t\t unsigned char *returnsha1)\n+{\n+\treturn write_sha1_file_core(buf, len, type, 0, returnsha1);\n+}\n+\n /*\n  * We need to unpack and recompress the object for writing\n  * it out to a different file.\n-- \n1.5.2.762.gd8c6-dirty\n"}]}