{"thread":{"id":"22472","subject":"[PATCH v3] fast-import: Stream very large blobs directly to pack","startedAt":"2010-02-01T17:27:35Z","lastAt":"2010-02-01T17:27:35Z","messageCount":1,"participants":["Shawn O. Pearce"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"133268","messageId":"20100201172735.GC9394@spearce.org","threadId":"22472","inReplyTo":null,"subject":"[PATCH v3] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-01T17:27:35Z","receivedAt":"2010-02-01T17:27:35Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If a blob is larger than the configured big-file-threshold, instead\nof reading it into a single buffer obtained from malloc, stream it\nonto the end of the current pack file.  Streaming the larger objects\ninto the pack avoids the 4+ GiB memory footprint that occurs when\nfast-import is processing 2+ GiB blobs.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n\n Changed --big-file-threshold option to be in bytes.\n Squashed the documentation patch from earlier today.\n\n Documentation/config.txt          |   14 +++\n Documentation/git-fast-import.txt |    6 ++\n fast-import.c                     |  179 +++++++++++++++++++++++++++++++++----\n t/t5705-clone-2gb.sh              |    2 +-\n t/t9300-fast-import.sh            |   46 ++++++++++\n 5 files changed, 227 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f7728ec..b16a20b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -409,6 +409,20 @@ You probably do not need to adjust this value.\n +\n Common unit suffixes of 'k', 'm', or 'g' are supported.\n \n+core.bigFileThreshold::\n+\tFiles larger than this size are stored deflated, without\n+\tattempting delta compression.  Storing large files without\n+\tdelta compression avoids excessive memory usage, at the\n+\tslight expense of increased disk usage.\n++\n+Default is 512 MiB on all platforms.  This should be reasonable\n+for most projects as source code and other text files can still\n+be delta compressed, but larger binary media files won't be.\n++\n+Common unit suffixes of 'k', 'm', or 'g' are supported.\n++\n+Currently only linkgit:git-fast-import[1] honors this setting.\n+\n core.excludesfile::\n \tIn addition to '.gitignore' (per-directory) and\n \t'.git/info/exclude', git looks into this file for patterns\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex e6d364f..79c5f69 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -50,6 +50,12 @@ OPTIONS\n \timporters may wish to lower this, such as to ensure the\n \tresulting packfiles fit on CDs.\n \n+--big-file-threshold=<n>::\n+\tMaximum size of a blob that fast-import will attempt to\n+\tcreate a delta for, expressed in bytes.  The default is 512m\n+\t(512 MiB).  Some importers may wish to lower this on systems\n+\twith constrained memory.\n+\n --depth=<n>::\n \tMaximum delta depth, for blob and tree deltification.\n \tDefault is 10.\ndiff --git a/fast-import.c b/fast-import.c\nindex 60d0aa2..2105310 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -280,6 +280,7 @@ struct recent_command\n /* Configured limits on output */\n static unsigned long max_depth = 10;\n static off_t max_packsize = (1LL << 32) - 1;\n+static uintmax_t big_file_threshold = 512 * 1024 * 1024;\n static int force_update;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n@@ -1003,7 +1004,7 @@ static void cycle_packfile(void)\n \n static size_t encode_header(\n \tenum object_type type,\n-\tsize_t size,\n+\tuintmax_t size,\n \tunsigned char *hdr)\n {\n \tint n = 1;\n@@ -1159,6 +1160,118 @@ static int store_object(\n \treturn 0;\n }\n \n+static void truncate_pack(off_t to)\n+{\n+\tif (ftruncate(pack_data->pack_fd, to)\n+\t || lseek(pack_data->pack_fd, to, SEEK_SET) != to)\n+\t\tdie_errno(\"cannot truncate pack to skip duplicate\");\n+\tpack_size = to;\n+}\n+\n+static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n+{\n+\tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n+\tunsigned char *in_buf = xmalloc(in_sz);\n+\tunsigned char *out_buf = xmalloc(out_sz);\n+\tstruct object_entry *e;\n+\tunsigned char sha1[20];\n+\tunsigned long hdrlen;\n+\toff_t offset;\n+\tgit_SHA_CTX c;\n+\tz_stream s;\n+\tint status = Z_OK;\n+\n+\t/* Determine if we should auto-checkpoint. */\n+\tif ((pack_size + 60 + len) > max_packsize\n+\t\t|| (pack_size + 60 + len) < pack_size)\n+\t\tcycle_packfile();\n+\n+\toffset = pack_size;\n+\n+\thdrlen = snprintf((char *)out_buf, out_sz, \"blob %\" PRIuMAX, len) + 1;\n+\tif (out_sz <= hdrlen)\n+\t\tdie(\"impossibly large object header\");\n+\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, out_buf, hdrlen);\n+\n+\tmemset(&s, 0, sizeof(s));\n+\tdeflateInit(&s, pack_compression_level);\n+\n+\thdrlen = encode_header(OBJ_BLOB, len, out_buf);\n+\tif (out_sz <= hdrlen)\n+\t\tdie(\"impossibly large object header\");\n+\n+\ts.next_out = out_buf + hdrlen;\n+\ts.avail_out = out_sz - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tif (0 < len && !s.avail_in) {\n+\t\t\tsize_t cnt = in_sz < len ? in_sz : (size_t)len;\n+\t\t\tsize_t n = fread(in_buf, 1, cnt, stdin);\n+\t\t\tif (!n && feof(stdin))\n+\t\t\t\tdie(\"EOF in data (%\" PRIuMAX \" bytes remaining)\", len);\n+\n+\t\t\tgit_SHA1_Update(&c, in_buf, n);\n+\t\t\ts.next_in = in_buf;\n+\t\t\ts.avail_in = n;\n+\t\t\tlen -= n;\n+\t\t}\n+\n+\t\tstatus = deflate(&s, len ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tsize_t n = s.next_out - out_buf;\n+\t\t\twrite_or_die(pack_data->pack_fd, out_buf, n);\n+\t\t\tpack_size += n;\n+\t\t\ts.next_out = out_buf;\n+\t\t\ts.avail_out = out_sz;\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tdeflateEnd(&s);\n+\tgit_SHA1_Final(sha1, &c);\n+\n+\tif (sha1out)\n+\t\thashcpy(sha1out, sha1);\n+\n+\te = insert_object(sha1);\n+\n+\tif (mark)\n+\t\tinsert_mark(mark, e);\n+\n+\tif (e->offset) {\n+\t\tduplicate_count_by_type[OBJ_BLOB]++;\n+\t\ttruncate_pack(offset);\n+\n+\t} else if (find_sha1_pack(sha1, packed_git)) {\n+\t\te->type = OBJ_BLOB;\n+\t\te->pack_id = MAX_PACK_ID;\n+\t\te->offset = 1; /* just not zero! */\n+\t\tduplicate_count_by_type[OBJ_BLOB]++;\n+\t\ttruncate_pack(offset);\n+\n+\t} else {\n+\t\te->depth = 0;\n+\t\te->type = OBJ_BLOB;\n+\t\te->pack_id = pack_id;\n+\t\te->offset = offset;\n+\t\tobject_count++;\n+\t\tobject_count_by_type[OBJ_BLOB]++;\n+\t}\n+\n+\tfree(in_buf);\n+\tfree(out_buf);\n+}\n+\n /* All calls must be guarded by find_object() or find_mark() to\n  * ensure the 'struct object_entry' passed was written by this\n  * process instance.  We unpack the entry by the offset, avoiding\n@@ -1704,7 +1817,7 @@ static void parse_mark(void)\n \t\tnext_mark = 0;\n }\n \n-static void parse_data(struct strbuf *sb)\n+static int parse_data(struct strbuf *sb, uintmax_t limit, uintmax_t *len_res)\n {\n \tstrbuf_reset(sb);\n \n@@ -1728,9 +1841,15 @@ static void parse_data(struct strbuf *sb)\n \t\tfree(term);\n \t}\n \telse {\n-\t\tsize_t n = 0, length;\n+\t\tuintmax_t len = strtoumax(command_buf.buf + 5, NULL, 10);\n+\t\tsize_t n = 0, length = (size_t)len;\n \n-\t\tlength = strtoul(command_buf.buf + 5, NULL, 10);\n+\t\tif (limit && limit < len) {\n+\t\t\t*len_res = len;\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (length < len)\n+\t\t\tdie(\"data is too large to use in this context\");\n \n \t\twhile (n < length) {\n \t\t\tsize_t s = strbuf_fread(sb, length - n, stdin);\n@@ -1742,6 +1861,7 @@ static void parse_data(struct strbuf *sb)\n \t}\n \n \tskip_optional_lf();\n+\treturn 1;\n }\n \n static int validate_raw_date(const char *src, char *result, int maxlen)\n@@ -1806,14 +1926,32 @@ static char *parse_ident(const char *buf)\n \treturn ident;\n }\n \n-static void parse_new_blob(void)\n+static void parse_and_store_blob(\n+\tstruct last_object *last,\n+\tunsigned char *sha1out,\n+\tuintmax_t mark)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tuintmax_t len;\n \n+\tif (parse_data(&buf, big_file_threshold, &len))\n+\t\tstore_object(OBJ_BLOB, &buf, last, sha1out, mark);\n+\telse {\n+\t\tif (last) {\n+\t\t\tstrbuf_release(&last->data);\n+\t\t\tlast->offset = 0;\n+\t\t\tlast->depth = 0;\n+\t\t}\n+\t\tstream_blob(len, sha1out, mark);\n+\t\tskip_optional_lf();\n+\t}\n+}\n+\n+static void parse_new_blob(void)\n+{\n \tread_next_command();\n \tparse_mark();\n-\tparse_data(&buf);\n-\tstore_object(OBJ_BLOB, &buf, &last_blob, NULL, next_mark);\n+\tparse_and_store_blob(&last_blob, NULL, next_mark);\n }\n \n static void unload_one_branch(void)\n@@ -1924,15 +2062,12 @@ static void file_change_m(struct branch *b)\n \t\t * another repository.\n \t\t */\n \t} else if (inline_data) {\n-\t\tstatic struct strbuf buf = STRBUF_INIT;\n-\n \t\tif (p != uq.buf) {\n \t\t\tstrbuf_addstr(&uq, p);\n \t\t\tp = uq.buf;\n \t\t}\n \t\tread_next_command();\n-\t\tparse_data(&buf);\n-\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t\tparse_and_store_blob(&last_blob, sha1, 0);\n \t} else if (oe) {\n \t\tif (oe->type != OBJ_BLOB)\n \t\t\tdie(\"Not a blob (actually a %s): %s\",\n@@ -2058,15 +2193,12 @@ static void note_change_n(struct branch *b)\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tstatic struct strbuf buf = STRBUF_INIT;\n-\n \t\tif (p != uq.buf) {\n \t\t\tstrbuf_addstr(&uq, p);\n \t\t\tp = uq.buf;\n \t\t}\n \t\tread_next_command();\n-\t\tparse_data(&buf);\n-\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t\tparse_and_store_blob(&last_blob, sha1, 0);\n \t} else if (oe) {\n \t\tif (oe->type != OBJ_BLOB)\n \t\t\tdie(\"Not a blob (actually a %s): %s\",\n@@ -2232,7 +2364,7 @@ static void parse_new_commit(void)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n-\tparse_data(&msg);\n+\tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n \tmerge_list = parse_merge(&merge_count);\n@@ -2353,7 +2485,7 @@ static void parse_new_tag(void)\n \t\ttagger = NULL;\n \n \t/* tag payload/message */\n-\tparse_data(&msg);\n+\tparse_data(&msg, 0, NULL);\n \n \t/* build the tag object */\n \tstrbuf_reset(&new_data);\n@@ -2473,11 +2605,15 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tpack_compression_seen = 1;\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"core.bigfilethreshold\")) {\n+\t\tlong n = git_config_int(k, v);\n+\t\tbig_file_threshold = 0 < n ? n : 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n static const char fast_import_usage[] =\n-\"git fast-import [--date-format=f] [--max-pack-size=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]\";\n+\"git fast-import [--date-format=f] [--max-pack-size=n] [--big-file-threshold=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]\";\n \n int main(int argc, const char **argv)\n {\n@@ -2518,7 +2654,12 @@ int main(int argc, const char **argv)\n \t\t}\n \t\telse if (!prefixcmp(a, \"--max-pack-size=\"))\n \t\t\tmax_packsize = strtoumax(a + 16, NULL, 0) * 1024 * 1024;\n-\t\telse if (!prefixcmp(a, \"--depth=\")) {\n+\t\telse if (!prefixcmp(a, \"--big-file-threshold=\")) {\n+\t\t\tunsigned long v;\n+\t\t\tif (!git_parse_ulong(a + 21, &v))\n+\t\t\t\tusage(fast_import_usage);\n+\t\t\tbig_file_threshold = v;\n+\t\t} else if (!prefixcmp(a, \"--depth=\")) {\n \t\t\tmax_depth = strtoul(a + 8, NULL, 0);\n \t\t\tif (max_depth > MAX_DEPTH)\n \t\t\t\tdie(\"--depth cannot exceed %u\", MAX_DEPTH);\ndiff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh\nindex 9f52154..adfaae8 100755\n--- a/t/t5705-clone-2gb.sh\n+++ b/t/t5705-clone-2gb.sh\n@@ -31,7 +31,7 @@ test_expect_success 'setup' '\n \t echo \"data 5\" &&\n \t echo \">2gb\" &&\n \t cat commit) |\n-\tgit fast-import &&\n+\tgit fast-import --big-file-threshold=2 &&\n \ttest ! -f exit-status\n \n '\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex b49815d..513db86 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1254,4 +1254,50 @@ test_expect_success \\\n \t'Q: verify note for third commit' \\\n \t'git cat-file blob refs/notes/foobar:$commit3 >actual && test_cmp expect actual'\n \n+##\n+## R: very large blobs\n+##\n+blobsize=$((2*1024*1024 + 53))\n+test-genrandom bar $blobsize >expect\n+cat >input <<INPUT_END\n+commit refs/heads/big-file\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+data <<COMMIT\n+R - big file\n+COMMIT\n+\n+M 644 inline big1\n+data $blobsize\n+INPUT_END\n+cat expect >>input\n+cat >>input <<INPUT_END\n+M 644 inline big2\n+data $blobsize\n+INPUT_END\n+cat expect >>input\n+echo >>input\n+\n+test_expect_success \\\n+\t'R: blob bigger than threshold' \\\n+\t'test_create_repo R &&\n+\t git --git-dir=R/.git fast-import --big-file-threshold=1 <input'\n+test_expect_success \\\n+\t'R: verify created pack' \\\n+\t': >verify &&\n+\t for p in R/.git/objects/pack/*.pack;\n+\t do\n+\t   git verify-pack -v $p >>verify || exit;\n+\t done'\n+test_expect_success \\\n+\t'R: verify written objects' \\\n+\t'git --git-dir=R/.git cat-file blob big-file:big1 >actual &&\n+\t test_cmp expect actual &&\n+\t a=$(git --git-dir=R/.git rev-parse big-file:big1) &&\n+\t b=$(git --git-dir=R/.git rev-parse big-file:big2) &&\n+\t test $a = $b'\n+test_expect_success \\\n+\t'R: blob appears only once' \\\n+\t'n=$(grep $a verify | wc -l) &&\n+\t test 1 = $n'\n+\n test_done\n-- \n1.7.0.rc0.170.g7207c\n\n-- \nShawn.\n"}]}