{"thread":{"id":"22431","subject":"[PATCH] fast-import: Stream very large blobs directly to pack","startedAt":"2010-01-29T01:23:50Z","lastAt":"2010-02-04T02:30:56Z","messageCount":27,"participants":["Shawn O. Pearce","Nicolas Pitre","Junio C Hamano","Jakub Narebski","Sverre Rabbelier","A Large Angry SCM"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"132930","messageId":"20100129012350.GD20488@spearce.org","threadId":"22431","inReplyTo":null,"subject":"[PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T01:23:50Z","receivedAt":"2010-01-29T01:23:50Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If a blob is larger than the configured big-file-threshold, instead\nof reading it into a single buffer obtained from malloc, stream it\nonto the end of the current pack file. Streaming the larger objects\ninto the pack avoids the 4+ GiB memory footprint that occurs when\nfast-import is processing 2+ GiB blobs.\n\nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-fast-import.txt | 6 ++\n fast-import.c | 174 +++++++++++++++++++++++++++++++++----\n t/t5705-clone-2gb.sh | 2 +-\n t/t9300-fast-import.sh | 46 ++++++++++\n 4 files changed, 210 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex e6d364f..16d3596 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -50,6 +50,12 @@ OPTIONS\n \timporters may wish to lower this, such as to ensure the\n \tresulting packfiles fit on CDs.\n \n+--big-file-threshold=::\n+\tMaximum size of a blob that fast-import will attempt to\n+\tcreate a delta for, expressed in MiB. The default is 512.\n+\tSome importers may wish to lower this on systems with\n+\tconstrained memory.\n+\n --depth=::\n \tMaximum delta depth, for blob and tree deltification.\n \tDefault is 10.\ndiff --git a/fast-import.c b/fast-import.c\nindex 60d0aa2..8114f93 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -280,6 +280,7 @@ struct recent_command\n /* Configured limits on output */\n static unsigned long max_depth = 10;\n static off_t max_packsize = (1LL << 32) - 1;\n+static uintmax_t big_file_threshold = 512 * 1024 * 1024;\n static int force_update;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n@@ -1003,7 +1004,7 @@ static void cycle_packfile(void)\n \n static size_t encode_header(\n \tenum object_type type,\n-\tsize_t size,\n+\tuintmax_t size,\n \tunsigned char *hdr)\n {\n \tint n = 1;\n@@ -1159,6 +1160,121 @@ static int store_object(\n \treturn 0;\n }\n \n+static void truncate_pack(off_t to)\n+{\n+\tif (ftruncate(pack_data->pack_fd, to)\n+\t || lseek(pack_data->pack_fd, to, SEEK_SET) != to)\n+\t\tdie_errno(\"cannot truncate pack to skip duplicate\");\n+\tpack_size = to;\n+}\n+\n+static void stream_blob(\n+\tuintmax_t len,\n+\tunsigned char *sha1out,\n+\tuintmax_t mark)\n+{\n+\tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n+\tunsigned char *in_buf = xmalloc(in_sz);\n+\tunsigned char *out_buf = xmalloc(out_sz);\n+\tstruct object_entry *e;\n+\tunsigned char sha1[20];\n+\tunsigned long hdrlen;\n+\toff_t offset;\n+\tgit_SHA_CTX c;\n+\tz_stream s;\n+\tint status = Z_OK;\n+\n+\t/* Determine if we should auto-checkpoint. */\n+\tif ((pack_size + 60 + len) > max_packsize\n+\t\t|| (pack_size + 60 + len) < pack_size)\n+\t\tcycle_packfile();\n+\n+\toffset = pack_size;\n+\n+\thdrlen = snprintf((char *)out_buf, out_sz, \"blob %\" PRIuMAX, len) + 1;\n+\tif (out_sz <= hdrlen)\n+\t\tdie(\"impossibly large object header\");\n+\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, out_buf, hdrlen);\n+\n+\tmemset(&s, 0, sizeof(s));\n+\tdeflateInit(&s, pack_compression_level);\n+\n+\thdrlen = encode_header(OBJ_BLOB, len, out_buf);\n+\tif (out_sz <= hdrlen)\n+\t\tdie(\"impossibly large object header\");\n+\n+\ts.next_out = out_buf + hdrlen;\n+\ts.avail_out = out_sz - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tif (0 < len && !s.avail_in) {\n+\t\t\tsize_t cnt = in_sz < len ? in_sz : (size_t)len;\n+\t\t\tsize_t n = fread(in_buf, 1, cnt, stdin);\n+\t\t\tif (!n && feof(stdin))\n+\t\t\t\tdie(\"EOF in data (%\" PRIuMAX \" bytes remaining)\", len);\n+\n+\t\t\tgit_SHA1_Update(&c, in_buf, n);\n+\t\t\ts.next_in = in_buf;\n+\t\t\ts.avail_in = n;\n+\t\t\tlen -= n;\n+\t\t}\n+\n+\t\tstatus = deflate(&s, len ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tsize_t n = s.next_out - out_buf;\n+\t\t\twrite_or_die(pack_data->pack_fd, out_buf, n);\n+\t\t\tpack_size += n;\n+\t\t\ts.next_out = out_buf;\n+\t\t\ts.avail_out = out_sz;\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tdeflateEnd(&s);\n+\tgit_SHA1_Final(sha1, &c);\n+\n+\tif (sha1out)\n+\t\thashcpy(sha1out, sha1);\n+\n+\te = insert_object(sha1);\n+\n+\tif (mark)\n+\t\tinsert_mark(mark, e);\n+\n+\tif (e->offset) {\n+\t\tduplicate_count_by_type[OBJ_BLOB]++;\n+\t\ttruncate_pack(offset);\n+\n+\t} else if (find_sha1_pack(sha1, packed_git)) {\n+\t\te->type = OBJ_BLOB;\n+\t\te->pack_id = MAX_PACK_ID;\n+\t\te->offset = 1; /* just not zero! */\n+\t\tduplicate_count_by_type[OBJ_BLOB]++;\n+\t\ttruncate_pack(offset);\n+\n+\t} else {\n+\t\te->depth = 0;\n+\t\te->type = OBJ_BLOB;\n+\t\te->pack_id = pack_id;\n+\t\te->offset = offset;\n+\t\tobject_count++;\n+\t\tobject_count_by_type[OBJ_BLOB]++;\n+\t}\n+\n+\tfree(in_buf);\n+\tfree(out_buf);\n+}\n+\n /* All calls must be guarded by find_object() or find_mark() to\n * ensure the 'struct object_entry' passed was written by this\n * process instance. We unpack the entry by the offset, avoiding\n@@ -1704,7 +1820,7 @@ static void parse_mark(void)\n \t\tnext_mark = 0;\n }\n \n-static void parse_data(struct strbuf *sb)\n+static int parse_data(struct strbuf *sb, uintmax_t limit, uintmax_t *len_res)\n {\n \tstrbuf_reset(sb);\n \n@@ -1728,9 +1844,15 @@ static void parse_data(struct strbuf *sb)\n \t\tfree(term);\n \t}\n \telse {\n-\t\tsize_t n = 0, length;\n+\t\tuintmax_t len = strtoumax(command_buf.buf + 5, NULL, 10);\n+\t\tsize_t n = 0, length = (size_t)len;\n \n-\t\tlength = strtoul(command_buf.buf + 5, NULL, 10);\n+\t\tif (limit && limit < len) {\n+\t\t\t*len_res = len;\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (length < len)\n+\t\t\tdie(\"data is too large to use in this context\");\n \n \t\twhile (n < length) {\n \t\t\tsize_t s = strbuf_fread(sb, length - n, stdin);\n@@ -1742,6 +1864,7 @@ static void parse_data(struct strbuf *sb)\n \t}\n \n \tskip_optional_lf();\n+\treturn 1;\n }\n \n static int validate_raw_date(const char *src, char *result, int maxlen)\n@@ -1806,14 +1929,31 @@ static char *parse_ident(const char *buf)\n \treturn ident;\n }\n \n-static void parse_new_blob(void)\n+static void parse_and_store_blob(\n+\tstruct last_object *last,\n+\tunsigned char *sha1out,\n+\tuintmax_t mark)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tuintmax_t len;\n+\n+\tif (parse_data(&buf, big_file_threshold, &len))\n+\t\tstore_object(OBJ_BLOB, &buf, last, sha1out, mark);\n+\telse {\n+\t\tif (last) {\n+\t\t\tstrbuf_release(&last->data);\n+\t\t\tlast->offset = 0;\n+\t\t\tlast->depth = 0;\n+\t\t}\n+\t\tstream_blob(len, sha1out, mark);\n+\t}\n+}\n \n+static void parse_new_blob(void)\n+{\n \tread_next_command();\n \tparse_mark();\n-\tparse_data(&buf);\n-\tstore_object(OBJ_BLOB, &buf, &last_blob, NULL, next_mark);\n+\tparse_and_store_blob(&last_blob, NULL, next_mark);\n }\n \n static void unload_one_branch(void)\n@@ -1924,15 +2064,12 @@ static void file_change_m(struct branch *b)\n \t\t * another repository.\n \t\t */\n \t} else if (inline_data) {\n-\t\tstatic struct strbuf buf = STRBUF_INIT;\n-\n \t\tif (p != uq.buf) {\n \t\t\tstrbuf_addstr(&uq, p);\n \t\t\tp = uq.buf;\n \t\t}\n \t\tread_next_command();\n-\t\tparse_data(&buf);\n-\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t\tparse_and_store_blob(&last_blob, sha1, 0);\n \t} else if (oe) {\n \t\tif (oe->type != OBJ_BLOB)\n \t\t\tdie(\"Not a blob (actually a %s): %s\",\n@@ -2058,15 +2195,12 @@ static void note_change_n(struct branch *b)\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tstatic struct strbuf buf = STRBUF_INIT;\n-\n \t\tif (p != uq.buf) {\n \t\t\tstrbuf_addstr(&uq, p);\n \t\t\tp = uq.buf;\n \t\t}\n \t\tread_next_command();\n-\t\tparse_data(&buf);\n-\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t\tparse_and_store_blob(&last_blob, sha1, 0);\n \t} else if (oe) {\n \t\tif (oe->type != OBJ_BLOB)\n \t\t\tdie(\"Not a blob (actually a %s): %s\",\n@@ -2232,7 +2366,7 @@ static void parse_new_commit(void)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n-\tparse_data(&msg);\n+\tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n \tmerge_list = parse_merge(&merge_count);\n@@ -2353,7 +2487,7 @@ static void parse_new_tag(void)\n \t\ttagger = NULL;\n \n \t/* tag payload/message */\n-\tparse_data(&msg);\n+\tparse_data(&msg, 0, NULL);\n \n \t/* build the tag object */\n \tstrbuf_reset(&new_data);\n@@ -2473,6 +2607,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tpack_compression_seen = 1;\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"core.bigfilethreshold\")) {\n+\t\tlong n = git_config_int(k, v);\n+\t\tbig_file_threshold = 0 < n ? n : 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -2518,6 +2656,8 @@ int main(int argc, const char **argv)\n \t\t}\n \t\telse if (!prefixcmp(a, \"--max-pack-size=\"))\n \t\t\tmax_packsize = strtoumax(a + 16, NULL, 0) * 1024 * 1024;\n+\t\telse if (!prefixcmp(a, \"--big-file-threshold=\"))\n+\t\t\tbig_file_threshold = strtoumax(a + 21, NULL, 0) * 1024 * 1024;\n \t\telse if (!prefixcmp(a, \"--depth=\")) {\n \t\t\tmax_depth = strtoul(a + 8, NULL, 0);\n \t\t\tif (max_depth > MAX_DEPTH)\ndiff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh\nindex 9f52154..adfaae8 100755\n--- a/t/t5705-clone-2gb.sh\n+++ b/t/t5705-clone-2gb.sh\n@@ -31,7 +31,7 @@ test_expect_success 'setup' '\n \t echo \"data 5\" &&\n \t echo \">2gb\" &&\n \t cat commit) |\n-\tgit fast-import &&\n+\tgit fast-import --big-file-threshold=2 &&\n \ttest ! -f exit-status\n \n '\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex b49815d..513db86 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1254,4 +1254,50 @@ test_expect_success \\\n \t'Q: verify note for third commit' \\\n \t'git cat-file blob refs/notes/foobar:$commit3 >actual && test_cmp expect actual'\n \n+##\n+## R: very large blobs\n+##\n+blobsize=$((2*1024*1024 + 53))\n+test-genrandom bar $blobsize >expect\n+cat >input < $GIT_COMMITTER_DATE\n+data <>input\n+cat >>input <>input\n+echo >>input\n+\n+test_expect_success \\\n+\t'R: blob bigger than threshold' \\\n+\t'test_create_repo R &&\n+\t git --git-dir=R/.git fast-import --big-file-threshold=1 verify &&\n+\t for p in R/.git/objects/pack/*.pack;\n+\t do\n+\t git verify-pack -v $p >>verify || exit;\n+\t done'\n+test_expect_success \\\n+\t'R: verify written objects' \\\n+\t'git --git-dir=R/.git cat-file blob big-file:big1 >actual &&\n+\t test_cmp expect actual &&\n+\t a=$(git --git-dir=R/.git rev-parse big-file:big1) &&\n+\t b=$(git --git-dir=R/.git rev-parse big-file:big2) &&\n+\t test $a = $b'\n+test_expect_success \\\n+\t'R: blob appears only once' \\\n+\t'n=$(grep $a verify | wc -l) &&\n+\t test 1 = $n'\n+\n test_done\n-- \n1.7.0.rc0.170.g7207c\n\n-- \nShawn.\n"},{"id":"132933","messageId":"alpine.LFD.2.00.1001282125410.1681@xanadu.home","threadId":"22431","inReplyTo":"20100129012350.GD20488@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-01-29T02:33:23Z","receivedAt":"2010-01-29T02:33:23Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 28 Jan 2010, Shawn O. Pearce wrote:\n\n> If a blob is larger than the configured big-file-threshold, instead\n> of reading it into a single buffer obtained from malloc, stream it\n> onto the end of the current pack file. Streaming the larger objects\n> into the pack avoids the 4+ GiB memory footprint that occurs when\n> fast-import is processing 2+ GiB blobs.\n\nYeah. I've had that item on my todo list for ages now. This \nbig-file-threshold principle has to be applied to 'git add' too so a big \nblob is stored in pack file form right away, and used to bypass delta \nsearching in 'git pack-objects', used to skip the diff machinery, and so \non.\n\n\nNicolas\n"},{"id":"132935","messageId":"20100129023701.GE20488@spearce.org","threadId":"22431","inReplyTo":"alpine.LFD.2.00.1001282125410.1681@xanadu.home","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T02:37:01Z","receivedAt":"2010-01-29T02:37:01Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> On Thu, 28 Jan 2010, Shawn O. Pearce wrote:\n> \n> > If a blob is larger than the configured big-file-threshold, instead\n> > of reading it into a single buffer obtained from malloc, stream it\n> > onto the end of the current pack file. Streaming the larger objects\n> > into the pack avoids the 4+ GiB memory footprint that occurs when\n> > fast-import is processing 2+ GiB blobs.\n> \n> Yeah. I've had that item on my todo list for ages now. This \n> big-file-threshold principle has to be applied to 'git add' too so a big \n> blob is stored in pack file form right away, and used to bypass delta \n> searching in 'git pack-objects', used to skip the diff machinery, and so \n> on.\n\nYea, there are a lot of places we should improve for bigger files.\ngfi just happened to be the first one I got a bug report on from\na user...\n\n-- \nShawn.\n"},{"id":"132939","messageId":"7vockdjx6w.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"20100129012350.GD20488@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-29T05:29:27Z","receivedAt":"2010-01-29T05:29:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> +static void stream_blob(\n> +\tuintmax_t len,\n> +\tunsigned char *sha1out,\n> +\tuintmax_t mark)\n\nA funny way to indent and line wrap...\n\n> +{\n> + ...\n> +\t/* Determine if we should auto-checkpoint. */\n> +\tif ((pack_size + 60 + len) > max_packsize\n> +\t\t|| (pack_size + 60 + len) < pack_size)\n> +\t\tcycle_packfile();\n\nWhat's \"60\" in this math?\n\nIf the data is not compressible, we could even grow and the end result\nmight be more than (pack_size + len), busting max_packsize. As we are\nstreaming out, we cannot say \"oops, let me try again after truncating and\nclosing the current file and then opening a new file\", and instead may\nhave to copy the data from the current one to a new one, and truncate the\ncurrent one. Is this something worth worrying about?\n"},{"id":"132966","messageId":"20100129152254.GC21821@spearce.org","threadId":"22431","inReplyTo":"7vockdjx6w.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T15:22:54Z","receivedAt":"2010-01-29T15:22:54Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > +static void stream_blob(\n> > +\tuintmax_t len,\n> > +\tunsigned char *sha1out,\n> > +\tuintmax_t mark)\n> \n> A funny way to indent and line wrap...\n\nConsistent with something else in the same file that has a similar\ntask... store_object(). I literally just copied the lines from\nthere and pasted here.\n \n> > +\t/* Determine if we should auto-checkpoint. */\n> > +\tif ((pack_size + 60 + len) > max_packsize\n> > +\t\t|| (pack_size + 60 + len) < pack_size)\n> > +\t\tcycle_packfile();\n> \n> What's \"60\" in this math?\n\nIIRC 60 is 20 bytes for the SHA-1 footer, and another 40 padding\nfor the object header, and the deflate() wrapping.\n\nAgain, the 60 was stolen from the existing store_object(), which\nalready has the same assumption. Only there I think we have len as\nthe fully compressed version, so the deflate() wrapping is already\nbeing accounted for.\n \n> If the data is not compressible, we could even grow and the end result\n> might be more than (pack_size + len), busting max_packsize.\n\nYea, that's a good point. I probably should run the length through\ndeflateBound() and use that here in the test. But I didn't think it\nwould make a huge difference. If the file isn't compressible what\nis the real increment over the uncompressed size? There's a header\nand footer, and instructions in the stream to indicate literal data,\nbut its not like its doubling in size.\n\n> As we are\n> streaming out, we cannot say \"oops, let me try again after truncating and\n> closing the current file and then opening a new file\", and instead may\n> have to copy the data from the current one to a new one, and truncate the\n> current one. Is this something worth worrying about?\n\nHmm. I'm not that worried about it, but then there's the case of\na blob that is larger than the max pack size. We can't store that,\nperiod. Should we try to exceed max pack size for that one object?\nOr should we refuse?\n\nThe major reason for this test is to ensure an object offset\nstarts before the 4 GiB boundary so idx v1 can hold the offset.\nA much less important reason is to try to support older 32 bit\nfilesystems which can't do more than 2 GiB per file.\n\n-- \nShawn.\n"},{"id":"132968","messageId":"20100129163838.GD21821@spearce.org","threadId":"22431","inReplyTo":"20100129152254.GC21821@spearce.org","subject":"[PATCH v2] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T16:38:38Z","receivedAt":"2010-01-29T16:38:38Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If a blob is larger than the configured big-file-threshold, instead\nof reading it into a single buffer obtained from malloc, stream it\nonto the end of the current pack file. Streaming the larger objects\ninto the pack avoids the 4+ GiB memory footprint that occurs when\nfast-import is processing 2+ GiB blobs.\n\nSigned-off-by: Shawn O. Pearce \n---\n\n Fixed a missing skip_optional_lf() at the end of the large blob\n command case.\n\n Documentation/git-fast-import.txt | 6 ++\n fast-import.c | 172 +++++++++++++++++++++++++++++++++----\n t/t5705-clone-2gb.sh | 2 +-\n t/t9300-fast-import.sh | 46 ++++++++++\n 4 files changed, 208 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex e6d364f..16d3596 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -50,6 +50,12 @@ OPTIONS\n \timporters may wish to lower this, such as to ensure the\n \tresulting packfiles fit on CDs.\n \n+--big-file-threshold=::\n+\tMaximum size of a blob that fast-import will attempt to\n+\tcreate a delta for, expressed in MiB. The default is 512.\n+\tSome importers may wish to lower this on systems with\n+\tconstrained memory.\n+\n --depth=::\n \tMaximum delta depth, for blob and tree deltification.\n \tDefault is 10.\ndiff --git a/fast-import.c b/fast-import.c\nindex 60d0aa2..8055f73 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -280,6 +280,7 @@ struct recent_command\n /* Configured limits on output */\n static unsigned long max_depth = 10;\n static off_t max_packsize = (1LL << 32) - 1;\n+static uintmax_t big_file_threshold = 512 * 1024 * 1024;\n static int force_update;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n@@ -1003,7 +1004,7 @@ static void cycle_packfile(void)\n \n static size_t encode_header(\n \tenum object_type type,\n-\tsize_t size,\n+\tuintmax_t size,\n \tunsigned char *hdr)\n {\n \tint n = 1;\n@@ -1159,6 +1160,118 @@ static int store_object(\n \treturn 0;\n }\n \n+static void truncate_pack(off_t to)\n+{\n+\tif (ftruncate(pack_data->pack_fd, to)\n+\t || lseek(pack_data->pack_fd, to, SEEK_SET) != to)\n+\t\tdie_errno(\"cannot truncate pack to skip duplicate\");\n+\tpack_size = to;\n+}\n+\n+static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n+{\n+\tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n+\tunsigned char *in_buf = xmalloc(in_sz);\n+\tunsigned char *out_buf = xmalloc(out_sz);\n+\tstruct object_entry *e;\n+\tunsigned char sha1[20];\n+\tunsigned long hdrlen;\n+\toff_t offset;\n+\tgit_SHA_CTX c;\n+\tz_stream s;\n+\tint status = Z_OK;\n+\n+\t/* Determine if we should auto-checkpoint. */\n+\tif ((pack_size + 60 + len) > max_packsize\n+\t\t|| (pack_size + 60 + len) < pack_size)\n+\t\tcycle_packfile();\n+\n+\toffset = pack_size;\n+\n+\thdrlen = snprintf((char *)out_buf, out_sz, \"blob %\" PRIuMAX, len) + 1;\n+\tif (out_sz <= hdrlen)\n+\t\tdie(\"impossibly large object header\");\n+\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, out_buf, hdrlen);\n+\n+\tmemset(&s, 0, sizeof(s));\n+\tdeflateInit(&s, pack_compression_level);\n+\n+\thdrlen = encode_header(OBJ_BLOB, len, out_buf);\n+\tif (out_sz <= hdrlen)\n+\t\tdie(\"impossibly large object header\");\n+\n+\ts.next_out = out_buf + hdrlen;\n+\ts.avail_out = out_sz - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tif (0 < len && !s.avail_in) {\n+\t\t\tsize_t cnt = in_sz < len ? in_sz : (size_t)len;\n+\t\t\tsize_t n = fread(in_buf, 1, cnt, stdin);\n+\t\t\tif (!n && feof(stdin))\n+\t\t\t\tdie(\"EOF in data (%\" PRIuMAX \" bytes remaining)\", len);\n+\n+\t\t\tgit_SHA1_Update(&c, in_buf, n);\n+\t\t\ts.next_in = in_buf;\n+\t\t\ts.avail_in = n;\n+\t\t\tlen -= n;\n+\t\t}\n+\n+\t\tstatus = deflate(&s, len ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tsize_t n = s.next_out - out_buf;\n+\t\t\twrite_or_die(pack_data->pack_fd, out_buf, n);\n+\t\t\tpack_size += n;\n+\t\t\ts.next_out = out_buf;\n+\t\t\ts.avail_out = out_sz;\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tdeflateEnd(&s);\n+\tgit_SHA1_Final(sha1, &c);\n+\n+\tif (sha1out)\n+\t\thashcpy(sha1out, sha1);\n+\n+\te = insert_object(sha1);\n+\n+\tif (mark)\n+\t\tinsert_mark(mark, e);\n+\n+\tif (e->offset) {\n+\t\tduplicate_count_by_type[OBJ_BLOB]++;\n+\t\ttruncate_pack(offset);\n+\n+\t} else if (find_sha1_pack(sha1, packed_git)) {\n+\t\te->type = OBJ_BLOB;\n+\t\te->pack_id = MAX_PACK_ID;\n+\t\te->offset = 1; /* just not zero! */\n+\t\tduplicate_count_by_type[OBJ_BLOB]++;\n+\t\ttruncate_pack(offset);\n+\n+\t} else {\n+\t\te->depth = 0;\n+\t\te->type = OBJ_BLOB;\n+\t\te->pack_id = pack_id;\n+\t\te->offset = offset;\n+\t\tobject_count++;\n+\t\tobject_count_by_type[OBJ_BLOB]++;\n+\t}\n+\n+\tfree(in_buf);\n+\tfree(out_buf);\n+}\n+\n /* All calls must be guarded by find_object() or find_mark() to\n * ensure the 'struct object_entry' passed was written by this\n * process instance. We unpack the entry by the offset, avoiding\n@@ -1704,7 +1817,7 @@ static void parse_mark(void)\n \t\tnext_mark = 0;\n }\n \n-static void parse_data(struct strbuf *sb)\n+static int parse_data(struct strbuf *sb, uintmax_t limit, uintmax_t *len_res)\n {\n \tstrbuf_reset(sb);\n \n@@ -1728,9 +1841,15 @@ static void parse_data(struct strbuf *sb)\n \t\tfree(term);\n \t}\n \telse {\n-\t\tsize_t n = 0, length;\n+\t\tuintmax_t len = strtoumax(command_buf.buf + 5, NULL, 10);\n+\t\tsize_t n = 0, length = (size_t)len;\n \n-\t\tlength = strtoul(command_buf.buf + 5, NULL, 10);\n+\t\tif (limit && limit < len) {\n+\t\t\t*len_res = len;\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (length < len)\n+\t\t\tdie(\"data is too large to use in this context\");\n \n \t\twhile (n < length) {\n \t\t\tsize_t s = strbuf_fread(sb, length - n, stdin);\n@@ -1742,6 +1861,7 @@ static void parse_data(struct strbuf *sb)\n \t}\n \n \tskip_optional_lf();\n+\treturn 1;\n }\n \n static int validate_raw_date(const char *src, char *result, int maxlen)\n@@ -1806,14 +1926,32 @@ static char *parse_ident(const char *buf)\n \treturn ident;\n }\n \n-static void parse_new_blob(void)\n+static void parse_and_store_blob(\n+\tstruct last_object *last,\n+\tunsigned char *sha1out,\n+\tuintmax_t mark)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tuintmax_t len;\n \n+\tif (parse_data(&buf, big_file_threshold, &len))\n+\t\tstore_object(OBJ_BLOB, &buf, last, sha1out, mark);\n+\telse {\n+\t\tif (last) {\n+\t\t\tstrbuf_release(&last->data);\n+\t\t\tlast->offset = 0;\n+\t\t\tlast->depth = 0;\n+\t\t}\n+\t\tstream_blob(len, sha1out, mark);\n+\t\tskip_optional_lf();\n+\t}\n+}\n+\n+static void parse_new_blob(void)\n+{\n \tread_next_command();\n \tparse_mark();\n-\tparse_data(&buf);\n-\tstore_object(OBJ_BLOB, &buf, &last_blob, NULL, next_mark);\n+\tparse_and_store_blob(&last_blob, NULL, next_mark);\n }\n \n static void unload_one_branch(void)\n@@ -1924,15 +2062,12 @@ static void file_change_m(struct branch *b)\n \t\t * another repository.\n \t\t */\n \t} else if (inline_data) {\n-\t\tstatic struct strbuf buf = STRBUF_INIT;\n-\n \t\tif (p != uq.buf) {\n \t\t\tstrbuf_addstr(&uq, p);\n \t\t\tp = uq.buf;\n \t\t}\n \t\tread_next_command();\n-\t\tparse_data(&buf);\n-\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t\tparse_and_store_blob(&last_blob, sha1, 0);\n \t} else if (oe) {\n \t\tif (oe->type != OBJ_BLOB)\n \t\t\tdie(\"Not a blob (actually a %s): %s\",\n@@ -2058,15 +2193,12 @@ static void note_change_n(struct branch *b)\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tstatic struct strbuf buf = STRBUF_INIT;\n-\n \t\tif (p != uq.buf) {\n \t\t\tstrbuf_addstr(&uq, p);\n \t\t\tp = uq.buf;\n \t\t}\n \t\tread_next_command();\n-\t\tparse_data(&buf);\n-\t\tstore_object(OBJ_BLOB, &buf, &last_blob, sha1, 0);\n+\t\tparse_and_store_blob(&last_blob, sha1, 0);\n \t} else if (oe) {\n \t\tif (oe->type != OBJ_BLOB)\n \t\t\tdie(\"Not a blob (actually a %s): %s\",\n@@ -2232,7 +2364,7 @@ static void parse_new_commit(void)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n-\tparse_data(&msg);\n+\tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n \tmerge_list = parse_merge(&merge_count);\n@@ -2353,7 +2485,7 @@ static void parse_new_tag(void)\n \t\ttagger = NULL;\n \n \t/* tag payload/message */\n-\tparse_data(&msg);\n+\tparse_data(&msg, 0, NULL);\n \n \t/* build the tag object */\n \tstrbuf_reset(&new_data);\n@@ -2473,6 +2605,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tpack_compression_seen = 1;\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"core.bigfilethreshold\")) {\n+\t\tlong n = git_config_int(k, v);\n+\t\tbig_file_threshold = 0 < n ? n : 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -2518,6 +2654,8 @@ int main(int argc, const char **argv)\n \t\t}\n \t\telse if (!prefixcmp(a, \"--max-pack-size=\"))\n \t\t\tmax_packsize = strtoumax(a + 16, NULL, 0) * 1024 * 1024;\n+\t\telse if (!prefixcmp(a, \"--big-file-threshold=\"))\n+\t\t\tbig_file_threshold = strtoumax(a + 21, NULL, 0) * 1024 * 1024;\n \t\telse if (!prefixcmp(a, \"--depth=\")) {\n \t\t\tmax_depth = strtoul(a + 8, NULL, 0);\n \t\t\tif (max_depth > MAX_DEPTH)\ndiff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh\nindex 9f52154..adfaae8 100755\n--- a/t/t5705-clone-2gb.sh\n+++ b/t/t5705-clone-2gb.sh\n@@ -31,7 +31,7 @@ test_expect_success 'setup' '\n \t echo \"data 5\" &&\n \t echo \">2gb\" &&\n \t cat commit) |\n-\tgit fast-import &&\n+\tgit fast-import --big-file-threshold=2 &&\n \ttest ! -f exit-status\n \n '\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex b49815d..513db86 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1254,4 +1254,50 @@ test_expect_success \\\n \t'Q: verify note for third commit' \\\n \t'git cat-file blob refs/notes/foobar:$commit3 >actual && test_cmp expect actual'\n \n+##\n+## R: very large blobs\n+##\n+blobsize=$((2*1024*1024 + 53))\n+test-genrandom bar $blobsize >expect\n+cat >input < $GIT_COMMITTER_DATE\n+data <>input\n+cat >>input <>input\n+echo >>input\n+\n+test_expect_success \\\n+\t'R: blob bigger than threshold' \\\n+\t'test_create_repo R &&\n+\t git --git-dir=R/.git fast-import --big-file-threshold=1 verify &&\n+\t for p in R/.git/objects/pack/*.pack;\n+\t do\n+\t git verify-pack -v $p >>verify || exit;\n+\t done'\n+test_expect_success \\\n+\t'R: verify written objects' \\\n+\t'git --git-dir=R/.git cat-file blob big-file:big1 >actual &&\n+\t test_cmp expect actual &&\n+\t a=$(git --git-dir=R/.git rev-parse big-file:big1) &&\n+\t b=$(git --git-dir=R/.git rev-parse big-file:big2) &&\n+\t test $a = $b'\n+test_expect_success \\\n+\t'R: blob appears only once' \\\n+\t'n=$(grep $a verify | wc -l) &&\n+\t test 1 = $n'\n+\n test_done\n-- \n1.7.0.rc0.170.g7207c\n\n-- \nShawn.\n"},{"id":"132972","messageId":"m3aavwbwaz.fsf@localhost.localdomain","threadId":"22431","inReplyTo":"20100129163838.GD21821@spearce.org","subject":"Re: [PATCH v2] fast-import: Stream very large blobs directly to pack","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-29T18:29:25Z","receivedAt":"2010-01-29T18:29:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Documentation/git-fast-import.txt | 6 ++\n> fast-import.c | 172 +++++++++++++++++++++++++++++++++----\n> t/t5705-clone-2gb.sh | 2 +-\n> t/t9300-fast-import.sh | 46 ++++++++++\n> 4 files changed, 208 insertions(+), 18 deletions(-)\n\n> +--big-file-threshold=::\n> +\tMaximum size of a blob that fast-import will attempt to\n> +\tcreate a delta for, expressed in MiB. The default is 512.\n> +\tSome importers may wish to lower this on systems with\n> +\tconstrained memory.\n> +\n\nShouldn't there be config variable corresponding to this command line\noption, so that it can be set once for [constrained] system?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"132973","messageId":"20100129183024.GA22101@spearce.org","threadId":"22431","inReplyTo":"m3aavwbwaz.fsf@localhost.localdomain","subject":"Re: [PATCH v2] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T18:30:24Z","receivedAt":"2010-01-29T18:30:24Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > Documentation/git-fast-import.txt | 6 ++\n> > fast-import.c | 172 +++++++++++++++++++++++++++++++++----\n> > t/t5705-clone-2gb.sh | 2 +-\n> > t/t9300-fast-import.sh | 46 ++++++++++\n> > 4 files changed, 208 insertions(+), 18 deletions(-)\n> \n> > +--big-file-threshold=::\n> > +\tMaximum size of a blob that fast-import will attempt to\n> > +\tcreate a delta for, expressed in MiB. The default is 512.\n> > +\tSome importers may wish to lower this on systems with\n> > +\tconstrained memory.\n> > +\n> \n> Shouldn't there be config variable corresponding to this command line\n> option, so that it can be set once for [constrained] system?\n\nImplemented as core.bigFileThreshold in this patch... but I didn't\ndocument it...\n\n-- \nShawn.\n"},{"id":"132974","messageId":"fabb9a1e1001291035l5cd09cb6q1f12037f96dce8a1@mail.gmail.com","threadId":"22431","inReplyTo":"20100129012350.GD20488@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-01-29T18:35:31Z","receivedAt":"2010-01-29T18:35:31Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Fri, Jan 29, 2010 at 02:23, Shawn O. Pearce wrote:\n> index 60d0aa2..8114f93 100644\n\nLooks like you based it off an old version of fast-import, at least\nnot on what is in master atm, since that has my fast-import series.\nRebasing shouldn't be too hard, you'll just have to move the option\nparsing to the appropriate function.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"132975","messageId":"20100129183705.GB22101@spearce.org","threadId":"22431","inReplyTo":"fabb9a1e1001291035l5cd09cb6q1f12037f96dce8a1@mail.gmail.com","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T18:37:05Z","receivedAt":"2010-01-29T18:37:05Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sverre Rabbelier wrote:\n> On Fri, Jan 29, 2010 at 02:23, Shawn O. Pearce wrote:\n> > index 60d0aa2..8114f93 100644\n> \n> Looks like you based it off an old version of fast-import, at least\n> not on what is in master atm, since that has my fast-import series.\n> Rebasing shouldn't be too hard, you'll just have to move the option\n> parsing to the appropriate function.\n\nI was intentionally slating this for maint, to fix a bug a user\nreported when handling large streams.\n\nBut yea... I guess I also owe Junio a rebased form for master so\nhe has less merge conflicts to resolve.\n\n-- \nShawn.\n"},{"id":"132976","messageId":"fabb9a1e1001291041w48776835m9b291923fbb859c3@mail.gmail.com","threadId":"22431","inReplyTo":"20100129183705.GB22101@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-01-29T18:41:10Z","receivedAt":"2010-01-29T18:41:10Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Fri, Jan 29, 2010 at 19:37, Shawn O. Pearce wrote:\n> I was intentionally slating this for maint, to fix a bug a user\n> reported when handling large streams.\n\nAh, sorry, did not notice that.\n\nSpeaking of which, how does one figure out what commit a patch\nis/could be based on? Couldn't we include that in the part behind the\ntriple dash?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"132977","messageId":"20100129184403.GC22101@spearce.org","threadId":"22431","inReplyTo":"fabb9a1e1001291041w48776835m9b291923fbb859c3@mail.gmail.com","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-29T18:44:03Z","receivedAt":"2010-01-29T18:44:03Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sverre Rabbelier wrote:\n> On Fri, Jan 29, 2010 at 19:37, Shawn O. Pearce wrote:\n> > I was intentionally slating this for maint, to fix a bug a user\n> > reported when handling large streams.\n> \n> Ah, sorry, did not notice that.\n> \n> Speaking of which, how does one figure out what commit a patch\n> is/could be based on? Couldn't we include that in the part behind the\n> triple dash?\n\nYou have to tell the other guy in your cover letter section. You can\ntry to guess if the index information is accurate by grepping\naround a log output looking for that blob sha-1 and then finding\nwhat branch that's in. Its not pretty.\n\nIIRC other DVCS systems include the ancestor identity in their\npatch output. Git doesn't bother, because in a multi-patch series\nthe 2nd (and subsequent) patches would have useless id information.\n\n-- \nShawn.\n"},{"id":"133035","messageId":"4B63691F.4000507@gmail.com","threadId":"22431","inReplyTo":"20100129183024.GA22101@spearce.org","subject":"Re: [PATCH v2] fast-import: Stream very large blobs directly to pack","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-01-29T23:02:55Z","receivedAt":"2010-01-29T23:02:55Z","isPatch":true,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n...\n> Implemented as core.bigFileThreshold in this patch... but I didn't\n> document it...\n\nBad dog! No biscuit!\n"},{"id":"133075","messageId":"7vockc45ut.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"20100129183705.GB22101@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-30T03:41:14Z","receivedAt":"2010-01-30T03:41:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Sverre Rabbelier wrote:\n>> On Fri, Jan 29, 2010 at 02:23, Shawn O. Pearce wrote:\n>> > index 60d0aa2..8114f93 100644\n>> \n>> Looks like you based it off an old version of fast-import, at least\n>> not on what is in master atm, since that has my fast-import series.\n>> Rebasing shouldn't be too hard, you'll just have to move the option\n>> parsing to the appropriate function.\n>\n> I was intentionally slating this for maint, to fix a bug a user\n> reported when handling large streams.\n\nI personally see that as adding a new feature (especially with new option\nand config).\n\n> But yea... I guess I also owe Junio a rebased form for master so\n> he has less merge conflicts to resolve.\n\nThanks.\n"},{"id":"133092","messageId":"7vmxzw15dt.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"7vockc45ut.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-30T06:19:42Z","receivedAt":"2010-01-30T06:19:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Shawn O. Pearce\" writes:\n>\n>> I was intentionally slating this for maint, to fix a bug a user\n>> reported when handling large streams.\n>\n> I personally see that as adding a new feature (especially with new option\n> and config).\n\nSorry, but I take it back. The new codepath triggers even without any\nexplicit request and _fixes_ the situation where old code simply failed,\nso it is worth queuing for the maintenance track.\n\nDo you want to do the deflatebound thing, or are we Ok without?\n"},{"id":"133096","messageId":"7vfx5oysbc.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"4B63691F.4000507@gmail.com","subject":"Re: [PATCH v2] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-30T07:17:59Z","receivedAt":"2010-01-30T07:17:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM writes:\n\n> Shawn O. Pearce wrote:\n> ...\n>> Implemented as core.bigFileThreshold in this patch... but I didn't\n>> document it...\n>\n> Bad dog! No biscuit!\n\n:-)\n\nPatches welcome.\n"},{"id":"133098","messageId":"7v3a1oyrkp.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"7vmxzw15dt.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-30T07:33:58Z","receivedAt":"2010-01-30T07:33:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I'll queued this with a result of my conflict resolution to 'pu' for now\nbut please double check after I push it out.\n\nYou may want to add the new option to the output from \"cmd -h\" and\nprobably description of the configuration in the doc before any of this\ngets official.\n"},{"id":"133253","messageId":"20100201152324.GD8916@spearce.org","threadId":"22431","inReplyTo":"7vmxzw15dt.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-01T15:23:24Z","receivedAt":"2010-02-01T15:23:24Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> Junio C Hamano writes:\n> > \"Shawn O. Pearce\" writes:\n> >\n> >> I was intentionally slating this for maint, to fix a bug a user\n> >> reported when handling large streams.\n> >\n> > I personally see that as adding a new feature (especially with new option\n> > and config).\n> \n> Sorry, but I take it back. The new codepath triggers even without any\n> explicit request and _fixes_ the situation where old code simply failed,\n> so it is worth queuing for the maintenance track.\n\nThanks.\n \n> Do you want to do the deflatebound thing, or are we Ok without?\n\nNo, I think we're OK without it. The pack size limit stuff is only\na rough guess anyway. I never meant for us to be sticking to it to\nwithin a byte. It was only meant to keep us from shoving a 100 MiB\nblob onto the end of a pack once we got close to the size requested\nby the user.\n\n-- \nShawn.\n"},{"id":"133254","messageId":"20100201152826.GE8916@spearce.org","threadId":"22431","inReplyTo":"7v3a1oyrkp.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-01T15:28:26Z","receivedAt":"2010-02-01T15:28:26Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> I'll queued this with a result of my conflict resolution to 'pu' for now\n> but please double check after I push it out.\n\nThe strtoumax call got messed up. Squash this into your merge:\n\ndiff --git a/fast-import.c b/fast-import.c\nindex e6ebcf6..9c65a24 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)\n \tif (!prefixcmp(option, \"max-pack-size=\")) {\n \t\toption_max_pack_size(option + 14);\n \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n-\t\tbig_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;\n+\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n \t} else if (!prefixcmp(option, \"depth=\")) {\n \t\toption_depth(option + 6);\n \t} else if (!prefixcmp(option, \"active-branches=\")) {\n \n> You may want to add the new option to the output from \"cmd -h\" and\n> probably description of the configuration in the doc before any of this\n> gets official.\n\nI'll send an additional patch in a minute with these documentation\nrelated updates.\n\n-- \nShawn.\n"},{"id":"133255","messageId":"20100201154133.GF8916@spearce.org","threadId":"22431","inReplyTo":"7v3a1oyrkp.fsf@alter.siamese.dyndns.org","subject":"[PATCH] fast-import: Document the core.bigFileThreshold configuration setting","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-01T15:41:33Z","receivedAt":"2010-02-01T15:41:33Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Signed-off-by: Shawn O. Pearce \n---\n Junio C Hamano wrote:\n > You may want to add the new option to the output from \"cmd -h\" and\n > probably description of the configuration in the doc before any of this\n > gets official.\n\n Doc patch. :-)\n\n Feel free to squash, or to apply as its own change, I don't care\n either way.\n\n Documentation/config.txt | 14 ++++++++++++++\n fast-import.c | 2 +-\n 2 files changed, 15 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex f7728ec..b16a20b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -409,6 +409,20 @@ You probably do not need to adjust this value.\n +\n Common unit suffixes of 'k', 'm', or 'g' are supported.\n \n+core.bigFileThreshold::\n+\tFiles larger than this size are stored deflated, without\n+\tattempting delta compression. Storing large files without\n+\tdelta compression avoids excessive memory usage, at the\n+\tslight expense of increased disk usage.\n++\n+Default is 512 MiB on all platforms. This should be reasonable\n+for most projects as source code and other text files can still\n+be delta compressed, but larger binary media files won't be.\n++\n+Common unit suffixes of 'k', 'm', or 'g' are supported.\n++\n+Currently only linkgit:git-fast-import[1] honors this setting.\n+\n core.excludesfile::\n \tIn addition to '.gitignore' (per-directory) and\n \t'.git/info/exclude', git looks into this file for patterns\ndiff --git a/fast-import.c b/fast-import.c\nindex 8055f73..1bcf3bb 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2613,7 +2613,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n }\n \n static const char fast_import_usage[] =\n-\"git fast-import [--date-format=f] [--max-pack-size=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]\";\n+\"git fast-import [--date-format=f] [--max-pack-size=n] [--big-file-threshold=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]\";\n \n int main(int argc, const char **argv)\n {\n-- \n1.7.0.rc0.170.g7207c\n\n-- \nShawn.\n"},{"id":"133283","messageId":"7v8wbcyarb.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"20100201152826.GE8916@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-01T20:14:00Z","receivedAt":"2010-02-01T20:14:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Junio C Hamano wrote:\n>> I'll queued this with a result of my conflict resolution to 'pu' for now\n>> but please double check after I push it out.\n>\n> The strtoumax call got messed up. Squash this into your merge:\n>\n> diff --git a/fast-import.c b/fast-import.c\n> index e6ebcf6..9c65a24 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)\n> \tif (!prefixcmp(option, \"max-pack-size=\")) {\n> \t\toption_max_pack_size(option + 14);\n> \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n> -\t\tbig_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;\n> +\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n> \t} else if (!prefixcmp(option, \"depth=\")) {\n> \t\toption_depth(option + 6);\n> \t} else if (!prefixcmp(option, \"active-branches=\")) {\n\nAh, of course. Sorry for not being careful and thanks for pointing it\nout.\n"},{"id":"133567","messageId":"7v7hqtzrmq.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"20100201152826.GE8916@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-04T02:01:01Z","receivedAt":"2010-02-04T02:01:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> The strtoumax call got messed up. Squash this into your merge:\n>\n> diff --git a/fast-import.c b/fast-import.c\n> index e6ebcf6..9c65a24 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)\n> \tif (!prefixcmp(option, \"max-pack-size=\")) {\n> \t\toption_max_pack_size(option + 14);\n> \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n> -\t\tbig_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;\n> +\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n> \t} else if (!prefixcmp(option, \"depth=\")) {\n> \t\toption_depth(option + 6);\n> \t} else if (!prefixcmp(option, \"active-branches=\")) {\n> \n>> You may want to add the new option to the output from \"cmd -h\" and\n>> probably description of the configuration in the doc before any of this\n>> gets official.\n>\n> I'll send an additional patch in a minute with these documentation\n> related updates.\n\nWell, well, well....\n\nThe documentation says this is counted in bytes, but somehow neither of us\nfound the above \" * 1024 * 1024\" suspicious.\n\nShouldn't it be at least like this? It would probably be a good idea to\nuse git_parse_ulong() or somesuch while we are at it.\n\n fast-import.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex ca21082..ea1ac0f 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)\n \tif (!prefixcmp(option, \"max-pack-size=\")) {\n \t\toption_max_pack_size(option + 14);\n \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n-\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n+\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0);\n \t} else if (!prefixcmp(option, \"depth=\")) {\n \t\toption_depth(option + 6);\n \t} else if (!prefixcmp(option, \"active-branches=\")) {\n"},{"id":"133568","messageId":"20100204020756.GP14799@spearce.org","threadId":"22431","inReplyTo":"7v7hqtzrmq.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-04T02:07:56Z","receivedAt":"2010-02-04T02:07:56Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \n> Well, well, well....\n> \n> The documentation says this is counted in bytes, but somehow neither of us\n> found the above \" * 1024 * 1024\" suspicious.\n> \n> Shouldn't it be at least like this? It would probably be a good idea to\n> use git_parse_ulong() or somesuch while we are at it.\n> \n> fast-import.c | 2 +-\n> 1 files changed, 1 insertions(+), 1 deletions(-)\n> \n> diff --git a/fast-import.c b/fast-import.c\n> index ca21082..ea1ac0f 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)\n> \tif (!prefixcmp(option, \"max-pack-size=\")) {\n> \t\toption_max_pack_size(option + 14);\n> \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n> -\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n> +\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0);\n\nIn my v3 patch I thought I replaced this code with:\n\n+ else if (!prefixcmp(a, \"--big-file-threshold=\")) {\n+ unsigned long v;\n+ if (!git_parse_ulong(a + 21, &v))\n+ usage(fast_import_usage);\n+ big_file_threshold = v;\n\nSo we relied on git_parse_ulong to handle unit suffixes as well.\n\n-- \nShawn.\n"},{"id":"133570","messageId":"7vock5yby2.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"20100204020756.GP14799@spearce.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-04T02:25:09Z","receivedAt":"2010-02-04T02:25:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> In my v3 patch I thought I replaced this code with:\n>\n> + else if (!prefixcmp(a, \"--big-file-threshold=\")) {\n> + unsigned long v;\n> + if (!git_parse_ulong(a + 21, &v))\n> + usage(fast_import_usage);\n> + big_file_threshold = v;\n>\n> So we relied on git_parse_ulong to handle unit suffixes as well.\n\nYeah, you did; but it didn't carried through the merge process across the\ncode restructure to add \"option_blah\" stuff. Sorry about that.\n\nLooking at the output from\n\n $ git grep -n -e ' \\* 1024 \\* 1024' -- '*.c'\n\nI noticed another issue. Don't we need the same thing for max_packsize?\nOr is that _too much_ of a backward incompatible change that we should\nwait until 1.7.1?\n"},{"id":"133571","messageId":"7vk4utybur.fsf@alter.siamese.dyndns.org","threadId":"22431","inReplyTo":"7vock5yby2.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-04T02:27:08Z","receivedAt":"2010-02-04T02:27:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Shawn O. Pearce\" writes:\n>\n>> In my v3 patch I thought I replaced this code with:\n>>\n>> + else if (!prefixcmp(a, \"--big-file-threshold=\")) {\n>> + unsigned long v;\n>> + if (!git_parse_ulong(a + 21, &v))\n>> + usage(fast_import_usage);\n>> + big_file_threshold = v;\n>>\n>> So we relied on git_parse_ulong to handle unit suffixes as well.\n>\n> Yeah, you did; but it didn't carried through the merge process across the\n> code restructure to add \"option_blah\" stuff. Sorry about that.\n\nAnd this is a fix-up for the mismerge. I didn't touch max-pack-size stuff.\n\n fast-import.c | 5 ++++-\n 1 files changed, 4 insertions(+), 1 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex ca21082..a6730d0 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2800,7 +2800,10 @@ static int parse_one_option(const char *option)\n \tif (!prefixcmp(option, \"max-pack-size=\")) {\n \t\toption_max_pack_size(option + 14);\n \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n-\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n+\t\tunsigned long v;\n+\t\tif (!git_parse_ulong(option + 19, &v))\n+\t\t\treturn 0;\n+\t\tbig_file_threshold = v;\n \t} else if (!prefixcmp(option, \"depth=\")) {\n \t\toption_depth(option + 6);\n \t} else if (!prefixcmp(option, \"active-branches=\")) {\n"},{"id":"133572","messageId":"alpine.LFD.2.00.1002032126050.1681@xanadu.home","threadId":"22431","inReplyTo":"7v7hqtzrmq.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-04T02:28:15Z","receivedAt":"2010-02-04T02:28:15Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 3 Feb 2010, Junio C Hamano wrote:\n\n> \"Shawn O. Pearce\" writes:\n> \n> > The strtoumax call got messed up. Squash this into your merge:\n> >\n> > diff --git a/fast-import.c b/fast-import.c\n> > index e6ebcf6..9c65a24 100644\n> > --- a/fast-import.c\n> > +++ b/fast-import.c\n> > @@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)\n> > \tif (!prefixcmp(option, \"max-pack-size=\")) {\n> > \t\toption_max_pack_size(option + 14);\n> > \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n> > -\t\tbig_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;\n> > +\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n> > \t} else if (!prefixcmp(option, \"depth=\")) {\n> > \t\toption_depth(option + 6);\n> > \t} else if (!prefixcmp(option, \"active-branches=\")) {\n> > \n> >> You may want to add the new option to the output from \"cmd -h\" and\n> >> probably description of the configuration in the doc before any of this\n> >> gets official.\n> >\n> > I'll send an additional patch in a minute with these documentation\n> > related updates.\n> \n> Well, well, well....\n> \n> The documentation says this is counted in bytes, but somehow neither of us\n> found the above \" * 1024 * 1024\" suspicious.\n> \n> Shouldn't it be at least like this? It would probably be a good idea to\n> use git_parse_ulong() or somesuch while we are at it.\n\nYes, definitely. I'm about to post a patch moving --max-pack-size in \nthat direction too. I just had to fix a couple other unsuspected issues \nto get there though. Patches will follow shortly.\n\n\nNicolas\n"},{"id":"133573","messageId":"20100204023056.GQ14799@spearce.org","threadId":"22431","inReplyTo":"7vk4utybur.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] fast-import: Stream very large blobs directly to pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-04T02:30:56Z","receivedAt":"2010-02-04T02:30:56Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> And this is a fix-up for the mismerge. I didn't touch max-pack-size stuff.\n> \n> fast-import.c | 5 ++++-\n> 1 files changed, 4 insertions(+), 1 deletions(-)\n> \n> diff --git a/fast-import.c b/fast-import.c\n> index ca21082..a6730d0 100644\n> --- a/fast-import.c\n> +++ b/fast-import.c\n> @@ -2800,7 +2800,10 @@ static int parse_one_option(const char *option)\n> \tif (!prefixcmp(option, \"max-pack-size=\")) {\n> \t\toption_max_pack_size(option + 14);\n> \t} else if (!prefixcmp(option, \"big-file-threshold=\")) {\n> -\t\tbig_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;\n> +\t\tunsigned long v;\n> +\t\tif (!git_parse_ulong(option + 19, &v))\n> +\t\t\treturn 0;\n> +\t\tbig_file_threshold = v;\n\nYup, looks good to me.\n\n-- \nShawn.\n"}]}