{"thread":{"id":"30629","subject":"[PATCH] pack-objects: use streaming interface for reading large loose blobs","startedAt":"2012-05-26T10:28:01Z","lastAt":"2012-05-29T17:56:18Z","messageCount":2,"participants":["Nguyễn Thái Ngọc Duy","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"192232","messageId":"1338028081-22638-1-git-send-email-pclouds@gmail.com","threadId":"30629","inReplyTo":null,"subject":"[PATCH] pack-objects: use streaming interface for reading large loose blobs","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-05-26T10:28:01Z","receivedAt":"2012-05-26T10:28:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"git usually streams large blobs directly to packs. But there are cases\nwhere git can create large loose blobs (unpack-objects or hash-object\nover pipe). Or they can come from other git implementations.\ncore.bigfilethreshold can also be lowered down and introduce a new\nwave of large loose blobs.\n\nUse streaming interface to read/compress/write these blobs in one\ngo. Fall back to normal way if somehow streaming interface cannot be\nused.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n On top of ng/pack-objects-cleanup. Changes since the last version is\n we do not rely on close_istream(NULL); any more.\n\n builtin/pack-objects.c | 73 +++++++++++++++++++++++++++++++++++++++++++++-----\n t/t1050-large.sh       | 12 +++++++++\n 2 files changed, 79 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ccfcbad..f334820 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -16,6 +16,7 @@\n #include \"list-objects.h\"\n #include \"progress.h\"\n #include \"refs.h\"\n+#include \"streaming.h\"\n #include \"thread-utils.h\"\n \n static const char *pack_usage[] = {\n@@ -150,6 +151,46 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \treturn stream.total_out;\n }\n \n+static unsigned long write_large_blob_data(struct git_istream *st, struct sha1file *f,\n+\t\t\t\t\t   const unsigned char *sha1)\n+{\n+\tgit_zstream stream;\n+\tunsigned char ibuf[1024 * 16];\n+\tunsigned char obuf[1024 * 16];\n+\tunsigned long olen = 0;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_deflate_init(&stream, pack_compression_level);\n+\n+\tfor (;;) {\n+\t\tssize_t readlen;\n+\t\tint zret = Z_OK;\n+\t\treadlen = read_istream(st, ibuf, sizeof(ibuf));\n+\t\tif (readlen == -1)\n+\t\t\tdie(_(\"unable to read %s\"), sha1_to_hex(sha1));\n+\n+\t\tstream.next_in = ibuf;\n+\t\tstream.avail_in = readlen;\n+\t\twhile ((stream.avail_in || readlen == 0) &&\n+\t\t       (zret == Z_OK || zret == Z_BUF_ERROR)) {\n+\t\t\tstream.next_out = obuf;\n+\t\t\tstream.avail_out = sizeof(obuf);\n+\t\t\tzret = git_deflate(&stream, readlen ? 0 : Z_FINISH);\n+\t\t\tsha1write(f, obuf, stream.next_out - obuf);\n+\t\t\tolen += stream.next_out - obuf;\n+\t\t}\n+\t\tif (stream.avail_in)\n+\t\t\tdie(_(\"deflate error (%d)\"), zret);\n+\t\tif (readlen == 0) {\n+\t\t\tif (zret != Z_STREAM_END)\n+\t\t\t\tdie(_(\"deflate error (%d)\"), zret);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tgit_deflate_end(&stream);\n+\treturn olen;\n+}\n+\n /*\n  * we are going to reuse the existing object data as is.  make\n  * sure it is not corrupt.\n@@ -208,11 +249,18 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \tunsigned hdrlen;\n \tenum object_type type;\n \tvoid *buf;\n+\tstruct git_istream *st = NULL;\n \n \tif (!usable_delta) {\n-\t\tbuf = read_sha1_file(entry->idx.sha1, &type, &size);\n-\t\tif (!buf)\n-\t\t\tdie(\"unable to read %s\", sha1_to_hex(entry->idx.sha1));\n+\t\tif (entry->type == OBJ_BLOB &&\n+\t\t    entry->size > big_file_threshold &&\n+\t\t    (st = open_istream(entry->idx.sha1, &type, &size, NULL)) != NULL)\n+\t\t\tbuf = NULL;\n+\t\telse {\n+\t\t\tbuf = read_sha1_file(entry->idx.sha1, &type, &size);\n+\t\t\tif (!buf)\n+\t\t\t\tdie(_(\"unable to read %s\"), sha1_to_hex(entry->idx.sha1));\n+\t\t}\n \t\t/*\n \t\t * make sure no cached delta data remains from a\n \t\t * previous attempt before a pack split occurred.\n@@ -233,7 +281,9 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t}\n \n-\tif (entry->z_delta_size)\n+\tif (st)\t/* large blob case, just assume we don't compress well */\n+\t\tdatalen = size;\n+\telse if (entry->z_delta_size)\n \t\tdatalen = entry->z_delta_size;\n \telse\n \t\tdatalen = do_compress(&buf, size);\n@@ -256,6 +306,8 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \t\twhile (ofs >>= 7)\n \t\t\tdheader[--pos] = 128 | (--ofs & 127);\n \t\tif (limit && hdrlen + sizeof(dheader) - pos + datalen + 20 >= limit) {\n+\t\t\tif (st)\n+\t\t\t\tclose_istream(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -268,6 +320,8 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \t\t * an additional 20 bytes for the base sha1.\n \t\t */\n \t\tif (limit && hdrlen + 20 + datalen + 20 >= limit) {\n+\t\t\tif (st)\n+\t\t\t\tclose_istream(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -276,13 +330,20 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \t\thdrlen += 20;\n \t} else {\n \t\tif (limit && hdrlen + datalen + 20 >= limit) {\n+\t\t\tif (st)\n+\t\t\t\tclose_istream(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n \t\tsha1write(f, header, hdrlen);\n \t}\n-\tsha1write(f, buf, datalen);\n-\tfree(buf);\n+\tif (st) {\n+\t\tdatalen = write_large_blob_data(st, f, entry->idx.sha1);\n+\t\tclose_istream(st);\n+\t} else {\n+\t\tsha1write(f, buf, datalen);\n+\t\tfree(buf);\n+\t}\n \n \treturn hdrlen + datalen;\n }\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 55ed955..313889b 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -134,6 +134,18 @@ test_expect_success 'repack' '\n \tgit repack -ad\n '\n \n+test_expect_success 'pack-objects with large loose object' '\n+\tSHA1=`git hash-object huge` &&\n+\ttest_create_repo loose &&\n+\techo $SHA1 | git pack-objects --stdout |\n+\t\tGIT_ALLOC_LIMIT=0 GIT_DIR=loose/.git git unpack-objects &&\n+\techo $SHA1 | GIT_DIR=loose/.git git pack-objects pack &&\n+\ttest_create_repo packed &&\n+\tmv pack-* packed/.git/objects/pack &&\n+\tGIT_DIR=packed/.git git cat-file blob $SHA1 >actual &&\n+\tcmp huge actual\n+'\n+\n test_expect_success 'tar achiving' '\n \tgit archive --format=tar HEAD >/dev/null\n '\n-- \n1.7.10.2.549.g9354186\n"},{"id":"192419","messageId":"7vd35m6fct.fsf@alter.siamese.dyndns.org","threadId":"30629","inReplyTo":"1338028081-22638-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] pack-objects: use streaming interface for reading large loose blobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-29T17:56:18Z","receivedAt":"2012-05-29T17:56:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:\n\n> git usually streams large blobs directly to packs. But there are cases\n> where git can create large loose blobs (unpack-objects or hash-object\n> over pipe). Or they can come from other git implementations.\n> core.bigfilethreshold can also be lowered down and introduce a new\n> wave of large loose blobs.\n>\n> Use streaming interface to read/compress/write these blobs in one\n> go. Fall back to normal way if somehow streaming interface cannot be\n> used.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> ---\n>  On top of ng/pack-objects-cleanup. Changes since the last version is\n>  we do not rely on close_istream(NULL); any more.\n\nThis version looks much cleaner, at least to me, as the logic in the\nfunction always switches upon \"st\", using it as \"are we reading the\ndata from the stream incrementally?\"  boolean.\n\nThanks, will queue.\n"}]}