{"thread":{"id":"23432","subject":"[PATCH v2 1/2] index-pack: smarter memory usage when resolving deltas","startedAt":"2010-04-12T16:11:07Z","lastAt":"2010-04-12T16:11:07Z","messageCount":1,"participants":["Nicolas Pitre"],"isPatch":true,"patchVersion":2,"patchTotal":2},"messages":[{"id":"139355","messageId":"alpine.LFD.2.00.1004121208560.7232@xanadu.home","threadId":"23432","inReplyTo":null,"subject":"[PATCH v2 1/2] index-pack: smarter memory usage when resolving deltas","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-12T16:11:07Z","receivedAt":"2010-04-12T16:11:07Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\nIn the same spirit as commit 9892bebafe, let's avoid allocating the full\nbuffer for the deflated data in get_data_from_pack() in order to inflate\nit.  Let's read and inflate the data in chunks instead to reduce memory\nusage.\n\nSigned-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n builtin/index-pack.c |   46 +++++++++++++++++++++++++---------------------\n 1 files changed, 25 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex b4cf8c5..127e713 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -359,34 +359,38 @@ static void *get_data_from_pack(struct object_entry *obj)\n {\n \toff_t from = obj[0].idx.offset + obj[0].hdr_size;\n \tunsigned long len = obj[1].idx.offset - from;\n-\tunsigned long rdy = 0;\n-\tunsigned char *src, *data;\n+\tunsigned char *data, *inbuf;\n \tz_stream stream;\n-\tint st;\n+\tint status;\n \n-\tsrc = xmalloc(len);\n-\tdata = src;\n-\tdo {\n-\t\tssize_t n = pread(pack_fd, data + rdy, len - rdy, from + rdy);\n-\t\tif (n < 0)\n-\t\t\tdie_errno(\"cannot pread pack file\");\n-\t\tif (!n)\n-\t\t\tdie(\"premature end of pack file, %lu bytes missing\",\n-\t\t\t    len - rdy);\n-\t\trdy += n;\n-\t} while (rdy < len);\n \tdata = xmalloc(obj->size);\n+\tinbuf = xmalloc((len < 64*1024) ? len : 64*1024);\n+\n \tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n \tstream.next_out = data;\n \tstream.avail_out = obj->size;\n-\tstream.next_in = src;\n-\tstream.avail_in = len;\n-\tgit_inflate_init(&stream);\n-\twhile ((st = git_inflate(&stream, Z_FINISH)) == Z_OK);\n-\tgit_inflate_end(&stream);\n-\tif (st != Z_STREAM_END || stream.total_out != obj->size)\n+\n+\tdo {\n+\t\tssize_t n = (len < 64*1024) ? len : 64*1024;\n+\t\tn = pread(pack_fd, inbuf, n, from);\n+\t\tif (n < 0)\n+\t\t\tdie_errno(\"cannot pread pack file\");\n+\t\tif (!n)\n+\t\t\tdie(\"premature end of pack file, %lu bytes missing\", len);\n+\t\tfrom += n;\n+\t\tlen -= n;\n+\t\tstream.next_in = inbuf;\n+\t\tstream.avail_in = n;\n+\t\tstatus = git_inflate(&stream, 0);\n+\t} while (len && status == Z_OK && !stream.avail_in);\n+\n+\t/* This has been inflated OK when first encountered, so... */\n+\tif (status != Z_STREAM_END || stream.total_out != obj->size)\n \t\tdie(\"serious inflate inconsistency\");\n-\tfree(src);\n+\n+\tgit_inflate_end(&stream);\n+\tfree(inbuf);\n \treturn data;\n }\n \n-- \n1.7.1.rc1.237.ge1730.dirty\n"}]}