{"thread":{"id":"27599","subject":"[PATCH] unpack_sha1_file(): zlib can only process 4GB at a time","startedAt":"2011-06-09T20:15:30Z","lastAt":"2011-06-09T22:28:39Z","messageCount":2,"participants":["Junio C Hamano","Sverre Rabbelier"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"169811","messageId":"7vpqmmwyl9.fsf@alter.siamese.dyndns.org","threadId":"27599","inReplyTo":null,"subject":"[PATCH] unpack_sha1_file(): zlib can only process 4GB at a time","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-09T20:15:30Z","receivedAt":"2011-06-09T20:15:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The same theme as \"unpack-objects\" patch.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n * On a beefy enough machine, you could register a large binary blob\n   and run fsck, fetch, or push to play with that object with these\n   patches, but there are many more places that aren't safe.\n\n   I seem to have caught a nasty cold, so this will be the end of the\n   series from me for today.  At least that is what I plan for now.\n\n sha1_file.c |   29 ++++++++++++++++++++---------\n 1 files changed, 20 insertions(+), 9 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 12a166f..ab9e962 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1257,7 +1257,7 @@ static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned lon\n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n \tstream->next_in = map;\n-\tstream->avail_in = mapsize;\n+\tstream->avail_in = zlib_buf_cap(mapsize);\n \tstream->next_out = buffer;\n \tstream->avail_out = bufsiz;\n \n@@ -1291,11 +1291,12 @@ static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned lon\n \treturn 0;\n }\n \n-static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size, const unsigned char *sha1)\n+static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long mapsize,\n+\t\t\t      unsigned long size, const unsigned char *sha1)\n {\n-\tint bytes = strlen(buffer) + 1;\n+\tunsigned long bytes = strlen(buffer) + 1;\n \tunsigned char *buf = xmallocz(size);\n-\tunsigned long n;\n+\tunsigned long n, bytes_to_produce, bytes_to_consume;\n \tint status = Z_OK;\n \n \tn = stream->total_out - bytes;\n@@ -1303,6 +1304,8 @@ static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size\n \t\tn = size;\n \tmemcpy(buf, (char *) buffer + bytes, n);\n \tbytes = n;\n+\n+\tbytes_to_consume = mapsize - stream->total_in;\n \tif (bytes <= size) {\n \t\t/*\n \t\t * The above condition must be (bytes <= size), not\n@@ -1318,17 +1321,25 @@ static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size\n \t\t * went well with status == Z_STREAM_END at the end.\n \t\t */\n \t\tstream->next_out = buf + bytes;\n-\t\tstream->avail_out = size - bytes;\n-\t\twhile (status == Z_OK)\n+\t\tbytes_to_produce = size - bytes;\n+\n+\t\twhile (status == Z_OK || status == Z_BUF_ERROR) {\n+\t\t\tunsigned char *in0 = stream->next_in;\n+\t\t\tunsigned char *out0 = stream->next_out;\n+\t\t\tstream->avail_in = zlib_buf_cap(bytes_to_consume);\n+\t\t\tstream->avail_out = zlib_buf_cap(bytes_to_produce);\n \t\t\tstatus = git_inflate(stream, Z_FINISH);\n+\t\t\tbytes_to_produce -= stream->next_out - out0;\n+\t\t\tbytes_to_consume -= stream->next_in - in0;\n+\t\t}\n \t}\n-\tif (status == Z_STREAM_END && !stream->avail_in) {\n+\tif (status == Z_STREAM_END && !bytes_to_consume) {\n \t\tgit_inflate_end(stream);\n \t\treturn buf;\n \t}\n \n \tif (status < 0)\n-\t\terror(\"corrupt loose object '%s'\", sha1_to_hex(sha1));\n+\t\terror(\"corrupt loose object '%s' %d\", sha1_to_hex(sha1), status);\n \telse if (stream->avail_in)\n \t\terror(\"garbage at end of loose object '%s'\",\n \t\t      sha1_to_hex(sha1));\n@@ -1397,7 +1408,7 @@ static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type\n \tif (ret < Z_OK || (*type = parse_sha1_header(hdr, size)) < 0)\n \t\treturn NULL;\n \n-\treturn unpack_sha1_rest(&stream, hdr, *size, sha1);\n+\treturn unpack_sha1_rest(&stream, hdr, mapsize, *size, sha1);\n }\n \n unsigned long get_size_from_delta(struct packed_git *p,\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169819","messageId":"BANLkTimtaFGhrzTVsg0=gods5Kza7tZ9=A@mail.gmail.com","threadId":"27599","inReplyTo":"7vpqmmwyl9.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] unpack_sha1_file(): zlib can only process 4GB at a time","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-06-09T22:28:39Z","receivedAt":"2011-06-09T22:28:39Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Thu, Jun 9, 2011 at 22:15, Junio C Hamano <gitster@pobox.com> wrote:\n>  * On a beefy enough machine, you could register a large binary blob\n>   and run fsck, fetch, or push to play with that object with these\n>   patches, but there are many more places that aren't safe.\n\nDo you know what those places are, and do you plan to fix them when\nyou get better? If not, should we record those places somewhere (if\nnothing else just on the list)?\n\n-- \nCheers,\n\nSverre Rabbelier\n"}]}