{"thread":{"id":"27601","subject":"[PATCH 0/6] zlib only processes 4GB at a time","startedAt":"2011-06-10T20:15:42Z","lastAt":"2011-06-13T11:56:56Z","messageCount":16,"participants":["Junio C Hamano","Thiago Farina","Erik Faye-Lund","Matthieu Moy","Jonathan Nieder"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"169853","messageId":"1307736948-16956-1-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":null,"subject":"[PATCH 0/6] zlib only processes 4GB at a time","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:42Z","receivedAt":"2011-06-10T20:15:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This is a re-roll from yesterday's series, taking a very different\napproach. Instead of fixing call sites one by one, the goal is to define\nour own API that can give an illusion to the callers that they can feed\nand receive data whose size can fit within \"unsigned long\", so that they\ncan set up the avail_in field of z_stream (now git_zstream) to the total\nsize, and expect it to fall down to zero when everything is consumed by\nthe library (vice versa for avail_out).\n\nThe first 5 patches clean up the existing callers that directly call into\nzlib to call git_inflate and git_deflate wrappers.\n\nPatch 6 introduces git_zstream that is a thin wrapper around z_stream,\nbut barfs when the caller gives a buffer larger than 4GB that would fit\nthe underlying zlib calling convention.\n\nThe next step would be to tweak zlib_post_call(), git_inflate() and\ngit_deflate() functions to internally loop and call underlying inflate()\nand deflate() when the incoming buffers are larger than 4GB, but that\npart is not done in this series (yet).\n\nJunio C Hamano (6):\n  zlib: refactor error message formatter\n  zlib: wrap remaining calls to direct inflate/inflateEnd\n  zlib: wrap inflateInit2 used to accept only for gzip format\n  zlib: wrap deflate side of the API\n  zlib: wrap deflateBound() too\n  zlib: zlib can only process 4GB at a time\n\n archive-zip.c            |   10 +-\n builtin/apply.c          |    2 +-\n builtin/index-pack.c     |   12 ++--\n builtin/pack-objects.c   |   18 ++--\n builtin/unpack-objects.c |    2 +-\n cache.h                  |   30 +++++--\n diff.c                   |   10 +-\n fast-import.c            |   30 +++---\n http-backend.c           |   11 +--\n http-push.c              |   16 ++--\n http.h                   |    2 +-\n pack-check.c             |    4 +-\n remote-curl.c            |   14 +--\n sha1_file.c              |   28 +++---\n zlib.c                   |  218 +++++++++++++++++++++++++++++++++++++++-------\n 15 files changed, 283 insertions(+), 124 deletions(-)\n\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169854","messageId":"1307736948-16956-2-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 1/6] zlib: refactor error message formatter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:43Z","receivedAt":"2011-06-10T20:15:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Before refactoring the main part of the wrappers, first move the\nlogic to convert error status that come back from zlib to string\nto a helper function.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n zlib.c |   73 +++++++++++++++++++++++++++++++++------------------------------\n 1 files changed, 38 insertions(+), 35 deletions(-)\n\ndiff --git a/zlib.c b/zlib.c\nindex c4d58da..be9d7e9 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -4,58 +4,61 @@\n  */\n #include \"cache.h\"\n \n-void git_inflate_init(z_streamp strm)\n+static const char *zerr_to_string(int status)\n {\n-\tconst char *err;\n-\n-\tswitch (inflateInit(strm)) {\n-\tcase Z_OK:\n-\t\treturn;\n-\n+\tswitch (status) {\n \tcase Z_MEM_ERROR:\n-\t\terr = \"out of memory\";\n-\t\tbreak;\n+\t\treturn \"out of memory\";\n \tcase Z_VERSION_ERROR:\n-\t\terr = \"wrong version\";\n-\t\tbreak;\n+\t\treturn \"wrong version\";\n+\tcase Z_NEED_DICT:\n+\t\treturn \"needs dictionary\";\n+\tcase Z_DATA_ERROR:\n+\t\treturn \"data stream error\";\n+\tcase Z_STREAM_ERROR:\n+\t\treturn \"stream consistency error\";\n \tdefault:\n-\t\terr = \"error\";\n+\t\treturn \"unknown error\";\n \t}\n-\tdie(\"inflateInit: %s (%s)\", err, strm->msg ? strm->msg : \"no message\");\n }\n \n-void git_inflate_end(z_streamp strm)\n+void git_inflate_init(z_streamp strm)\n {\n-\tif (inflateEnd(strm) != Z_OK)\n-\t\terror(\"inflateEnd: %s\", strm->msg ? strm->msg : \"failed\");\n+\tint status = inflateInit(strm);\n+\n+\tif (status == Z_OK)\n+\t\treturn;\n+\tdie(\"inflateInit: %s (%s)\", zerr_to_string(status),\n+\t    strm->msg ? strm->msg : \"no message\");\n }\n \n-int git_inflate(z_streamp strm, int flush)\n+void git_inflate_end(z_streamp strm)\n {\n-\tint ret = inflate(strm, flush);\n-\tconst char *err;\n+\tint status = inflateEnd(strm);\n \n-\tswitch (ret) {\n-\t/* Out of memory is fatal. */\n-\tcase Z_MEM_ERROR:\n-\t\tdie(\"inflate: out of memory\");\n+\tif (status == Z_OK)\n+\t\treturn;\n+\terror(\"inflateEnd: %s (%s)\", zerr_to_string(status),\n+\t      strm->msg ? strm->msg : \"no message\");\n+}\n \n-\t/* Data corruption errors: we may want to recover from them (fsck) */\n-\tcase Z_NEED_DICT:\n-\t\terr = \"needs dictionary\"; break;\n-\tcase Z_DATA_ERROR:\n-\t\terr = \"data stream error\"; break;\n-\tcase Z_STREAM_ERROR:\n-\t\terr = \"stream consistency error\"; break;\n-\tdefault:\n-\t\terr = \"unknown error\"; break;\n+int git_inflate(z_streamp strm, int flush)\n+{\n+\tint status = inflate(strm, flush);\n \n+\tswitch (status) {\n \t/* Z_BUF_ERROR: normal, needs more space in the output buffer */\n \tcase Z_BUF_ERROR:\n \tcase Z_OK:\n \tcase Z_STREAM_END:\n-\t\treturn ret;\n+\t\treturn status;\n+\n+\tcase Z_MEM_ERROR:\n+\t\tdie(\"inflate: out of memory\");\n+\tdefault:\n+\t\tbreak;\n \t}\n-\terror(\"inflate: %s (%s)\", err, strm->msg ? strm->msg : \"no message\");\n-\treturn ret;\n+\terror(\"inflate: %s (%s)\", zerr_to_string(status),\n+\t      strm->msg ? strm->msg : \"no message\");\n+\treturn status;\n }\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169855","messageId":"1307736948-16956-3-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 2/6] zlib: wrap remaining calls to direct inflate/inflateEnd","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:44Z","receivedAt":"2011-06-10T20:15:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Two callsites in http-backend.c to inflate() and inflateEnd()\nwere not using git_ prefixed versions.  After this, running\n\n    $ find all objects -print | xargs nm -ugo | grep inflate\n\nshows only zlib.c makes direct calls to zlib for inflate operation,\nexcept for a singlecall to inflateInit2 in http-backend.c\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n http-backend.c |    4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex 8501504..c74cb03 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -296,7 +296,7 @@ static void inflate_request(const char *prog_name, int out)\n \t\t\tstream.next_out = out_buf;\n \t\t\tstream.avail_out = sizeof(out_buf);\n \n-\t\t\tret = inflate(&stream, Z_NO_FLUSH);\n+\t\t\tret = git_inflate(&stream, Z_NO_FLUSH);\n \t\t\tif (ret != Z_OK && ret != Z_STREAM_END)\n \t\t\t\tdie(\"zlib error inflating request, result %d\", ret);\n \n@@ -311,7 +311,7 @@ static void inflate_request(const char *prog_name, int out)\n \t}\n \n done:\n-\tinflateEnd(&stream);\n+\tgit_inflate_end(&stream);\n \tclose(out);\n }\n \n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169858","messageId":"1307736948-16956-4-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 3/6] zlib: wrap inflateInit2 used to accept only for gzip format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:45Z","receivedAt":"2011-06-10T20:15:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"http-backend.c uses inflateInit2() to tell the library that it wants to\naccept only gzip format. Wrap it in a helper function so that readers do\nnot have to wonder what the magic numbers 15 and 16 are for.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n cache.h        |    1 +\n http-backend.c |    5 +----\n zlib.c         |   15 +++++++++++++++\n 3 files changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ce73e1f..50f09d0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -21,6 +21,7 @@\n #endif\n \n void git_inflate_init(z_streamp strm);\n+void git_inflate_init_gzip_only(z_streamp strm);\n void git_inflate_end(z_streamp strm);\n int git_inflate(z_streamp strm, int flush);\n \ndiff --git a/http-backend.c b/http-backend.c\nindex c74cb03..ab5015d 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -275,12 +275,9 @@ static void inflate_request(const char *prog_name, int out)\n \tunsigned char in_buf[8192];\n \tunsigned char out_buf[8192];\n \tunsigned long cnt = 0;\n-\tint ret;\n \n \tmemset(&stream, 0, sizeof(stream));\n-\tret = inflateInit2(&stream, (15 + 16));\n-\tif (ret != Z_OK)\n-\t\tdie(\"cannot start zlib inflater, zlib err %d\", ret);\n+\tgit_inflate_init_gzip_only(&stream);\n \n \twhile (1) {\n \t\tssize_t n = xread(0, in_buf, sizeof(in_buf));\ndiff --git a/zlib.c b/zlib.c\nindex be9d7e9..b613cbd 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -32,6 +32,21 @@ void git_inflate_init(z_streamp strm)\n \t    strm->msg ? strm->msg : \"no message\");\n }\n \n+void git_inflate_init_gzip_only(z_streamp strm)\n+{\n+\t/*\n+\t * Use default 15 bits, +16 is to accept only gzip and to\n+\t * yield Z_DATA_ERROR when fed zlib format.\n+\t */\n+\tconst int windowBits = 15 + 16;\n+\tint status = inflateInit2(strm, windowBits);\n+\n+\tif (status == Z_OK)\n+\t\treturn;\n+\tdie(\"inflateInit2: %s (%s)\", zerr_to_string(status),\n+\t    strm->msg ? strm->msg : \"no message\");\n+}\n+\n void git_inflate_end(z_streamp strm)\n {\n \tint status = inflateEnd(strm);\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169857","messageId":"1307736948-16956-5-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 4/6] zlib: wrap deflate side of the API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:46Z","receivedAt":"2011-06-10T20:15:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Wrap deflateInit, deflate, and deflateEnd for everybody, and the sole use\nof deflateInit2 in remote-curl.c to tell the library to use gzip header\nand trailer in git_deflate_init_gzip().\n\nThere is only one caller that cares about the status from deflateEnd().\nIntroduce git_deflate_end_gently() to let that sole caller retrieve the\nstatus and act on it (i.e. die) for now, but we would probably want to\nmake inflate_end/deflate_end die when they ran out of memory and get\nrid of the _gently() kind.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n archive-zip.c          |    6 ++--\n builtin/index-pack.c   |    6 ++--\n builtin/pack-objects.c |    6 ++--\n cache.h                |    6 ++++\n diff.c                 |    6 ++--\n fast-import.c          |   22 ++++++++--------\n http-push.c            |   12 ++++----\n remote-curl.c          |   10 ++-----\n sha1_file.c            |   10 ++++----\n zlib.c                 |   62 ++++++++++++++++++++++++++++++++++++++++++++++++\n 10 files changed, 105 insertions(+), 41 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex cf28504..081ff83 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -96,7 +96,7 @@ static void *zlib_deflate(void *data, unsigned long size,\n \tint result;\n \n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, compression_level);\n+\tgit_deflate_init(&stream, compression_level);\n \tmaxsize = deflateBound(&stream, size);\n \tbuffer = xmalloc(maxsize);\n \n@@ -106,7 +106,7 @@ static void *zlib_deflate(void *data, unsigned long size,\n \tstream.avail_out = maxsize;\n \n \tdo {\n-\t\tresult = deflate(&stream, Z_FINISH);\n+\t\tresult = git_deflate(&stream, Z_FINISH);\n \t} while (result == Z_OK);\n \n \tif (result != Z_STREAM_END) {\n@@ -114,7 +114,7 @@ static void *zlib_deflate(void *data, unsigned long size,\n \t\treturn NULL;\n \t}\n \n-\tdeflateEnd(&stream);\n+\tgit_deflate_end(&stream);\n \t*compressed_size = stream.total_out;\n \n \treturn buffer;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 31f001f..dbfb5da 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -671,21 +671,21 @@ static int write_compressed(struct sha1file *f, void *in, unsigned int size)\n \tunsigned char outbuf[4096];\n \n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, zlib_compression_level);\n+\tgit_deflate_init(&stream, zlib_compression_level);\n \tstream.next_in = in;\n \tstream.avail_in = size;\n \n \tdo {\n \t\tstream.next_out = outbuf;\n \t\tstream.avail_out = sizeof(outbuf);\n-\t\tstatus = deflate(&stream, Z_FINISH);\n+\t\tstatus = git_deflate(&stream, Z_FINISH);\n \t\tsha1write(f, outbuf, sizeof(outbuf) - stream.avail_out);\n \t} while (status == Z_OK);\n \n \tif (status != Z_STREAM_END)\n \t\tdie(\"unable to deflate appended object (%d)\", status);\n \tsize = stream.total_out;\n-\tdeflateEnd(&stream);\n+\tgit_deflate_end(&stream);\n \treturn size;\n }\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f402a84..61f9755 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -131,7 +131,7 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \tunsigned long maxsize;\n \n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, pack_compression_level);\n+\tgit_deflate_init(&stream, pack_compression_level);\n \tmaxsize = deflateBound(&stream, size);\n \n \tin = *pptr;\n@@ -142,9 +142,9 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \tstream.avail_in = size;\n \tstream.next_out = out;\n \tstream.avail_out = maxsize;\n-\twhile (deflate(&stream, Z_FINISH) == Z_OK)\n+\twhile (git_deflate(&stream, Z_FINISH) == Z_OK)\n \t\t; /* nothing */\n-\tdeflateEnd(&stream);\n+\tgit_deflate_end(&stream);\n \n \tfree(in);\n \treturn stream.total_out;\ndiff --git a/cache.h b/cache.h\nindex 50f09d0..5eb61ac 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -25,6 +25,12 @@ void git_inflate_init_gzip_only(z_streamp strm);\n void git_inflate_end(z_streamp strm);\n int git_inflate(z_streamp strm, int flush);\n \n+void git_deflate_init(z_streamp strm, int level);\n+void git_deflate_init_gzip(z_streamp strm, int level);\n+void git_deflate_end(z_streamp strm);\n+int git_deflate_end_gently(z_streamp strm);\n+int git_deflate(z_streamp strm, int flush);\n+\n #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n #define DTYPE(de)\t((de)->d_type)\n #else\ndiff --git a/diff.c b/diff.c\nindex 559bf57..c15c70f 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1732,7 +1732,7 @@ static unsigned char *deflate_it(char *data,\n \tz_stream stream;\n \n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, zlib_compression_level);\n+\tgit_deflate_init(&stream, zlib_compression_level);\n \tbound = deflateBound(&stream, size);\n \tdeflated = xmalloc(bound);\n \tstream.next_out = deflated;\n@@ -1740,9 +1740,9 @@ static unsigned char *deflate_it(char *data,\n \n \tstream.next_in = (unsigned char *)data;\n \tstream.avail_in = size;\n-\twhile (deflate(&stream, Z_FINISH) == Z_OK)\n+\twhile (git_deflate(&stream, Z_FINISH) == Z_OK)\n \t\t; /* nothing */\n-\tdeflateEnd(&stream);\n+\tgit_deflate_end(&stream);\n \t*result_size = stream.total_out;\n \treturn deflated;\n }\ndiff --git a/fast-import.c b/fast-import.c\nindex 78d9786..42979e6 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1050,7 +1050,7 @@ static int store_object(\n \t\tdelta = NULL;\n \n \tmemset(&s, 0, sizeof(s));\n-\tdeflateInit(&s, pack_compression_level);\n+\tgit_deflate_init(&s, pack_compression_level);\n \tif (delta) {\n \t\ts.next_in = delta;\n \t\ts.avail_in = deltalen;\n@@ -1060,9 +1060,9 @@ static int store_object(\n \t}\n \ts.avail_out = deflateBound(&s, s.avail_in);\n \ts.next_out = out = xmalloc(s.avail_out);\n-\twhile (deflate(&s, Z_FINISH) == Z_OK)\n-\t\t/* nothing */;\n-\tdeflateEnd(&s);\n+\twhile (git_deflate(&s, Z_FINISH) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_deflate_end(&s);\n \n \t/* Determine if we should auto-checkpoint. */\n \tif ((max_packsize && (pack_size + 60 + s.total_out) > max_packsize)\n@@ -1078,14 +1078,14 @@ static int store_object(\n \t\t\tdelta = NULL;\n \n \t\t\tmemset(&s, 0, sizeof(s));\n-\t\t\tdeflateInit(&s, pack_compression_level);\n+\t\t\tgit_deflate_init(&s, pack_compression_level);\n \t\t\ts.next_in = (void *)dat->buf;\n \t\t\ts.avail_in = dat->len;\n \t\t\ts.avail_out = deflateBound(&s, s.avail_in);\n \t\t\ts.next_out = out = xrealloc(out, s.avail_out);\n-\t\t\twhile (deflate(&s, Z_FINISH) == Z_OK)\n-\t\t\t\t/* nothing */;\n-\t\t\tdeflateEnd(&s);\n+\t\t\twhile (git_deflate(&s, Z_FINISH) == Z_OK)\n+\t\t\t\t; /* nothing */\n+\t\t\tgit_deflate_end(&s);\n \t\t}\n \t}\n \n@@ -1187,7 +1187,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tcrc32_begin(pack_file);\n \n \tmemset(&s, 0, sizeof(s));\n-\tdeflateInit(&s, pack_compression_level);\n+\tgit_deflate_init(&s, pack_compression_level);\n \n \thdrlen = encode_in_pack_object_header(OBJ_BLOB, len, out_buf);\n \tif (out_sz <= hdrlen)\n@@ -1209,7 +1209,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \t\t\tlen -= n;\n \t\t}\n \n-\t\tstatus = deflate(&s, len ? 0 : Z_FINISH);\n+\t\tstatus = git_deflate(&s, len ? 0 : Z_FINISH);\n \n \t\tif (!s.avail_out || status == Z_STREAM_END) {\n \t\t\tsize_t n = s.next_out - out_buf;\n@@ -1228,7 +1228,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \t\t\tdie(\"unexpected deflate failure: %d\", status);\n \t\t}\n \t}\n-\tdeflateEnd(&s);\n+\tgit_deflate_end(&s);\n \tgit_SHA1_Final(sha1, &c);\n \n \tif (sha1out)\ndiff --git a/http-push.c b/http-push.c\nindex d18346c..46e68c8 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -359,7 +359,7 @@ static void start_put(struct transfer_request *request)\n \n \t/* Set it up */\n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, zlib_compression_level);\n+\tgit_deflate_init(&stream, zlib_compression_level);\n \tsize = deflateBound(&stream, len + hdrlen);\n \tstrbuf_init(&request->buffer.buf, size);\n \trequest->buffer.posn = 0;\n@@ -371,15 +371,15 @@ static void start_put(struct transfer_request *request)\n \t/* First header.. */\n \tstream.next_in = (void *)hdr;\n \tstream.avail_in = hdrlen;\n-\twhile (deflate(&stream, 0) == Z_OK)\n-\t\t/* nothing */;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n \n \t/* Then the data itself.. */\n \tstream.next_in = unpacked;\n \tstream.avail_in = len;\n-\twhile (deflate(&stream, Z_FINISH) == Z_OK)\n-\t\t/* nothing */;\n-\tdeflateEnd(&stream);\n+\twhile (git_deflate(&stream, Z_FINISH) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_deflate_end(&stream);\n \tfree(unpacked);\n \n \trequest->buffer.buf.len = stream.total_out;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 775d614..3c7621a 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -475,11 +475,7 @@ static int post_rpc(struct rpc_state *rpc)\n \t\tint ret;\n \n \t\tmemset(&stream, 0, sizeof(stream));\n-\t\tret = deflateInit2(&stream, Z_BEST_COMPRESSION,\n-\t\t\t\tZ_DEFLATED, (15 + 16),\n-\t\t\t\t8, Z_DEFAULT_STRATEGY);\n-\t\tif (ret != Z_OK)\n-\t\t\tdie(\"cannot deflate request; zlib init error %d\", ret);\n+\t\tgit_deflate_init_gzip(&stream, Z_BEST_COMPRESSION);\n \t\tsize = deflateBound(&stream, rpc->len);\n \t\tgzip_body = xmalloc(size);\n \n@@ -488,11 +484,11 @@ static int post_rpc(struct rpc_state *rpc)\n \t\tstream.next_out = (unsigned char *)gzip_body;\n \t\tstream.avail_out = size;\n \n-\t\tret = deflate(&stream, Z_FINISH);\n+\t\tret = git_deflate(&stream, Z_FINISH);\n \t\tif (ret != Z_STREAM_END)\n \t\t\tdie(\"cannot deflate request; zlib deflate error %d\", ret);\n \n-\t\tret = deflateEnd(&stream);\n+\t\tret = git_deflate_end_gently(&stream);\n \t\tif (ret != Z_OK)\n \t\t\tdie(\"cannot deflate request; zlib end error %d\", ret);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1a7e410..0eefb61 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2442,7 +2442,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \n \t/* Set it up */\n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, zlib_compression_level);\n+\tgit_deflate_init(&stream, zlib_compression_level);\n \tstream.next_out = compressed;\n \tstream.avail_out = sizeof(compressed);\n \tgit_SHA1_Init(&c);\n@@ -2450,8 +2450,8 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \t/* First header.. */\n \tstream.next_in = (unsigned char *)hdr;\n \tstream.avail_in = hdrlen;\n-\twhile (deflate(&stream, 0) == Z_OK)\n-\t\t/* nothing */;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n \tgit_SHA1_Update(&c, hdr, hdrlen);\n \n \t/* Then the data itself.. */\n@@ -2459,7 +2459,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \tstream.avail_in = len;\n \tdo {\n \t\tunsigned char *in0 = stream.next_in;\n-\t\tret = deflate(&stream, Z_FINISH);\n+\t\tret = git_deflate(&stream, Z_FINISH);\n \t\tgit_SHA1_Update(&c, in0, stream.next_in - in0);\n \t\tif (write_buffer(fd, compressed, stream.next_out - compressed) < 0)\n \t\t\tdie(\"unable to write sha1 file\");\n@@ -2469,7 +2469,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \n \tif (ret != Z_STREAM_END)\n \t\tdie(\"unable to deflate new object %s (%d)\", sha1_to_hex(sha1), ret);\n-\tret = deflateEnd(&stream);\n+\tret = git_deflate_end_gently(&stream);\n \tif (ret != Z_OK)\n \t\tdie(\"deflateEnd on object %s failed (%d)\", sha1_to_hex(sha1), ret);\n \tgit_SHA1_Final(parano_sha1, &c);\ndiff --git a/zlib.c b/zlib.c\nindex b613cbd..ee47a3a 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -77,3 +77,65 @@ int git_inflate(z_streamp strm, int flush)\n \t      strm->msg ? strm->msg : \"no message\");\n \treturn status;\n }\n+\n+void git_deflate_init(z_streamp strm, int level)\n+{\n+\tint status = deflateInit(strm, level);\n+\n+\tif (status == Z_OK)\n+\t\treturn;\n+\tdie(\"deflateInit: %s (%s)\", zerr_to_string(status),\n+\t    strm->msg ? strm->msg : \"no message\");\n+}\n+\n+void git_deflate_init_gzip(z_streamp strm, int level)\n+{\n+\t/*\n+\t * Use default 15 bits, +16 is to generate gzip header/trailer\n+\t * instead of the zlib wrapper.\n+\t */\n+\tconst int windowBits = 15 + 16;\n+\tint status = deflateInit2(strm, level,\n+\t\t\t\t  Z_DEFLATED, windowBits,\n+\t\t\t\t  8, Z_DEFAULT_STRATEGY);\n+\tif (status == Z_OK)\n+\t\treturn;\n+\tdie(\"deflateInit2: %s (%s)\", zerr_to_string(status),\n+\t    strm->msg ? strm->msg : \"no message\");\n+}\n+\n+void git_deflate_end(z_streamp strm)\n+{\n+\tint status = deflateEnd(strm);\n+\n+\tif (status == Z_OK)\n+\t\treturn;\n+\terror(\"deflateEnd: %s (%s)\", zerr_to_string(status),\n+\t      strm->msg ? strm->msg : \"no message\");\n+}\n+\n+int git_deflate_end_gently(z_streamp strm)\n+{\n+\treturn deflateEnd(strm);\n+}\n+\n+int git_deflate(z_streamp strm, int flush)\n+{\n+\tint status = deflate(strm, flush);\n+\n+\tswitch (status) {\n+\t/* Z_BUF_ERROR: normal, needs more space in the output buffer */\n+\tcase Z_BUF_ERROR:\n+\tcase Z_OK:\n+\tcase Z_STREAM_END:\n+\t\treturn status;\n+\n+\tcase Z_MEM_ERROR:\n+\t\tdie(\"deflate: out of memory\");\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\terror(\"deflate: %s (%s)\", zerr_to_string(status),\n+\t      strm->msg ? strm->msg : \"no message\");\n+\treturn status;\n+}\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169856","messageId":"1307736948-16956-6-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 5/6] zlib: wrap deflateBound() too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:47Z","receivedAt":"2011-06-10T20:15:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Signed-off-by: Junio C Hamano <gitster@pobox.com>\n---\n archive-zip.c          |    2 +-\n builtin/pack-objects.c |    2 +-\n cache.h                |    4 +---\n diff.c                 |    2 +-\n fast-import.c          |    4 ++--\n http-push.c            |    2 +-\n remote-curl.c          |    2 +-\n zlib.c                 |    9 +++++++++\n 8 files changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 081ff83..8fbac27 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -97,7 +97,7 @@ static void *zlib_deflate(void *data, unsigned long size,\n \n \tmemset(&stream, 0, sizeof(stream));\n \tgit_deflate_init(&stream, compression_level);\n-\tmaxsize = deflateBound(&stream, size);\n+\tmaxsize = git_deflate_bound(&stream, size);\n \tbuffer = xmalloc(maxsize);\n \n \tstream.next_in = data;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 61f9755..8981dd6 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -132,7 +132,7 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \n \tmemset(&stream, 0, sizeof(stream));\n \tgit_deflate_init(&stream, pack_compression_level);\n-\tmaxsize = deflateBound(&stream, size);\n+\tmaxsize = git_deflate_bound(&stream, size);\n \n \tin = *pptr;\n \tout = xmalloc(maxsize);\ndiff --git a/cache.h b/cache.h\nindex 5eb61ac..85ac5ec 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -16,9 +16,6 @@\n #endif\n \n #include <zlib.h>\n-#if defined(NO_DEFLATE_BOUND) || ZLIB_VERNUM < 0x1200\n-#define deflateBound(c,s)  ((s) + (((s) + 7) >> 3) + (((s) + 63) >> 6) + 11)\n-#endif\n \n void git_inflate_init(z_streamp strm);\n void git_inflate_init_gzip_only(z_streamp strm);\n@@ -30,6 +27,7 @@ void git_deflate_init_gzip(z_streamp strm, int level);\n void git_deflate_end(z_streamp strm);\n int git_deflate_end_gently(z_streamp strm);\n int git_deflate(z_streamp strm, int flush);\n+unsigned long git_deflate_bound(z_streamp, unsigned long);\n \n #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n #define DTYPE(de)\t((de)->d_type)\ndiff --git a/diff.c b/diff.c\nindex c15c70f..bac9a4b 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1733,7 +1733,7 @@ static unsigned char *deflate_it(char *data,\n \n \tmemset(&stream, 0, sizeof(stream));\n \tgit_deflate_init(&stream, zlib_compression_level);\n-\tbound = deflateBound(&stream, size);\n+\tbound = git_deflate_bound(&stream, size);\n \tdeflated = xmalloc(bound);\n \tstream.next_out = deflated;\n \tstream.avail_out = bound;\ndiff --git a/fast-import.c b/fast-import.c\nindex 42979e6..e4116a4 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1058,7 +1058,7 @@ static int store_object(\n \t\ts.next_in = (void *)dat->buf;\n \t\ts.avail_in = dat->len;\n \t}\n-\ts.avail_out = deflateBound(&s, s.avail_in);\n+\ts.avail_out = git_deflate_bound(&s, s.avail_in);\n \ts.next_out = out = xmalloc(s.avail_out);\n \twhile (git_deflate(&s, Z_FINISH) == Z_OK)\n \t\t; /* nothing */\n@@ -1081,7 +1081,7 @@ static int store_object(\n \t\t\tgit_deflate_init(&s, pack_compression_level);\n \t\t\ts.next_in = (void *)dat->buf;\n \t\t\ts.avail_in = dat->len;\n-\t\t\ts.avail_out = deflateBound(&s, s.avail_in);\n+\t\t\ts.avail_out = git_deflate_bound(&s, s.avail_in);\n \t\t\ts.next_out = out = xrealloc(out, s.avail_out);\n \t\t\twhile (git_deflate(&s, Z_FINISH) == Z_OK)\n \t\t\t\t; /* nothing */\ndiff --git a/http-push.c b/http-push.c\nindex 46e68c8..ecd67cf 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -360,7 +360,7 @@ static void start_put(struct transfer_request *request)\n \t/* Set it up */\n \tmemset(&stream, 0, sizeof(stream));\n \tgit_deflate_init(&stream, zlib_compression_level);\n-\tsize = deflateBound(&stream, len + hdrlen);\n+\tsize = git_deflate_bound(&stream, len + hdrlen);\n \tstrbuf_init(&request->buffer.buf, size);\n \trequest->buffer.posn = 0;\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 3c7621a..13d8cee 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -476,7 +476,7 @@ static int post_rpc(struct rpc_state *rpc)\n \n \t\tmemset(&stream, 0, sizeof(stream));\n \t\tgit_deflate_init_gzip(&stream, Z_BEST_COMPRESSION);\n-\t\tsize = deflateBound(&stream, rpc->len);\n+\t\tsize = git_deflate_bound(&stream, rpc->len);\n \t\tgzip_body = xmalloc(size);\n \n \t\tstream.next_in = (unsigned char *)rpc->buf;\ndiff --git a/zlib.c b/zlib.c\nindex ee47a3a..8f19d2f 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -78,6 +78,15 @@ int git_inflate(z_streamp strm, int flush)\n \treturn status;\n }\n \n+#if defined(NO_DEFLATE_BOUND) || ZLIB_VERNUM < 0x1200\n+#define deflateBound(c,s)  ((s) + (((s) + 7) >> 3) + (((s) + 63) >> 6) + 11)\n+#endif\n+\n+unsigned long git_deflate_bound(z_streamp strm, unsigned long size)\n+{\n+\treturn deflateBound(strm, size);\n+}\n+\n void git_deflate_init(z_streamp strm, int level)\n {\n \tint status = deflateInit(strm, level);\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169859","messageId":"1307736948-16956-7-git-send-email-gitster@pobox.com","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T20:15:48Z","receivedAt":"2011-06-10T20:15:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The size of objects we read from the repository and data we try to put\ninto the repository are represented in \"unsigned long\", so that on larger\narchitectures we can handle objects that weigh more than 4GB.\n\nBut the interface defined in zlib.h to communicate with inflate/deflate\nlimits avail_in (how many bytes of input are we calling zlib with) and\navail_out (how many bytes of output from zlib are we ready to accept)\nfields effectively to 4GB by defining their type to be uInt.\n\nIn many places in our code, we allocate a large buffer (e.g. mmap'ing a\nlarge loose object file) and tell zlib its size by assigning the size to\navail_in field of the stream, but that will truncate the high octets of\nthe real size. The worst part of this story is that we often pass around\nz_stream (the state object used by zlib) to keep track of the number of\nused bytes in input/output buffer by inspecting these two fields, which\npractically limits our callchain to the same 4GB limit.\n\nWrap z_stream in another structure git_zstream that can express avail_in\nand avail_out in unsigned long. For now, just die() when the caller gives\na size that cannot be given to a single zlib call. In later patches in the\nseries, we would make git_inflate() and git_deflate() internally loop to\ngive callers an illusion that our \"improved\" version of zlib interface can\noperate on a buffer larger than 4GB in one go.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n archive-zip.c            |    2 +-\n builtin/apply.c          |    2 +-\n builtin/index-pack.c     |    6 +-\n builtin/pack-objects.c   |   10 ++--\n builtin/unpack-objects.c |    2 +-\n cache.h                  |   35 +++++++++-----\n diff.c                   |    2 +-\n fast-import.c            |    4 +-\n http-backend.c           |    2 +-\n http-push.c              |    2 +-\n http.h                   |    2 +-\n pack-check.c             |    4 +-\n remote-curl.c            |    2 +-\n sha1_file.c              |   18 ++++----\n zlib.c                   |  119 +++++++++++++++++++++++++++++++++++-----------\n 15 files changed, 142 insertions(+), 70 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 8fbac27..72d55a5 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -90,7 +90,7 @@ static void copy_le32(unsigned char *dest, unsigned int n)\n static void *zlib_deflate(void *data, unsigned long size,\n \t\tint compression_level, unsigned long *compressed_size)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tunsigned long maxsize;\n \tvoid *buffer;\n \tint result;\ndiff --git a/builtin/apply.c b/builtin/apply.c\nindex 530d4bb..f2edc52 100644\n--- a/builtin/apply.c\n+++ b/builtin/apply.c\n@@ -1634,7 +1634,7 @@ static inline int metadata_changes(struct patch *patch)\n static char *inflate_it(const void *data, unsigned long size,\n \t\t\tunsigned long inflated_size)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tvoid *out;\n \tint st;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex dbfb5da..f65cb37 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -265,7 +265,7 @@ static void unlink_base_data(struct base_data *c)\n static void *unpack_entry_data(unsigned long offset, unsigned long size)\n {\n \tint status;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tvoid *buf = xmalloc(size);\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -355,7 +355,7 @@ static void *get_data_from_pack(struct object_entry *obj)\n \toff_t from = obj[0].idx.offset + obj[0].hdr_size;\n \tunsigned long len = obj[1].idx.offset - from;\n \tunsigned char *data, *inbuf;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tint status;\n \n \tdata = xmalloc(obj->size);\n@@ -666,7 +666,7 @@ static void parse_pack_objects(unsigned char *sha1)\n \n static int write_compressed(struct sha1file *f, void *in, unsigned int size)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tint status;\n \tunsigned char outbuf[4096];\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8981dd6..c6e2d87 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -126,7 +126,7 @@ static void *get_delta(struct object_entry *entry)\n \n static unsigned long do_compress(void **pptr, unsigned long size)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tvoid *in, *out;\n \tunsigned long maxsize;\n \n@@ -160,7 +160,7 @@ static int check_pack_inflate(struct packed_git *p,\n \t\toff_t len,\n \t\tunsigned long expect)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tunsigned char fakebuf[4096], *in;\n \tint st;\n \n@@ -187,12 +187,12 @@ static void copy_pack_data(struct sha1file *f,\n \t\toff_t len)\n {\n \tunsigned char *in;\n-\tunsigned int avail;\n+\tunsigned long avail;\n \n \twhile (len) {\n \t\tin = use_pack(p, w_curs, offset, &avail);\n \t\tif (avail > len)\n-\t\t\tavail = (unsigned int)len;\n+\t\t\tavail = (unsigned long)len;\n \t\tsha1write(f, in, avail);\n \t\toffset += avail;\n \t\tlen -= avail;\n@@ -994,7 +994,7 @@ static void check_object(struct object_entry *entry)\n \t\tconst unsigned char *base_ref = NULL;\n \t\tstruct object_entry *base_entry;\n \t\tunsigned long used, used_0;\n-\t\tunsigned int avail;\n+\t\tunsigned long avail;\n \t\toff_t ofs;\n \t\tunsigned char *buf, c;\n \ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f63973c..14e04e6 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -90,7 +90,7 @@ static void use(int bytes)\n \n static void *get_data(unsigned long size)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tvoid *buf = xmalloc(size);\n \n \tmemset(&stream, 0, sizeof(stream));\ndiff --git a/cache.h b/cache.h\nindex 85ac5ec..1784bab 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -16,18 +16,27 @@\n #endif\n \n #include <zlib.h>\n-\n-void git_inflate_init(z_streamp strm);\n-void git_inflate_init_gzip_only(z_streamp strm);\n-void git_inflate_end(z_streamp strm);\n-int git_inflate(z_streamp strm, int flush);\n-\n-void git_deflate_init(z_streamp strm, int level);\n-void git_deflate_init_gzip(z_streamp strm, int level);\n-void git_deflate_end(z_streamp strm);\n-int git_deflate_end_gently(z_streamp strm);\n-int git_deflate(z_streamp strm, int flush);\n-unsigned long git_deflate_bound(z_streamp, unsigned long);\n+typedef struct git_zstream {\n+\tz_stream z;\n+\tunsigned long avail_in;\n+\tunsigned long avail_out;\n+\tunsigned long total_in;\n+\tunsigned long total_out;\n+\tunsigned char *next_in;\n+\tunsigned char *next_out;\n+} git_zstream;\n+\n+void git_inflate_init(git_zstream *);\n+void git_inflate_init_gzip_only(git_zstream *);\n+void git_inflate_end(git_zstream *);\n+int git_inflate(git_zstream *, int flush);\n+\n+void git_deflate_init(git_zstream *, int level);\n+void git_deflate_init_gzip(git_zstream *, int level);\n+void git_deflate_end(git_zstream *);\n+int git_deflate_end_gently(git_zstream *);\n+int git_deflate(git_zstream *, int flush);\n+unsigned long git_deflate_bound(git_zstream *, unsigned long);\n \n #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n #define DTYPE(de)\t((de)->d_type)\n@@ -991,7 +1000,7 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n extern void pack_report(void);\n extern int open_pack_index(struct packed_git *);\n extern void close_pack_index(struct packed_git *);\n-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned int *);\n+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n extern void unuse_pack(struct pack_window **);\n extern void free_pack_by_name(const char *);\ndiff --git a/diff.c b/diff.c\nindex bac9a4b..431873d 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1729,7 +1729,7 @@ static unsigned char *deflate_it(char *data,\n {\n \tint bound;\n \tunsigned char *deflated;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \n \tmemset(&stream, 0, sizeof(stream));\n \tgit_deflate_init(&stream, zlib_compression_level);\ndiff --git a/fast-import.c b/fast-import.c\nindex e4116a4..1d5e333 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1017,7 +1017,7 @@ static int store_object(\n \tunsigned char sha1[20];\n \tunsigned long hdrlen, deltalen;\n \tgit_SHA_CTX c;\n-\tz_stream s;\n+\tgit_zstream s;\n \n \thdrlen = sprintf((char *)hdr,\"%s %lu\", typename(type),\n \t\t(unsigned long)dat->len) + 1;\n@@ -1163,7 +1163,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \toff_t offset;\n \tgit_SHA_CTX c;\n \tgit_SHA_CTX pack_file_ctx;\n-\tz_stream s;\n+\tgit_zstream s;\n \tint status = Z_OK;\n \n \t/* Determine if we should auto-checkpoint. */\ndiff --git a/http-backend.c b/http-backend.c\nindex ab5015d..59ad7da 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -271,7 +271,7 @@ static struct rpc_service *select_service(const char *name)\n \n static void inflate_request(const char *prog_name, int out)\n {\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tunsigned char in_buf[8192];\n \tunsigned char out_buf[8192];\n \tunsigned long cnt = 0;\ndiff --git a/http-push.c b/http-push.c\nindex ecd67cf..1e8a830 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -352,7 +352,7 @@ static void start_put(struct transfer_request *request)\n \tunsigned long len;\n \tint hdrlen;\n \tssize_t size;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \n \tunpacked = read_sha1_file(request->obj->sha1, &type, &len);\n \thdrlen = sprintf(hdr, \"%s %lu\", typename(type), len) + 1;\ndiff --git a/http.h b/http.h\nindex e9ed3c2..a39304a 100644\n--- a/http.h\n+++ b/http.h\n@@ -172,7 +172,7 @@ struct http_object_request {\n \tunsigned char sha1[20];\n \tunsigned char real_sha1[20];\n \tgit_SHA_CTX c;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tint zret;\n \tint rename;\n \tstruct active_request_slot *slot;\ndiff --git a/pack-check.c b/pack-check.c\nindex a1a5216..0c19b6e 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -26,7 +26,7 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs,\n \tuint32_t data_crc = crc32(0, NULL, 0);\n \n \tdo {\n-\t\tunsigned int avail;\n+\t\tunsigned long avail;\n \t\tvoid *data = use_pack(p, w_curs, offset, &avail);\n \t\tif (avail > len)\n \t\t\tavail = len;\n@@ -61,7 +61,7 @@ static int verify_packfile(struct packed_git *p,\n \n \tgit_SHA1_Init(&ctx);\n \tdo {\n-\t\tunsigned int remaining;\n+\t\tunsigned long remaining;\n \t\tunsigned char *in = use_pack(p, w_curs, offset, &remaining);\n \t\toffset += remaining;\n \t\tif (!pack_sig_ofs)\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 13d8cee..bc48a36 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -471,7 +471,7 @@ static int post_rpc(struct rpc_state *rpc)\n \t\t * the transfer time.\n \t\t */\n \t\tsize_t size;\n-\t\tz_stream stream;\n+\t\tgit_zstream stream;\n \t\tint ret;\n \n \t\tmemset(&stream, 0, sizeof(stream));\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 0eefb61..94d4319 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -833,7 +833,7 @@ static int in_window(struct pack_window *win, off_t offset)\n unsigned char *use_pack(struct packed_git *p,\n \t\tstruct pack_window **w_cursor,\n \t\toff_t offset,\n-\t\tunsigned int *left)\n+\t\tunsigned long *left)\n {\n \tstruct pack_window *win = *w_cursor;\n \n@@ -1244,7 +1244,7 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \treturn used;\n }\n \n-static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n+static int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n {\n \tunsigned long size, used;\n \tstatic const char valid_loose_object_type[8] = {\n@@ -1291,7 +1291,7 @@ static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned lon\n \treturn 0;\n }\n \n-static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size, const unsigned char *sha1)\n+static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)\n {\n \tint bytes = strlen(buffer) + 1;\n \tunsigned char *buf = xmallocz(size);\n@@ -1390,7 +1390,7 @@ static int parse_sha1_header(const char *hdr, unsigned long *sizep)\n static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n {\n \tint ret;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tchar hdr[8192];\n \n \tret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));\n@@ -1406,7 +1406,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n {\n \tconst unsigned char *data;\n \tunsigned char delta_head[20], *in;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tint st;\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -1528,7 +1528,7 @@ static int unpack_object_header(struct packed_git *p,\n \t\t\t\tunsigned long *sizep)\n {\n \tunsigned char *base;\n-\tunsigned int left;\n+\tunsigned long left;\n \tunsigned long used;\n \tenum object_type type;\n \n@@ -1641,7 +1641,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \t\t\t\t    unsigned long size)\n {\n \tint st;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tunsigned char *buffer, *in;\n \n \tbuffer = xmallocz(size);\n@@ -2074,7 +2074,7 @@ static int sha1_loose_object_info(const unsigned char *sha1, unsigned long *size\n \tint status;\n \tunsigned long mapsize, size;\n \tvoid *map;\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tchar hdr[32];\n \n \tmap = map_sha1_file(sha1, &mapsize);\n@@ -2425,7 +2425,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n {\n \tint fd, ret;\n \tunsigned char compressed[4096];\n-\tz_stream stream;\n+\tgit_zstream stream;\n \tgit_SHA_CTX c;\n \tunsigned char parano_sha1[20];\n \tchar *filename;\ndiff --git a/zlib.c b/zlib.c\nindex 8f19d2f..fe537e3 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -22,45 +22,90 @@ static const char *zerr_to_string(int status)\n \t}\n }\n \n-void git_inflate_init(z_streamp strm)\n+/*\n+ * avail_in and avail_out in zlib are counted in uInt, which typically\n+ * limits the size of the buffer we can use to 4GB when interacting\n+ * with zlib in a single call to inflate/deflate.\n+ */\n+#define ZLIB_BUF_MAX ((uInt)-1)\n+static inline uInt zlib_buf_cap(unsigned long len)\n+{\n+\tif (ZLIB_BUF_MAX < len)\n+\t\tdie(\"working buffer for zlib too large\");\n+\treturn len;\n+}\n+\n+static void zlib_pre_call(git_zstream *s)\n+{\n+\ts->z.next_in = s->next_in;\n+\ts->z.next_out = s->next_out;\n+\ts->z.total_in = s->total_in;\n+\ts->z.total_out = s->total_out;\n+\ts->z.avail_in = zlib_buf_cap(s->avail_in);\n+\ts->z.avail_out = zlib_buf_cap(s->avail_out);\n+}\n+\n+static void zlib_post_call(git_zstream *s)\n+{\n+\ts->next_in = s->z.next_in;\n+\ts->next_out = s->z.next_out;\n+\ts->total_in = s->z.total_in;\n+\ts->total_out = s->z.total_out;\n+\ts->avail_in = s->z.avail_in;\n+\ts->avail_out = s->z.avail_out;\n+}\n+\n+void git_inflate_init(git_zstream *strm)\n {\n-\tint status = inflateInit(strm);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = inflateInit(&strm->z);\n+\tzlib_post_call(strm);\n \tif (status == Z_OK)\n \t\treturn;\n \tdie(\"inflateInit: %s (%s)\", zerr_to_string(status),\n-\t    strm->msg ? strm->msg : \"no message\");\n+\t    strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-void git_inflate_init_gzip_only(z_streamp strm)\n+void git_inflate_init_gzip_only(git_zstream *strm)\n {\n \t/*\n \t * Use default 15 bits, +16 is to accept only gzip and to\n \t * yield Z_DATA_ERROR when fed zlib format.\n \t */\n \tconst int windowBits = 15 + 16;\n-\tint status = inflateInit2(strm, windowBits);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = inflateInit2(&strm->z, windowBits);\n+\tzlib_post_call(strm);\n \tif (status == Z_OK)\n \t\treturn;\n \tdie(\"inflateInit2: %s (%s)\", zerr_to_string(status),\n-\t    strm->msg ? strm->msg : \"no message\");\n+\t    strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-void git_inflate_end(z_streamp strm)\n+void git_inflate_end(git_zstream *strm)\n {\n-\tint status = inflateEnd(strm);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = inflateEnd(&strm->z);\n+\tzlib_post_call(strm);\n \tif (status == Z_OK)\n \t\treturn;\n \terror(\"inflateEnd: %s (%s)\", zerr_to_string(status),\n-\t      strm->msg ? strm->msg : \"no message\");\n+\t      strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-int git_inflate(z_streamp strm, int flush)\n+int git_inflate(git_zstream *strm, int flush)\n {\n-\tint status = inflate(strm, flush);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = inflate(&strm->z, flush);\n+\tzlib_post_call(strm);\n \tswitch (status) {\n \t/* Z_BUF_ERROR: normal, needs more space in the output buffer */\n \tcase Z_BUF_ERROR:\n@@ -74,7 +119,7 @@ int git_inflate(z_streamp strm, int flush)\n \t\tbreak;\n \t}\n \terror(\"inflate: %s (%s)\", zerr_to_string(status),\n-\t      strm->msg ? strm->msg : \"no message\");\n+\t      strm->z.msg ? strm->z.msg : \"no message\");\n \treturn status;\n }\n \n@@ -82,56 +127,74 @@ int git_inflate(z_streamp strm, int flush)\n #define deflateBound(c,s)  ((s) + (((s) + 7) >> 3) + (((s) + 63) >> 6) + 11)\n #endif\n \n-unsigned long git_deflate_bound(z_streamp strm, unsigned long size)\n+unsigned long git_deflate_bound(git_zstream *strm, unsigned long size)\n {\n-\treturn deflateBound(strm, size);\n+\treturn deflateBound(&strm->z, size);\n }\n \n-void git_deflate_init(z_streamp strm, int level)\n+void git_deflate_init(git_zstream *strm, int level)\n {\n-\tint status = deflateInit(strm, level);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = deflateInit(&strm->z, level);\n+\tzlib_post_call(strm);\n \tif (status == Z_OK)\n \t\treturn;\n \tdie(\"deflateInit: %s (%s)\", zerr_to_string(status),\n-\t    strm->msg ? strm->msg : \"no message\");\n+\t    strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-void git_deflate_init_gzip(z_streamp strm, int level)\n+void git_deflate_init_gzip(git_zstream *strm, int level)\n {\n \t/*\n \t * Use default 15 bits, +16 is to generate gzip header/trailer\n \t * instead of the zlib wrapper.\n \t */\n \tconst int windowBits = 15 + 16;\n-\tint status = deflateInit2(strm, level,\n+\tint status;\n+\n+\tzlib_pre_call(strm);\n+\tstatus = deflateInit2(&strm->z, level,\n \t\t\t\t  Z_DEFLATED, windowBits,\n \t\t\t\t  8, Z_DEFAULT_STRATEGY);\n+\tzlib_post_call(strm);\n \tif (status == Z_OK)\n \t\treturn;\n \tdie(\"deflateInit2: %s (%s)\", zerr_to_string(status),\n-\t    strm->msg ? strm->msg : \"no message\");\n+\t    strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-void git_deflate_end(z_streamp strm)\n+void git_deflate_end(git_zstream *strm)\n {\n-\tint status = deflateEnd(strm);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = deflateEnd(&strm->z);\n+\tzlib_post_call(strm);\n \tif (status == Z_OK)\n \t\treturn;\n \terror(\"deflateEnd: %s (%s)\", zerr_to_string(status),\n-\t      strm->msg ? strm->msg : \"no message\");\n+\t      strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-int git_deflate_end_gently(z_streamp strm)\n+int git_deflate_end_gently(git_zstream *strm)\n {\n-\treturn deflateEnd(strm);\n+\tint status;\n+\n+\tzlib_pre_call(strm);\n+\tstatus = deflateEnd(&strm->z);\n+\tzlib_post_call(strm);\n+\treturn status;\n }\n \n-int git_deflate(z_streamp strm, int flush)\n+int git_deflate(git_zstream *strm, int flush)\n {\n-\tint status = deflate(strm, flush);\n+\tint status;\n \n+\tzlib_pre_call(strm);\n+\tstatus = deflate(&strm->z, flush);\n+\tzlib_post_call(strm);\n \tswitch (status) {\n \t/* Z_BUF_ERROR: normal, needs more space in the output buffer */\n \tcase Z_BUF_ERROR:\n@@ -145,6 +208,6 @@ int git_deflate(z_streamp strm, int flush)\n \t\tbreak;\n \t}\n \terror(\"deflate: %s (%s)\", zerr_to_string(status),\n-\t      strm->msg ? strm->msg : \"no message\");\n+\t      strm->z.msg ? strm->z.msg : \"no message\");\n \treturn status;\n }\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169866","messageId":"BANLkTinz9VAL_j9hZeF=kuAPwU1+0yEyRw@mail.gmail.com","threadId":"27601","inReplyTo":"1307736948-16956-5-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 4/6] zlib: wrap deflate side of the API","fromName":"Thiago Farina","fromEmail":"tfransosi@gmail.com","sentAt":"2011-06-10T22:23:47Z","receivedAt":"2011-06-10T22:23:47Z","isPatch":true,"sender":{"key":"tfransosi@gmail.com","avatar":"https://avatars.githubusercontent.com/u/970071?v=4"},"body":"On Fri, Jun 10, 2011 at 5:15 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> +void git_deflate_init_gzip(z_streamp strm, int level)\n> +{\n> +       /*\n> +        * Use default 15 bits, +16 is to generate gzip header/trailer\n> +        * instead of the zlib wrapper.\n> +        */\n> +       const int windowBits = 15 + 16;\nWas this style intentional?\n"},{"id":"169868","messageId":"7vvcwduwaf.fsf@alter.siamese.dyndns.org","threadId":"27601","inReplyTo":"BANLkTinz9VAL_j9hZeF=kuAPwU1+0yEyRw@mail.gmail.com","subject":"Re: [PATCH 4/6] zlib: wrap deflate side of the API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T23:00:24Z","receivedAt":"2011-06-10T23:00:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thiago Farina <tfransosi@gmail.com> writes:\n\n> On Fri, Jun 10, 2011 at 5:15 PM, Junio C Hamano <gitster@pobox.com> wrote:\n>> +void git_deflate_init_gzip(z_streamp strm, int level)\n>> +{\n>> +       /*\n>> +        * Use default 15 bits, +16 is to generate gzip header/trailer\n>> +        * instead of the zlib wrapper.\n>> +        */\n>> +       const int windowBits = 15 + 16;\n> Was this style intentional?\n\nSorry, but what style specifically do you find questionable?\n"},{"id":"169870","messageId":"7vei31uu38.fsf@alter.siamese.dyndns.org","threadId":"27601","inReplyTo":"1307736948-16956-1-git-send-email-gitster@pobox.com","subject":"[PATCH 7/6] zlib: allow feeding more than 4GB in one go","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-10T23:47:55Z","receivedAt":"2011-06-10T23:47:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> The next step would be to tweak zlib_post_call(), git_inflate() and\n> git_deflate() functions to internally loop and call underlying inflate()\n> and deflate() when the incoming buffers are larger than 4GB, but that\n> part is not done in this series (yet).\n\nAnd this is such a patch.\n\nI had a draft of this one ready when I sent the previous 6-patch series,\nbut it took embarrassingly long time for me to figure out what went wrong\nin this patch before I realized that Z_FINISH should not be relayed to the\nunderlying inflate/deflate if we are splitting the request into multiple\nphases.\n\nI consider it still rough, but with this on top of v1.7.5.4, together with\nthe previous 6 patches, it seems to pass a trivial \"have a large blob,\nadd, commit, pack, verify, index-pack, fsck\" tests, without using either\nof the recent \"let's pass large blob to fast-import\" (in 'master') nor\n\"let's stream a blob out without expanding it in core first\" (in 'next')\nchanges.\n\n-- test script to be stored in /var/tmp/junk or somewhere --\n#!/bin/sh\n\nset -x\n\nrm -fr .git && git init || exit\n\necho | dd bs=1M seek=4800 of=a.big\nsleep 2\ngit add a.big\n\neval $(git ls-files -s a.big |\n    sed -e 's/^[0-7]* \\([0-9a-f][0-9a-f]\\)\\([0-9a-f]*\\) .*/h=\\1 a=\\2/')\necho $h $a\n\nls -l .git/objects/$h/$a || exit\n\ngit commit -m initial || exit\n\ngit fsck || exit\n\ngit repack -a -d || exit\n\ngit fsck || exit\n\nf=$(find .git/objects/?? -type f)\ntest -z \"$f\" || exit\n\np=$(find .git/objects/pack -type f -name '*.pack') &&\ni=${p%.pack}.idx &&\n\ntest -f $p &&\ntest -f $i || exit\n\nmv $p $i . &&\npp=$(basename $p) &&\nii=$(basename $i) &&\ntest -f $pp &&\ntest -f $ii || exit\n\ngit verify-pack $pp || exit\n\ngit unpack-objects <$pp || exit\n\nls -l .git/objects/$h/$a || exit\n\nrm -fr .git/objects/?? || exit\n\ngit index-pack --stdin <$pp || exit\ntest -f \"$i\" || exit\ncmp $i $ii || exit\n\ngit cat-file blob \"$h$a\" >b.big || exit\ncmp a.big b.big\n\necho all done.\nexit 0\n-- test script ends here --\n\n-- >8 --\nUpdate zlib_post_call() that adjusts the wrapper's notion of avail_in and\navail_out to what came back from zlib, so that the callers can feed\nbuffers larger than than 4GB to the API.\n\nWhen underlying inflate/deflate stopped processing because we fed a buffer\nlarger than 4GB limit, detect that case, update the state variables, and\nlet the zlib function work another round.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n zlib.c |   78 +++++++++++++++++++++++++++++++++++++++++++++++----------------\n 1 files changed, 58 insertions(+), 20 deletions(-)\n\ndiff --git a/zlib.c b/zlib.c\nindex fe537e3..3c63d48 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -27,12 +27,11 @@ static const char *zerr_to_string(int status)\n  * limits the size of the buffer we can use to 4GB when interacting\n  * with zlib in a single call to inflate/deflate.\n  */\n-#define ZLIB_BUF_MAX ((uInt)-1)\n+/* #define ZLIB_BUF_MAX ((uInt)-1) */\n+#define ZLIB_BUF_MAX ((uInt) 1024 * 1024 * 1024) /* 1GB */\n static inline uInt zlib_buf_cap(unsigned long len)\n {\n-\tif (ZLIB_BUF_MAX < len)\n-\t\tdie(\"working buffer for zlib too large\");\n-\treturn len;\n+\treturn (ZLIB_BUF_MAX < len) ? ZLIB_BUF_MAX : len;\n }\n \n static void zlib_pre_call(git_zstream *s)\n@@ -47,12 +46,22 @@ static void zlib_pre_call(git_zstream *s)\n \n static void zlib_post_call(git_zstream *s)\n {\n+\tunsigned long bytes_consumed;\n+\tunsigned long bytes_produced;\n+\n+\tbytes_consumed = s->z.next_in - s->next_in;\n+\tbytes_produced = s->z.next_out - s->next_out;\n+\tif (s->z.total_out != s->total_out + bytes_produced)\n+\t\tdie(\"BUG: total_out mismatch\");\n+\tif (s->z.total_in != s->total_in + bytes_consumed)\n+\t\tdie(\"BUG: total_in mismatch\");\n+\n+\ts->total_out = s->z.total_out;\n+\ts->total_in = s->z.total_in;\n \ts->next_in = s->z.next_in;\n \ts->next_out = s->z.next_out;\n-\ts->total_in = s->z.total_in;\n-\ts->total_out = s->z.total_out;\n-\ts->avail_in = s->z.avail_in;\n-\ts->avail_out = s->z.avail_out;\n+\ts->avail_in -= bytes_consumed;\n+\ts->avail_out -= bytes_produced;\n }\n \n void git_inflate_init(git_zstream *strm)\n@@ -103,18 +112,32 @@ int git_inflate(git_zstream *strm, int flush)\n {\n \tint status;\n \n-\tzlib_pre_call(strm);\n-\tstatus = inflate(&strm->z, flush);\n-\tzlib_post_call(strm);\n+\tfor (;;) {\n+\t\tzlib_pre_call(strm);\n+\t\t/* Never say Z_FINISH unless we are feeding everything */\n+\t\tstatus = inflate(&strm->z,\n+\t\t\t\t (strm->z.avail_in != strm->avail_in)\n+\t\t\t\t ? 0 : flush);\n+\t\tif (status == Z_MEM_ERROR)\n+\t\t\tdie(\"inflate: out of memory\");\n+\t\tzlib_post_call(strm);\n+\n+\t\t/*\n+\t\t * Let zlib work another round, while we can still\n+\t\t * make progress.\n+\t\t */\n+\t\tif ((strm->avail_out && !strm->z.avail_out) &&\n+\t\t    (status == Z_OK || status == Z_BUF_ERROR))\n+\t\t\tcontinue;\n+\t\tbreak;\n+\t}\n+\n \tswitch (status) {\n \t/* Z_BUF_ERROR: normal, needs more space in the output buffer */\n \tcase Z_BUF_ERROR:\n \tcase Z_OK:\n \tcase Z_STREAM_END:\n \t\treturn status;\n-\n-\tcase Z_MEM_ERROR:\n-\t\tdie(\"inflate: out of memory\");\n \tdefault:\n \t\tbreak;\n \t}\n@@ -192,18 +215,33 @@ int git_deflate(git_zstream *strm, int flush)\n {\n \tint status;\n \n-\tzlib_pre_call(strm);\n-\tstatus = deflate(&strm->z, flush);\n-\tzlib_post_call(strm);\n+\tfor (;;) {\n+\t\tzlib_pre_call(strm);\n+\n+\t\t/* Never say Z_FINISH unless we are feeding everything */\n+\t\tstatus = deflate(&strm->z,\n+\t\t\t\t (strm->z.avail_in != strm->avail_in)\n+\t\t\t\t ? 0 : flush);\n+\t\tif (status == Z_MEM_ERROR)\n+\t\t\tdie(\"deflate: out of memory\");\n+\t\tzlib_post_call(strm);\n+\n+\t\t/*\n+\t\t * Let zlib work another round, while we can still\n+\t\t * make progress.\n+\t\t */\n+\t\tif ((strm->avail_out && !strm->z.avail_out) &&\n+\t\t    (status == Z_OK || status == Z_BUF_ERROR))\n+\t\t\tcontinue;\n+\t\tbreak;\n+\t}\n+\n \tswitch (status) {\n \t/* Z_BUF_ERROR: normal, needs more space in the output buffer */\n \tcase Z_BUF_ERROR:\n \tcase Z_OK:\n \tcase Z_STREAM_END:\n \t\treturn status;\n-\n-\tcase Z_MEM_ERROR:\n-\t\tdie(\"deflate: out of memory\");\n \tdefault:\n \t\tbreak;\n \t}\n-- \n1.7.6.rc1.118.ge175b4a\n"},{"id":"169900","messageId":"BANLkTikmLDZj2qdkmF-kBUkB33o9EjtBpg@mail.gmail.com","threadId":"27601","inReplyTo":"1307736948-16956-7-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2011-06-12T20:43:42Z","receivedAt":"2011-06-12T20:43:42Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Fri, Jun 10, 2011 at 10:15 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> The size of objects we read from the repository and data we try to put\n> into the repository are represented in \"unsigned long\", so that on larger\n> architectures we can handle objects that weigh more than 4GB.\n\nshouldn't this be \"size_t\" instead of \"unsigned long\"? \"unsigned long\"\nis 32bit on Win64, whereas size_t is 64bit:\n\nhttp://msdn.microsoft.com/en-us/library/s3f49ktz(v=vs.80).aspx\nhttp://msdn.microsoft.com/en-us/library/aa384083(VS.85).aspx\n"},{"id":"169903","messageId":"7v4o3uspiy.fsf@alter.siamese.dyndns.org","threadId":"27601","inReplyTo":"BANLkTikmLDZj2qdkmF-kBUkB33o9EjtBpg@mail.gmail.com","subject":"Re: [PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-12T21:33:57Z","receivedAt":"2011-06-12T21:33:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Erik Faye-Lund <kusmabite@gmail.com> writes:\n\n> On Fri, Jun 10, 2011 at 10:15 PM, Junio C Hamano <gitster@pobox.com> wrote:\n>> The size of objects we read from the repository and data we try to put\n>> into the repository are represented in \"unsigned long\", so that on larger\n>> architectures we can handle objects that weigh more than 4GB.\n>\n> shouldn't this be \"size_t\" instead of \"unsigned long\"?\n\nNo, this must be unsigned long as that is the internal type we use.\n\nImplementation of git on 32-bit platforms has always been limited to 4GB\nfrom day one. This topic is not about changing it.\n"},{"id":"169904","messageId":"vpqfwneraea.fsf@bauges.imag.fr","threadId":"27601","inReplyTo":"7v4o3uspiy.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-12T21:46:05Z","receivedAt":"2011-06-12T21:46:05Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Implementation of git on 32-bit platforms has always been limited to 4GB\n> from day one. This topic is not about changing it.\n\nErik's remark was about Win64, not about 32-bit.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169931","messageId":"BANLkTi=sT_LxRaJSM3Cj-QkSwqGan29K7A@mail.gmail.com","threadId":"27601","inReplyTo":"7v4o3uspiy.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2011-06-13T11:17:00Z","receivedAt":"2011-06-13T11:17:00Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Sun, Jun 12, 2011 at 11:33 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Erik Faye-Lund <kusmabite@gmail.com> writes:\n>\n>> On Fri, Jun 10, 2011 at 10:15 PM, Junio C Hamano <gitster@pobox.com> wrote:\n>>> The size of objects we read from the repository and data we try to put\n>>> into the repository are represented in \"unsigned long\", so that on larger\n>>> architectures we can handle objects that weigh more than 4GB.\n>>\n>> shouldn't this be \"size_t\" instead of \"unsigned long\"?\n>\n> No, this must be unsigned long as that is the internal type we use.\n\nI'm not sure I even understand what you mean here. \"unsigned long\" is\nthe only choice because it's the type we use? That's sounds pretty\nclose to tautology to my ears.\n\nLooking a bit more at the code, it seems that we currently use\n\"unsigned long\" a lot of places where \"size_t\" should have been used.\nAnd this series is about changing places where \"unsigned int\" is being\nused instead of \"unsigned long\". This sounds backwards to me;\nshouldn't all code that deals with sizes (both the ones that are\n\"unsigned int\" AND the ones that are \"unsigned long\") be changed to\nsize_t instead?\n\n> Implementation of git on 32-bit platforms has always been limited to 4GB\n> from day one. This topic is not about changing it.\n\nAs Matthieu pointed out, my comment wasn't about 32-bit systems;\n\"unsigned long\" is 32-bit even on 64-bit versions on Windows. I\nbelieve TortoiseGit builds Git for Windows as 64-bit, so AFAIK this\nwould still be a problem for them.\n"},{"id":"169932","messageId":"20110613115222.GA10501@elie","threadId":"27601","inReplyTo":"BANLkTi=sT_LxRaJSM3Cj-QkSwqGan29K7A@mail.gmail.com","subject":"Re: [PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-06-13T11:52:22Z","receivedAt":"2011-06-13T11:52:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Erik Faye-Lund wrote:\n\n> Looking a bit more at the code, it seems that we currently use\n> \"unsigned long\" a lot of places where \"size_t\" should have been used.\n\nAs Junio's comment about 32-bit platforms hints, when it comes to be\nthe time to change all those places, wouldn't off_t be a better\nchoice?\n\nThanks --- it is exciting to watch this topic move forward, inch by\ninch.  And thanks for the hint about invalid word size == sizeof(long)\nassumptions on Windows.\n"},{"id":"169933","messageId":"7vsjreq707.fsf@alter.siamese.dyndns.org","threadId":"27601","inReplyTo":"BANLkTi=sT_LxRaJSM3Cj-QkSwqGan29K7A@mail.gmail.com","subject":"Re: [PATCH 6/6] zlib: zlib can only process 4GB at a time","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-13T11:56:56Z","receivedAt":"2011-06-13T11:56:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Erik Faye-Lund <kusmabite@gmail.com> writes:\n\n> On Sun, Jun 12, 2011 at 11:33 PM, Junio C Hamano <gitster@pobox.com> wrote:\n>> Erik Faye-Lund <kusmabite@gmail.com> writes:\n>>\n>>> On Fri, Jun 10, 2011 at 10:15 PM, Junio C Hamano <gitster@pobox.com> wrote:\n>>>> The size of objects we read from the repository and data we try to put\n>>>> into the repository are represented in \"unsigned long\", so that on larger\n>>>> architectures we can handle objects that weigh more than 4GB.\n>>>\n>>> shouldn't this be \"size_t\" instead of \"unsigned long\"?\n>>\n>> No, this must be unsigned long as that is the internal type we use.\n\nThere are two unrelated issues you have to address if your \"unsigned long\"\nis 32-bit and you want to handle more than 4GB data in git.\n\nWhen git holds repository data in core, it always has represented it as a\npair of <pointer to the beginning of memory block that holds data, length>\nwhere the length is \"unsigned long\" from day one.  See read_sha1_file() in\nread-cache.c that appears in e83c516 (Initial revision of \"git\", the\ninformation manager from hell, 2005-04-07). This limits you to 4GB if your\n\"unsigned long\" is 32-bit.\n\nThe right type to use in order to enable more platforms to go beyond 4GB\nmight be to use uintmax_t, but the series you are commenting on however is\nnot about changing that.\n\nWe have another problem stemming from the way in which we incorrectly used\nzlib API even on a platform where \"unsigned long\" is capable to express\nsize beyond 4GB. In many places, we set up the state object used by zlib\nAPI (i.e. z_stream) to point at the \"pointer to the beginning of memory\nblock\" with its \"next_in\" field, and \"length\" with its \"avail_in\" field,\npass that object around in the callchain, and expect that by making\nrepeated call to zlib, \"next_in\" would eventually progress to the end of\nthe data we have in core while \"avail_in\" would fall to zero when all data\nis processed. The \"avail_in\" field zlib API gives us however is uInt which\nis 32-bit, so this expectation is incorrect. If you have 4G+32 bytes of\ndata, for example, we only feed 32 bytes and stop, barfing on \"corrupt\"\ndata.\n\nThat is the issue this series is about. The approach of the series takes\nis to wrap zlib's state object with our own, that has our own \"avail_in\"\nfield (by the way, the same issue exists in \"next_out/avail_out\" on the\noutput side) that uses the same type of \"length\" used in other parts of\nour system.\n\nThe type of the \"avail_in\" and \"avail_out\" fields in the wrapper needs to\nbe updated to match that type when you address the \"other\" issue to update\nall the internal \"length\" from \"unsigned long\" to \"uintmax_t\", but not\nbefore. And updating the rest of the system to \"uintmax_t\" is not part of\nthe scope of this series.\n"}]}