[PATCH] unpack-objects: zlib can only process 4GB at a time
- From
Junio C Hamano <gitster@pobox.com>
- Date
- Jun 9, 2011, 20:11 UTC
- Message-ID
- <7vzklqwyr5.fsf@alter.siamese.dyndns.org>
- In-Reply-To
- <7v1uz42liw.fsf@alter.siamese.dyndns.org>
This codepath tried to tell zlib that we have allocated a large enough output buffer and expected a single call to inflate() to succeed, but zlib cannot expand into a buffer that is larger than 4GB at a time.
Ideally we should be transferring a better cue between send-pack and receive-pack (or fetch-pack and upload-pack) to tell that we would be better off not to explode into individual loose objects, but that is not an excuse to leave our use of zlib broken and fail to unpack large objects in a pack even on a beefy machine with enough memory.
Signed-off-by: Junio C Hamano <gitster@pobox.com> --- builtin/unpack-objects.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c index f63973c..b208d6e 100644 --- a/builtin/unpack-objects.c +++ b/builtin/unpack-objects.c @@ -92,18 +92,22 @@ static void *get_data(unsigned long size) { z_stream stream; void *buf = xmalloc(size); + unsigned long bytes_to_inflate; memset(&stream, 0, sizeof(stream)); stream.next_out = buf; - stream.avail_out = size; + bytes_to_inflate = size; + stream.avail_out = zlib_buf_cap(bytes_to_inflate); stream.next_in = fill(1); stream.avail_in = len; git_inflate_init(&stream); for (;;) { + unsigned char *out0 = stream.next_out; int ret = git_inflate(&stream, 0); use(len - stream.avail_in); + bytes_to_inflate -= (stream.next_out - out0); if (stream.total_out == size && ret == Z_STREAM_END) break; if (ret != Z_OK) { @@ -117,6 +121,7 @@ static void *get_data(unsigned long size) } stream.next_in = fill(1); stream.avail_in = len; + stream.avail_out = zlib_buf_cap(bytes_to_inflate); } git_inflate_end(&stream); return buf;
-- 1.7.6.rc1.118.ge175b4a