git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/6] zlib only processes 4GB at a time

From
Junio C Hamano <gitster@pobox.com>
Date
Jun 10, 2011, 20:15 UTC
Message-ID
<1307736948-16956-1-git-send-email-gitster@pobox.com>

This is a re-roll from yesterday's series, taking a very different approach. Instead of fixing call sites one by one, the goal is to define our own API that can give an illusion to the callers that they can feed and receive data whose size can fit within "unsigned long", so that they can set up the avail_in field of z_stream (now git_zstream) to the total size, and expect it to fall down to zero when everything is consumed by the library (vice versa for avail_out).

The first 5 patches clean up the existing callers that directly call into zlib to call git_inflate and git_deflate wrappers.

Patch 6 introduces git_zstream that is a thin wrapper around z_stream, but barfs when the caller gives a buffer larger than 4GB that would fit the underlying zlib calling convention.

The next step would be to tweak zlib_post_call(), git_inflate() and git_deflate() functions to internally loop and call underlying inflate() and deflate() when the incoming buffers are larger than 4GB, but that part is not done in this series (yet).

Junio C Hamano (6):
  zlib: refactor error message formatter
  zlib: wrap remaining calls to direct inflate/inflateEnd
  zlib: wrap inflateInit2 used to accept only for gzip format
  zlib: wrap deflate side of the API
  zlib: wrap deflateBound() too
  zlib: zlib can only process 4GB at a time
 archive-zip.c            |   10 +-
 builtin/apply.c          |    2 +-
 builtin/index-pack.c     |   12 ++--
 builtin/pack-objects.c   |   18 ++--
 builtin/unpack-objects.c |    2 +-
 cache.h                  |   30 +++++--
 diff.c                   |   10 +-
 fast-import.c            |   30 +++---
 http-backend.c           |   11 +--
 http-push.c              |   16 ++--
 http.h                   |    2 +-
 pack-check.c             |    4 +-
 remote-curl.c            |   14 +--
 sha1_file.c              |   28 +++---
 zlib.c                   |  218 +++++++++++++++++++++++++++++++++++++++-------
 15 files changed, 283 insertions(+), 124 deletions(-)
-- 
1.7.6.rc1.118.ge175b4a
Next: Junio C Hamano
Message 1 of 16 in “zlib only processes 4GB at a time”
  1. 0/6 zlib only processes 4GB at a timeJunio C Hamano, Jun 10, 2011
  2. 1/6 zlib: refactor error message formatterJunio C Hamano, Jun 10, 2011
  3. 2/6 zlib: wrap remaining calls to direct inflate/inflateEndJunio C Hamano, Jun 10, 2011
  4. 3/6 zlib: wrap inflateInit2 used to accept only for gzip formatJunio C Hamano, Jun 10, 2011
  5. 4/6 zlib: wrap deflate side of the APIJunio C Hamano, Jun 10, 2011
  6. Thiago FarinaJun 10, 2011
  7. Junio C HamanoJun 10, 2011
  8. 5/6 zlib: wrap deflateBound() tooJunio C Hamano, Jun 10, 2011
  9. 6/6 zlib: zlib can only process 4GB at a timeJunio C Hamano, Jun 10, 2011
  10. Erik Faye-LundJun 12, 2011
  11. Junio C HamanoJun 12, 2011
  12. Matthieu MoyJun 12, 2011
  13. Erik Faye-LundJun 13, 2011
  14. Jonathan NiederJun 13, 2011
  15. Junio C HamanoJun 13, 2011
  16. 7/6 zlib: allow feeding more than 4GB in one goJunio C Hamano, Jun 10, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.