git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/4] archive-zip: in write_zip_entry() call git_deflate() in a loop

From
René Scharfe <l.s.r@web.de>
Date
Aug 7, 2025, 19:51 UTC
Message-ID
<da21879f-5a73-41e1-9fc9-1ab128d8717d@web.de>
In-Reply-To
<20250804-toon-archive-zip-fix-v1-3-ca89858e5eaa@iotcl.com>
On 8/4/25 6:56 PM, Toon Claes wrote:
Show 49 quoted lines
> The function git_deflate() might not complete to deflate all the input
> data in one go. While the function is already being called in a loop,
> every loop fresh data is read from the stream. This is not correct,
> because input data might get lost.
> 
> As we see in many other callsites, git_deflate() should be called in a
> loop on the existing input to make it process all the input data.
> 
> Add in a nested loop around git_deflate() to process the input buffer
> completely, before continuing the parent loop that reads from more data
> from the input stream.
> 
> Co-authored-by: Justin Tobler <jltobler@gmail.com>
> Signed-off-by: Toon Claes <toon@iotcl.com>
> ---
>  archive-zip.c | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
> 
> diff --git a/archive-zip.c b/archive-zip.c
> index d41a12de5f..25a0224130 100644
> --- a/archive-zip.c
> +++ b/archive-zip.c
> @@ -471,15 +471,17 @@ static int write_zip_entry(struct archiver_args *args,
>  
>  			zstream.next_in = buf;
>  			zstream.avail_in = readlen;
> -			zstream.next_out = compressed;
> -			zstream.avail_out = sizeof(compressed);
> -			result = git_deflate(&zstream, 0);
> -			if (result != Z_OK)
> -				die(_("deflate error (%d)"), result);
> -			out_len = zstream.next_out - compressed;
> +			do {
> +				zstream.next_out = compressed;
> +				zstream.avail_out = sizeof(compressed);
> +				result = git_deflate(&zstream, 0);
> +				if (result != Z_OK)
> +					die(_("deflate error (%d)"), result);
> +				out_len = zstream.next_out - compressed;
>  
> -			write_or_die(1, compressed, out_len);
> -			compressed_size += out_len;
> +				write_or_die(1, compressed, out_len);
> +				compressed_size += out_len;
> +			} while (zstream.avail_out == 0);
>  		}
>  		close_istream(stream);
>  		if (readlen)
> 

Makes sense. If deflate somehow fills the output buffer (with internally pending data, I suppose -- the fresh input data alone is not enough), this clears it and lets it go another round without feeding new input.

The mistake was thinking that the existence of deflateBound(), which gives a maximum deflated size for any given input, implies a similarly tight limit on individual chunks of input, which makes no sense in hindsight.

René
Previous: Toon ClaesNext: Toon Claes
Message 7 of 9 in “Fix archiving some corner-case files into zip”
  1. 0/4 Fix archiving some corner-case files into zipToon Claes, Aug 4, 2025
  2. 1/4 archive-zip: deduplicate code setting output buffer in write_zip_entry()Toon Claes, Aug 4, 2025
  3. René ScharfeAug 7, 2025
  4. 2/4 archive-zip: remove unneccesarry condition in write_zip_entry()Toon Claes, Aug 4, 2025
  5. René ScharfeAug 7, 2025
  6. 3/4 archive-zip: in write_zip_entry() call git_deflate() in a loopToon Claes, Aug 4, 2025
  7. René ScharfeAug 7, 2025
  8. 4/4 archive-zip: move git_deflate() with Z_FINISH into the loopToon Claes, Aug 4, 2025
  9. René ScharfeAug 7, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.