Re: [PATCH v3 4/5] archive-zip: support archives bigger than 4GB

From: René Scharfe <l.s.r@web.de>
Date: Apr 27, 2017, 19:54 UTC
Message-ID: <a9cb6572-500e-bbc6-2aac-7cb940d4b171@web.de>
In-Reply-To: <alpine.DEB.2.11.1704270552590.4681@perkele.intern.softwolves.pp.se>

Am 27.04.2017 um 06:57 schrieb Peter Krefting:

> René Scharfe:
>> Windows XP.  Don't laugh. ;)
>
> You can always install 7-zip or something to extract on XP.

Sure, but if we were to start emitting zip64 records regardless of the size of entries then we'd break compatibility. We should have a very good reason for doing that. (I don't see the need so far.)

Show 5 quoted lines

>> What do other archivers do?
> 
> You should compare with what bsdtar (libarchive) does in zip64 mode. It 
> also only ever does streaming mode (with data descriptors and such), and 
> it does zip64.

Good idea. They do the same as InfoZIP (whose behavior I copied in the series), i.e. emit zip64 records only for big files by default:

https://github.com/libarchive/libarchive/blob/master/libarchive/archive_write_set_format_zip.c#L722

They set the bar much higher for the uncompressed size of streamed files; they emit zip64 extras only for sizes bigger than 0xff000000.

Show 7 quoted lines

>> But I think a more important question is: Can the generated files be 
>> extracted by popular tools (most importantly Windows' built-in 
>> functionality, I guess)?
> 
> OK, so only enable zip64 mode if there are files >4G or the archive ends 
> up being >4G. But the question is how we can tell, especially in 
> streaming mode, and especially if data descriptors are magical...

The type of descriptor to use depends on the presence of 64-bit sizes in a zip64 extra for that record. For streaming compression some kind of threshold lower than 0xffffffff needs to be set, because deflate can increase the size of the result.

René