Re: bug: git-archive does not use the zip64 extension for archives with more than 16k entries

From: Johannes Schauer <josch@debian.org>
Date: Aug 13, 2015, 02:25 UTC
Message-ID: <20150813022545.30116.44787@localhost>
In-Reply-To: <55CBA140.7050301@web.de>

Hi,

Quoting René Scharfe (2015-08-12 21:40:48)

Show 8 quoted lines

> Am 11.08.2015 um 12:40 schrieb Johannes Schauer:
> > for repositories with more than 16k files and folders, git-archive will create
> > zip files which store the wrong number of entries. That is, it stores the
> > number of entries modulo 16k. This will break unpackers that do not include
> > code to support this brokenness.
> 
> The limit is rather 65535 entries, isn't it?  The entries field has two 
> bytes, and they are used fully.

seems to be 65535 indeed.

I just forwarded the number Dieter Baron (libzip contributor) told me when they replied (off list) to my bug report against libzip: http://nih.at/listarchive/libzip-discuss/msg00554.html

But reading https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64 the limit indeed seems to be 65535.

> Which programs are affected? InfoZIP Zip 3.0 and 7-Zip 9.20 seem to handle an
> archive with more entries just fine.  The built-in functionality of Windows
> 10 doesn't.

In my case I discovered this because libzip http://nih.at/libzip does not implement reading an archive with more than 65535 entries without zip64.

> Besides, 64K entries should be enough for anybody. ;-)

:P

> Seriously, though: What kind of repository has that many files and uses the
> ZIP format to distribute snapshots?  Just curious.

I have not searched for any.

In my case I was using git to keep track of the modifications our tools do to a directory of files to detect regressions. These tools are also able to read data from zip archives instead from a directory. I created the zip archive using git-archive because the files already were in git so that seemed most convenient to me. That's when I discovered the problem because our tools use libzip. The easy workaround was to use another packager instead of git-archive. We use the zip format because Windows has support for it.

Show 6 quoted lines

> > Instead, git-archive should use the zip64 extension to handle more than 16k
> > files and folders correctly.
> 
> That seems to be what InfoZIP does and Windows 10 handles it just fine. If
> lower Windows versions and other popular extractors can unzip such archives
> as well then this might indeed be the way to go.

The wikipedia page above claims that windows versions starting with vista have support for zip64. It also lists some other software with support for it.

Thanks!

cheers, josch