Re: [PATCH 1/3] t5004: test ZIP archives with many entries

> Am 23.08.2015 um 07:54 schrieb Eric Sunshine:
>> On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe <l.s.r@web.de> wrote:
>>> +test_lazy_prereq ZIPINFO '
>>> +       n=$("$ZIPINFO" "$TEST_DIRECTORY"/t5004/empty.zip | sed -n "2s/.* //p")
>>> +       test "x$n" = "x0"
>>> +'
>>
>> Unfortunately, this sed expression isn't portable due to dissimilar
>> output of various zipinfo implementations. On Linux, the output of
>> zipinfo is:
>>
>>      $ zipinfo t/t5004/empty.zip
>>      Archive:  t/t5004/empty.zip
>>      Zip file size: 62 bytes, number of entries: 0
>>      Empty zipfile.
>>      $
>>
>> however, on Mac OS X:
>>
>>      $ zipinfo t/t5004/empty.zip
>>      Archive:  t/t5004/empty.zip   62 bytes   0 files
>>      Empty zipfile.
>>      $
>>
>> and on FreeBSD, the zipinfo command seems to have been removed
>> altogether in favor of "unzip -Z" (emulate zipinfo).
>
> I suspected that zipinfo's output might be formatted differently on
> different platforms and tried to guard against it by checking for the
> number zero there. Git's ZIP file creation is platform independent
> (modulo bugs), so having a test run at least somewhere should
> suffice. In theory.
>
> We could add support for the one-line-summary variant on OS X easily,
> though.

>> One might hope that "unzip -Z" would be a reasonable replacement for
>> zipinfo, however, it is apparently only partially implemented on
>> FreeBSD, and requires that -1 be passed, as well. Even with "unzip -Z
>> -1", there are issues. The output on Linux and Mac OS X is:
>>
>>      $ unzip -Z -1 t/t5004/empty.zip
>>      Empty zipfile.
>>      $
>>
>> but FreeBSD differs:
>>
>>      $ unzip -Z -1 t/t5004/empty.zip
>>      $
>>
>> With a non-empty zip file, the output is identical on all platforms:
>>
>>      $ unzip -Z -1 twofiles.zip
>>      file1
>>      file2
>>      $
>>
>> So, if you combine that with "wc -l" or test_line_count, you may have
>> a portable and reliable entry counter.
>
> Counting all entries is slow, and more importantly it's not what we
> want. In this test we need the number of entries recorded in the ZIP
> directory, not the actual number of entries found by scanning the
> archive, or the directory.

> On Linux "unzip -Z -1 many.zip | wc -l" reports 65792 even before
> adding ZIP64 support; only without -1 we get the interesting numbers
> (specifically with "unzip -Z many.zip | sed -n '2p;$p'"):
>
>     Zip file size: 6841366 bytes, number of entries: 256
>     65792 files, 0 bytes uncompressed, 0 bytes compressed: 0.0%
>
>> With these three patches applied, Mac OS X has trouble with 'many.zip':
>>
>>      $ unzip -Z -1 many.zip
>>      warning [many.zip]:  76 extra bytes at beginning or within zipfile
>>        (attempting to process anyway)
>>      error [many.zip]:  reported length of central directory is
>>        -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
>>        zipfile?).  Compensating...
>>      00/
>>      00/00
>>      ...
>>      ff/ff
>>      error: expected central file header signature not found (file
>>        #65793). (please check that you have transferred or created the
>>        zipfile in the appropriate BINARY mode and that you have compiled
>>        UnZip properly)
>>
>> And FreeBSD doesn't like it either:
>>
>>      $ unzip -Z -1 many.zip
>>      unzip: Invalid central directory signature
>>      $
>>
>
> Looks like they don't support ZIP64. Or I got some of the fields wrong
> after all.

> https://en.wikipedia.org/wiki/Zip_%28file_format%29#ZIP64 says: "OS X
> Yosemite does support the creation of ZIP64 archives, but does not
> support unzipping these archives using the shipped unzip command-line
> utility or graphical Archive Utility.[citation needed]".
>
> How does unzip react to a ZIP file with more than 65535 entries that
> was created natively on these platforms? And what does zipinfo (a real
> one, without -1) report at the top for such files?