Re: [PATCH 1/3] t5004: test ZIP archives with many entries

From: René Scharfe <l.s.r@web.de>
Date: Aug 23, 2015, 09:29 UTC
Message-ID: <trinity-6e67d416-0a61-4e73-9779-63519dd83fdb-1440322151491@3capp-webde-bs47>
In-Reply-To: <CAPig+cSy+c9mOGOTN9e4xfLrvPc8nv7e0T_4PDA-vB-otwrvjw@mail.gmail.com>

Am 23.08.2015 um 07:54 schrieb Eric Sunshine:

Show 35 quoted lines

> On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe <l.s.r@web.de> wrote:
>> diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh
>> index 654adda..c6bd729 100755
>> --- a/t/t5004-archive-corner-cases.sh
>> +++ b/t/t5004-archive-corner-cases.sh
>> @@ -115,4 +115,44 @@ test_expect_success 'archive empty subtree by direct pathspec' '
>>          check_dir extract sub
>>   '
>>
>> +ZIPINFO=zipinfo
>> +
>> +test_lazy_prereq ZIPINFO '
>> +       n=$("$ZIPINFO" "$TEST_DIRECTORY"/t5004/empty.zip | sed -n "2s/.* //p")
>> +       test "x$n" = "x0"
>> +'
> 
> Unfortunately, this sed expression isn't portable due to dissimilar
> output of various zipinfo implementations. On Linux, the output of
> zipinfo is:
> 
>      $ zipinfo t/t5004/empty.zip
>      Archive:  t/t5004/empty.zip
>      Zip file size: 62 bytes, number of entries: 0
>      Empty zipfile.
>      $
> 
> however, on Mac OS X:
> 
>      $ zipinfo t/t5004/empty.zip
>      Archive:  t/t5004/empty.zip   62 bytes   0 files
>      Empty zipfile.
>      $
> 
> and on FreeBSD, the zipinfo command seems to have been removed
> altogether in favor of "unzip -Z" (emulate zipinfo).

Thanks for your thorough checks!

I suspected that zipinfo's output might be formatted differently on different platforms and tried to guard against it by checking for the number zero there. Git's ZIP file creation is platform independent (modulo bugs), so having a test run at least somewhere should suffice. In theory.

We could add support for the one-line-summary variant on OS X easily, though.

Show 23 quoted lines

> One might hope that "unzip -Z" would be a reasonable replacement for
> zipinfo, however, it is apparently only partially implemented on
> FreeBSD, and requires that -1 be passed, as well. Even with "unzip -Z
> -1", there are issues. The output on Linux and Mac OS X is:
> 
>      $ unzip -Z -1 t/t5004/empty.zip
>      Empty zipfile.
>      $
> 
> but FreeBSD differs:
> 
>      $ unzip -Z -1 t/t5004/empty.zip
>      $
> 
> With a non-empty zip file, the output is identical on all platforms:
> 
>      $ unzip -Z -1 twofiles.zip
>      file1
>      file2
>      $
> 
> So, if you combine that with "wc -l" or test_line_count, you may have
> a portable and reliable entry counter.

Counting all entries is slow, and more importantly it's not what we want. In this test we need the number of entries recorded in the ZIP directory, not the actual number of entries found by scanning the archive, or the directory.

On Linux "unzip -Z -1 many.zip | wc -l" reports 65792 even before adding ZIP64 support; only without -1 we get the interesting numbers (specifically with "unzip -Z many.zip | sed -n '2p;$p'"):

    Zip file size: 6841366 bytes, number of entries: 256
    65792 files, 0 bytes uncompressed, 0 bytes compressed: 0.0%

Show 23 quoted lines

> With these three patches applied, Mac OS X has trouble with 'many.zip':
> 
>      $ unzip -Z -1 many.zip
>      warning [many.zip]:  76 extra bytes at beginning or within zipfile
>        (attempting to process anyway)
>      error [many.zip]:  reported length of central directory is
>        -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
>        zipfile?).  Compensating...
>      00/
>      00/00
>      ...
>      ff/ff
>      error: expected central file header signature not found (file
>        #65793). (please check that you have transferred or created the
>        zipfile in the appropriate BINARY mode and that you have compiled
>        UnZip properly)
> 
> And FreeBSD doesn't like it either:
> 
>      $ unzip -Z -1 many.zip
>      unzip: Invalid central directory signature
>      $
>

Looks like they don't support ZIP64. Or I got some of the fields wrong after all.

https://en.wikipedia.org/wiki/Zip_%28file_format%29#ZIP64 says: "OS X Yosemite does support the creation of ZIP64 archives, but does not support unzipping these archives using the shipped unzip command-line utility or graphical Archive Utility.[citation needed]".

How does unzip react to a ZIP file with more than 65535 entries that was created natively on these platforms? And what does zipinfo (a real one, without -1) report at the top for such files?

Thanks, René