Re: [PATCH 1/3] t5004: test ZIP archives with many entries

From: Eric Sunshine <sunshine@sunshineco.com>
Date: Aug 23, 2015, 05:54 UTC
Message-ID: <CAPig+cSy+c9mOGOTN9e4xfLrvPc8nv7e0T_4PDA-vB-otwrvjw@mail.gmail.com>
In-Reply-To: <55D8C824.6000704@web.de>

On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe <l.s.r@web.de> wrote:

Show 31 quoted lines

> A ZIP file directory has a 16-bit field for the number of entries it
> contains.  There are 64-bit extensions to deal with that.  Demonstrate
> that git archive --format=zip currently doesn't use them and instead
> overflows the field.
>
> InfoZIP's unzip doesn't care about this field and extracts all files
> anyway.  Software that uses the directory for presenting a filesystem
> like view quickly -- notably Windows -- depends on it, but doesn't
> lend itself to an automatic test case easily.  Use InfoZIP's zipinfo,
> which probably isn't available everywhere but at least can provides
> *some* way to check this field.
>
> To speed things up a bit create and commit only a subset of the files
> and build a fake tree out of duplicates and pass that to git archive.
>
> Signed-off-by: Rene Scharfe <l.s.r@web.de>
> ---
> diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh
> index 654adda..c6bd729 100755
> --- a/t/t5004-archive-corner-cases.sh
> +++ b/t/t5004-archive-corner-cases.sh
> @@ -115,4 +115,44 @@ test_expect_success 'archive empty subtree by direct pathspec' '
>         check_dir extract sub
>  '
>
> +ZIPINFO=zipinfo
> +
> +test_lazy_prereq ZIPINFO '
> +       n=$("$ZIPINFO" "$TEST_DIRECTORY"/t5004/empty.zip | sed -n "2s/.* //p")
> +       test "x$n" = "x0"
> +'

Unfortunately, this sed expression isn't portable due to dissimilar output of various zipinfo implementations. On Linux, the output of zipinfo is:

    $ zipinfo t/t5004/empty.zip
    Archive:  t/t5004/empty.zip
    Zip file size: 62 bytes, number of entries: 0
    Empty zipfile.
    $

however, on Mac OS X:

    $ zipinfo t/t5004/empty.zip
    Archive:  t/t5004/empty.zip   62 bytes   0 files
    Empty zipfile.
    $

and on FreeBSD, the zipinfo command seems to have been removed altogether in favor of "unzip -Z" (emulate zipinfo).

One might hope that "unzip -Z" would be a reasonable replacement for zipinfo, however, it is apparently only partially implemented on FreeBSD, and requires that -1 be passed, as well. Even with "unzip -Z -1", there are issues. The output on Linux and Mac OS X is:

    $ unzip -Z -1 t/t5004/empty.zip
    Empty zipfile.
    $

but FreeBSD differs:

    $ unzip -Z -1 t/t5004/empty.zip
    $

With a non-empty zip file, the output is identical on all platforms:

    $ unzip -Z -1 twofiles.zip
    file1
    file2
    $

So, if you combine that with "wc -l" or test_line_count, you may have a portable and reliable entry counter.

More below...

Show 30 quoted lines

> +test_expect_failure ZIPINFO 'zip archive with many entries' '
> +       # add a directory with 256 files
> +       mkdir 00 &&
> +       for a in 0 1 2 3 4 5 6 7 8 9 a b c d e f
> +       do
> +               for b in 0 1 2 3 4 5 6 7 8 9 a b c d e f
> +               do
> +                       : >00/$a$b
> +               done
> +       done &&
> +       git add 00 &&
> +       git commit -m "256 files in 1 directory" &&
> +
> +       # duplicate it to get 65536 files in 256 directories
> +       subtree=$(git write-tree --prefix=00/) &&
> +       for c in 0 1 2 3 4 5 6 7 8 9 a b c d e f
> +       do
> +               for d in 0 1 2 3 4 5 6 7 8 9 a b c d e f
> +               do
> +                       echo "040000 tree $subtree      $c$d"
> +               done
> +       done >tree &&
> +       tree=$(git mktree <tree) &&
> +
> +       # zip them
> +       git archive -o many.zip $tree &&
> +
> +       # check the number of entries in the ZIP file directory
> +       expr 65536 + 256 >expect &&
> +       "$ZIPINFO" many.zip | head -2 | sed -n "2s/.* //p" >actual &&

With these three patches applied, Mac OS X has trouble with 'many.zip':

    $ unzip -Z -1 many.zip
    warning [many.zip]:  76 extra bytes at beginning or within zipfile
      (attempting to process anyway)
    error [many.zip]:  reported length of central directory is
      -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
      zipfile?).  Compensating...
    00/
    00/00
    ...
    ff/ff
    error: expected central file header signature not found (file
      #65793). (please check that you have transferred or created the
      zipfile in the appropriate BINARY mode and that you have compiled
      UnZip properly)

And FreeBSD doesn't like it either:

    $ unzip -Z -1 many.zip
    unzip: Invalid central directory signature
    $

Show 6 quoted lines

> +       test_cmp expect actual
> +'
> +
>  test_done
> --
> 2.5.0