From: Torsten Bögershausen <tboegi@web.de>
Date: Sun, 30 Apr 2017 05:31:37 GMT
Subject: Re: [PATCH v3 0/5] archive-zip: support files and archives bigger than 4GB
Message-ID: <e30554f3-1aa3-acea-500b-6392fce902be@web.de>
In-Reply-To: <edf33657-f74b-3cd5-44a7-8e16231bd978@web.de>



On 30/04/17 00:28, René Scharfe wrote:
> Am 29.04.2017 um 23:00 schrieb Torsten Bögershausen:
>> This fails here under Mac OS:
>> commit 4cdf3f9d84568da72f1dcade812de7a42ecb6d15
>> Author: René Scharfe <l.s.r@web.de>
>> Date:   Mon Apr 24 19:33:34 2017 +0200
>>
>>      archive-zip: support files bigger than 4GB
>>
>> ---------------------------
>> Parts of t5004.log, hope this is helpful:
>>
>> "$GIT_UNZIP" -t many-big.zip
>>
>> Archive:  many-big.zip
>> warning [many-big.zip]:  577175 extra bytes at beginning or within zipfile
>>    (attempting to process anyway)
>> error [many-big.zip]:  start of central directory not found;
>>    zipfile corrupt.
>>    (please check that you have transferred or created the zipfile in the
>>    appropriate BINARY mode and that you have compiled UnZip properly)
>> not ok 12 - zip archive bigger than 4GB
>> #	
>> #		# build string containing 65536 characters
>
> Which version of unzip do you have (unzip -v, look for ZIP64_SUPPORT)?
> It seems that (some version of?) OS X ships with an older unzip which
> can't handle big files:
>
>    https://superuser.com/questions/114011/extract-large-zip-file-50-gb-on-mac-os-x
>
> Is the following check (zip archive with files bigger than 4GB) skipped,
> e.g. because ZIPINFO is missing?  Otherwise I would expect it to fail as
> well.
>
> René
>

Sorry, I was not looking careful enough, the macro `$GIT_UNZIP`
gave the impression that an unzip provided by Git (or the Git test framework) 
was used :-(

$ which unzip
/usr/bin/unzip

$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1) for Unix 
on Aug  1 2015.

UnZip special compilation options:
         COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
         SET_DIR_ATTRIB
         TIMESTAMP
         USE_EF_UT_TIME
         USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
         USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
         VMS_TEXT_CONV
         [decryption, version 2.9 of 05 May 2000]

UnZip and ZipInfo environment options:
            UNZIP:  [none]
         UNZIPOPT:  [none]
          ZIPINFO:  [none]
       ZIPINFOOPT:  [none]

-------------------
And here is the longer log:
not ok 12 - zip archive bigger than 4GB
#
#               # build string containing 65536 characters
# 
s=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef &&
# 
s=$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s &&
# 
s=$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s &&
#
#               # create blob with a length of 65536 + 1 bytes
#               blob=$(echo $s | git hash-object -w --stdin) &&
#
#               # create tree containing 65500 entries of that blob
#               for i in $(test_seq 1 65500)
#               do
#                       echo "100644 blob $blob $i"
#               done >tree &&
#               tree=$(git mktree <tree) &&
#
#               # zip it, creating an archive a bit bigger than 4GB
#               git archive -0 -o many-big.zip $tree &&
#
#               "$GIT_UNZIP" -t many-big.zip 9999 65500 &&
#               "$GIT_UNZIP" -t many-big.zip
#

skipping test: zip archive with files bigger than 4GB
         # Pack created with:
         #   dd if=/dev/zero of=file bs=1M count=4100 && git hash-object -w file
         mkdir -p .git/objects/pack &&
         (
                 cd .git/objects/pack &&
                 "$GIT_UNZIP" "$TEST_DIRECTORY"/t5004/big-pack.zip
         ) &&
         blob=754a93d6fada4c6873360e6cb4b209132271ab0e &&
         size=$(expr 4100 "*" 1024 "*" 1024) &&

         # create a tree containing the file
         tree=$(echo "100644 blob $blob  big-file" | git mktree) &&

         # zip it, creating an archive with a file bigger than 4GB
         git archive -o big.zip $tree &&

         "$GIT_UNZIP" -t big.zip &&
         "$ZIPINFO" big.zip >big.lst &&
         grep $size big.lst

ok 13 # skip zip archive with files bigger than 4GB (missing ZIPINFO of 
EXPENSIVE,UNZIP,ZIPINFO)

# failed 1 among 13 test(s)
1..13