{"thread":{"id":"40054","subject":"bug: git-archive does not use the zip64 extension for archives with more than 16k entries","startedAt":"2015-08-11T10:40:56Z","lastAt":"2015-08-28T16:47:30Z","messageCount":18,"participants":["Johannes Schauer","René Scharfe","Duy Nguyen","Eric Sunshine","Johannes Löthberg","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"267821","messageId":"20150811104056.16465.58131@localhost","threadId":"40054","inReplyTo":null,"subject":"bug: git-archive does not use the zip64 extension for archives with more than 16k entries","fromName":"Johannes Schauer","fromEmail":"josch@debian.org","sentAt":"2015-08-11T10:40:56Z","receivedAt":"2015-08-11T10:40:56Z","isPatch":false,"sender":{"key":"josch@debian.org","avatar":null},"body":"Hi,\n\nfor repositories with more than 16k files and folders, git-archive will create\nzip files which store the wrong number of entries. That is, it stores the\nnumber of entries modulo 16k. This will break unpackers that do not include\ncode to support this brokenness.\n\nInstead, git-archive should use the zip64 extension to handle more than 16k\nfiles and folders correctly.\n\nThanks!\n\ncheers, josch\n"},{"id":"267941","messageId":"55CBA140.7050301@web.de","threadId":"40054","inReplyTo":"20150811104056.16465.58131@localhost","subject":"Re: bug: git-archive does not use the zip64 extension for archives with more than 16k entries","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-08-12T19:40:48Z","receivedAt":"2015-08-12T19:40:48Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 11.08.2015 um 12:40 schrieb Johannes Schauer:\n> Hi,\n>\n> for repositories with more than 16k files and folders, git-archive will create\n> zip files which store the wrong number of entries. That is, it stores the\n> number of entries modulo 16k. This will break unpackers that do not include\n> code to support this brokenness.\n\nThe limit is rather 65535 entries, isn't it? The entries field has two \nbytes, and they are used fully.\n\nWhich programs are affected? InfoZIP Zip 3.0 and 7-Zip 9.20 seem to \nhandle an archive with more entries just fine. The built-in \nfunctionality of Windows 10 doesn't.\n\nBesides, 64K entries should be enough for anybody. ;-)\n\nSeriously, though: What kind of repository has that many files and uses \nthe ZIP format to distribute snapshots? Just curious.\n\n> Instead, git-archive should use the zip64 extension to handle more than 16k\n> files and folders correctly.\n\nThat seems to be what InfoZIP does and Windows 10 handles it just fine. \n If lower Windows versions and other popular extractors can unzip such \narchives as well then this might indeed be the way to go.\n\nThanks,\nRené\n"},{"id":"267967","messageId":"20150813022545.30116.44787@localhost","threadId":"40054","inReplyTo":"55CBA140.7050301@web.de","subject":"Re: bug: git-archive does not use the zip64 extension for archives with more than 16k entries","fromName":"Johannes Schauer","fromEmail":"josch@debian.org","sentAt":"2015-08-13T02:25:45Z","receivedAt":"2015-08-13T02:25:45Z","isPatch":false,"sender":{"key":"josch@debian.org","avatar":null},"body":"Hi,\n\nQuoting René Scharfe (2015-08-12 21:40:48)\n> Am 11.08.2015 um 12:40 schrieb Johannes Schauer:\n> > for repositories with more than 16k files and folders, git-archive will create\n> > zip files which store the wrong number of entries. That is, it stores the\n> > number of entries modulo 16k. This will break unpackers that do not include\n> > code to support this brokenness.\n> \n> The limit is rather 65535 entries, isn't it? The entries field has two \n> bytes, and they are used fully.\n\nseems to be 65535 indeed.\n\nI just forwarded the number Dieter Baron (libzip contributor) told me when they\nreplied (off list) to my bug report against libzip:\nhttp://nih.at/listarchive/libzip-discuss/msg00554.html\n\nBut reading https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64 the limit\nindeed seems to be 65535.\n\n> Which programs are affected? InfoZIP Zip 3.0 and 7-Zip 9.20 seem to handle an\n> archive with more entries just fine. The built-in functionality of Windows\n> 10 doesn't.\n\nIn my case I discovered this because libzip http://nih.at/libzip does not\nimplement reading an archive with more than 65535 entries without zip64.\n\n> Besides, 64K entries should be enough for anybody. ;-)\n\n:P\n\n> Seriously, though: What kind of repository has that many files and uses the\n> ZIP format to distribute snapshots? Just curious.\n\nI have not searched for any.\n\nIn my case I was using git to keep track of the modifications our tools do to a\ndirectory of files to detect regressions. These tools are also able to read\ndata from zip archives instead from a directory. I created the zip archive\nusing git-archive because the files already were in git so that seemed most\nconvenient to me. That's when I discovered the problem because our tools use\nlibzip. The easy workaround was to use another packager instead of git-archive.\nWe use the zip format because Windows has support for it.\n\n> > Instead, git-archive should use the zip64 extension to handle more than 16k\n> > files and folders correctly.\n> \n> That seems to be what InfoZIP does and Windows 10 handles it just fine. If\n> lower Windows versions and other popular extractors can unzip such archives\n> as well then this might indeed be the way to go.\n\nThe wikipedia page above claims that windows versions starting with vista have\nsupport for zip64. It also lists some other software with support for it.\n\nThanks!\n\ncheers, josch\n"},{"id":"268118","messageId":"CACsJy8AuuD-mk54Hx3tqBRLkEC5Phq04r98UaBVtk15tOsBT3Q@mail.gmail.com","threadId":"40054","inReplyTo":"55CBA140.7050301@web.de","subject":"Re: bug: git-archive does not use the zip64 extension for archives with more than 16k entries","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-15T08:40:32Z","receivedAt":"2015-08-15T08:40:32Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 13, 2015 at 2:40 AM, René Scharfe wrote:\n> Seriously, though: What kind of repository has that many files and uses the\n> ZIP format to distribute snapshots? Just curious.\n\nNot the \"uses the zip format\" part, but at least webkit and gentoo-x86\nboth exceed 64k limit. Even if we don't support this case, we probably\nshould error out instead of producing corrupt archives.\n-- \nDuy\n"},{"id":"268515","messageId":"55D8C824.6000704@web.de","threadId":"40054","inReplyTo":"20150813022545.30116.44787@localhost","subject":"[PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-08-22T19:06:12Z","receivedAt":"2015-08-22T19:06:12Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"A ZIP file directory has a 16-bit field for the number of entries it\ncontains. There are 64-bit extensions to deal with that. Demonstrate\nthat git archive --format=zip currently doesn't use them and instead\noverflows the field.\n\nInfoZIP's unzip doesn't care about this field and extracts all files\nanyway. Software that uses the directory for presenting a filesystem\nlike view quickly -- notably Windows -- depends on it, but doesn't\nlend itself to an automatic test case easily. Use InfoZIP's zipinfo,\nwhich probably isn't available everywhere but at least can provides\n*some* way to check this field.\n\nTo speed things up a bit create and commit only a subset of the files\nand build a fake tree out of duplicates and pass that to git archive.\n\nSigned-off-by: Rene Scharfe \n---\n t/t5004-archive-corner-cases.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh\nindex 654adda..c6bd729 100755\n--- a/t/t5004-archive-corner-cases.sh\n+++ b/t/t5004-archive-corner-cases.sh\n@@ -115,4 +115,44 @@ test_expect_success 'archive empty subtree by direct pathspec' '\n \tcheck_dir extract sub\n '\n \n+ZIPINFO=zipinfo\n+\n+test_lazy_prereq ZIPINFO '\n+\tn=$(\"$ZIPINFO\" \"$TEST_DIRECTORY\"/t5004/empty.zip | sed -n \"2s/.* //p\")\n+\ttest \"x$n\" = \"x0\"\n+'\n+\n+test_expect_failure ZIPINFO 'zip archive with many entries' '\n+\t# add a directory with 256 files\n+\tmkdir 00 &&\n+\tfor a in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n+\tdo\n+\t\tfor b in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n+\t\tdo\n+\t\t\t: >00/$a$b\n+\t\tdone\n+\tdone &&\n+\tgit add 00 &&\n+\tgit commit -m \"256 files in 1 directory\" &&\n+\n+\t# duplicate it to get 65536 files in 256 directories\n+\tsubtree=$(git write-tree --prefix=00/) &&\n+\tfor c in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n+\tdo\n+\t\tfor d in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n+\t\tdo\n+\t\t\techo \"040000 tree $subtree\t$c$d\"\n+\t\tdone\n+\tdone >tree &&\n+\ttree=$(git mktree expect &&\n+\t\"$ZIPINFO\" many.zip | head -2 | sed -n \"2s/.* //p\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.5.0\n"},{"id":"268516","messageId":"55D8C837.6070601@web.de","threadId":"40054","inReplyTo":"20150813022545.30116.44787@localhost","subject":"[PATCH 2/3] archive-zip: use a local variable to store the creator version","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-08-22T19:06:31Z","receivedAt":"2015-08-22T19:06:31Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Use a simpler conditional right next to the code which makes a higher\ncreator version necessary -- namely symlink handling and support for\nexecutable files -- instead of a long line with a ternary operator.\nThe resulting code has more lines but is simpler and allows reuse of\nthe value easily.\n\nSigned-off-by: Rene Scharfe \n---\n archive-zip.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex ae3d67f..2a76156 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -223,6 +223,7 @@ static int write_zip_entry(struct archiver_args *args,\n \tunsigned long size;\n \tint is_binary = -1;\n \tconst char *path_without_prefix = path + args->baselen;\n+\tunsigned int creator_version = 0;\n \n \tcrc = crc32(0, NULL, 0);\n \n@@ -251,6 +252,8 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tmethod = 0;\n \t\tattr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :\n \t\t\t(mode & 0111) ? ((mode) << 16) : 0;\n+\t\tif (S_ISLNK(mode) || (mode & 0111))\n+\t\t\tcreator_version = 0x0317;\n \t\tif (S_ISREG(mode) && args->compression_level != 0 && size > 0)\n \t\t\tmethod = 8;\n \n@@ -303,8 +306,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t}\n \n \tcopy_le32(dirent.magic, 0x02014b50);\n-\tcopy_le16(dirent.creator_version,\n-\t\tS_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0);\n+\tcopy_le16(dirent.creator_version, creator_version);\n \tcopy_le16(dirent.version, 10);\n \tcopy_le16(dirent.flags, flags);\n \tcopy_le16(dirent.compression_method, method);\n-- \n2.5.0\n"},{"id":"268517","messageId":"55D8C845.3090908@web.de","threadId":"40054","inReplyTo":"20150813022545.30116.44787@localhost","subject":"[PATCH 3/3] archive-zip: support more than 65535 entries","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-08-22T19:06:45Z","receivedAt":"2015-08-22T19:06:45Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Support more than 65535 entries cleanly by writing a \"zip64 end of\ncentral directory record\" (with a 64-bit field for the number of\nentries) before the usual \"end of central directory record\" (which\ncontains only a 16-bit field). InfoZIP's zip does the same.\nArchives with 65535 or less entries are not affected.\n\nPrograms that extract all files like InfoZIP's zip and 7-Zip\nignored the field and could extract all files already. Software\nthat relies on the ZIP file directory to show a list of contained\nfiles quickly to simulate to normal directory like Windows'\nbuilt-in ZIP functionality only saw a subset of the included files.\n\nWindows supports ZIP64 since Vista according to\nhttps://en.wikipedia.org/wiki/Zip_%28file_format%29#ZIP64.\n\nSuggested-by: Johannes Schauer \nSigned-off-by: Rene Scharfe \n---\n archive-zip.c | 93 +++++++++++++++++++++++++++++++++++++++--\n t/t5004-archive-corner-cases.sh | 2 +-\n 2 files changed, 91 insertions(+), 4 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 2a76156..9db4735 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -16,7 +16,9 @@ static unsigned int zip_dir_size;\n \n static unsigned int zip_offset;\n static unsigned int zip_dir_offset;\n-static unsigned int zip_dir_entries;\n+static uint64_t zip_dir_entries;\n+\n+static unsigned int max_creator_version;\n \n #define ZIP_DIRECTORY_MIN_SIZE\t(1024 * 1024)\n #define ZIP_STREAM\t(1 << 3)\n@@ -86,6 +88,28 @@ struct zip_extra_mtime {\n \tunsigned char _end[1];\n };\n \n+struct zip64_dir_trailer {\n+\tunsigned char magic[4];\n+\tunsigned char record_size[8];\n+\tunsigned char creator_version[2];\n+\tunsigned char version[2];\n+\tunsigned char disk[4];\n+\tunsigned char directory_start_disk[4];\n+\tunsigned char entries_on_this_disk[8];\n+\tunsigned char entries[8];\n+\tunsigned char size[8];\n+\tunsigned char offset[8];\n+\tunsigned char _end[1];\n+};\n+\n+struct zip64_dir_trailer_locator {\n+\tunsigned char magic[4];\n+\tunsigned char disk[4];\n+\tunsigned char offset[8];\n+\tunsigned char number_of_disks[4];\n+\tunsigned char _end[1];\n+};\n+\n /*\n * On ARM, padding is added at the end of the struct, so a simple\n * sizeof(struct ...) reports two bytes more than the payload size\n@@ -98,6 +122,12 @@ struct zip_extra_mtime {\n #define ZIP_EXTRA_MTIME_SIZE\toffsetof(struct zip_extra_mtime, _end)\n #define ZIP_EXTRA_MTIME_PAYLOAD_SIZE \\\n \t(ZIP_EXTRA_MTIME_SIZE - offsetof(struct zip_extra_mtime, flags))\n+#define ZIP64_DIR_TRAILER_SIZE\toffsetof(struct zip64_dir_trailer, _end)\n+#define ZIP64_DIR_TRAILER_RECORD_SIZE \\\n+\t(ZIP64_DIR_TRAILER_SIZE - \\\n+\t offsetof(struct zip64_dir_trailer, creator_version))\n+#define ZIP64_DIR_TRAILER_LOCATOR_SIZE \\\n+\toffsetof(struct zip64_dir_trailer_locator, _end)\n \n static void copy_le16(unsigned char *dest, unsigned int n)\n {\n@@ -113,6 +143,31 @@ static void copy_le32(unsigned char *dest, unsigned int n)\n \tdest[3] = 0xff & (n >> 030);\n }\n \n+static void copy_le64(unsigned char *dest, uint64_t n)\n+{\n+\tdest[0] = 0xff & n;\n+\tdest[1] = 0xff & (n >> 010);\n+\tdest[2] = 0xff & (n >> 020);\n+\tdest[3] = 0xff & (n >> 030);\n+\tdest[4] = 0xff & (n >> 040);\n+\tdest[5] = 0xff & (n >> 050);\n+\tdest[6] = 0xff & (n >> 060);\n+\tdest[7] = 0xff & (n >> 070);\n+}\n+\n+static uint64_t clamp_max(uint64_t n, uint64_t max, int *clamped)\n+{\n+\tif (n <= max)\n+\t\treturn n;\n+\t*clamped = 1;\n+\treturn max;\n+}\n+\n+static void copy_le16_clamp(unsigned char *dest, uint64_t n, int *clamped)\n+{\n+\tcopy_le16(dest, clamp_max(n, 0xffff, clamped));\n+}\n+\n static void *zlib_deflate_raw(void *data, unsigned long size,\n \t\t\t int compression_level,\n \t\t\t unsigned long *compressed_size)\n@@ -282,6 +337,9 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\t\tsha1_to_hex(sha1));\n \t}\n \n+\tif (creator_version > max_creator_version)\n+\t\tmax_creator_version = creator_version;\n+\n \tif (buffer && method == 8) {\n \t\tout = deflated = zlib_deflate_raw(buffer, size,\n \t\t\t\t\t\t args->compression_level,\n@@ -439,20 +497,49 @@ static int write_zip_entry(struct archiver_args *args,\n \treturn 0;\n }\n \n+static void write_zip64_trailer(void)\n+{\n+\tstruct zip64_dir_trailer trailer64;\n+\tstruct zip64_dir_trailer_locator locator64;\n+\n+\tcopy_le32(trailer64.magic, 0x06064b50);\n+\tcopy_le64(trailer64.record_size, ZIP64_DIR_TRAILER_RECORD_SIZE);\n+\tcopy_le16(trailer64.creator_version, max_creator_version);\n+\tcopy_le16(trailer64.version, 45);\n+\tcopy_le32(trailer64.disk, 0);\n+\tcopy_le32(trailer64.directory_start_disk, 0);\n+\tcopy_le64(trailer64.entries_on_this_disk, zip_dir_entries);\n+\tcopy_le64(trailer64.entries, zip_dir_entries);\n+\tcopy_le64(trailer64.size, zip_dir_offset);\n+\tcopy_le64(trailer64.offset, zip_offset);\n+\n+\tcopy_le32(locator64.magic, 0x07064b50);\n+\tcopy_le32(locator64.disk, 0);\n+\tcopy_le64(locator64.offset, zip_offset + zip_dir_offset);\n+\tcopy_le32(locator64.number_of_disks, 1);\n+\n+\twrite_or_die(1, &trailer64, ZIP64_DIR_TRAILER_SIZE);\n+\twrite_or_die(1, &locator64, ZIP64_DIR_TRAILER_LOCATOR_SIZE);\n+}\n+\n static void write_zip_trailer(const unsigned char *sha1)\n {\n \tstruct zip_dir_trailer trailer;\n+\tint clamped = 0;\n \n \tcopy_le32(trailer.magic, 0x06054b50);\n \tcopy_le16(trailer.disk, 0);\n \tcopy_le16(trailer.directory_start_disk, 0);\n-\tcopy_le16(trailer.entries_on_this_disk, zip_dir_entries);\n-\tcopy_le16(trailer.entries, zip_dir_entries);\n+\tcopy_le16_clamp(trailer.entries_on_this_disk, zip_dir_entries,\n+\t\t\t&clamped);\n+\tcopy_le16_clamp(trailer.entries, zip_dir_entries, &clamped);\n \tcopy_le32(trailer.size, zip_dir_offset);\n \tcopy_le32(trailer.offset, zip_offset);\n \tcopy_le16(trailer.comment_length, sha1 ? GIT_SHA1_HEXSZ : 0);\n \n \twrite_or_die(1, zip_dir, zip_dir_offset);\n+\tif (clamped)\n+\t\twrite_zip64_trailer();\n \twrite_or_die(1, &trailer, ZIP_DIR_TRAILER_SIZE);\n \tif (sha1)\n \t\twrite_or_die(1, sha1_to_hex(sha1), GIT_SHA1_HEXSZ);\ndiff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh\nindex c6bd729..cca2338 100755\n--- a/t/t5004-archive-corner-cases.sh\n+++ b/t/t5004-archive-corner-cases.sh\n@@ -122,7 +122,7 @@ test_lazy_prereq ZIPINFO '\n \ttest \"x$n\" = \"x0\"\n '\n \n-test_expect_failure ZIPINFO 'zip archive with many entries' '\n+test_expect_success ZIPINFO 'zip archive with many entries' '\n \t# add a directory with 256 files\n \tmkdir 00 &&\n \tfor a in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n-- \n2.5.0\n"},{"id":"268523","messageId":"CAPig+cSy+c9mOGOTN9e4xfLrvPc8nv7e0T_4PDA-vB-otwrvjw@mail.gmail.com","threadId":"40054","inReplyTo":"55D8C824.6000704@web.de","subject":"Re: [PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-23T05:54:22Z","receivedAt":"2015-08-23T05:54:22Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe wrote:\n> A ZIP file directory has a 16-bit field for the number of entries it\n> contains. There are 64-bit extensions to deal with that. Demonstrate\n> that git archive --format=zip currently doesn't use them and instead\n> overflows the field.\n>\n> InfoZIP's unzip doesn't care about this field and extracts all files\n> anyway. Software that uses the directory for presenting a filesystem\n> like view quickly -- notably Windows -- depends on it, but doesn't\n> lend itself to an automatic test case easily. Use InfoZIP's zipinfo,\n> which probably isn't available everywhere but at least can provides\n> *some* way to check this field.\n>\n> To speed things up a bit create and commit only a subset of the files\n> and build a fake tree out of duplicates and pass that to git archive.\n>\n> Signed-off-by: Rene Scharfe \n> ---\n> diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh\n> index 654adda..c6bd729 100755\n> --- a/t/t5004-archive-corner-cases.sh\n> +++ b/t/t5004-archive-corner-cases.sh\n> @@ -115,4 +115,44 @@ test_expect_success 'archive empty subtree by direct pathspec' '\n> check_dir extract sub\n> '\n>\n> +ZIPINFO=zipinfo\n> +\n> +test_lazy_prereq ZIPINFO '\n> + n=$(\"$ZIPINFO\" \"$TEST_DIRECTORY\"/t5004/empty.zip | sed -n \"2s/.* //p\")\n> + test \"x$n\" = \"x0\"\n> +'\n\nUnfortunately, this sed expression isn't portable due to dissimilar\noutput of various zipinfo implementations. On Linux, the output of\nzipinfo is:\n\n $ zipinfo t/t5004/empty.zip\n Archive: t/t5004/empty.zip\n Zip file size: 62 bytes, number of entries: 0\n Empty zipfile.\n $\n\nhowever, on Mac OS X:\n\n $ zipinfo t/t5004/empty.zip\n Archive: t/t5004/empty.zip 62 bytes 0 files\n Empty zipfile.\n $\n\nand on FreeBSD, the zipinfo command seems to have been removed\naltogether in favor of \"unzip -Z\" (emulate zipinfo).\n\nOne might hope that \"unzip -Z\" would be a reasonable replacement for\nzipinfo, however, it is apparently only partially implemented on\nFreeBSD, and requires that -1 be passed, as well. Even with \"unzip -Z\n-1\", there are issues. The output on Linux and Mac OS X is:\n\n $ unzip -Z -1 t/t5004/empty.zip\n Empty zipfile.\n $\n\nbut FreeBSD differs:\n\n $ unzip -Z -1 t/t5004/empty.zip\n $\n\nWith a non-empty zip file, the output is identical on all platforms:\n\n $ unzip -Z -1 twofiles.zip\n file1\n file2\n $\n\nSo, if you combine that with \"wc -l\" or test_line_count, you may have\na portable and reliable entry counter.\n\nMore below...\n\n> +test_expect_failure ZIPINFO 'zip archive with many entries' '\n> + # add a directory with 256 files\n> + mkdir 00 &&\n> + for a in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n> + do\n> + for b in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n> + do\n> + : >00/$a$b\n> + done\n> + done &&\n> + git add 00 &&\n> + git commit -m \"256 files in 1 directory\" &&\n> +\n> + # duplicate it to get 65536 files in 256 directories\n> + subtree=$(git write-tree --prefix=00/) &&\n> + for c in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n> + do\n> + for d in 0 1 2 3 4 5 6 7 8 9 a b c d e f\n> + do\n> + echo \"040000 tree $subtree $c$d\"\n> + done\n> + done >tree &&\n> + tree=$(git mktree +\n> + # zip them\n> + git archive -o many.zip $tree &&\n> +\n> + # check the number of entries in the ZIP file directory\n> + expr 65536 + 256 >expect &&\n> + \"$ZIPINFO\" many.zip | head -2 | sed -n \"2s/.* //p\" >actual &&\n\nWith these three patches applied, Mac OS X has trouble with 'many.zip':\n\n $ unzip -Z -1 many.zip\n warning [many.zip]: 76 extra bytes at beginning or within zipfile\n (attempting to process anyway)\n error [many.zip]: reported length of central directory is\n -76 bytes too long (Atari STZip zipfile? J.H.Holm ZIPSPLIT 1.1\n zipfile?). Compensating...\n 00/\n 00/00\n ...\n ff/ff\n error: expected central file header signature not found (file\n #65793). (please check that you have transferred or created the\n zipfile in the appropriate BINARY mode and that you have compiled\n UnZip properly)\n\nAnd FreeBSD doesn't like it either:\n\n $ unzip -Z -1 many.zip\n unzip: Invalid central directory signature\n $\n\n> + test_cmp expect actual\n> +'\n> +\n> test_done\n> --\n> 2.5.0\n"},{"id":"268524","messageId":"trinity-6e67d416-0a61-4e73-9779-63519dd83fdb-1440322151491@3capp-webde-bs47","threadId":"40054","inReplyTo":"CAPig+cSy+c9mOGOTN9e4xfLrvPc8nv7e0T_4PDA-vB-otwrvjw@mail.gmail.com","subject":"Re: [PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-08-23T09:29:11Z","receivedAt":"2015-08-23T09:29:11Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.08.2015 um 07:54 schrieb Eric Sunshine:\n> On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe wrote:\n>> diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh\n>> index 654adda..c6bd729 100755\n>> --- a/t/t5004-archive-corner-cases.sh\n>> +++ b/t/t5004-archive-corner-cases.sh\n>> @@ -115,4 +115,44 @@ test_expect_success 'archive empty subtree by direct pathspec' '\n>> check_dir extract sub\n>> '\n>>\n>> +ZIPINFO=zipinfo\n>> +\n>> +test_lazy_prereq ZIPINFO '\n>> + n=$(\"$ZIPINFO\" \"$TEST_DIRECTORY\"/t5004/empty.zip | sed -n \"2s/.* //p\")\n>> + test \"x$n\" = \"x0\"\n>> +'\n> \n> Unfortunately, this sed expression isn't portable due to dissimilar\n> output of various zipinfo implementations. On Linux, the output of\n> zipinfo is:\n> \n> $ zipinfo t/t5004/empty.zip\n> Archive: t/t5004/empty.zip\n> Zip file size: 62 bytes, number of entries: 0\n> Empty zipfile.\n> $\n> \n> however, on Mac OS X:\n> \n> $ zipinfo t/t5004/empty.zip\n> Archive: t/t5004/empty.zip 62 bytes 0 files\n> Empty zipfile.\n> $\n> \n> and on FreeBSD, the zipinfo command seems to have been removed\n> altogether in favor of \"unzip -Z\" (emulate zipinfo).\n\nThanks for your thorough checks!\n\nI suspected that zipinfo's output might be formatted differently on\ndifferent platforms and tried to guard against it by checking for the\nnumber zero there. Git's ZIP file creation is platform independent\n(modulo bugs), so having a test run at least somewhere should\nsuffice. In theory.\n\nWe could add support for the one-line-summary variant on OS X easily,\nthough.\n\n> One might hope that \"unzip -Z\" would be a reasonable replacement for\n> zipinfo, however, it is apparently only partially implemented on\n> FreeBSD, and requires that -1 be passed, as well. Even with \"unzip -Z\n> -1\", there are issues. The output on Linux and Mac OS X is:\n> \n> $ unzip -Z -1 t/t5004/empty.zip\n> Empty zipfile.\n> $\n> \n> but FreeBSD differs:\n> \n> $ unzip -Z -1 t/t5004/empty.zip\n> $\n> \n> With a non-empty zip file, the output is identical on all platforms:\n> \n> $ unzip -Z -1 twofiles.zip\n> file1\n> file2\n> $\n> \n> So, if you combine that with \"wc -l\" or test_line_count, you may have\n> a portable and reliable entry counter.\n\nCounting all entries is slow, and more importantly it's not what we\nwant. In this test we need the number of entries recorded in the ZIP\ndirectory, not the actual number of entries found by scanning the\narchive, or the directory.\n\nOn Linux \"unzip -Z -1 many.zip | wc -l\" reports 65792 even before\nadding ZIP64 support; only without -1 we get the interesting numbers\n(specifically with \"unzip -Z many.zip | sed -n '2p;$p'\"):\n\n Zip file size: 6841366 bytes, number of entries: 256\n 65792 files, 0 bytes uncompressed, 0 bytes compressed: 0.0%\n\n> With these three patches applied, Mac OS X has trouble with 'many.zip':\n> \n> $ unzip -Z -1 many.zip\n> warning [many.zip]: 76 extra bytes at beginning or within zipfile\n> (attempting to process anyway)\n> error [many.zip]: reported length of central directory is\n> -76 bytes too long (Atari STZip zipfile? J.H.Holm ZIPSPLIT 1.1\n> zipfile?). Compensating...\n> 00/\n> 00/00\n> ...\n> ff/ff\n> error: expected central file header signature not found (file\n> #65793). (please check that you have transferred or created the\n> zipfile in the appropriate BINARY mode and that you have compiled\n> UnZip properly)\n> \n> And FreeBSD doesn't like it either:\n> \n> $ unzip -Z -1 many.zip\n> unzip: Invalid central directory signature\n> $\n> \n\nLooks like they don't support ZIP64. Or I got some of the fields wrong\nafter all.\n\nhttps://en.wikipedia.org/wiki/Zip_%28file_format%29#ZIP64 says: \"OS X\nYosemite does support the creation of ZIP64 archives, but does not\nsupport unzipping these archives using the shipped unzip command-line\nutility or graphical Archive Utility.[citation needed]\".\n\nHow does unzip react to a ZIP file with more than 65535 entries that\nwas created natively on these platforms? And what does zipinfo (a real\none, without -1) report at the top for such files?\n\nThanks,\nRené\n"},{"id":"268525","messageId":"55D993F8.4080506@web.de","threadId":"40054","inReplyTo":"trinity-6e67d416-0a61-4e73-9779-63519dd83fdb-1440322151491@3capp-webde-bs47","subject":"Eric Sunshine mail delivery failure","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-08-23T09:35:52Z","receivedAt":"2015-08-23T09:35:52Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Eric, hope you see this reply on the list. Direct replies to \nsunshine@sunshineco.com are rejected by my mail provider on submit in \nThunderbird with the following message:\n\n Requested action not taken: mailbox unavailable\n invalid DNS MX or A/AAAA resource record.\n\nAnd with this one when using their web interface:\n\n A message that you sent could not be delivered to one or more of\n its recipients. This is a permanent error. The following address\n failed:\n\n \"sunshine@sunshineco.com\":\n no valid MX hosts found\n\nIt seems web.de wants you to get an AAAA record before I'm allowed to \nsend mails to you. Sounds crazy. Sorry about that. Time to find a\nbetter provider, I guess. :-(\n\nRené\n"},{"id":"268534","messageId":"20150823171622.GA28700@zorg.kyriasis.com","threadId":"40054","inReplyTo":"55D993F8.4080506@web.de","subject":"Re: Eric Sunshine mail delivery failure","fromName":"Johannes Löthberg","fromEmail":"johannes@kyriasis.com","sentAt":"2015-08-23T17:16:22Z","receivedAt":"2015-08-23T17:16:22Z","isPatch":false,"sender":{"key":"johannes@kyriasis.com","avatar":"https://gravatar.com/avatar/af2dea1b1759403329a1b7eaa28a07b5c5d1901e9f55723103d6324ffb7737ae?d=mp&s=160"},"body":"On 23/08, René Scharfe wrote:\n>Eric, hope you see this reply on the list. Direct replies to \n>sunshine@sunshineco.com are rejected by my mail provider on submit in \n>Thunderbird with the following message:\n>\n> Requested action not taken: mailbox unavailable\n> invalid DNS MX or A/AAAA resource record.\n>\n>And with this one when using their web interface:\n>\n> A message that you sent could not be delivered to one or more of\n> its recipients. This is a permanent error. The following address\n> failed:\n>\n> \"sunshine@sunshineco.com\":\n> no valid MX hosts found\n>\n>It seems web.de wants you to get an AAAA record before I'm allowed to \n>send mails to you. Sounds crazy. Sorry about that. Time to find a\n>better provider, I guess. :-(\n>\n\nJust an A record would be enough. The issue is that mail.sunshineco.com \nhas neither an A nor an AAAA record, it is a CNAME to sunshineco.com, \nwhich is invalid according to RFC2181.\n\n-- \nSincerely,\n Johannes Löthberg\n PGP Key ID: 0x50FB9B273A9D0BB5\n https://theos.kyriasis.com/~kyrias/\n"},{"id":"268536","messageId":"CAPig+cSNSfpt7gOLvz7P4oDrNF5fTQ38v1pfncJU3h7a6FjMyQ@mail.gmail.com","threadId":"40054","inReplyTo":"trinity-6e67d416-0a61-4e73-9779-63519dd83fdb-1440322151491@3capp-webde-bs47","subject":"Re: [PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-23T17:45:26Z","receivedAt":"2015-08-23T17:45:26Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 23, 2015 at 5:29 AM, \"René Scharfe\" wrote:\n> Am 23.08.2015 um 07:54 schrieb Eric Sunshine:\n>> On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe wrote:\n>>> +test_lazy_prereq ZIPINFO '\n>>> + n=$(\"$ZIPINFO\" \"$TEST_DIRECTORY\"/t5004/empty.zip | sed -n \"2s/.* //p\")\n>>> + test \"x$n\" = \"x0\"\n>>> +'\n>>\n>> Unfortunately, this sed expression isn't portable due to dissimilar\n>> output of various zipinfo implementations. On Linux, the output of\n>> zipinfo is:\n>>\n>> $ zipinfo t/t5004/empty.zip\n>> Archive: t/t5004/empty.zip\n>> Zip file size: 62 bytes, number of entries: 0\n>> Empty zipfile.\n>> $\n>>\n>> however, on Mac OS X:\n>>\n>> $ zipinfo t/t5004/empty.zip\n>> Archive: t/t5004/empty.zip 62 bytes 0 files\n>> Empty zipfile.\n>> $\n>>\n>> and on FreeBSD, the zipinfo command seems to have been removed\n>> altogether in favor of \"unzip -Z\" (emulate zipinfo).\n>\n> I suspected that zipinfo's output might be formatted differently on\n> different platforms and tried to guard against it by checking for the\n> number zero there. Git's ZIP file creation is platform independent\n> (modulo bugs), so having a test run at least somewhere should\n> suffice. In theory.\n>\n> We could add support for the one-line-summary variant on OS X easily,\n> though.\n\nProbably, although it's looking like testing on Mac OS X won't be\nfruitful (see below).\n\n>> One might hope that \"unzip -Z\" would be a reasonable replacement for\n>> zipinfo, however, it is apparently only partially implemented on\n>> FreeBSD, and requires that -1 be passed, as well. Even with \"unzip -Z\n>> -1\", there are issues. The output on Linux and Mac OS X is:\n>>\n>> $ unzip -Z -1 t/t5004/empty.zip\n>> Empty zipfile.\n>> $\n>>\n>> but FreeBSD differs:\n>>\n>> $ unzip -Z -1 t/t5004/empty.zip\n>> $\n>>\n>> With a non-empty zip file, the output is identical on all platforms:\n>>\n>> $ unzip -Z -1 twofiles.zip\n>> file1\n>> file2\n>> $\n>>\n>> So, if you combine that with \"wc -l\" or test_line_count, you may have\n>> a portable and reliable entry counter.\n>\n> Counting all entries is slow, and more importantly it's not what we\n> want. In this test we need the number of entries recorded in the ZIP\n> directory, not the actual number of entries found by scanning the\n> archive, or the directory.\n\nAh, right. The commit message did state this clearly enough...\n\n> On Linux \"unzip -Z -1 many.zip | wc -l\" reports 65792 even before\n> adding ZIP64 support; only without -1 we get the interesting numbers\n> (specifically with \"unzip -Z many.zip | sed -n '2p;$p'\"):\n>\n> Zip file size: 6841366 bytes, number of entries: 256\n> 65792 files, 0 bytes uncompressed, 0 bytes compressed: 0.0%\n>\n>> With these three patches applied, Mac OS X has trouble with 'many.zip':\n>>\n>> $ unzip -Z -1 many.zip\n>> warning [many.zip]: 76 extra bytes at beginning or within zipfile\n>> (attempting to process anyway)\n>> error [many.zip]: reported length of central directory is\n>> -76 bytes too long (Atari STZip zipfile? J.H.Holm ZIPSPLIT 1.1\n>> zipfile?). Compensating...\n>> 00/\n>> 00/00\n>> ...\n>> ff/ff\n>> error: expected central file header signature not found (file\n>> #65793). (please check that you have transferred or created the\n>> zipfile in the appropriate BINARY mode and that you have compiled\n>> UnZip properly)\n>>\n>> And FreeBSD doesn't like it either:\n>>\n>> $ unzip -Z -1 many.zip\n>> unzip: Invalid central directory signature\n>> $\n>>\n>\n> Looks like they don't support ZIP64. Or I got some of the fields wrong\n> after all.\n\nA >65536 file zip created on Mac OS X with Mac's \"zip\" command given\nto \"unzip\" or \"zipinfo\" results in exactly the same warnings/errors as\nabove (including the bit about \"76 extra bytes\" and \"-76 bytes too\nlong\"), so it doesn't seem to be a problem with your implementation.\n\n> https://en.wikipedia.org/wiki/Zip_%28file_format%29#ZIP64 says: \"OS X\n> Yosemite does support the creation of ZIP64 archives, but does not\n> support unzipping these archives using the shipped unzip command-line\n> utility or graphical Archive Utility.[citation needed]\".\n>\n> How does unzip react to a ZIP file with more than 65535 entries that\n> was created natively on these platforms? And what does zipinfo (a real\n> one, without -1) report at the top for such files?\n\nOn Mac OS X, unzip does extract all the files (although complains as\nnoted above). zipinfo caps out at reporting 65535 for the number of\nfiles (although it lists them all fine). With the warnings/errors\nfiltered out for clarity:\n\n $ zipinfo biggy.zip\n Archive: biggy.zip 9642874 bytes 65535 files\n ...\n"},{"id":"268539","messageId":"CAPig+cR3zZK5BJmG0S2K0PLcY9p-1Ko4ynR9GzM2wLq8xjn36g@mail.gmail.com","threadId":"40054","inReplyTo":"20150823171622.GA28700@zorg.kyriasis.com","subject":"Re: Eric Sunshine mail delivery failure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-23T18:24:06Z","receivedAt":"2015-08-23T18:24:06Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 23, 2015 at 1:16 PM, Johannes Löthberg\n wrote:\n> On 23/08, René Scharfe wrote:\n>> Eric, hope you see this reply on the list. Direct replies to\n>> sunshine@sunshineco.com are rejected by my mail provider on submit in\n>> Thunderbird with the following message:\n>>\n>> Requested action not taken: mailbox unavailable\n>> invalid DNS MX or A/AAAA resource record.\n>>\n>> And with this one when using their web interface:\n>>\n>> A message that you sent could not be delivered to one or more of\n>> its recipients. This is a permanent error. The following address\n>> failed:\n>>\n>> \"sunshine@sunshineco.com\":\n>> no valid MX hosts found\n>>\n>> It seems web.de wants you to get an AAAA record before I'm allowed to send\n>> mails to you.\n>\n> Just an A record would be enough. The issue is that mail.sunshineco.com has\n> neither an A nor an AAAA record, it is a CNAME to sunshineco.com, which is\n> invalid according to RFC2181.\n\nInterestingly, the default configuration for all domains managed by\nthis service provider is for the mailhost to be a CNAME. While the\nrestriction in section 10.3 of RFC2181 makes sense as a way to avoid\nextra \"network burden\", in practice, email services seem to be pretty\nrelaxed about it, and follow the CNAME indirection as needed.\n\nI suppose it's possible that web.de is being extra strict (although it\nseems that such strictness would be painful for its users), or this\ncould just be a temporary DNS lookup failure. It's hard to tell based\nupon the errors René reported.\n\nI did change the CNAME to an A just in case, though who knows how long\nit will take for the change to propagate over to web.de's server.\n"},{"id":"268541","messageId":"CAPig+cS+sDQV0O=pZXL9sw8ww39J=asxrbNm28jG0VSFhXtmtA@mail.gmail.com","threadId":"40054","inReplyTo":"CA+EOSBmk2cdQe3owaXgkYAgTZqpUFa=J8g5FYq28-=VhDcJ4EA@mail.gmail.com","subject":"Re: Eric Sunshine mail delivery failure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-23T18:48:08Z","receivedAt":"2015-08-23T18:48:08Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 23, 2015 at 2:36 PM, Elia Pinto wrote:\n> Il 23/Ago/2015 20:26, \"Eric Sunshine\" ha scritto:\n>> On Sun, Aug 23, 2015 at 1:16 PM, Johannes Löthberg\n>> wrote:\n>> > Just an A record would be enough. The issue is that mail.sunshineco.com\n>> > has\n>> > neither an A nor an AAAA record, it is a CNAME to sunshineco.com, which\n>> > is\n>> > invalid according to RFC2181.\n>>\n>> Interestingly, the default configuration for all domains managed by\n>> this service provider is for the mailhost to be a CNAME. While the\n>> restriction in section 10.3 of RFC2181 makes sense as a way to avoid\n>> extra \"network burden\", in practice, email services seem to be pretty\n>> relaxed about it, and follow the CNAME indirection as needed.\n>>\n>> I suppose it's possible that web.de is being extra strict (although it\n>> seems that such strictness would be painful for its users), or this\n>> could just be a temporary DNS lookup failure. It's hard to tell based\n>> upon the errors René reported.\n>>\n>> I did change the CNAME to an A just in case, though who knows how long\n>> it will take for the change to propagate over to web.de's server.\n> Anyone can check Here https://dnschecker.org/#CNAME/Mail.sunshineco.com\n> It would fail with your change\n\nInteresting service; thanks for the pointer. However, since it's just\nquerying a random set of DNS servers, it's not necessarily indicative\nof whether the change has actually propagated to the DNS server(s)\nanswering web.de's mail server's queries. Local configuration (TTL's,\netc.) on those servers or anywhere in between, as well as network\nconditions, could impact propagation to an unknown degree.\n"},{"id":"268542","messageId":"CAPig+cSSO3ZTauM1hgUV=govWdmde0ds-fyt7QdHQauiwSmQBw@mail.gmail.com","threadId":"40054","inReplyTo":"CAPig+cS+sDQV0O=pZXL9sw8ww39J=asxrbNm28jG0VSFhXtmtA@mail.gmail.com","subject":"Re: Eric Sunshine mail delivery failure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-23T18:57:56Z","receivedAt":"2015-08-23T18:57:56Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 23, 2015 at 2:48 PM, Eric Sunshine wrote:\n> On Sun, Aug 23, 2015 at 2:36 PM, Elia Pinto wrote:\n>> Il 23/Ago/2015 20:26, \"Eric Sunshine\" ha scritto:\n>>> I did change the CNAME to an A just in case, though who knows how long\n>>> it will take for the change to propagate over to web.de's server.\n>> Anyone can check Here https://dnschecker.org/#CNAME/Mail.sunshineco.com\n>> It would fail with your change\n>\n> Interesting service; thanks for the pointer. However, since it's just\n> querying a random set of DNS servers, it's not necessarily indicative\n> of whether the change has actually propagated to the DNS server(s)\n> answering web.de's mail server's queries. Local configuration (TTL's,\n> etc.) on those servers or anywhere in between, as well as network\n> conditions, could impact propagation to an unknown degree.\n\nAlso, the propagation time of the A record can be quite different from\nthe point at which the CNAME record finally expires (based upon its\nTTL, which may differ dramatically from server to server), so the\nabove CNAME query may continue to succeed long after the A record has\npropagated.\n"},{"id":"268841","messageId":"xmqqwpwf2z8c.fsf@gitster.mtv.corp.google.com","threadId":"40054","inReplyTo":"CAPig+cSNSfpt7gOLvz7P4oDrNF5fTQ38v1pfncJU3h7a6FjMyQ@mail.gmail.com","subject":"Re: [PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-08-28T15:45:39Z","receivedAt":"2015-08-28T15:45:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Sun, Aug 23, 2015 at 5:29 AM, \"René Scharfe\" wrote:\n>> I suspected that zipinfo's output might be formatted differently on\n>> different platforms and tried to guard against it by checking for the\n>> number zero there. Git's ZIP file creation is platform independent\n>> (modulo bugs), so having a test run at least somewhere should\n>> suffice. In theory.\n>>\n>> We could add support for the one-line-summary variant on OS X easily,\n>> though.\n>\n> Probably, although it's looking like testing on Mac OS X won't be\n> fruitful (see below).\n\nCan we move this topic forward by introducing a new prerequisite\nZIPINFO and used at the beginning of these tests (make it a lazy\nprereq)? Run zipinfo on a trivial archive and see if its output is\nsomething we recognize to decide if the platform supports that\nZIPINFO prerequisite and do this test only on them.\n\nAfter all, what _is_ being tested, i.e. our archive creation, would\nnot change across platforms, so having a test that runs on a known\nsubset of platforms is better than not having anything at all.\n\nThanks.\n"},{"id":"268842","messageId":"xmqqsi732yod.fsf@gitster.mtv.corp.google.com","threadId":"40054","inReplyTo":"xmqqwpwf2z8c.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-08-28T15:57:38Z","receivedAt":"2015-08-28T15:57:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Eric Sunshine writes:\n>\n>> On Sun, Aug 23, 2015 at 5:29 AM, \"René Scharfe\" wrote:\n>>> I suspected that zipinfo's output might be formatted differently on\n>>> different platforms and tried to guard against it by checking for the\n>>> number zero there. Git's ZIP file creation is platform independent\n>>> (modulo bugs), so having a test run at least somewhere should\n>>> suffice. In theory.\n>>>\n>>> We could add support for the one-line-summary variant on OS X easily,\n>>> though.\n>>\n>> Probably, although it's looking like testing on Mac OS X won't be\n>> fruitful (see below).\n>\n> Can we move this topic forward by introducing a new prerequisite\n> ZIPINFO and used at the beginning of these tests (make it a lazy\n> prereq)? Run zipinfo on a trivial archive and see if its output is\n> something we recognize to decide if the platform supports that\n> ZIPINFO prerequisite and do this test only on them.\n\nHeh, that is exactly what the patch under discussion does. So...\n\n> After all, what _is_ being tested, i.e. our archive creation, would\n> not change across platforms, so having a test that runs on a known\n> subset of platforms is better than not having anything at all.\n>\n> Thanks.\n\n...I'd say we can take this patch as-is, and those who want to have\na working test on MacOS can come up with an enhancement to the way\nthe script parses output from zipinfo that would also work on their\nplatforms.\n\nThanks and sorry for the noise ;-)\n"},{"id":"268845","messageId":"CAPig+cQzpo=dXxaWmR6AZ2UerZXYzTfMXZQwckrkPmWdDp_wqQ@mail.gmail.com","threadId":"40054","inReplyTo":"xmqqsi732yod.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/3] t5004: test ZIP archives with many entries","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-28T16:47:30Z","receivedAt":"2015-08-28T16:47:30Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Aug 28, 2015 at 11:57 AM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>> Eric Sunshine writes:\n>>> On Sun, Aug 23, 2015 at 5:29 AM, \"René Scharfe\" wrote:\n>>>> I suspected that zipinfo's output might be formatted differently on\n>>>> different platforms and tried to guard against it by checking for the\n>>>> number zero there. Git's ZIP file creation is platform independent\n>>>> (modulo bugs), so having a test run at least somewhere should\n>>>> suffice. In theory.\n>>>>\n>>>> We could add support for the one-line-summary variant on OS X easily,\n>>>> though.\n>>>\n>>> Probably, although it's looking like testing on Mac OS X won't be\n>>> fruitful (see below).\n>>\n>> Can we move this topic forward by introducing a new prerequisite\n>> ZIPINFO and used at the beginning of these tests (make it a lazy\n>> prereq)? Run zipinfo on a trivial archive and see if its output is\n>> something we recognize to decide if the platform supports that\n>> ZIPINFO prerequisite and do this test only on them.\n>\n> Heh, that is exactly what the patch under discussion does. So...\n>\n>> After all, what _is_ being tested, i.e. our archive creation, would\n>> not change across platforms, so having a test that runs on a known\n>> subset of platforms is better than not having anything at all.\n>\n> ...I'd say we can take this patch as-is, and those who want to have\n> a working test on MacOS can come up with an enhancement to the way\n> the script parses output from zipinfo that would also work on their\n> platforms.\n\nRight, the new test is correctly skipped on Mac OS X and FreeBSD, so\nthe patch is suitable as-is. We might, however, want to augment the\ncommit message with some of the knowledge learned from this thread.\nPerhaps modify the last sentence of the second paragraph and then\ninsert additional information following it, like this?\n\n ... at least provides\n *some* way to check this field, although presently only on Linux.\n\n zipinfo on current Mac OS X (Yosemite 10.10.5) does not support\n this field, and, when encountered, caps the printed file count at\n 65535 (and spits out warnings and errors), thus is not useful for\n testing. (Its output also differs from zipinfo on Linux, thus\n requires changes to the 'sed' recognition and extraction\n expressions, but that's a minor issue.)\n\n zipinfo on FreeBSD seems to have been retired altogether in favor\n of \"unzip -Z\", however, only in the emasculated form \"unzip -Z\n -1\" which lists archive entries but does not provide a file\n count, thus is not useful for this test.\n\n(I also snuck a s/can// fix in there for the last sentence of the\nsecond paragraph.)\n"}]}