{"thread":{"id":"31231","subject":"git archive --format zip utf-8 issues","startedAt":"2012-08-10T21:58:12Z","lastAt":"2012-09-24T18:13:04Z","messageCount":21,"participants":["Sven Strickroth","Junio C Hamano","René Scharfe","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"196846","messageId":"502583F4.8030308@tu-clausthal.de","threadId":"31231","inReplyTo":null,"subject":"git archive --format zip utf-8 issues","fromName":"Sven Strickroth","fromEmail":"sven.strickroth@tu-clausthal.de","sentAt":"2012-08-10T21:58:12Z","receivedAt":"2012-08-10T21:58:12Z","isPatch":false,"sender":{"key":"sven.strickroth@tu-clausthal.de","avatar":null},"body":"Hi,\n\nwhen I create a git repository, add a file containing utf-8 characters\nor umlauts (like öäü.txt), commit and then export the HEAD revision to a\nzip archive using \"git archive --format zip -o 1.zip HEAD\", the zip file\ncontains incorrect filenames:\n\n$ unzip -l 1.zip\nArchive: 1.zip\n4490a6dab1df5404f91ab3eb871f133154bff0bf\n Length Date Time Name\n--------- ---------- ----- ----\n 6 2012-08-10 23:41 +?+?++.txt\n--------- -------\n 6 1 file\n-- \nBest regards,\n Sven Strickroth\n PGP key id F5A9D4C4 @ any key-server\n"},{"id":"196849","messageId":"7vtxwagy9f.fsf@alter.siamese.dyndns.org","threadId":"31231","inReplyTo":"502583F4.8030308@tu-clausthal.de","subject":"Re: git archive --format zip utf-8 issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-10T22:47:40Z","receivedAt":"2012-08-10T22:47:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sven Strickroth writes:\n\n> when I create a git repository, add a file containing utf-8 characters\n> or umlauts (like öäü.txt), commit and then export the HEAD revision to a\n> zip archive using \"git archive --format zip -o 1.zip HEAD\", the zip file\n> contains incorrect filenames:\n\nMy reading of archive-zip.c seems to suggest that we write out\nwhatever pathname you have in the tree, so a pathname encoded in\nUTF-8 will be literally written out in the resulting zip archive.\n\nDo you know in what encoding the pathnames are _expected_ to be\nstored in zip archives? Random documentation seems to suggest that\nthere is no standard encoding, e.g. http://docs.python.org/library/zipfile.html\nsays:\n\n There is no official file name encoding for ZIP files. If you\n have unicode file names, you must convert them to byte strings\n in your desired encoding before passing them to write(). WinZip\n interprets all file names as encoded in CP437, also known as DOS\n Latin.\n\nwhich may explain it.\n\nIt may not be a bad idea for \"git archive --format=zip\" to\n\n (1) check if pathname is a correct UTF-8; and\n (2) check if it can be reencoded to latin-1\n\nand if (and only if) both are true, automatically re-encode the path\nto latin-1.\n\nOf course, \"git archive --format=zip --path-reencode=utf8-to-latin1\"\nwould be the most generic way to do this.\n"},{"id":"196850","messageId":"50259EEC.2020701@tu-clausthal.de","threadId":"31231","inReplyTo":"7vtxwagy9f.fsf@alter.siamese.dyndns.org","subject":"Re: git archive --format zip utf-8 issues","fromName":"Sven Strickroth","fromEmail":"sven.strickroth@tu-clausthal.de","sentAt":"2012-08-10T23:53:16Z","receivedAt":"2012-08-10T23:53:16Z","isPatch":false,"sender":{"key":"sven.strickroth@tu-clausthal.de","avatar":null},"body":"Am 11.08.2012 00:47 schrieb Junio C Hamano:\n> Do you know in what encoding the pathnames are _expected_ to be\n> stored in zip archives?\n\nre-encoding to latin1 does not always work and may break double byte\ntotally (e.g. chinese or japanese).\n\nPKZIP APPNOTE seems to be the zip standard and it specifies a utf-8\nflag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT\n> A. Local file header:\n> general purpose bit flag: (2 bytes)\n> Bit 11: Language encoding flag (EFS). If this bit is\n> set, the filename and comment fields for this file\n> must be encoded using UTF-8. (see APPENDIX D)\n\n-- \nBest regards,\n Sven Strickroth\n PGP key id F5A9D4C4 @ any key-server\n"},{"id":"196871","messageId":"5026C649.2090700@lsrfire.ath.cx","threadId":"31231","inReplyTo":"7vtxwagy9f.fsf@alter.siamese.dyndns.org","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-08-11T20:53:29Z","receivedAt":"2012-08-11T20:53:29Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 11.08.2012 00:47, schrieb Junio C Hamano:\n> Sven Strickroth writes:\n>\n>> when I create a git repository, add a file containing utf-8 characters\n>> or umlauts (like öäü.txt), commit and then export the HEAD revision to a\n>> zip archive using \"git archive --format zip -o 1.zip HEAD\", the zip file\n>> contains incorrect filenames:\n>\n> My reading of archive-zip.c seems to suggest that we write out\n> whatever pathname you have in the tree, so a pathname encoded in\n> UTF-8 will be literally written out in the resulting zip archive.\n\nSorry for my imperialistic attitude of \"ASCII filenames should be enough \nfor everybody\". Laziness..\n\n> Do you know in what encoding the pathnames are _expected_ to be\n> stored in zip archives? Random documentation seems to suggest that\n> there is no standard encoding, e.g. http://docs.python.org/library/zipfile.html\n> says:\n>\n> There is no official file name encoding for ZIP files. If you\n> have unicode file names, you must convert them to byte strings\n> in your desired encoding before passing them to write(). WinZip\n> interprets all file names as encoded in CP437, also known as DOS\n> Latin.\n>\n> which may explain it.\n\nhttp://www.pkware.com/documents/casestudies/APPNOTE.TXT is the standard \ndocument, as Sven noted, and it says that filenames are encoded in code \npage 437, or optionally UTF-8 (a later addition). Discussions like \nhttp://stackoverflow.com/questions/106367/ seem to indicate that at \nleast some archivers use the local code page as well.\n\n> It may not be a bad idea for \"git archive --format=zip\" to\n>\n> (1) check if pathname is a correct UTF-8; and\n> (2) check if it can be reencoded to latin-1\n>\n> and if (and only if) both are true, automatically re-encode the path\n> to latin-1.\n\nThe standard says we need to convert to CP437, or to UTF-8, or provide \nboth versions. A more interesting question is: What's supported by which \nprograms?\n\nThe ZIP functionality built into Windows 7 doesn't seem to work with \nUTF-8 encoded filenames (except for those that only use the ASCII \nsubset), and to ignore the UTF-8 part if both are given. Handling \numlauts should be possible anyway, because they are on code page 437, \nbut for other characters we'd have to aim for compatibility with other \nprograms like Info-ZIP and 7-Zip.\n\nHow do we know which encoding was used for a filename?\n\n> Of course, \"git archive --format=zip --path-reencode=utf8-to-latin1\"\n> would be the most generic way to do this.\n\nI really hope we can make do without additional options.\n\nRené\n"},{"id":"196872","messageId":"5026C651.8050705@lsrfire.ath.cx","threadId":"31231","inReplyTo":"50259EEC.2020701@tu-clausthal.de","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-08-11T20:53:37Z","receivedAt":"2012-08-11T20:53:37Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 11.08.2012 01:53, schrieb Sven Strickroth:\n> Am 11.08.2012 00:47 schrieb Junio C Hamano:\n>> Do you know in what encoding the pathnames are _expected_ to be\n>> stored in zip archives?\n>\n> re-encoding to latin1 does not always work and may break double byte\n> totally (e.g. chinese or japanese).\n>\n> PKZIP APPNOTE seems to be the zip standard and it specifies a utf-8\n> flag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT\n>> A. Local file header:\n>> general purpose bit flag: (2 bytes)\n>> Bit 11: Language encoding flag (EFS). If this bit is\n>> set, the filename and comment fields for this file\n>> must be encoded using UTF-8. (see APPENDIX D)\n\nYes, that's one of the two methods for supporting UTF-8 filenames \ndescribed there.\n\nThe other method involves writing extra ZIP header fields and was \ninvented by Info-ZIP. They don't use it consistently anymore, though \n(from zip -h2):\n\n \"Zip now stores UTF-8 in entry path and comment fields on systems\n where UTF-8 char set is default, such as most modern Unix, and\n and on other systems in new extra fields with escaped versions in\n entry path and comment fields for backward compatibility.\"\n\nRené\n"},{"id":"196873","messageId":"5026D081.2040906@tu-clausthal.de","threadId":"31231","inReplyTo":"5026C649.2090700@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"Sven Strickroth","fromEmail":"sven.strickroth@tu-clausthal.de","sentAt":"2012-08-11T21:37:05Z","receivedAt":"2012-08-11T21:37:05Z","isPatch":false,"sender":{"key":"sven.strickroth@tu-clausthal.de","avatar":null},"body":"Am 11.08.2012 22:53 schrieb René Scharfe:\n> The standard says we need to convert to CP437, or to UTF-8, or provide \n> both versions. A more interesting question is: What's supported by which \n> programs?\n> \n> The ZIP functionality built into Windows 7 doesn't seem to work with \n> UTF-8 encoded filenames (except for those that only use the ASCII \n> subset), and to ignore the UTF-8 part if both are given.\n\nI played a bit with the git source code and found out, that\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex f5af81f..e0ccb4f 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -257,7 +257,7 @@ static int write_zip_entry(struct archiver_args *args,\n \tcopy_le16(dirent.creator_version,\n \t\tS_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0);\n \tcopy_le16(dirent.version, 10);\n-\tcopy_le16(dirent.flags, flags);\n+\tcopy_le16(dirent.flags, flags+2048);\n \tcopy_le16(dirent.compression_method, method);\n \tcopy_le16(dirent.mtime, zip_time);\n \tcopy_le16(dirent.mdate, zip_date);\n--\nworks with 7-zip, however, not with Windows 7 build-in zip.\n\nIf I create a zip file with 7-zip which contains umlauts and other\nunicode chars like (國立1-кккк.txt) the Windows 7 build-in zip displays\nthem correctly, too.\n\n-- \nBest regards,\n Sven Strickroth\n PGP key id F5A9D4C4 @ any key-server\n"},{"id":"196874","messageId":"7vlihkhhwc.fsf@alter.siamese.dyndns.org","threadId":"31231","inReplyTo":"5026C651.8050705@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-12T04:08:03Z","receivedAt":"2012-08-12T04:08:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n>> PKZIP APPNOTE seems to be the zip standard and it specifies a utf-8\n>> flag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT\n>>> A. Local file header:\n>>> general purpose bit flag: (2 bytes)\n>>> Bit 11: Language encoding flag (EFS). If this bit is\n>>> set, the filename and comment fields for this file\n>>> must be encoded using UTF-8. (see APPENDIX D)\n>\n> Yes, that's one of the two methods for supporting UTF-8 filenames\n> described there.\n>\n> The other method involves writing extra ZIP header fields and was\n> invented by Info-ZIP. They don't use it consistently anymore, though\n> (from zip -h2):\n>\n> \"Zip now stores UTF-8 in entry path and comment fields on systems\n> where UTF-8 char set is default, such as most modern Unix, and\n> and on other systems in new extra fields with escaped versions in\n> entry path and comment fields for backward compatibility.\"\n\nThanks; so if we adopt one of these methods, the readers that matter\nwill be happy? And if so, which one? Or both?\n"},{"id":"196875","messageId":"7vhas8hgzl.fsf@alter.siamese.dyndns.org","threadId":"31231","inReplyTo":"5026C649.2090700@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-12T04:27:42Z","receivedAt":"2012-08-12T04:27:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> ... A more interesting question is: What's supported by\n> which programs?\n\nYes, that is the most interesting question.\n\n>> Of course, \"git archive --format=zip --path-reencode=utf8-to-latin1\"\n>> would be the most generic way to do this.\n>\n> I really hope we can make do without additional options.\n\nWe need to at least know the path encoding used in the tree objects,\nand I'd be OK with a solution that assumes a single encoding is used\nfor the entire tree.\n\nWe would eventually need to also know the encoding used on the local\nworking tree (i.e. in what encoding paths are returned from\nreaddir() and the pathspec the user gives us from the command line),\nand iconv it to the tree objects encoding for the project when\ncreating a cache_entry object to be fed to add_to_index(), and iconv\nit back from the tree objects encoding to the working tree encoding\nin write_entry(), but that is a longer term direction. For now, in\norder to address the immediate issue, we only need the tree object\nencoding, which should default to UTF-8 for interoperability.\n\nSo \"git archive --format=zip --in-object-path-encoding=big5\" for a\nproject whose tree object pathnames are in that encoding (and we\nalways record paths in UTF-8 when writing zipfiles) should be the\nminimal that we need for now.\n\nOptionally, with a configuration variable i18n.inObjectPathEncoding\n(as opposed to the eventual i18n.worktreePathEncoding) set to big5,\nusers of such a project can say \"git archive --format=zip\" without\nthe \"--in-object-path-encoding\" option.\n\nConsidering that zip is a format meant for exchange, I'd think we\nwould be fine to always write in UTF-8 and leaving the readers\nresponsible for converting the pathname while extracting. If a\nmajor zip extractor is incapable of handling UTF-8 (or even if\ncapable it is cumbersome, for that matter), we may end up having to\nadd \"--in-archive-path-encoding=UTF-8\" option to \"git archive\", with\nassociated \"zip.archivePathEncoding\" variable, though.\n"},{"id":"198131","messageId":"20120830222603.GA20289@sigill.intra.peff.net","threadId":"31231","inReplyTo":"5026D081.2040906@tu-clausthal.de","subject":"Re: git archive --format zip utf-8 issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-08-30T22:26:03Z","receivedAt":"2012-08-30T22:26:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 11, 2012 at 11:37:05PM +0200, Sven Strickroth wrote:\n\n> Am 11.08.2012 22:53 schrieb René Scharfe:\n> > The standard says we need to convert to CP437, or to UTF-8, or provide \n> > both versions. A more interesting question is: What's supported by which \n> > programs?\n> > \n> > The ZIP functionality built into Windows 7 doesn't seem to work with \n> > UTF-8 encoded filenames (except for those that only use the ASCII \n> > subset), and to ignore the UTF-8 part if both are given.\n> \n> I played a bit with the git source code and found out, that\n> \n> diff --git a/archive-zip.c b/archive-zip.c\n> index f5af81f..e0ccb4f 100644\n> --- a/archive-zip.c\n> +++ b/archive-zip.c\n> @@ -257,7 +257,7 @@ static int write_zip_entry(struct archiver_args *args,\n> \tcopy_le16(dirent.creator_version,\n> \t\tS_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0);\n> \tcopy_le16(dirent.version, 10);\n> -\tcopy_le16(dirent.flags, flags);\n> +\tcopy_le16(dirent.flags, flags+2048);\n> \tcopy_le16(dirent.compression_method, method);\n> \tcopy_le16(dirent.mtime, zip_time);\n> \tcopy_le16(dirent.mdate, zip_date);\n> --\n> works with 7-zip, however, not with Windows 7 build-in zip.\n> \n> If I create a zip file with 7-zip which contains umlauts and other\n> unicode chars like (國立1-кккк.txt) the Windows 7 build-in zip displays\n> them correctly, too.\n\nPing on this stalled discussion.\n\nIt seems like there are two separate issues here:\n\n 1. Knowing the encoding of pathnames in the repository.\n\n 2. Setting the right flags in zip output.\n\nA full solution would handle both parts, but let's ignore (1) for a\nmoment, and assume we have utf-8 (or can massage into utf-8 from an\nencoding specified by the user).\n\nIt seems like just setting the magic utf-8 flag would be the only thing\nwe need to do, according to the standard. But according to discussions\nreferenced elsewhere in this thread, that flag was invented only in\n2007, so we may be dealing with older implementations (I have no idea\nhow common they would be; that may be the problem with Windows 7's zip\nyou are seeing). We could re-encode to cp437, which the standard\nspecifies, but apparently some implementations do not respect that\n(and use a local code page instead). And it cannot represent all utf-8\ncharacters, anyway.\n\nIt sounds like 7-zip has figured out a more portable solution. Can you\nshow us a sample of 7-zip's output with utf-8 characters to compare to\nwhat git generates? I wonder if it is using a combination of methods.\n\n-Peff\n"},{"id":"198340","messageId":"5046634A.4020608@lsrfire.ath.cx","threadId":"31231","inReplyTo":"20120830222603.GA20289@sigill.intra.peff.net","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-04T20:23:38Z","receivedAt":"2012-09-04T20:23:38Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 31.08.2012 00:26, schrieb Jeff King:\n> Ping on this stalled discussion.\n\nSorry, I got distracted by other stuff again. I did some experiments,\nthough, and here's a preliminary result.\n\n> It seems like there are two separate issues here:\n> \n> 1. Knowing the encoding of pathnames in the repository.\n> \n> 2. Setting the right flags in zip output.\n> \n> A full solution would handle both parts, but let's ignore (1) for a\n> moment, and assume we have utf-8 (or can massage into utf-8 from an\n> encoding specified by the user).\n\nYes, good thinking. Re-encoding may be beneficial for tar files as well,\nbut we can ignore that point for the moment.\n\n> It seems like just setting the magic utf-8 flag would be the only thing\n> we need to do, according to the standard. But according to discussions\n> referenced elsewhere in this thread, that flag was invented only in\n> 2007, so we may be dealing with older implementations (I have no idea\n> how common they would be; that may be the problem with Windows 7's zip\n> you are seeing). We could re-encode to cp437, which the standard\n> specifies, but apparently some implementations do not respect that\n> (and use a local code page instead). And it cannot represent all utf-8\n> characters, anyway.\n\nYes, we could do that, plus adding an extra field with a UTF-8 version of\nthe path. That's the legacy method invented by Info-ZIP. They switched\nto using the new flag on Linux at least, though.\n\n> It sounds like 7-zip has figured out a more portable solution. Can you\n> show us a sample of 7-zip's output with utf-8 characters to compare to\n> what git generates? I wonder if it is using a combination of methods.\n\nI'm not so sure they produce more portable files. I created an archive\nwith files named jaя.txt, smørrebrød.txt, süd.txt and €uro.txt with 7-Zip\non Windows 7 and while unzip on Ubuntu 12.04 managed to recreate the\ncyrillic character and the Euro symbol, it mangled the slashed o and the\numlaut.\n\nWith the following patch I could create archives with git on Linux and\nmsysgit and extract them flawlessly on Windows with 7-Zip and with\nInfo-ZIP unzip on Linux, but not with unzip on Windows, where it mangled\nall non-ASCII characters.\n\nThis gets confusing; it would help to have a compatibility matrix for all\nintersting extractors and character classes -- for each proposed solution\nor archiver we'd like to imitate.\n\nBut now for the patch, which is a bit confusing as well. I'm curious to\nhear about results for more platforms, extractors and character classes.\nBased on that we can see if we need to generate the extra fields instead\nof relying on the new flag.\n\n-- >8 --\nSubject: [PATCH] archive-zip: support UTF-8 paths\n\nSet general purpose flag 11 if we encounter a path that contains\nnon-ASCII characters. We assume that all paths are given as UTF-8; no\nconversion is done.\n\nThe flag seems to be ignored by unzip unless we also mark the archive\nentry as coming from a Unix system. This is done by setting the field\ncreator_version (\"version made by\" in the standard[1]) to 0x03NN.\n\nThe NN part represents the version of the standard supported by us, and\nthis patch sets it to 3f (for version 6.3) for Unix paths. We keep\ncreator_version set to 0 (FAT filesystem, standard version 0) in the\nnon-special cases, as before.\n\nBut when we declare a file to have a Unix path, then we have to set the\nfile mode as well, or unzip will extract the files with the permission\nset 0000, i.e. inaccessible by all.\n\n[1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT\n\nSigned-off-by: Rene Scharfe \n---\n archive-zip.c | 27 +++++++++++++++++++++------\n 1 file changed, 21 insertions(+), 6 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex f5af81f..928da1d 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -4,6 +4,8 @@\n #include \"cache.h\"\n #include \"archive.h\"\n #include \"streaming.h\"\n+#include \"commit.h\"\n+#include \"utf8.h\"\n \n static int zip_date;\n static int zip_time;\n@@ -16,7 +18,8 @@ static unsigned int zip_dir_offset;\n static unsigned int zip_dir_entries;\n \n #define ZIP_DIRECTORY_MIN_SIZE\t(1024 * 1024)\n-#define ZIP_STREAM (8)\n+#define ZIP_STREAM\t(1 << 3)\n+#define ZIP_UTF8\t(1 << 11)\n \n struct zip_local_header {\n \tunsigned char magic[4];\n@@ -173,7 +176,8 @@ static int write_zip_entry(struct archiver_args *args,\n {\n \tstruct zip_local_header header;\n \tstruct zip_dir_header dirent;\n-\tunsigned long attr2;\n+\tunsigned int creator_version = 0;\n+\tunsigned long attr2 = 0;\n \tunsigned long compressed_size;\n \tunsigned long crc;\n \tunsigned long direntsize;\n@@ -187,6 +191,13 @@ static int write_zip_entry(struct archiver_args *args,\n \n \tcrc = crc32(0, NULL, 0);\n \n+\tif (has_non_ascii(path)) {\n+\t\tif (is_utf8(path))\n+\t\t\tflags |= ZIP_UTF8;\n+\t\telse\n+\t\t\twarning(\"Path is not valid UTF-8: %s\", path);\n+\t}\n+\n \tif (pathlen > 0xffff) {\n \t\treturn error(\"path too long (%d chars, SHA1: %s): %s\",\n \t\t\t\t(int)pathlen, sha1_to_hex(sha1), path);\n@@ -204,10 +215,15 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tenum object_type type = sha1_object_info(sha1, &size);\n \n \t\tmethod = 0;\n-\t\tattr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :\n-\t\t\t(mode & 0111) ? ((mode) << 16) : 0;\n \t\tif (S_ISREG(mode) && args->compression_level != 0 && size > 0)\n \t\t\tmethod = 8;\n+\t\tif (S_ISLNK(mode) || (mode & 0111) || (flags & ZIP_UTF8)) {\n+\t\t\tcreator_version = 0x033f;\n+\t\t\tattr2 = mode;\n+\t\t\tif (S_ISLNK(mode))\n+\t\t\t\tattr2 |= 0777;\n+\t\t\tattr2 <<= 16;\n+\t\t}\n \t\tcompressed_size = size;\n \n \t\tif (S_ISREG(mode) && type == OBJ_BLOB && !args->convert &&\n@@ -254,8 +270,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t}\n \n \tcopy_le32(dirent.magic, 0x02014b50);\n-\tcopy_le16(dirent.creator_version,\n-\t\tS_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0);\n+\tcopy_le16(dirent.creator_version, creator_version);\n \tcopy_le16(dirent.version, 10);\n \tcopy_le16(dirent.flags, flags);\n \tcopy_le16(dirent.compression_method, method);\n-- \n1.7.11.msysgit.1.1.gbf71771\n"},{"id":"198344","messageId":"7vehmh8prt.fsf@alter.siamese.dyndns.org","threadId":"31231","inReplyTo":"5046634A.4020608@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-04T21:03:02Z","receivedAt":"2012-09-04T21:03:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> But now for the patch, which is a bit confusing as well. I'm curious to\n> hear about results for more platforms, extractors and character classes.\n> Based on that we can see if we need to generate the extra fields instead\n> of relying on the new flag.\n\nThanks for keeping the ball rolling.\n\n> Subject: [PATCH] archive-zip: support UTF-8 paths\n>\n> Set general purpose flag 11 if we encounter a path that contains\n> non-ASCII characters. We assume that all paths are given as UTF-8; no\n> conversion is done.\n>\n> The flag seems to be ignored by unzip unless we also mark the archive\n> entry as coming from a Unix system. This is done by setting the field\n> creator_version (\"version made by\" in the standard[1]) to 0x03NN.\n>\n> The NN part represents the version of the standard supported by us, and\n> this patch sets it to 3f (for version 6.3) for Unix paths. We keep\n> creator_version set to 0 (FAT filesystem, standard version 0) in the\n> non-special cases, as before.\n>\n> But when we declare a file to have a Unix path, then we have to set the\n> file mode as well, or unzip will extract the files with the permission\n> set 0000, i.e. inaccessible by all.\n>\n> [1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT\n>\n> Signed-off-by: Rene Scharfe \n> ---\n> archive-zip.c | 27 +++++++++++++++++++++------\n> 1 file changed, 21 insertions(+), 6 deletions(-)\n>\n> diff --git a/archive-zip.c b/archive-zip.c\n> index f5af81f..928da1d 100644\n> --- a/archive-zip.c\n> +++ b/archive-zip.c\n> @@ -4,6 +4,8 @@\n> #include \"cache.h\"\n> #include \"archive.h\"\n> #include \"streaming.h\"\n> +#include \"commit.h\"\n> +#include \"utf8.h\"\n> \n> static int zip_date;\n> static int zip_time;\n> @@ -16,7 +18,8 @@ static unsigned int zip_dir_offset;\n> static unsigned int zip_dir_entries;\n> \n> #define ZIP_DIRECTORY_MIN_SIZE\t(1024 * 1024)\n> -#define ZIP_STREAM (8)\n> +#define ZIP_STREAM\t(1 << 3)\n> +#define ZIP_UTF8\t(1 << 11)\n> \n> struct zip_local_header {\n> \tunsigned char magic[4];\n> @@ -173,7 +176,8 @@ static int write_zip_entry(struct archiver_args *args,\n> {\n> \tstruct zip_local_header header;\n> \tstruct zip_dir_header dirent;\n> -\tunsigned long attr2;\n> +\tunsigned int creator_version = 0;\n> +\tunsigned long attr2 = 0;\n> \tunsigned long compressed_size;\n> \tunsigned long crc;\n> \tunsigned long direntsize;\n> @@ -187,6 +191,13 @@ static int write_zip_entry(struct archiver_args *args,\n> \n> \tcrc = crc32(0, NULL, 0);\n> \n> +\tif (has_non_ascii(path)) {\n\nDo we want to treat \\033 as \"ascii\" in this codepath? The function\nprimarily is used by the log formatter to see if we need 8-bit CTE\nwhen writing out in the e-mail format.\n\n> +\t\tif (is_utf8(path))\n> +\t\t\tflags |= ZIP_UTF8;\n> +\t\telse\n> +\t\t\twarning(\"Path is not valid UTF-8: %s\", path);\n> +\t}\n> +\n> \tif (pathlen > 0xffff) {\n> \t\treturn error(\"path too long (%d chars, SHA1: %s): %s\",\n> \t\t\t\t(int)pathlen, sha1_to_hex(sha1), path);\n> @@ -204,10 +215,15 @@ static int write_zip_entry(struct archiver_args *args,\n> \t\tenum object_type type = sha1_object_info(sha1, &size);\n> \n> \t\tmethod = 0;\n> -\t\tattr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :\n> -\t\t\t(mode & 0111) ? ((mode) << 16) : 0;\n> \t\tif (S_ISREG(mode) && args->compression_level != 0 && size > 0)\n> \t\t\tmethod = 8;\n> +\t\tif (S_ISLNK(mode) || (mode & 0111) || (flags & ZIP_UTF8)) {\n> +\t\t\tcreator_version = 0x033f;\n> +\t\t\tattr2 = mode;\n> +\t\t\tif (S_ISLNK(mode))\n> +\t\t\t\tattr2 |= 0777;\n> +\t\t\tattr2 <<= 16;\n> +\t\t}\n> \t\tcompressed_size = size;\n> \n> \t\tif (S_ISREG(mode) && type == OBJ_BLOB && !args->convert &&\n> @@ -254,8 +270,7 @@ static int write_zip_entry(struct archiver_args *args,\n> \t}\n> \n> \tcopy_le32(dirent.magic, 0x02014b50);\n> -\tcopy_le16(dirent.creator_version,\n> -\t\tS_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0);\n> +\tcopy_le16(dirent.creator_version, creator_version);\n> \tcopy_le16(dirent.version, 10);\n> \tcopy_le16(dirent.flags, flags);\n> \tcopy_le16(dirent.compression_method, method);\n"},{"id":"198398","messageId":"5047A9C0.9020200@lsrfire.ath.cx","threadId":"31231","inReplyTo":"7vehmh8prt.fsf@alter.siamese.dyndns.org","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-05T19:36:32Z","receivedAt":"2012-09-05T19:36:32Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.09.2012 23:03, schrieb Junio C Hamano:\n> René Scharfe writes:\n>> +\tif (has_non_ascii(path)) {\n>\n> Do we want to treat \\033 as \"ascii\" in this codepath? The function\n> primarily is used by the log formatter to see if we need 8-bit CTE\n> when writing out in the e-mail format.\n\nArgh, yes, I'd think so. The function name mislead me. This won't \nmatter for compatibility testing, but should be corrected before inclusion.\n\nJust checked: unzip strips the ESC character when extracting (whether \nthe UTF-8 flag is set or not) and 7-Zip replaces it with an underscore. \nThe built-in ZIP extractor of Windows 7 skips such a file; it doesn't \neven show up in archive directory listings.\n\nRené\n"},{"id":"199392","messageId":"5058CE49.3070108@lsrfire.ath.cx","threadId":"31231","inReplyTo":"5047A9C0.9020200@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-18T19:40:57Z","receivedAt":"2012-09-18T19:40:57Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Hello again,\n\nso two weeks have passed, and I've moved at a glacial pace towards a \nmethod how to measure compatibility of our generated ZIP files. Sorry, \nI just keep getting distracted.\n\nAnyway, the idea is to have a bunch of files with names using different \nscripts, zip them with several packers (including git archive), unzip \nthem and compare the result with the original files.\n\nAs test corpus I used files named like the pangrams on this UTF-8 \nsampler page, the exact commands are attached:\n\n http://www.columbia.edu/~fdc/utf8/index.html#quickbrownfox\n\nThe numbers below are how many lines the output of diff -ru contains for \nthis pair of packer and unpacker. There are 37 files, so the worst \nresult is 74 lines of difference (\"Only in [...]\" for both sides), while \n0 indicates a perfect score.\n\nHmm, come to think of it, an empty directory would show up as 37, so \nthis metric is not ideal. A better one would be to simply give one \npoint for each correctly unpacked file.\n\n Windows Info-ZIP unzip\n 7-Zip PeaZip builtin Linux msysgit Windows\n7-Zip 9.20 0 0 46 26 43 43\nPeaZip 4.7.1 win64 0 0 46 26 42 42\nInfo-ZIP zip 3.0 Linux 0 0 72 0 43 43\nInfo-ZIP zip 3.0 Windows 45 45 n/a 0 43 43\ngit-master 72 72 72 60 72 72\ngit-master-patch1 0 0 72 60 72 72\ngit-master-patch2 0 0 72 0 72 72\ngit-v1.7.11.msysgit.1 72 72 72 60 72 72\ngit-v1.7.11.msysgit.1-patch1 0 0 72 60 72 72\ngit-v1.7.11.msysgit.1-patch2 0 0 72 0 72 72\n\nInfo-ZIP's programs don't work too well on Windows. The built-in \nunzipper of Windows 7 even refuses to open the file created by the \nWindows version of zip. Speaking of which, this is the worst of the \nunpackers.\n\nWith the two patches applied, we can say \"use 7-Zip or PeaZip on Windows \nand unzip on Linux\" and filenames with all tested characters will be \npreserved. I was surprised to see this working fine with msysgit like \nthat, even though no reencoding is introduced by the patches.\n\nI wonder what 7-Zip and PeaZip do that gives them a slightly nicer score \nwith the Windows-internal unzipper. Umlauts, Nordic characters and \naccents are preserved by that combination. It seems that unzip on Linux \nfails to unpack exactly these names, so perhaps they employ a dirty \ntrick like using the local encoding in the ZIP file, which makes it \nunportable.\n\nI'll reply with the two patches, which contain basically the same code \nas the previous patch, only split up. The second one declares that \nfilenames with UTF-8 encoding came from Unix (instead of FAT), which \nmakes unzip happy. This, however, implies that we contain Unix \npermissions for these entries, which is a bit ugly.\n\nRené\n\n\n#!/bin/sh\n(\n\tmkdir pangrams\n\tcd pangrams\n\n\techo English >\"The quick brown fox jumps over the lazy dog\"\n\techo Irish 1 >\"An ḃfuil do Ä‹roí ag bualaḋ ó ḟaitíos an Ä¡rá a á¹�eall\"\n\techo Irish 2 >\"lena ṗóg éada ó ṡlí do leasa ṫú\"\n\techo Irish 3 >\"D'ḟuascail Ã�osa Úrá¹�ac na hÓiÄ¡e Beannaiṫe pór\"\n\techo Irish 4 >\"Éava agus Ã�ḋaiá¹�\"\n\techo Dutch >\"Pa's wijze lynx bezag vroom het fikse aquaduct\"\n\techo German 1 >\"Falsches Ãœben von Xylophonmusik quält\"\n\techo German 2 >\"jeden größeren Zwerg\"\n\techo Norwegian >\"BlÃ¥bærsyltetøy\"\n\techo Danish >\"Høj bly gom vandt fræk sexquiz pÃ¥ wc\"\n\techo Swedish >\"Flygande bäckasiner söka strax hwila pÃ¥ mjuka tuvor\"\n\techo Icelandic >\"Sævör grét áðan því úlpan var ónýt\"\n\techo Finnish >\"Törkylempijävongahdus\"\n\techo Polish >\"Pchnąć w tÄ™ łódź jeża lub osiem skrzyÅ„ fig\"\n\techo Czech >\"PříliÅ¡ žluÅ¥ouÄ�ký kůň úpÄ›l Ä�ábelské kódy\"\n\techo Slovak 1 >\"Starý kôň na hÅ•be kníh žuje tíško povädnuté ruže\"\n\techo Slovak 2 >\"na stĺpe sa Ä�ateľ uÄ�í kvákaÅ¥ novú ódu o živote\"\n\techo monotonic Greek >\"ξεσκεπάζω την ψυχοφθόÏ�α βδελυγμία\"\n\techo polytonic Greek >\"ξεσκεπάζω τὴν ψυχοφθόÏ�α βδελυγμία\"\n\techo Russian >\"Съешь же ещё Ñ�тих мÑ�гких французÑ�ких булок да выпей чаю\"\n\techo Bulgarian 1 >\"Жълтата дюлÑ� беше щаÑ�тлива\"\n\techo Bulgarian 2 >\"че пухът, който цъфна, замръзна като гьон\"\n\techo Northern Sami >\"Vuol Ruoŧa geÄ‘ggiid leat máŋga luosa ja Ä�uovžža\"\n\techo Hungarian >\"Ã�rvíztűrÅ‘ tükörfúrógép\"\n\techo Spanish 1 >\"El pingüino Wenceslao hizo kilómetros bajo exhaustiva\"\n\techo Spanish 2 >\"lluvia y frío añoraba a su querido cachorro\"\n\techo Portuguese 1 >\"O próximo vôo à noite sobre o Atlântico\"\n\techo Portuguese 2 >\"põe freqüentemente o único médico\"\n\techo French 1 >\"Les naïfs ægithales hâtifs pondant à Noël où il gèle\"\n\techo French 2 >\"sont sûrs d'être déçus en voyant leurs drôles\"\n\techo French 3 >\"d'Å“ufs abîmés\"\n\techo Esperanto >\"EÄ¥oÅ�anÄ�o ĉiuĵaÅ­de\"\n\techo Hebrew >\"×–×” ×›×™×£ סת×� לשמוע ×�יך תנצח קרפד ×¢×¥ טוב בגן\"\n\techo Hiragana 1 >\"ã�„ã‚�ã�¯ã�«ã�»ã�¸ã�©ã€€ã�¡ã‚Šã�¬ã‚‹ã‚’\"\n\techo Hiragana 2 >\"ã‚�ã�Œã‚ˆã�Ÿã‚Œã�žã€€ã�¤ã�­ã�ªã‚‰ã‚€\"\n\techo Hiragana 3 >\"ã�†ã‚�ã�®ã�Šã��ã‚„ã�¾ã€€ã�‘ã�µã�“ã�ˆã�¦\"\n\techo Hiragana 4 >\"ã�‚ã�•ã��ゆã‚�ã�¿ã�˜ã€€ã‚‘ã�²ã‚‚ã�›ã�š\"\n)\n"},{"id":"199395","messageId":"5058CFB0.6040506@lsrfire.ath.cx","threadId":"31231","inReplyTo":"5058CE49.3070108@lsrfire.ath.cx","subject":"[PATCH 1/2] archive-zip: support UTF-8 paths","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-18T19:46:56Z","receivedAt":"2012-09-18T19:46:56Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Set general purpose flag 11 if we encounter a path that contains\nnon-ASCII characters. We assume that all paths are given as UTF-8; no\nconversion is done.\n\nSigned-off-by: Rene Scharfe \n---\nChanges from previous version: Stop using has_non_ascii(), which does\nslightly too much for our purposes, and split off creator version\nchange into a separate patch.\n\n archive-zip.c | 22 +++++++++++++++++++++-\n 1 file changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex f5af81f..0f763e8 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -4,6 +4,7 @@\n #include \"cache.h\"\n #include \"archive.h\"\n #include \"streaming.h\"\n+#include \"utf8.h\"\n \n static int zip_date;\n static int zip_time;\n@@ -16,7 +17,8 @@ static unsigned int zip_dir_offset;\n static unsigned int zip_dir_entries;\n \n #define ZIP_DIRECTORY_MIN_SIZE\t(1024 * 1024)\n-#define ZIP_STREAM (8)\n+#define ZIP_STREAM\t(1 << 3)\n+#define ZIP_UTF8\t(1 << 11)\n \n struct zip_local_header {\n \tunsigned char magic[4];\n@@ -164,6 +166,17 @@ static void set_zip_header_data_desc(struct zip_local_header *header,\n \tcopy_le32(header->size, size);\n }\n \n+static int has_only_ascii(const char *s)\n+{\n+\tfor (;;) {\n+\t\tint c = *s++;\n+\t\tif (c == '\\0')\n+\t\t\treturn 1;\n+\t\tif (!isascii(c))\n+\t\t\treturn 0;\n+\t}\n+}\n+\n #define STREAM_BUFFER_SIZE (1024 * 16)\n \n static int write_zip_entry(struct archiver_args *args,\n@@ -187,6 +200,13 @@ static int write_zip_entry(struct archiver_args *args,\n \n \tcrc = crc32(0, NULL, 0);\n \n+\tif (!has_only_ascii(path)) {\n+\t\tif (is_utf8(path))\n+\t\t\tflags |= ZIP_UTF8;\n+\t\telse\n+\t\t\twarning(\"Path is not valid UTF-8: %s\", path);\n+\t}\n+\n \tif (pathlen > 0xffff) {\n \t\treturn error(\"path too long (%d chars, SHA1: %s): %s\",\n \t\t\t\t(int)pathlen, sha1_to_hex(sha1), path);\n-- \n1.7.12\n"},{"id":"199399","messageId":"5058D13C.8050203@lsrfire.ath.cx","threadId":"31231","inReplyTo":"5058CE49.3070108@lsrfire.ath.cx","subject":"[PATCH 2/2] archive-zip: declare creator to be Unix for UTF-8 paths","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-18T19:53:32Z","receivedAt":"2012-09-18T19:53:32Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"The UTF-8 flag seems to be ignored by unzip unless we also mark the\narchive entry as coming from a Unix system. This is done by setting the\nfield creator_version (\"version made by\" in the standard[1]) to 0x03NN.\n\nThe NN part represents the version of the standard supported by us, and\nthis patch sets it to 3f (for version 6.3) for Unix paths. We keep\ncreator_version set to 0 (FAT filesystem, standard version 0) in the\nnon-special cases, as before.\n\nBut when we declare a file to have a Unix path, then we have to set the\nfile mode as well, or unzip will extract the files with the permission\nset 0000, i.e. inaccessible by all.\n\n[1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT\n---\nNo sign-off for this, yet. Perhaps there is a better way to convince\nunzip to respect the flag? And if not, do we need to offer umask\nsettings for ZIP as well as we have for tar? And perhaps declare all\nfiles as being from a Unix filesystem, for consistency?\n\n archive-zip.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 0f763e8..e9b3dc9 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -186,7 +186,8 @@ static int write_zip_entry(struct archiver_args *args,\n {\n \tstruct zip_local_header header;\n \tstruct zip_dir_header dirent;\n-\tunsigned long attr2;\n+\tunsigned int creator_version = 0;\n+\tunsigned long attr2 = 0;\n \tunsigned long compressed_size;\n \tunsigned long crc;\n \tunsigned long direntsize;\n@@ -224,10 +225,15 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tenum object_type type = sha1_object_info(sha1, &size);\n \n \t\tmethod = 0;\n-\t\tattr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :\n-\t\t\t(mode & 0111) ? ((mode) << 16) : 0;\n \t\tif (S_ISREG(mode) && args->compression_level != 0 && size > 0)\n \t\t\tmethod = 8;\n+\t\tif (S_ISLNK(mode) || (mode & 0111) || (flags & ZIP_UTF8)) {\n+\t\t\tcreator_version = 0x033f;\n+\t\t\tattr2 = mode;\n+\t\t\tif (S_ISLNK(mode))\n+\t\t\t\tattr2 |= 0777;\n+\t\t\tattr2 <<= 16;\n+\t\t}\n \t\tcompressed_size = size;\n \n \t\tif (S_ISREG(mode) && type == OBJ_BLOB && !args->convert &&\n@@ -274,8 +280,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t}\n \n \tcopy_le32(dirent.magic, 0x02014b50);\n-\tcopy_le16(dirent.creator_version,\n-\t\tS_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0);\n+\tcopy_le16(dirent.creator_version, creator_version);\n \tcopy_le16(dirent.version, 10);\n \tcopy_le16(dirent.flags, flags);\n \tcopy_le16(dirent.compression_method, method);\n-- \n1.7.12\n"},{"id":"199403","messageId":"5058D870.1030204@lsrfire.ath.cx","threadId":"31231","inReplyTo":"5058CE49.3070108@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-18T20:24:16Z","receivedAt":"2012-09-18T20:24:16Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.09.2012 21:40, schrieb René Scharfe:\n> Windows Info-ZIP unzip\n> 7-Zip PeaZip builtin Linux msysgit Windows\n> git-master-patch1 0 0 72 60 72 72\n> git-master-patch2 0 0 72 0 72 72\n\nOh, and when I wrote Windows, I meant a German Windows 7 Home Premium \nx64, and Linux is Ubuntu 12.04 x86.\n\nI've also attached generated ZIP files for the two archivers above, for \neasier testing. Let's see if they make it through.\n\nRené\n\n"},{"id":"199412","messageId":"7v1uhzkpcc.fsf@alter.siamese.dyndns.org","threadId":"31231","inReplyTo":"5058CE49.3070108@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-18T21:12:19Z","receivedAt":"2012-09-18T21:12:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> Windows Info-ZIP unzip\n> 7-Zip PeaZip builtin Linux msysgit Windows\n> 7-Zip 9.20 0 0 46 26 43 43\n> PeaZip 4.7.1 win64 0 0 46 26 42 42\n> Info-ZIP zip 3.0 Linux 0 0 72 0 43 43\n> Info-ZIP zip 3.0 Windows 45 45 n/a 0 43 43\n> ...\n> I wonder what 7-Zip and PeaZip do that gives them a slightly nicer\n> score with the Windows-internal unzipper. Umlauts, Nordic characters\n> and accents are preserved by that combination. It seems that unzip on\n> Linux fails to unpack exactly these names, so perhaps they employ a\n> dirty trick like using the local encoding in the ZIP file, which makes\n> it unportable.\n> ...\n\nThanks for this work.\n\nIt is kind of surprising that \"Windows builtin\" has very poor score\nextracting from the output of Zip tools running on Windows (I am\nlooking at 46, 46 and n/a over there). If you tell it to create an\narchive from its disk and then extract from it, I wonder what would\nhappen.\n\nDoes this result mean that practically nobody uses Zip archive with\nexotic letters in paths on that platform? I am not talking about\ndevelopers and savvy people who know where to download third-party\nZip archivers and how to install them. I am imagining a grandma who\nreceived an archive full of photos of her grandchild in her Outlook\nExpress or GMail inbox, clicked the attachment to download it, and\nis trying to view the photo inside.\n"},{"id":"199634","messageId":"505B91E9.7060208@lsrfire.ath.cx","threadId":"31231","inReplyTo":"7v1uhzkpcc.fsf@alter.siamese.dyndns.org","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-20T22:00:09Z","receivedAt":"2012-09-20T22:00:09Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.09.2012 23:12, schrieb Junio C Hamano:\n> René Scharfe writes:\n>\n>> Windows Info-ZIP unzip\n>> 7-Zip PeaZip builtin Linux msysgit Windows\n>> 7-Zip 9.20 0 0 46 26 43 43\n>> PeaZip 4.7.1 win64 0 0 46 26 42 42\n>> Info-ZIP zip 3.0 Linux 0 0 72 0 43 43\n>> Info-ZIP zip 3.0 Windows 45 45 n/a 0 43 43\n\n> It is kind of surprising that \"Windows builtin\" has very poor score\n> extracting from the output of Zip tools running on Windows (I am\n> looking at 46, 46 and n/a over there). If you tell it to create an\n> archive from its disk and then extract from it, I wonder what would\n> happen.\n\nI didn't include it as a packer because it refused to archive the \npangrams directory due to illegal characters in one of the filenames. \nWhen I just tried a bit harder, I had to delete all but 14 files with \nLatin script, accents etc. before I could zip the directory. I'll \ninclude these results in the next round.\n\nIt uses codepage 850 on my system (MSDOS Latin 1). I don't expect this \nto be portable.\n\n> Does this result mean that practically nobody uses Zip archive with\n> exotic letters in paths on that platform? I am not talking about\n> developers and savvy people who know where to download third-party\n> Zip archivers and how to install them. I am imagining a grandma who\n> received an archive full of photos of her grandchild in her Outlook\n> Express or GMail inbox, clicked the attachment to download it, and\n> is trying to view the photo inside.\n\nNot necessarily. Photos often have names like img_0123.jpg etc., which \nare handled just fine. And all family members probably use the same \ncodepage on their computers, so they're less likely to run into this \nproblem.\n\nBy the way, I found this bug asking for codepage support in unzip:\n\n https://bugs.launchpad.net/ubuntu/+source/unzip/+bug/580961\n\nMultiple codepages seem to be used for ZIP files in the wild, none of \nthem are supported by unzip on Linux, which only accepts ASCII or UTF-8.\n\nRené\n"},{"id":"199820","messageId":"506082A7.4010108@lsrfire.ath.cx","threadId":"31231","inReplyTo":"5058CE49.3070108@lsrfire.ath.cx","subject":"[PATCH 3/2] archive-zip: write extended timestamp","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-24T15:56:23Z","receivedAt":"2012-09-24T15:56:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"File modification times in ZIP files are encoded in DOS format: local\ntime with a granularity of two seconds. Add an extra field to all\narchive entries to also record the mtime in Unix' fashion, as UTC with\na granularity of one second.\n\nThis has the desirable side-effect of convincing Info-ZIP unzip 6.00\nto respect general purpose flag 11, which is used to indicate that a\nfile name is encoded in UTF-8. Any extra field would do, actually,\nbut the extended timestamp is a reasonably small one (22 bytes per\nentry). Archives created by Info-ZIP zip 3.0 contain it, too (but\nwith ctime and atime as well).\n\nSigned-off-by: Rene Scharfe \n---\n archive-zip.c | 27 ++++++++++++++++++++++++---\n 1 file changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 0f763e8..55f66b4 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -76,6 +76,14 @@ struct zip_dir_trailer {\n \tunsigned char _end[1];\n };\n \n+struct zip_extra_mtime {\n+\tunsigned char magic[2];\n+\tunsigned char extra_size[2];\n+\tunsigned char flags[1];\n+\tunsigned char mtime[4];\n+\tunsigned char _end[1];\n+};\n+\n /*\n * On ARM, padding is added at the end of the struct, so a simple\n * sizeof(struct ...) reports two bytes more than the payload size\n@@ -85,6 +93,9 @@ struct zip_dir_trailer {\n #define ZIP_DATA_DESC_SIZE\toffsetof(struct zip_data_desc, _end)\n #define ZIP_DIR_HEADER_SIZE\toffsetof(struct zip_dir_header, _end)\n #define ZIP_DIR_TRAILER_SIZE\toffsetof(struct zip_dir_trailer, _end)\n+#define ZIP_EXTRA_MTIME_SIZE\toffsetof(struct zip_extra_mtime, _end)\n+#define ZIP_EXTRA_MTIME_PAYLOAD_SIZE \\\n+\t(ZIP_EXTRA_MTIME_SIZE - offsetof(struct zip_extra_mtime, flags))\n \n static void copy_le16(unsigned char *dest, unsigned int n)\n {\n@@ -186,6 +197,7 @@ static int write_zip_entry(struct archiver_args *args,\n {\n \tstruct zip_local_header header;\n \tstruct zip_dir_header dirent;\n+\tstruct zip_extra_mtime extra;\n \tunsigned long attr2;\n \tunsigned long compressed_size;\n \tunsigned long crc;\n@@ -266,8 +278,13 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t}\n \t}\n \n+\tcopy_le16(extra.magic, 0x5455);\n+\tcopy_le16(extra.extra_size, ZIP_EXTRA_MTIME_PAYLOAD_SIZE);\n+\textra.flags[0] = 1;\t/* just mtime */\n+\tcopy_le32(extra.mtime, args->time);\n+\n \t/* make sure we have enough free space in the dictionary */\n-\tdirentsize = ZIP_DIR_HEADER_SIZE + pathlen;\n+\tdirentsize = ZIP_DIR_HEADER_SIZE + pathlen + ZIP_EXTRA_MTIME_SIZE;\n \twhile (zip_dir_size < zip_dir_offset + direntsize) {\n \t\tzip_dir_size += ZIP_DIRECTORY_MIN_SIZE;\n \t\tzip_dir = xrealloc(zip_dir, zip_dir_size);\n@@ -283,7 +300,7 @@ static int write_zip_entry(struct archiver_args *args,\n \tcopy_le16(dirent.mdate, zip_date);\n \tset_zip_dir_data_desc(&dirent, size, compressed_size, crc);\n \tcopy_le16(dirent.filename_length, pathlen);\n-\tcopy_le16(dirent.extra_length, 0);\n+\tcopy_le16(dirent.extra_length, ZIP_EXTRA_MTIME_SIZE);\n \tcopy_le16(dirent.comment_length, 0);\n \tcopy_le16(dirent.disk, 0);\n \tcopy_le16(dirent.attr1, 0);\n@@ -301,11 +318,13 @@ static int write_zip_entry(struct archiver_args *args,\n \telse\n \t\tset_zip_header_data_desc(&header, size, compressed_size, crc);\n \tcopy_le16(header.filename_length, pathlen);\n-\tcopy_le16(header.extra_length, 0);\n+\tcopy_le16(header.extra_length, ZIP_EXTRA_MTIME_SIZE);\n \twrite_or_die(1, &header, ZIP_LOCAL_HEADER_SIZE);\n \tzip_offset += ZIP_LOCAL_HEADER_SIZE;\n \twrite_or_die(1, path, pathlen);\n \tzip_offset += pathlen;\n+\twrite_or_die(1, &extra, ZIP_EXTRA_MTIME_SIZE);\n+\tzip_offset += ZIP_EXTRA_MTIME_SIZE;\n \tif (stream && method == 0) {\n \t\tunsigned char buf[STREAM_BUFFER_SIZE];\n \t\tssize_t readlen;\n@@ -402,6 +421,8 @@ static int write_zip_entry(struct archiver_args *args,\n \tzip_dir_offset += ZIP_DIR_HEADER_SIZE;\n \tmemcpy(zip_dir + zip_dir_offset, path, pathlen);\n \tzip_dir_offset += pathlen;\n+\tmemcpy(zip_dir + zip_dir_offset, &extra, ZIP_EXTRA_MTIME_SIZE);\n+\tzip_dir_offset += ZIP_EXTRA_MTIME_SIZE;\n \tzip_dir_entries++;\n \n \treturn 0;\n-- \n1.7.12\n"},{"id":"199821","messageId":"506082C9.6050604@lsrfire.ath.cx","threadId":"31231","inReplyTo":"505B91E9.7060208@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2012-09-24T15:56:57Z","receivedAt":"2012-09-24T15:56:57Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Hi,\n\nI found a way to make unzip respect the UTF-8 flag in ZIP files: \nApparently (from looking at the source) an extended field needs to be \npresent in order for it to even look at general purpose flag 11. I sent \na patch to add an extended timestamp field that fits the bill.\n\nHere are new numbers on ZIP international filename compatibility:\n\n\t\t7-Zip\tPeaZip\tbuiltin\tunzip\tunzip\tunzip\t7z\n\t\tWindows\tWindows\tWindows\tLinux\tmingw\tWindows\tLinux\ngit\tLinux\t1\t1\t1\t7\t1\t1\t1\ngit 1\tLinux\t37\t37\t1\t7\t1\t1\t37\ngit 2\tLinux\t37\t37\t1\t37\t1\t1\t37\ngit 3\tLinux\t37\t37\t1\t37\t15\t15\t37\ngit\tmingw\t1\t1\t1\t7\t1\t1\t1\ngit 1\tmingw\t37\t37\t1\t7\t1\t1\t37\ngit 2\tmingw\t37\t37\t1\t37\t1\t1\t37\ngit 3\tmingw\t37\t37\t1\t37\t15\t15\t37\n7-Zip\tWindows\t37\t37\t14\t24\t15\t15\t24\nPeaZip\tWindows\t37\t37\t14\t24\t15\t15\t24\nzip\tLinux\t37\t37\t1\t37\t15\t15\t37\nzip\tWindows\t14\t14\t0\t37\t15\t15\t1\nbuiltin\tWindows\t14\t14\t14\t1\t14\t14\t1\n\nThe test corpus still consists of 37 files based on the pangrams on the \nfollowing web page:\n\n\thttp://www.columbia.edu/~fdc/utf8/index.html#quickbrownfox\n\nThe files can be created using the attached script. It also provides a \ncheck command to count the files with correct names, and the results of \nthat for different ZIP extractors are give in the table. The built-in \nZIP functionality on Windows was only able to pack 14 of the 37 files, \nwhich explains the low score across the board for this packer.\n\n\"git 1\" is the patch \"archive-zip: support UTF-8 paths\" added, which \nlet's archive-zip make use of the UTF-8 flag. \"git 2\" is \"git 1\" plus \nthe patch \"archive-zip: declare creator to be Unix for UTF-8 paths\". \nBoth have been posted before. \"git 3\" is \"git 1\" plus the new patch \n\"archive-zip: write extended timestamp\".\n\nLet's drop patch 2 (Unix as creator) and keep patches 1 (UTF-8 flag) and \n3 (mtime field) to make archive-zip record non-ASCII filenames in a \nportable way. It's not perfect, but I don't know how to do any better \ngiven that Windows' built-in ZIP functionality expects filenames in the \nlocal code page and with an international audience for projects \ndistributing ZIP files.\n\nRené\n\n\n\n#!/bin/sh\n\nfiles() {\ncat <&2\n\texit 1\n\t;;\nesac\n"},{"id":"199834","messageId":"7vtxunz3v3.fsf@alter.siamese.dyndns.org","threadId":"31231","inReplyTo":"506082C9.6050604@lsrfire.ath.cx","subject":"Re: git archive --format zip utf-8 issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-24T18:13:04Z","receivedAt":"2012-09-24T18:13:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> \"git 1\" is the patch \"archive-zip: support UTF-8 paths\" added, which\n> let's archive-zip make use of the UTF-8 flag. \"git 2\" is \"git 1\" plus\n> the patch \"archive-zip: declare creator to be Unix for UTF-8\n> paths\". Both have been posted before. \"git 3\" is \"git 1\" plus the new\n> patch \"archive-zip: write extended timestamp\".\n>\n> Let's drop patch 2 (Unix as creator) and keep patches 1 (UTF-8 flag)\n> and 3 (mtime field) to make archive-zip record non-ASCII filenames in\n> a portable way. It's not perfect, but I don't know how to do any\n> better given that Windows' built-in ZIP functionality expects\n> filenames in the local code page and with an international audience\n> for projects distributing ZIP files.\n\nThanks.\n"}]}