# git archive --format zip utf-8 issues 21 messages from 2012-08-10 to 2012-09-24. Participants: Sven Strickroth, Junio C Hamano, René Scharfe, Jeff King. Thread: https://gitlist.dev/t/31231 ## Sven Strickroth, 2012-08-10 21:58 Subject: git archive --format zip utf-8 issues Message-ID: <502583F4.8030308@tu-clausthal.de> URL: https://gitlist.dev/e/502583F4.8030308%40tu-clausthal.de ``` Hi, when I create a git repository, add a file containing utf-8 characters or umlauts (like öäü.txt), commit and then export the HEAD revision to a zip archive using "git archive --format zip -o 1.zip HEAD", the zip file contains incorrect filenames: $ unzip -l 1.zip Archive: 1.zip 4490a6dab1df5404f91ab3eb871f133154bff0bf Length Date Time Name --------- ---------- ----- ---- 6 2012-08-10 23:41 +?+?++.txt --------- ------- 6 1 file -- Best regards, Sven Strickroth PGP key id F5A9D4C4 @ any key-server ``` ## Junio C Hamano, 2012-08-10 22:47 Subject: Re: git archive --format zip utf-8 issues Message-ID: <7vtxwagy9f.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vtxwagy9f.fsf%40alter.siamese.dyndns.org In-Reply-To: <502583F4.8030308@tu-clausthal.de> ``` Sven Strickroth writes: > when I create a git repository, add a file containing utf-8 characters > or umlauts (like öäü.txt), commit and then export the HEAD revision to a > zip archive using "git archive --format zip -o 1.zip HEAD", the zip file > contains incorrect filenames: My reading of archive-zip.c seems to suggest that we write out whatever pathname you have in the tree, so a pathname encoded in UTF-8 will be literally written out in the resulting zip archive. Do you know in what encoding the pathnames are _expected_ to be stored in zip archives? Random documentation seems to suggest that there is no standard encoding, e.g. http://docs.python.org/library/zipfile.html says: There is no official file name encoding for ZIP files. If you have unicode file names, you must convert them to byte strings in your desired encoding before passing them to write(). WinZip interprets all file names as encoded in CP437, also known as DOS Latin. which may explain it. It may not be a bad idea for "git archive --format=zip" to (1) check if pathname is a correct UTF-8; and (2) check if it can be reencoded to latin-1 and if (and only if) both are true, automatically re-encode the path to latin-1. Of course, "git archive --format=zip --path-reencode=utf8-to-latin1" would be the most generic way to do this. ``` ## Sven Strickroth, 2012-08-10 23:53 Subject: Re: git archive --format zip utf-8 issues Message-ID: <50259EEC.2020701@tu-clausthal.de> URL: https://gitlist.dev/e/50259EEC.2020701%40tu-clausthal.de In-Reply-To: <7vtxwagy9f.fsf@alter.siamese.dyndns.org> ``` Am 11.08.2012 00:47 schrieb Junio C Hamano: > Do you know in what encoding the pathnames are _expected_ to be > stored in zip archives? re-encoding to latin1 does not always work and may break double byte totally (e.g. chinese or japanese). PKZIP APPNOTE seems to be the zip standard and it specifies a utf-8 flag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT > A. Local file header: > general purpose bit flag: (2 bytes) > Bit 11: Language encoding flag (EFS). If this bit is > set, the filename and comment fields for this file > must be encoded using UTF-8. (see APPENDIX D) -- Best regards, Sven Strickroth PGP key id F5A9D4C4 @ any key-server ``` ## René Scharfe, 2012-08-11 20:53 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5026C649.2090700@lsrfire.ath.cx> URL: https://gitlist.dev/e/5026C649.2090700%40lsrfire.ath.cx In-Reply-To: <7vtxwagy9f.fsf@alter.siamese.dyndns.org> ``` Am 11.08.2012 00:47, schrieb Junio C Hamano: > Sven Strickroth writes: > >> when I create a git repository, add a file containing utf-8 characters >> or umlauts (like öäü.txt), commit and then export the HEAD revision to a >> zip archive using "git archive --format zip -o 1.zip HEAD", the zip file >> contains incorrect filenames: > > My reading of archive-zip.c seems to suggest that we write out > whatever pathname you have in the tree, so a pathname encoded in > UTF-8 will be literally written out in the resulting zip archive. Sorry for my imperialistic attitude of "ASCII filenames should be enough for everybody". Laziness.. > Do you know in what encoding the pathnames are _expected_ to be > stored in zip archives? Random documentation seems to suggest that > there is no standard encoding, e.g. http://docs.python.org/library/zipfile.html > says: > > There is no official file name encoding for ZIP files. If you > have unicode file names, you must convert them to byte strings > in your desired encoding before passing them to write(). WinZip > interprets all file names as encoded in CP437, also known as DOS > Latin. > > which may explain it. http://www.pkware.com/documents/casestudies/APPNOTE.TXT is the standard document, as Sven noted, and it says that filenames are encoded in code page 437, or optionally UTF-8 (a later addition). Discussions like http://stackoverflow.com/questions/106367/ seem to indicate that at least some archivers use the local code page as well. > It may not be a bad idea for "git archive --format=zip" to > > (1) check if pathname is a correct UTF-8; and > (2) check if it can be reencoded to latin-1 > > and if (and only if) both are true, automatically re-encode the path > to latin-1. The standard says we need to convert to CP437, or to UTF-8, or provide both versions. A more interesting question is: What's supported by which programs? The ZIP functionality built into Windows 7 doesn't seem to work with UTF-8 encoded filenames (except for those that only use the ASCII subset), and to ignore the UTF-8 part if both are given. Handling umlauts should be possible anyway, because they are on code page 437, but for other characters we'd have to aim for compatibility with other programs like Info-ZIP and 7-Zip. How do we know which encoding was used for a filename? > Of course, "git archive --format=zip --path-reencode=utf8-to-latin1" > would be the most generic way to do this. I really hope we can make do without additional options. René ``` ## René Scharfe, 2012-08-11 20:53 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5026C651.8050705@lsrfire.ath.cx> URL: https://gitlist.dev/e/5026C651.8050705%40lsrfire.ath.cx In-Reply-To: <50259EEC.2020701@tu-clausthal.de> ``` Am 11.08.2012 01:53, schrieb Sven Strickroth: > Am 11.08.2012 00:47 schrieb Junio C Hamano: >> Do you know in what encoding the pathnames are _expected_ to be >> stored in zip archives? > > re-encoding to latin1 does not always work and may break double byte > totally (e.g. chinese or japanese). > > PKZIP APPNOTE seems to be the zip standard and it specifies a utf-8 > flag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT >> A. Local file header: >> general purpose bit flag: (2 bytes) >> Bit 11: Language encoding flag (EFS). If this bit is >> set, the filename and comment fields for this file >> must be encoded using UTF-8. (see APPENDIX D) Yes, that's one of the two methods for supporting UTF-8 filenames described there. The other method involves writing extra ZIP header fields and was invented by Info-ZIP. They don't use it consistently anymore, though (from zip -h2): "Zip now stores UTF-8 in entry path and comment fields on systems where UTF-8 char set is default, such as most modern Unix, and and on other systems in new extra fields with escaped versions in entry path and comment fields for backward compatibility." René ``` ## Sven Strickroth, 2012-08-11 21:37 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5026D081.2040906@tu-clausthal.de> URL: https://gitlist.dev/e/5026D081.2040906%40tu-clausthal.de In-Reply-To: <5026C649.2090700@lsrfire.ath.cx> ``` Am 11.08.2012 22:53 schrieb René Scharfe: > The standard says we need to convert to CP437, or to UTF-8, or provide > both versions. A more interesting question is: What's supported by which > programs? > > The ZIP functionality built into Windows 7 doesn't seem to work with > UTF-8 encoded filenames (except for those that only use the ASCII > subset), and to ignore the UTF-8 part if both are given. I played a bit with the git source code and found out, that diff --git a/archive-zip.c b/archive-zip.c index f5af81f..e0ccb4f 100644 --- a/archive-zip.c +++ b/archive-zip.c @@ -257,7 +257,7 @@ static int write_zip_entry(struct archiver_args *args, copy_le16(dirent.creator_version, S_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0); copy_le16(dirent.version, 10); - copy_le16(dirent.flags, flags); + copy_le16(dirent.flags, flags+2048); copy_le16(dirent.compression_method, method); copy_le16(dirent.mtime, zip_time); copy_le16(dirent.mdate, zip_date); -- works with 7-zip, however, not with Windows 7 build-in zip. If I create a zip file with 7-zip which contains umlauts and other unicode chars like (國立1-кккк.txt) the Windows 7 build-in zip displays them correctly, too. -- Best regards, Sven Strickroth PGP key id F5A9D4C4 @ any key-server ``` ## Junio C Hamano, 2012-08-12 04:08 Subject: Re: git archive --format zip utf-8 issues Message-ID: <7vlihkhhwc.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vlihkhhwc.fsf%40alter.siamese.dyndns.org In-Reply-To: <5026C651.8050705@lsrfire.ath.cx> ``` René Scharfe writes: >> PKZIP APPNOTE seems to be the zip standard and it specifies a utf-8 >> flag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT >>> A. Local file header: >>> general purpose bit flag: (2 bytes) >>> Bit 11: Language encoding flag (EFS). If this bit is >>> set, the filename and comment fields for this file >>> must be encoded using UTF-8. (see APPENDIX D) > > Yes, that's one of the two methods for supporting UTF-8 filenames > described there. > > The other method involves writing extra ZIP header fields and was > invented by Info-ZIP. They don't use it consistently anymore, though > (from zip -h2): > > "Zip now stores UTF-8 in entry path and comment fields on systems > where UTF-8 char set is default, such as most modern Unix, and > and on other systems in new extra fields with escaped versions in > entry path and comment fields for backward compatibility." Thanks; so if we adopt one of these methods, the readers that matter will be happy? And if so, which one? Or both? ``` ## Junio C Hamano, 2012-08-12 04:27 Subject: Re: git archive --format zip utf-8 issues Message-ID: <7vhas8hgzl.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vhas8hgzl.fsf%40alter.siamese.dyndns.org In-Reply-To: <5026C649.2090700@lsrfire.ath.cx> ``` René Scharfe writes: > ... A more interesting question is: What's supported by > which programs? Yes, that is the most interesting question. >> Of course, "git archive --format=zip --path-reencode=utf8-to-latin1" >> would be the most generic way to do this. > > I really hope we can make do without additional options. We need to at least know the path encoding used in the tree objects, and I'd be OK with a solution that assumes a single encoding is used for the entire tree. We would eventually need to also know the encoding used on the local working tree (i.e. in what encoding paths are returned from readdir() and the pathspec the user gives us from the command line), and iconv it to the tree objects encoding for the project when creating a cache_entry object to be fed to add_to_index(), and iconv it back from the tree objects encoding to the working tree encoding in write_entry(), but that is a longer term direction. For now, in order to address the immediate issue, we only need the tree object encoding, which should default to UTF-8 for interoperability. So "git archive --format=zip --in-object-path-encoding=big5" for a project whose tree object pathnames are in that encoding (and we always record paths in UTF-8 when writing zipfiles) should be the minimal that we need for now. Optionally, with a configuration variable i18n.inObjectPathEncoding (as opposed to the eventual i18n.worktreePathEncoding) set to big5, users of such a project can say "git archive --format=zip" without the "--in-object-path-encoding" option. Considering that zip is a format meant for exchange, I'd think we would be fine to always write in UTF-8 and leaving the readers responsible for converting the pathname while extracting. If a major zip extractor is incapable of handling UTF-8 (or even if capable it is cumbersome, for that matter), we may end up having to add "--in-archive-path-encoding=UTF-8" option to "git archive", with associated "zip.archivePathEncoding" variable, though. ``` ## Jeff King, 2012-08-30 22:26 Subject: Re: git archive --format zip utf-8 issues Message-ID: <20120830222603.GA20289@sigill.intra.peff.net> URL: https://gitlist.dev/e/20120830222603.GA20289%40sigill.intra.peff.net In-Reply-To: <5026D081.2040906@tu-clausthal.de> ``` On Sat, Aug 11, 2012 at 11:37:05PM +0200, Sven Strickroth wrote: > Am 11.08.2012 22:53 schrieb René Scharfe: > > The standard says we need to convert to CP437, or to UTF-8, or provide > > both versions. A more interesting question is: What's supported by which > > programs? > > > > The ZIP functionality built into Windows 7 doesn't seem to work with > > UTF-8 encoded filenames (except for those that only use the ASCII > > subset), and to ignore the UTF-8 part if both are given. > > I played a bit with the git source code and found out, that > > diff --git a/archive-zip.c b/archive-zip.c > index f5af81f..e0ccb4f 100644 > --- a/archive-zip.c > +++ b/archive-zip.c > @@ -257,7 +257,7 @@ static int write_zip_entry(struct archiver_args *args, > copy_le16(dirent.creator_version, > S_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0); > copy_le16(dirent.version, 10); > - copy_le16(dirent.flags, flags); > + copy_le16(dirent.flags, flags+2048); > copy_le16(dirent.compression_method, method); > copy_le16(dirent.mtime, zip_time); > copy_le16(dirent.mdate, zip_date); > -- > works with 7-zip, however, not with Windows 7 build-in zip. > > If I create a zip file with 7-zip which contains umlauts and other > unicode chars like (國立1-кккк.txt) the Windows 7 build-in zip displays > them correctly, too. Ping on this stalled discussion. It seems like there are two separate issues here: 1. Knowing the encoding of pathnames in the repository. 2. Setting the right flags in zip output. A full solution would handle both parts, but let's ignore (1) for a moment, and assume we have utf-8 (or can massage into utf-8 from an encoding specified by the user). It seems like just setting the magic utf-8 flag would be the only thing we need to do, according to the standard. But according to discussions referenced elsewhere in this thread, that flag was invented only in 2007, so we may be dealing with older implementations (I have no idea how common they would be; that may be the problem with Windows 7's zip you are seeing). We could re-encode to cp437, which the standard specifies, but apparently some implementations do not respect that (and use a local code page instead). And it cannot represent all utf-8 characters, anyway. It sounds like 7-zip has figured out a more portable solution. Can you show us a sample of 7-zip's output with utf-8 characters to compare to what git generates? I wonder if it is using a combination of methods. -Peff ``` ## René Scharfe, 2012-09-04 20:23 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5046634A.4020608@lsrfire.ath.cx> URL: https://gitlist.dev/e/5046634A.4020608%40lsrfire.ath.cx In-Reply-To: <20120830222603.GA20289@sigill.intra.peff.net> ``` Am 31.08.2012 00:26, schrieb Jeff King: > Ping on this stalled discussion. Sorry, I got distracted by other stuff again. I did some experiments, though, and here's a preliminary result. > It seems like there are two separate issues here: > > 1. Knowing the encoding of pathnames in the repository. > > 2. Setting the right flags in zip output. > > A full solution would handle both parts, but let's ignore (1) for a > moment, and assume we have utf-8 (or can massage into utf-8 from an > encoding specified by the user). Yes, good thinking. Re-encoding may be beneficial for tar files as well, but we can ignore that point for the moment. > It seems like just setting the magic utf-8 flag would be the only thing > we need to do, according to the standard. But according to discussions > referenced elsewhere in this thread, that flag was invented only in > 2007, so we may be dealing with older implementations (I have no idea > how common they would be; that may be the problem with Windows 7's zip > you are seeing). We could re-encode to cp437, which the standard > specifies, but apparently some implementations do not respect that > (and use a local code page instead). And it cannot represent all utf-8 > characters, anyway. Yes, we could do that, plus adding an extra field with a UTF-8 version of the path. That's the legacy method invented by Info-ZIP. They switched to using the new flag on Linux at least, though. > It sounds like 7-zip has figured out a more portable solution. Can you > show us a sample of 7-zip's output with utf-8 characters to compare to > what git generates? I wonder if it is using a combination of methods. I'm not so sure they produce more portable files. I created an archive with files named jaя.txt, smørrebrød.txt, süd.txt and €uro.txt with 7-Zip on Windows 7 and while unzip on Ubuntu 12.04 managed to recreate the cyrillic character and the Euro symbol, it mangled the slashed o and the umlaut. With the following patch I could create archives with git on Linux and msysgit and extract them flawlessly on Windows with 7-Zip and with Info-ZIP unzip on Linux, but not with unzip on Windows, where it mangled all non-ASCII characters. This gets confusing; it would help to have a compatibility matrix for all intersting extractors and character classes -- for each proposed solution or archiver we'd like to imitate. But now for the patch, which is a bit confusing as well. I'm curious to hear about results for more platforms, extractors and character classes. Based on that we can see if we need to generate the extra fields instead of relying on the new flag. -- >8 -- Subject: [PATCH] archive-zip: support UTF-8 paths Set general purpose flag 11 if we encounter a path that contains non-ASCII characters. We assume that all paths are given as UTF-8; no conversion is done. The flag seems to be ignored by unzip unless we also mark the archive entry as coming from a Unix system. This is done by setting the field creator_version ("version made by" in the standard[1]) to 0x03NN. The NN part represents the version of the standard supported by us, and this patch sets it to 3f (for version 6.3) for Unix paths. We keep creator_version set to 0 (FAT filesystem, standard version 0) in the non-special cases, as before. But when we declare a file to have a Unix path, then we have to set the file mode as well, or unzip will extract the files with the permission set 0000, i.e. inaccessible by all. [1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT Signed-off-by: Rene Scharfe --- archive-zip.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/archive-zip.c b/archive-zip.c index f5af81f..928da1d 100644 --- a/archive-zip.c +++ b/archive-zip.c @@ -4,6 +4,8 @@ #include "cache.h" #include "archive.h" #include "streaming.h" +#include "commit.h" +#include "utf8.h" static int zip_date; static int zip_time; @@ -16,7 +18,8 @@ static unsigned int zip_dir_offset; static unsigned int zip_dir_entries; #define ZIP_DIRECTORY_MIN_SIZE (1024 * 1024) -#define ZIP_STREAM (8) +#define ZIP_STREAM (1 << 3) +#define ZIP_UTF8 (1 << 11) struct zip_local_header { unsigned char magic[4]; @@ -173,7 +176,8 @@ static int write_zip_entry(struct archiver_args *args, { struct zip_local_header header; struct zip_dir_header dirent; - unsigned long attr2; + unsigned int creator_version = 0; + unsigned long attr2 = 0; unsigned long compressed_size; unsigned long crc; unsigned long direntsize; @@ -187,6 +191,13 @@ static int write_zip_entry(struct archiver_args *args, crc = crc32(0, NULL, 0); + if (has_non_ascii(path)) { + if (is_utf8(path)) + flags |= ZIP_UTF8; + else + warning("Path is not valid UTF-8: %s", path); + } + if (pathlen > 0xffff) { return error("path too long (%d chars, SHA1: %s): %s", (int)pathlen, sha1_to_hex(sha1), path); @@ -204,10 +215,15 @@ static int write_zip_entry(struct archiver_args *args, enum object_type type = sha1_object_info(sha1, &size); method = 0; - attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) : - (mode & 0111) ? ((mode) << 16) : 0; if (S_ISREG(mode) && args->compression_level != 0 && size > 0) method = 8; + if (S_ISLNK(mode) || (mode & 0111) || (flags & ZIP_UTF8)) { + creator_version = 0x033f; + attr2 = mode; + if (S_ISLNK(mode)) + attr2 |= 0777; + attr2 <<= 16; + } compressed_size = size; if (S_ISREG(mode) && type == OBJ_BLOB && !args->convert && @@ -254,8 +270,7 @@ static int write_zip_entry(struct archiver_args *args, } copy_le32(dirent.magic, 0x02014b50); - copy_le16(dirent.creator_version, - S_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0); + copy_le16(dirent.creator_version, creator_version); copy_le16(dirent.version, 10); copy_le16(dirent.flags, flags); copy_le16(dirent.compression_method, method); -- 1.7.11.msysgit.1.1.gbf71771 ``` ## Junio C Hamano, 2012-09-04 21:03 Subject: Re: git archive --format zip utf-8 issues Message-ID: <7vehmh8prt.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vehmh8prt.fsf%40alter.siamese.dyndns.org In-Reply-To: <5046634A.4020608@lsrfire.ath.cx> ``` René Scharfe writes: > But now for the patch, which is a bit confusing as well. I'm curious to > hear about results for more platforms, extractors and character classes. > Based on that we can see if we need to generate the extra fields instead > of relying on the new flag. Thanks for keeping the ball rolling. > Subject: [PATCH] archive-zip: support UTF-8 paths > > Set general purpose flag 11 if we encounter a path that contains > non-ASCII characters. We assume that all paths are given as UTF-8; no > conversion is done. > > The flag seems to be ignored by unzip unless we also mark the archive > entry as coming from a Unix system. This is done by setting the field > creator_version ("version made by" in the standard[1]) to 0x03NN. > > The NN part represents the version of the standard supported by us, and > this patch sets it to 3f (for version 6.3) for Unix paths. We keep > creator_version set to 0 (FAT filesystem, standard version 0) in the > non-special cases, as before. > > But when we declare a file to have a Unix path, then we have to set the > file mode as well, or unzip will extract the files with the permission > set 0000, i.e. inaccessible by all. > > [1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT > > Signed-off-by: Rene Scharfe > --- > archive-zip.c | 27 +++++++++++++++++++++------ > 1 file changed, 21 insertions(+), 6 deletions(-) > > diff --git a/archive-zip.c b/archive-zip.c > index f5af81f..928da1d 100644 > --- a/archive-zip.c > +++ b/archive-zip.c > @@ -4,6 +4,8 @@ > #include "cache.h" > #include "archive.h" > #include "streaming.h" > +#include "commit.h" > +#include "utf8.h" > > static int zip_date; > static int zip_time; > @@ -16,7 +18,8 @@ static unsigned int zip_dir_offset; > static unsigned int zip_dir_entries; > > #define ZIP_DIRECTORY_MIN_SIZE (1024 * 1024) > -#define ZIP_STREAM (8) > +#define ZIP_STREAM (1 << 3) > +#define ZIP_UTF8 (1 << 11) > > struct zip_local_header { > unsigned char magic[4]; > @@ -173,7 +176,8 @@ static int write_zip_entry(struct archiver_args *args, > { > struct zip_local_header header; > struct zip_dir_header dirent; > - unsigned long attr2; > + unsigned int creator_version = 0; > + unsigned long attr2 = 0; > unsigned long compressed_size; > unsigned long crc; > unsigned long direntsize; > @@ -187,6 +191,13 @@ static int write_zip_entry(struct archiver_args *args, > > crc = crc32(0, NULL, 0); > > + if (has_non_ascii(path)) { Do we want to treat \033 as "ascii" in this codepath? The function primarily is used by the log formatter to see if we need 8-bit CTE when writing out in the e-mail format. > + if (is_utf8(path)) > + flags |= ZIP_UTF8; > + else > + warning("Path is not valid UTF-8: %s", path); > + } > + > if (pathlen > 0xffff) { > return error("path too long (%d chars, SHA1: %s): %s", > (int)pathlen, sha1_to_hex(sha1), path); > @@ -204,10 +215,15 @@ static int write_zip_entry(struct archiver_args *args, > enum object_type type = sha1_object_info(sha1, &size); > > method = 0; > - attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) : > - (mode & 0111) ? ((mode) << 16) : 0; > if (S_ISREG(mode) && args->compression_level != 0 && size > 0) > method = 8; > + if (S_ISLNK(mode) || (mode & 0111) || (flags & ZIP_UTF8)) { > + creator_version = 0x033f; > + attr2 = mode; > + if (S_ISLNK(mode)) > + attr2 |= 0777; > + attr2 <<= 16; > + } > compressed_size = size; > > if (S_ISREG(mode) && type == OBJ_BLOB && !args->convert && > @@ -254,8 +270,7 @@ static int write_zip_entry(struct archiver_args *args, > } > > copy_le32(dirent.magic, 0x02014b50); > - copy_le16(dirent.creator_version, > - S_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0); > + copy_le16(dirent.creator_version, creator_version); > copy_le16(dirent.version, 10); > copy_le16(dirent.flags, flags); > copy_le16(dirent.compression_method, method); ``` ## René Scharfe, 2012-09-05 19:36 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5047A9C0.9020200@lsrfire.ath.cx> URL: https://gitlist.dev/e/5047A9C0.9020200%40lsrfire.ath.cx In-Reply-To: <7vehmh8prt.fsf@alter.siamese.dyndns.org> ``` Am 04.09.2012 23:03, schrieb Junio C Hamano: > René Scharfe writes: >> + if (has_non_ascii(path)) { > > Do we want to treat \033 as "ascii" in this codepath? The function > primarily is used by the log formatter to see if we need 8-bit CTE > when writing out in the e-mail format. Argh, yes, I'd think so. The function name mislead me. This won't matter for compatibility testing, but should be corrected before inclusion. Just checked: unzip strips the ESC character when extracting (whether the UTF-8 flag is set or not) and 7-Zip replaces it with an underscore. The built-in ZIP extractor of Windows 7 skips such a file; it doesn't even show up in archive directory listings. René ``` ## René Scharfe, 2012-09-18 19:40 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5058CE49.3070108@lsrfire.ath.cx> URL: https://gitlist.dev/e/5058CE49.3070108%40lsrfire.ath.cx In-Reply-To: <5047A9C0.9020200@lsrfire.ath.cx> ``` Hello again, so two weeks have passed, and I've moved at a glacial pace towards a method how to measure compatibility of our generated ZIP files. Sorry, I just keep getting distracted. Anyway, the idea is to have a bunch of files with names using different scripts, zip them with several packers (including git archive), unzip them and compare the result with the original files. As test corpus I used files named like the pangrams on this UTF-8 sampler page, the exact commands are attached: http://www.columbia.edu/~fdc/utf8/index.html#quickbrownfox The numbers below are how many lines the output of diff -ru contains for this pair of packer and unpacker. There are 37 files, so the worst result is 74 lines of difference ("Only in [...]" for both sides), while 0 indicates a perfect score. Hmm, come to think of it, an empty directory would show up as 37, so this metric is not ideal. A better one would be to simply give one point for each correctly unpacked file. Windows Info-ZIP unzip 7-Zip PeaZip builtin Linux msysgit Windows 7-Zip 9.20 0 0 46 26 43 43 PeaZip 4.7.1 win64 0 0 46 26 42 42 Info-ZIP zip 3.0 Linux 0 0 72 0 43 43 Info-ZIP zip 3.0 Windows 45 45 n/a 0 43 43 git-master 72 72 72 60 72 72 git-master-patch1 0 0 72 60 72 72 git-master-patch2 0 0 72 0 72 72 git-v1.7.11.msysgit.1 72 72 72 60 72 72 git-v1.7.11.msysgit.1-patch1 0 0 72 60 72 72 git-v1.7.11.msysgit.1-patch2 0 0 72 0 72 72 Info-ZIP's programs don't work too well on Windows. The built-in unzipper of Windows 7 even refuses to open the file created by the Windows version of zip. Speaking of which, this is the worst of the unpackers. With the two patches applied, we can say "use 7-Zip or PeaZip on Windows and unzip on Linux" and filenames with all tested characters will be preserved. I was surprised to see this working fine with msysgit like that, even though no reencoding is introduced by the patches. I wonder what 7-Zip and PeaZip do that gives them a slightly nicer score with the Windows-internal unzipper. Umlauts, Nordic characters and accents are preserved by that combination. It seems that unzip on Linux fails to unpack exactly these names, so perhaps they employ a dirty trick like using the local encoding in the ZIP file, which makes it unportable. I'll reply with the two patches, which contain basically the same code as the previous patch, only split up. The second one declares that filenames with UTF-8 encoding came from Unix (instead of FAT), which makes unzip happy. This, however, implies that we contain Unix permissions for these entries, which is a bit ugly. René #!/bin/sh ( mkdir pangrams cd pangrams echo English >"The quick brown fox jumps over the lazy dog" echo Irish 1 >"An ḃfuil do Ä‹roí ag bualaḋ ó ḟaitíos an Ä¡rá a á¹�eall" echo Irish 2 >"lena ṗóg éada ó ṡlí do leasa ṫú" echo Irish 3 >"D'ḟuascail Ã�osa Úrá¹�ac na hÓiÄ¡e Beannaiṫe pór" echo Irish 4 >"Éava agus Ã�ḋaiá¹�" echo Dutch >"Pa's wijze lynx bezag vroom het fikse aquaduct" echo German 1 >"Falsches Ãœben von Xylophonmusik quält" echo German 2 >"jeden größeren Zwerg" echo Norwegian >"BlÃ¥bærsyltetøy" echo Danish >"Høj bly gom vandt fræk sexquiz pÃ¥ wc" echo Swedish >"Flygande bäckasiner söka strax hwila pÃ¥ mjuka tuvor" echo Icelandic >"Sævör grét áðan því úlpan var ónýt" echo Finnish >"Törkylempijävongahdus" echo Polish >"Pchnąć w tÄ™ łódź jeża lub osiem skrzyÅ„ fig" echo Czech >"PříliÅ¡ žluÅ¥ouÄ�ký kůň úpÄ›l Ä�ábelské kódy" echo Slovak 1 >"Starý kôň na hÅ•be kníh žuje tíško povädnuté ruže" echo Slovak 2 >"na stĺpe sa Ä�ateľ uÄ�í kvákaÅ¥ novú ódu o živote" echo monotonic Greek >"ξεσκεπάζω την ψυχοφθόÏ�α βδελυγμία" echo polytonic Greek >"ξεσκεπάζω τὴν ψυχοφθόÏ�α βδελυγμία" echo Russian >"Съешь же ещё Ñ�тих мÑ�гких французÑ�ких булок да выпей чаю" echo Bulgarian 1 >"Жълтата дюлÑ� беше щаÑ�тлива" echo Bulgarian 2 >"че пухът, който цъфна, замръзна като гьон" echo Northern Sami >"Vuol Ruoŧa geÄ‘ggiid leat máŋga luosa ja Ä�uovžža" echo Hungarian >"Ã�rvíztűrÅ‘ tükörfúrógép" echo Spanish 1 >"El pingüino Wenceslao hizo kilómetros bajo exhaustiva" echo Spanish 2 >"lluvia y frío añoraba a su querido cachorro" echo Portuguese 1 >"O próximo vôo à noite sobre o Atlântico" echo Portuguese 2 >"põe freqüentemente o único médico" echo French 1 >"Les naïfs ægithales hâtifs pondant à Noël où il gèle" echo French 2 >"sont sûrs d'être déçus en voyant leurs drôles" echo French 3 >"d'Å“ufs abîmés" echo Esperanto >"EÄ¥oÅ�anÄ�o ĉiuĵaÅ­de" echo Hebrew >"×–×” ×›×™×£ סת×� לשמוע ×�יך תנצח קרפד ×¢×¥ טוב בגן" echo Hiragana 1 >"ã�„ã‚�ã�¯ã�«ã�»ã�¸ã�©ã€€ã�¡ã‚Šã�¬ã‚‹ã‚’" echo Hiragana 2 >"ã‚�ã�Œã‚ˆã�Ÿã‚Œã�žã€€ã�¤ã�­ã�ªã‚‰ã‚€" echo Hiragana 3 >"ã�†ã‚�ã�®ã�Šã��ã‚„ã�¾ã€€ã�‘ã�µã�“ã�ˆã�¦" echo Hiragana 4 >"ã�‚ã�•ã��ゆã‚�ã�¿ã�˜ã€€ã‚‘ã�²ã‚‚ã�›ã�š" ) ``` ## René Scharfe, 2012-09-18 19:46 Subject: [PATCH 1/2] archive-zip: support UTF-8 paths Message-ID: <5058CFB0.6040506@lsrfire.ath.cx> URL: https://gitlist.dev/e/5058CFB0.6040506%40lsrfire.ath.cx In-Reply-To: <5058CE49.3070108@lsrfire.ath.cx> ``` Set general purpose flag 11 if we encounter a path that contains non-ASCII characters. We assume that all paths are given as UTF-8; no conversion is done. Signed-off-by: Rene Scharfe --- Changes from previous version: Stop using has_non_ascii(), which does slightly too much for our purposes, and split off creator version change into a separate patch. archive-zip.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/archive-zip.c b/archive-zip.c index f5af81f..0f763e8 100644 --- a/archive-zip.c +++ b/archive-zip.c @@ -4,6 +4,7 @@ #include "cache.h" #include "archive.h" #include "streaming.h" +#include "utf8.h" static int zip_date; static int zip_time; @@ -16,7 +17,8 @@ static unsigned int zip_dir_offset; static unsigned int zip_dir_entries; #define ZIP_DIRECTORY_MIN_SIZE (1024 * 1024) -#define ZIP_STREAM (8) +#define ZIP_STREAM (1 << 3) +#define ZIP_UTF8 (1 << 11) struct zip_local_header { unsigned char magic[4]; @@ -164,6 +166,17 @@ static void set_zip_header_data_desc(struct zip_local_header *header, copy_le32(header->size, size); } +static int has_only_ascii(const char *s) +{ + for (;;) { + int c = *s++; + if (c == '\0') + return 1; + if (!isascii(c)) + return 0; + } +} + #define STREAM_BUFFER_SIZE (1024 * 16) static int write_zip_entry(struct archiver_args *args, @@ -187,6 +200,13 @@ static int write_zip_entry(struct archiver_args *args, crc = crc32(0, NULL, 0); + if (!has_only_ascii(path)) { + if (is_utf8(path)) + flags |= ZIP_UTF8; + else + warning("Path is not valid UTF-8: %s", path); + } + if (pathlen > 0xffff) { return error("path too long (%d chars, SHA1: %s): %s", (int)pathlen, sha1_to_hex(sha1), path); -- 1.7.12 ``` ## René Scharfe, 2012-09-18 19:53 Subject: [PATCH 2/2] archive-zip: declare creator to be Unix for UTF-8 paths Message-ID: <5058D13C.8050203@lsrfire.ath.cx> URL: https://gitlist.dev/e/5058D13C.8050203%40lsrfire.ath.cx In-Reply-To: <5058CE49.3070108@lsrfire.ath.cx> ``` The UTF-8 flag seems to be ignored by unzip unless we also mark the archive entry as coming from a Unix system. This is done by setting the field creator_version ("version made by" in the standard[1]) to 0x03NN. The NN part represents the version of the standard supported by us, and this patch sets it to 3f (for version 6.3) for Unix paths. We keep creator_version set to 0 (FAT filesystem, standard version 0) in the non-special cases, as before. But when we declare a file to have a Unix path, then we have to set the file mode as well, or unzip will extract the files with the permission set 0000, i.e. inaccessible by all. [1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT --- No sign-off for this, yet. Perhaps there is a better way to convince unzip to respect the flag? And if not, do we need to offer umask settings for ZIP as well as we have for tar? And perhaps declare all files as being from a Unix filesystem, for consistency? archive-zip.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/archive-zip.c b/archive-zip.c index 0f763e8..e9b3dc9 100644 --- a/archive-zip.c +++ b/archive-zip.c @@ -186,7 +186,8 @@ static int write_zip_entry(struct archiver_args *args, { struct zip_local_header header; struct zip_dir_header dirent; - unsigned long attr2; + unsigned int creator_version = 0; + unsigned long attr2 = 0; unsigned long compressed_size; unsigned long crc; unsigned long direntsize; @@ -224,10 +225,15 @@ static int write_zip_entry(struct archiver_args *args, enum object_type type = sha1_object_info(sha1, &size); method = 0; - attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) : - (mode & 0111) ? ((mode) << 16) : 0; if (S_ISREG(mode) && args->compression_level != 0 && size > 0) method = 8; + if (S_ISLNK(mode) || (mode & 0111) || (flags & ZIP_UTF8)) { + creator_version = 0x033f; + attr2 = mode; + if (S_ISLNK(mode)) + attr2 |= 0777; + attr2 <<= 16; + } compressed_size = size; if (S_ISREG(mode) && type == OBJ_BLOB && !args->convert && @@ -274,8 +280,7 @@ static int write_zip_entry(struct archiver_args *args, } copy_le32(dirent.magic, 0x02014b50); - copy_le16(dirent.creator_version, - S_ISLNK(mode) || (S_ISREG(mode) && (mode & 0111)) ? 0x0317 : 0); + copy_le16(dirent.creator_version, creator_version); copy_le16(dirent.version, 10); copy_le16(dirent.flags, flags); copy_le16(dirent.compression_method, method); -- 1.7.12 ``` ## René Scharfe, 2012-09-18 20:24 Subject: Re: git archive --format zip utf-8 issues Message-ID: <5058D870.1030204@lsrfire.ath.cx> URL: https://gitlist.dev/e/5058D870.1030204%40lsrfire.ath.cx In-Reply-To: <5058CE49.3070108@lsrfire.ath.cx> ``` Am 18.09.2012 21:40, schrieb René Scharfe: > Windows Info-ZIP unzip > 7-Zip PeaZip builtin Linux msysgit Windows > git-master-patch1 0 0 72 60 72 72 > git-master-patch2 0 0 72 0 72 72 Oh, and when I wrote Windows, I meant a German Windows 7 Home Premium x64, and Linux is Ubuntu 12.04 x86. I've also attached generated ZIP files for the two archivers above, for easier testing. Let's see if they make it through. René ``` ## Junio C Hamano, 2012-09-18 21:12 Subject: Re: git archive --format zip utf-8 issues Message-ID: <7v1uhzkpcc.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v1uhzkpcc.fsf%40alter.siamese.dyndns.org In-Reply-To: <5058CE49.3070108@lsrfire.ath.cx> ``` René Scharfe writes: > Windows Info-ZIP unzip > 7-Zip PeaZip builtin Linux msysgit Windows > 7-Zip 9.20 0 0 46 26 43 43 > PeaZip 4.7.1 win64 0 0 46 26 42 42 > Info-ZIP zip 3.0 Linux 0 0 72 0 43 43 > Info-ZIP zip 3.0 Windows 45 45 n/a 0 43 43 > ... > I wonder what 7-Zip and PeaZip do that gives them a slightly nicer > score with the Windows-internal unzipper. Umlauts, Nordic characters > and accents are preserved by that combination. It seems that unzip on > Linux fails to unpack exactly these names, so perhaps they employ a > dirty trick like using the local encoding in the ZIP file, which makes > it unportable. > ... Thanks for this work. It is kind of surprising that "Windows builtin" has very poor score extracting from the output of Zip tools running on Windows (I am looking at 46, 46 and n/a over there). If you tell it to create an archive from its disk and then extract from it, I wonder what would happen. Does this result mean that practically nobody uses Zip archive with exotic letters in paths on that platform? I am not talking about developers and savvy people who know where to download third-party Zip archivers and how to install them. I am imagining a grandma who received an archive full of photos of her grandchild in her Outlook Express or GMail inbox, clicked the attachment to download it, and is trying to view the photo inside. ``` ## René Scharfe, 2012-09-20 22:00 Subject: Re: git archive --format zip utf-8 issues Message-ID: <505B91E9.7060208@lsrfire.ath.cx> URL: https://gitlist.dev/e/505B91E9.7060208%40lsrfire.ath.cx In-Reply-To: <7v1uhzkpcc.fsf@alter.siamese.dyndns.org> ``` Am 18.09.2012 23:12, schrieb Junio C Hamano: > René Scharfe writes: > >> Windows Info-ZIP unzip >> 7-Zip PeaZip builtin Linux msysgit Windows >> 7-Zip 9.20 0 0 46 26 43 43 >> PeaZip 4.7.1 win64 0 0 46 26 42 42 >> Info-ZIP zip 3.0 Linux 0 0 72 0 43 43 >> Info-ZIP zip 3.0 Windows 45 45 n/a 0 43 43 > It is kind of surprising that "Windows builtin" has very poor score > extracting from the output of Zip tools running on Windows (I am > looking at 46, 46 and n/a over there). If you tell it to create an > archive from its disk and then extract from it, I wonder what would > happen. I didn't include it as a packer because it refused to archive the pangrams directory due to illegal characters in one of the filenames. When I just tried a bit harder, I had to delete all but 14 files with Latin script, accents etc. before I could zip the directory. I'll include these results in the next round. It uses codepage 850 on my system (MSDOS Latin 1). I don't expect this to be portable. > Does this result mean that practically nobody uses Zip archive with > exotic letters in paths on that platform? I am not talking about > developers and savvy people who know where to download third-party > Zip archivers and how to install them. I am imagining a grandma who > received an archive full of photos of her grandchild in her Outlook > Express or GMail inbox, clicked the attachment to download it, and > is trying to view the photo inside. Not necessarily. Photos often have names like img_0123.jpg etc., which are handled just fine. And all family members probably use the same codepage on their computers, so they're less likely to run into this problem. By the way, I found this bug asking for codepage support in unzip: https://bugs.launchpad.net/ubuntu/+source/unzip/+bug/580961 Multiple codepages seem to be used for ZIP files in the wild, none of them are supported by unzip on Linux, which only accepts ASCII or UTF-8. René ``` ## René Scharfe, 2012-09-24 15:56 Subject: [PATCH 3/2] archive-zip: write extended timestamp Message-ID: <506082A7.4010108@lsrfire.ath.cx> URL: https://gitlist.dev/e/506082A7.4010108%40lsrfire.ath.cx In-Reply-To: <5058CE49.3070108@lsrfire.ath.cx> ``` File modification times in ZIP files are encoded in DOS format: local time with a granularity of two seconds. Add an extra field to all archive entries to also record the mtime in Unix' fashion, as UTC with a granularity of one second. This has the desirable side-effect of convincing Info-ZIP unzip 6.00 to respect general purpose flag 11, which is used to indicate that a file name is encoded in UTF-8. Any extra field would do, actually, but the extended timestamp is a reasonably small one (22 bytes per entry). Archives created by Info-ZIP zip 3.0 contain it, too (but with ctime and atime as well). Signed-off-by: Rene Scharfe --- archive-zip.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/archive-zip.c b/archive-zip.c index 0f763e8..55f66b4 100644 --- a/archive-zip.c +++ b/archive-zip.c @@ -76,6 +76,14 @@ struct zip_dir_trailer { unsigned char _end[1]; }; +struct zip_extra_mtime { + unsigned char magic[2]; + unsigned char extra_size[2]; + unsigned char flags[1]; + unsigned char mtime[4]; + unsigned char _end[1]; +}; + /* * On ARM, padding is added at the end of the struct, so a simple * sizeof(struct ...) reports two bytes more than the payload size @@ -85,6 +93,9 @@ struct zip_dir_trailer { #define ZIP_DATA_DESC_SIZE offsetof(struct zip_data_desc, _end) #define ZIP_DIR_HEADER_SIZE offsetof(struct zip_dir_header, _end) #define ZIP_DIR_TRAILER_SIZE offsetof(struct zip_dir_trailer, _end) +#define ZIP_EXTRA_MTIME_SIZE offsetof(struct zip_extra_mtime, _end) +#define ZIP_EXTRA_MTIME_PAYLOAD_SIZE \ + (ZIP_EXTRA_MTIME_SIZE - offsetof(struct zip_extra_mtime, flags)) static void copy_le16(unsigned char *dest, unsigned int n) { @@ -186,6 +197,7 @@ static int write_zip_entry(struct archiver_args *args, { struct zip_local_header header; struct zip_dir_header dirent; + struct zip_extra_mtime extra; unsigned long attr2; unsigned long compressed_size; unsigned long crc; @@ -266,8 +278,13 @@ static int write_zip_entry(struct archiver_args *args, } } + copy_le16(extra.magic, 0x5455); + copy_le16(extra.extra_size, ZIP_EXTRA_MTIME_PAYLOAD_SIZE); + extra.flags[0] = 1; /* just mtime */ + copy_le32(extra.mtime, args->time); + /* make sure we have enough free space in the dictionary */ - direntsize = ZIP_DIR_HEADER_SIZE + pathlen; + direntsize = ZIP_DIR_HEADER_SIZE + pathlen + ZIP_EXTRA_MTIME_SIZE; while (zip_dir_size < zip_dir_offset + direntsize) { zip_dir_size += ZIP_DIRECTORY_MIN_SIZE; zip_dir = xrealloc(zip_dir, zip_dir_size); @@ -283,7 +300,7 @@ static int write_zip_entry(struct archiver_args *args, copy_le16(dirent.mdate, zip_date); set_zip_dir_data_desc(&dirent, size, compressed_size, crc); copy_le16(dirent.filename_length, pathlen); - copy_le16(dirent.extra_length, 0); + copy_le16(dirent.extra_length, ZIP_EXTRA_MTIME_SIZE); copy_le16(dirent.comment_length, 0); copy_le16(dirent.disk, 0); copy_le16(dirent.attr1, 0); @@ -301,11 +318,13 @@ static int write_zip_entry(struct archiver_args *args, else set_zip_header_data_desc(&header, size, compressed_size, crc); copy_le16(header.filename_length, pathlen); - copy_le16(header.extra_length, 0); + copy_le16(header.extra_length, ZIP_EXTRA_MTIME_SIZE); write_or_die(1, &header, ZIP_LOCAL_HEADER_SIZE); zip_offset += ZIP_LOCAL_HEADER_SIZE; write_or_die(1, path, pathlen); zip_offset += pathlen; + write_or_die(1, &extra, ZIP_EXTRA_MTIME_SIZE); + zip_offset += ZIP_EXTRA_MTIME_SIZE; if (stream && method == 0) { unsigned char buf[STREAM_BUFFER_SIZE]; ssize_t readlen; @@ -402,6 +421,8 @@ static int write_zip_entry(struct archiver_args *args, zip_dir_offset += ZIP_DIR_HEADER_SIZE; memcpy(zip_dir + zip_dir_offset, path, pathlen); zip_dir_offset += pathlen; + memcpy(zip_dir + zip_dir_offset, &extra, ZIP_EXTRA_MTIME_SIZE); + zip_dir_offset += ZIP_EXTRA_MTIME_SIZE; zip_dir_entries++; return 0; -- 1.7.12 ``` ## René Scharfe, 2012-09-24 15:56 Subject: Re: git archive --format zip utf-8 issues Message-ID: <506082C9.6050604@lsrfire.ath.cx> URL: https://gitlist.dev/e/506082C9.6050604%40lsrfire.ath.cx In-Reply-To: <505B91E9.7060208@lsrfire.ath.cx> ``` Hi, I found a way to make unzip respect the UTF-8 flag in ZIP files: Apparently (from looking at the source) an extended field needs to be present in order for it to even look at general purpose flag 11. I sent a patch to add an extended timestamp field that fits the bill. Here are new numbers on ZIP international filename compatibility: 7-Zip PeaZip builtin unzip unzip unzip 7z Windows Windows Windows Linux mingw Windows Linux git Linux 1 1 1 7 1 1 1 git 1 Linux 37 37 1 7 1 1 37 git 2 Linux 37 37 1 37 1 1 37 git 3 Linux 37 37 1 37 15 15 37 git mingw 1 1 1 7 1 1 1 git 1 mingw 37 37 1 7 1 1 37 git 2 mingw 37 37 1 37 1 1 37 git 3 mingw 37 37 1 37 15 15 37 7-Zip Windows 37 37 14 24 15 15 24 PeaZip Windows 37 37 14 24 15 15 24 zip Linux 37 37 1 37 15 15 37 zip Windows 14 14 0 37 15 15 1 builtin Windows 14 14 14 1 14 14 1 The test corpus still consists of 37 files based on the pangrams on the following web page: http://www.columbia.edu/~fdc/utf8/index.html#quickbrownfox The files can be created using the attached script. It also provides a check command to count the files with correct names, and the results of that for different ZIP extractors are give in the table. The built-in ZIP functionality on Windows was only able to pack 14 of the 37 files, which explains the low score across the board for this packer. "git 1" is the patch "archive-zip: support UTF-8 paths" added, which let's archive-zip make use of the UTF-8 flag. "git 2" is "git 1" plus the patch "archive-zip: declare creator to be Unix for UTF-8 paths". Both have been posted before. "git 3" is "git 1" plus the new patch "archive-zip: write extended timestamp". Let's drop patch 2 (Unix as creator) and keep patches 1 (UTF-8 flag) and 3 (mtime field) to make archive-zip record non-ASCII filenames in a portable way. It's not perfect, but I don't know how to do any better given that Windows' built-in ZIP functionality expects filenames in the local code page and with an international audience for projects distributing ZIP files. René #!/bin/sh files() { cat <&2 exit 1 ;; esac ``` ## Junio C Hamano, 2012-09-24 18:13 Subject: Re: git archive --format zip utf-8 issues Message-ID: <7vtxunz3v3.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vtxunz3v3.fsf%40alter.siamese.dyndns.org In-Reply-To: <506082C9.6050604@lsrfire.ath.cx> ``` René Scharfe writes: > "git 1" is the patch "archive-zip: support UTF-8 paths" added, which > let's archive-zip make use of the UTF-8 flag. "git 2" is "git 1" plus > the patch "archive-zip: declare creator to be Unix for UTF-8 > paths". Both have been posted before. "git 3" is "git 1" plus the new > patch "archive-zip: write extended timestamp". > > Let's drop patch 2 (Unix as creator) and keep patches 1 (UTF-8 flag) > and 3 (mtime field) to make archive-zip record non-ASCII filenames in > a portable way. It's not perfect, but I don't know how to do any > better given that Windows' built-in ZIP functionality expects > filenames in the local code page and with an international audience > for projects distributing ZIP files. Thanks. ```