git archive --format zip utf-8 issues

Hi,

when I create a git repository, add a file containing utf-8 characters or umlauts (like öäü.txt), commit and then export the HEAD revision to a zip archive using "git archive --format zip -o 1.zip HEAD", the zip file contains incorrect filenames:

$ unzip -l 1.zip
Archive:  1.zip
4490a6dab1df5404f91ab3eb871f133154bff0bf
  Length      Date    Time    Name
---------  ---------- -----   ----
        6  2012-08-10 23:41   +?+?++.txt
---------                     -------
        6                     1 file

Sven Strickroth <sven.strickroth@tu-clausthal.de> writes:

My reading of archive-zip.c seems to suggest that we write out whatever pathname you have in the tree, so a pathname encoded in UTF-8 will be literally written out in the resulting zip archive.

Do you know in what encoding the pathnames are _expected_ to be stored in zip archives? Random documentation seems to suggest that there is no standard encoding, e.g. http://docs.python.org/library/zipfile.html says:

    There is no official file name encoding for ZIP files. If you
    have unicode file names, you must convert them to byte strings
    in your desired encoding before passing them to write(). WinZip
    interprets all file names as encoded in CP437, also known as DOS
    Latin.

which may explain it.

It may not be a bad idea for "git archive --format=zip" to

 (1) check if pathname is a correct UTF-8; and
 (2) check if it can be reencoded to latin-1

and if (and only if) both are true, automatically re-encode the path to latin-1.

Of course, "git archive --format=zip --path-reencode=utf8-to-latin1" would be the most generic way to do this.

Am 11.08.2012 00:47 schrieb Junio C Hamano:

re-encoding to latin1 does not always work and may break double byte totally (e.g. chinese or japanese).

PKZIP APPNOTE seems to be the zip standard and it specifies a utf-8 flag: http://www.pkware.com/documents/casestudies/APPNOTE.TXT

Am 11.08.2012 01:53, schrieb Sven Strickroth:

Yes, that's one of the two methods for supporting UTF-8 filenames described there.

The other method involves writing extra ZIP header fields and was invented by Info-ZIP. They don't use it consistently anymore, though (from zip -h2):

  "Zip now stores UTF-8 in entry path and comment fields on systems
   where UTF-8 char set is default, such as most modern Unix, and
   and on other systems in new extra fields with escaped versions in
   entry path and comment fields for backward compatibility."

René

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

Thanks; so if we adopt one of these methods, the readers that matter will be happy? And if so, which one? Or both?

Am 11.08.2012 00:47, schrieb Junio C Hamano:

Sorry for my imperialistic attitude of "ASCII filenames should be enough for everybody". Laziness..

http://www.pkware.com/documents/casestudies/APPNOTE.TXT is the standard document, as Sven noted, and it says that filenames are encoded in code page 437, or optionally UTF-8 (a later addition). Discussions like http://stackoverflow.com/questions/106367/ seem to indicate that at least some archivers use the local code page as well.

The standard says we need to convert to CP437, or to UTF-8, or provide both versions. A more interesting question is: What's supported by which programs?

The ZIP functionality built into Windows 7 doesn't seem to work with UTF-8 encoded filenames (except for those that only use the ASCII subset), and to ignore the UTF-8 part if both are given. Handling umlauts should be possible anyway, because they are on code page 437, but for other characters we'd have to aim for compatibility with other programs like Info-ZIP and 7-Zip.

How do we know which encoding was used for a filename?

I really hope we can make do without additional options.

René

Am 11.08.2012 22:53 schrieb René Scharfe:

I played a bit with the git source code and found out, that

On Sat, Aug 11, 2012 at 11:37:05PM +0200, Sven Strickroth wrote:

Ping on this stalled discussion.

It seems like there are two separate issues here:

  1. Knowing the encoding of pathnames in the repository.

  2. Setting the right flags in zip output.

A full solution would handle both parts, but let's ignore (1) for a moment, and assume we have utf-8 (or can massage into utf-8 from an encoding specified by the user).

It seems like just setting the magic utf-8 flag would be the only thing we need to do, according to the standard. But according to discussions referenced elsewhere in this thread, that flag was invented only in 2007, so we may be dealing with older implementations (I have no idea how common they would be; that may be the problem with Windows 7's zip you are seeing). We could re-encode to cp437, which the standard specifies, but apparently some implementations do not respect that (and use a local code page instead). And it cannot represent all utf-8 characters, anyway.

It sounds like 7-zip has figured out a more portable solution. Can you show us a sample of 7-zip's output with utf-8 characters to compare to what git generates? I wonder if it is using a combination of methods.

-Peff

Am 31.08.2012 00:26, schrieb Jeff King:

Sorry, I got distracted by other stuff again. I did some experiments, though, and here's a preliminary result.

Yes, good thinking. Re-encoding may be beneficial for tar files as well, but we can ignore that point for the moment.

Yes, we could do that, plus adding an extra field with a UTF-8 version of the path. That's the legacy method invented by Info-ZIP. They switched to using the new flag on Linux at least, though.

I'm not so sure they produce more portable files. I created an archive with files named jaя.txt, smørrebrød.txt, süd.txt and €uro.txt with 7-Zip on Windows 7 and while unzip on Ubuntu 12.04 managed to recreate the cyrillic character and the Euro symbol, it mangled the slashed o and the umlaut.

With the following patch I could create archives with git on Linux and msysgit and extract them flawlessly on Windows with 7-Zip and with Info-ZIP unzip on Linux, but not with unzip on Windows, where it mangled all non-ASCII characters.

This gets confusing; it would help to have a compatibility matrix for all intersting extractors and character classes -- for each proposed solution or archiver we'd like to imitate.

But now for the patch, which is a bit confusing as well. I'm curious to hear about results for more platforms, extractors and character classes. Based on that we can see if we need to generate the extra fields instead of relying on the new flag.

-- >8 --
Subject: [PATCH] archive-zip: support UTF-8 paths

Set general purpose flag 11 if we encounter a path that contains non-ASCII characters. We assume that all paths are given as UTF-8; no conversion is done.

The flag seems to be ignored by unzip unless we also mark the archive entry as coming from a Unix system. This is done by setting the field creator_version ("version made by" in the standard[1]) to 0x03NN.

The NN part represents the version of the standard supported by us, and this patch sets it to 3f (for version 6.3) for Unix paths. We keep creator_version set to 0 (FAT filesystem, standard version 0) in the non-special cases, as before.

But when we declare a file to have a Unix path, then we have to set the file mode as well, or unzip will extract the files with the permission set 0000, i.e. inaccessible by all.

[1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
 archive-zip.c | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

Thanks for keeping the ball rolling.

Do we want to treat \033 as "ascii" in this codepath? The function primarily is used by the log formatter to see if we need 8-bit CTE when writing out in the e-mail format.

Am 04.09.2012 23:03, schrieb Junio C Hamano:

Argh, yes, I'd think so. The function name mislead me. This won't matter for compatibility testing, but should be corrected before inclusion.

Just checked: unzip strips the ESC character when extracting (whether the UTF-8 flag is set or not) and 7-Zip replaces it with an underscore. The built-in ZIP extractor of Windows 7 skips such a file; it doesn't even show up in archive directory listings.

René

Hello again,

so two weeks have passed, and I've moved at a glacial pace towards a method how to measure compatibility of our generated ZIP files. Sorry, I just keep getting distracted.

Anyway, the idea is to have a bunch of files with names using different scripts, zip them with several packers (including git archive), unzip them and compare the result with the original files.

As test corpus I used files named like the pangrams on this UTF-8 sampler page, the exact commands are attached:

    http://www.columbia.edu/~fdc/utf8/index.html#quickbrownfox

The numbers below are how many lines the output of diff -ru contains for this pair of packer and unpacker. There are 37 files, so the worst result is 74 lines of difference ("Only in [...]" for both sides), while 0 indicates a perfect score.

Hmm, come to think of it, an empty directory would show up as 37, so this metric is not ideal. A better one would be to simply give one point for each correctly unpacked file.

                                          Windows    Info-ZIP unzip
                             7-Zip PeaZip builtin Linux msysgit Windows
7-Zip 9.20                      0      0      46    26      43      43
PeaZip 4.7.1 win64              0      0      46    26      42      42
Info-ZIP zip 3.0 Linux          0      0      72     0      43      43
Info-ZIP zip 3.0 Windows       45     45     n/a     0      43      43
git-master                     72     72      72    60      72      72
git-master-patch1               0      0      72    60      72      72
git-master-patch2               0      0      72     0      72      72
git-v1.7.11.msysgit.1          72     72      72    60      72      72
git-v1.7.11.msysgit.1-patch1    0      0      72    60      72      72
git-v1.7.11.msysgit.1-patch2    0      0      72     0      72      72

Info-ZIP's programs don't work too well on Windows. The built-in unzipper of Windows 7 even refuses to open the file created by the Windows version of zip. Speaking of which, this is the worst of the unpackers.

With the two patches applied, we can say "use 7-Zip or PeaZip on Windows and unzip on Linux" and filenames with all tested characters will be preserved. I was surprised to see this working fine with msysgit like that, even though no reencoding is introduced by the patches.

I wonder what 7-Zip and PeaZip do that gives them a slightly nicer score with the Windows-internal unzipper. Umlauts, Nordic characters and accents are preserved by that combination. It seems that unzip on Linux fails to unpack exactly these names, so perhaps they employ a dirty trick like using the local encoding in the ZIP file, which makes it unportable.

I'll reply with the two patches, which contain basically the same code as the previous patch, only split up. The second one declares that filenames with UTF-8 encoding came from Unix (instead of FAT), which makes unzip happy. This, however, implies that we contain Unix permissions for these entries, which is a bit ugly.

René

#!/bin/sh
(
	mkdir pangrams
	cd pangrams

	echo English >"The quick brown fox jumps over the lazy dog"
	echo Irish 1 >"An ḃfuil do ċroí ag bualaḋ ó ḟaitíos an ġrá a �eall"
	echo Irish 2 >"lena ṗóg éada ó ṡlí do leasa ṫú"
	echo Irish 3 >"D'ḟuascail �osa Úr�ac na hÓiġe Beannaiṫe pór"
	echo Irish 4 >"Éava agus �ḋai�"
	echo Dutch >"Pa's wijze lynx bezag vroom het fikse aquaduct"
	echo German 1 >"Falsches Üben von Xylophonmusik quält"
	echo German 2 >"jeden größeren Zwerg"
	echo Norwegian >"Blåbærsyltetøy"
	echo Danish >"Høj bly gom vandt fræk sexquiz på wc"
	echo Swedish >"Flygande bäckasiner söka strax hwila på mjuka tuvor"
	echo Icelandic >"Sævör grét áðan því úlpan var ónýt"
	echo Finnish >"Törkylempijävongahdus"
	echo Polish >"Pchnąć w tę łódź jeża lub osiem skrzyń fig"
	echo Czech >"Příliš žluťou�ký kůň úpěl �ábelské kódy"
	echo Slovak 1 >"Starý kôň na hŕbe kníh žuje tíško povädnuté ruže"
	echo Slovak 2 >"na stĺpe sa �ateľ u�í kvákať novú ódu o živote"
	echo monotonic Greek >"ξεσκεπάζω την ψυχοφθό�α βδελυγμία"
	echo polytonic Greek >"ξεσκεπάζω τὴν ψυχοφθό�α βδελυγμία"
	echo Russian >"Съешь же ещё �тих м�гких француз�ких булок да выпей чаю"
	echo Bulgarian 1 >"Жълтата дюл� беше ща�тлива"
	echo Bulgarian 2 >"че пухът, който цъфна, замръзна като гьон"
	echo Northern Sami >"Vuol Ruoŧa geđggiid leat máŋga luosa ja �uovžža"
	echo Hungarian >"�rvíztűrő tükörfúrógép"
	echo Spanish 1 >"El pingüino Wenceslao hizo kilómetros bajo exhaustiva"
	echo Spanish 2 >"lluvia y frío añoraba a su querido cachorro"
	echo Portuguese 1 >"O próximo vôo à noite sobre o Atlântico"
	echo Portuguese 2 >"põe freqüentemente o único médico"
	echo French 1 >"Les naïfs ægithales hâtifs pondant à Noël où il gèle"
	echo French 2 >"sont sûrs d'être déçus en voyant leurs drôles"
	echo French 3 >"d'œufs abîmés"
	echo Esperanto >"Eĥo�an�o ĉiuĵaŭde"
	echo Hebrew >"זה כיף סת� לשמוע �יך תנצח קרפד עץ טוב בגן"
	echo Hiragana 1 >"�������　�り�るを"
	echo Hiragana 2 >"��よ�れ�　���らむ"
	echo Hiragana 3 >"�����や�　�����"
	echo Hiragana 4 >"���ゆ���　ゑ�も��"
)

Set general purpose flag 11 if we encounter a path that contains non-ASCII characters. We assume that all paths are given as UTF-8; no conversion is done.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
Changes from previous version: Stop using has_non_ascii(), which does
slightly too much for our purposes, and split off creator version
change into a separate patch.

 archive-zip.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

The UTF-8 flag seems to be ignored by unzip unless we also mark the archive entry as coming from a Unix system. This is done by setting the field creator_version ("version made by" in the standard[1]) to 0x03NN.

The NN part represents the version of the standard supported by us, and this patch sets it to 3f (for version 6.3) for Unix paths. We keep creator_version set to 0 (FAT filesystem, standard version 0) in the non-special cases, as before.

But when we declare a file to have a Unix path, then we have to set the file mode as well, or unzip will extract the files with the permission set 0000, i.e. inaccessible by all.

[1] http://www.pkware.com/documents/casestudies/APPNOTE.TXT --- No sign-off for this, yet. Perhaps there is a better way to convince unzip to respect the flag? And if not, do we need to offer umask settings for ZIP as well as we have for tar? And perhaps declare all files as being from a Unix filesystem, for consistency?

 archive-zip.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

Am 18.09.2012 21:40, schrieb René Scharfe:

Oh, and when I wrote Windows, I meant a German Windows 7 Home Premium x64, and Linux is Ubuntu 12.04 x86.

I've also attached generated ZIP files for the two archivers above, for easier testing. Let's see if they make it through.

René

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

Thanks for this work.

It is kind of surprising that "Windows builtin" has very poor score extracting from the output of Zip tools running on Windows (I am looking at 46, 46 and n/a over there). If you tell it to create an archive from its disk and then extract from it, I wonder what would happen.

Does this result mean that practically nobody uses Zip archive with exotic letters in paths on that platform? I am not talking about developers and savvy people who know where to download third-party Zip archivers and how to install them. I am imagining a grandma who received an archive full of photos of her grandchild in her Outlook Express or GMail inbox, clicked the attachment to download it, and is trying to view the photo inside.

Am 18.09.2012 23:12, schrieb Junio C Hamano:

I didn't include it as a packer because it refused to archive the pangrams directory due to illegal characters in one of the filenames. When I just tried a bit harder, I had to delete all but 14 files with Latin script, accents etc. before I could zip the directory. I'll include these results in the next round.

It uses codepage 850 on my system (MSDOS Latin 1). I don't expect this to be portable.

Not necessarily. Photos often have names like img_0123.jpg etc., which are handled just fine. And all family members probably use the same codepage on their computers, so they're less likely to run into this problem.

By the way, I found this bug asking for codepage support in unzip:

   https://bugs.launchpad.net/ubuntu/+source/unzip/+bug/580961

Multiple codepages seem to be used for ZIP files in the wild, none of them are supported by unzip on Linux, which only accepts ASCII or UTF-8.

René

Hi,

I found a way to make unzip respect the UTF-8 flag in ZIP files: Apparently (from looking at the source) an extended field needs to be present in order for it to even look at general purpose flag 11. I sent a patch to add an extended timestamp field that fits the bill.

Here are new numbers on ZIP international filename compatibility:

		7-Zip	PeaZip	builtin	unzip	unzip	unzip	7z
		Windows	Windows	Windows	Linux	mingw	Windows	Linux
git	Linux	1	1	1	7	1	1	1
git 1	Linux	37	37	1	7	1	1	37
git 2	Linux	37	37	1	37	1	1	37
git 3	Linux	37	37	1	37	15	15	37
git	mingw	1	1	1	7	1	1	1
git 1	mingw	37	37	1	7	1	1	37
git 2	mingw	37	37	1	37	1	1	37
git 3	mingw	37	37	1	37	15	15	37
7-Zip	Windows	37	37	14	24	15	15	24
PeaZip	Windows	37	37	14	24	15	15	24
zip	Linux	37	37	1	37	15	15	37
zip	Windows	14	14	0	37	15	15	1
builtin	Windows	14	14	14	1	14	14	1

The test corpus still consists of 37 files based on the pangrams on the following web page:

	http://www.columbia.edu/~fdc/utf8/index.html#quickbrownfox

The files can be created using the attached script. It also provides a check command to count the files with correct names, and the results of that for different ZIP extractors are give in the table. The built-in ZIP functionality on Windows was only able to pack 14 of the 37 files, which explains the low score across the board for this packer.

"git 1" is the patch "archive-zip: support UTF-8 paths" added, which let's archive-zip make use of the UTF-8 flag. "git 2" is "git 1" plus the patch "archive-zip: declare creator to be Unix for UTF-8 paths". Both have been posted before. "git 3" is "git 1" plus the new patch "archive-zip: write extended timestamp".

Let's drop patch 2 (Unix as creator) and keep patches 1 (UTF-8 flag) and 3 (mtime field) to make archive-zip record non-ASCII filenames in a portable way. It's not perfect, but I don't know how to do any better given that Windows' built-in ZIP functionality expects filenames in the local code page and with an international audience for projects distributing ZIP files.

René

#!/bin/sh

files() { cat <<EOF pangrams/��よ�れ�　���らむ pangrams/�����や�　����� pangrams/�������　�り�るを pangrams/���ゆ���　ゑ�も�� pangrams/An ḃfuil do ċroí ag bualaḋ ó ḟaitíos an ġrá a �eall pangrams/�rvíztűrő tükörfúrógép pangrams/Blåbærsyltetøy pangrams/D'ḟuascail �osa Úr�ac na hÓiġe Beannaiṫe pór pangrams/d'œufs abîmés pangrams/Éava agus �ḋai� pangrams/Eĥo�an�o ĉiuĵaŭde pangrams/El pingüino Wenceslao hizo kilómetros bajo exhaustiva pangrams/Falsches Üben von Xylophonmusik quält pangrams/Flygande bäckasiner söka strax hwila på mjuka tuvor pangrams/Høj bly gom vandt fræk sexquiz på wc pangrams/jeden größeren Zwerg pangrams/lena ṗóg éada ó ṡlí do leasa ṫú pangrams/Les naïfs ægithales hâtifs pondant à Noël où il gèle pangrams/lluvia y frío añoraba a su querido cachorro pangrams/na stĺpe sa �ateľ u�í kvákať novú ódu o živote pangrams/O próximo vôo à noite sobre o Atlântico pangrams/Pa's wijze lynx bezag vroom het fikse aquaduct pangrams/Pchnąć w tę łódź jeża lub osiem skrzyń fig pangrams/põe freqüentemente o único médico pangrams/Příliš žluťou�ký kůň úpěl �ábelské kódy pangrams/Sævör grét áðan því úlpan var ónýt pangrams/sont sûrs d'être déçus en voyant leurs drôles pangrams/Starý kôň na hŕbe kníh žuje tíško povädnuté ruže pangrams/The quick brown fox jumps over the lazy dog pangrams/Törkylempijävongahdus pangrams/Vuol Ruoŧa geđggiid leat máŋga luosa ja �uovžža pangrams/זה כיף סת� לשמוע �יך תנצח קרפד עץ טוב בגן pangrams/ξεσκεπάζω την ψυχοφθό�α βδελυγμία pangrams/ξεσκεπάζω τὴν ψυχοφθό�α βδελυγμία pangrams/Жълтата дюл� беше ща�тлива pangrams/Съешь же ещё �тих м�гких француз�ких булок да выпей чаю pangrams/че пухът, който цъфна, замръзна като гьон EOF }

case "$1" in
create)
	mkdir -p pangrams
	files | while read file
	do
		touch "$file"
	done
	;;
check)
	files | while read file
	do
		test -f "$file" && echo "$file"
	done | wc -l
	;;
*)
	echo "Usage: $0 create | check" >&2
	exit 1
	;;
esac

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

Thanks.

File modification times in ZIP files are encoded in DOS format: local time with a granularity of two seconds. Add an extra field to all archive entries to also record the mtime in Unix' fashion, as UTC with a granularity of one second.

This has the desirable side-effect of convincing Info-ZIP unzip 6.00 to respect general purpose flag 11, which is used to indicate that a file name is encoded in UTF-8. Any extra field would do, actually, but the extended timestamp is a reasonably small one (22 bytes per entry). Archives created by Info-ZIP zip 3.0 contain it, too (but with ctime and atime as well).

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
 archive-zip.c | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

Yes, that is the most interesting question.

We need to at least know the path encoding used in the tree objects, and I'd be OK with a solution that assumes a single encoding is used for the entire tree.

We would eventually need to also know the encoding used on the local working tree (i.e. in what encoding paths are returned from readdir() and the pathspec the user gives us from the command line), and iconv it to the tree objects encoding for the project when creating a cache_entry object to be fed to add_to_index(), and iconv it back from the tree objects encoding to the working tree encoding in write_entry(), but that is a longer term direction. For now, in order to address the immediate issue, we only need the tree object encoding, which should default to UTF-8 for interoperability.

So "git archive --format=zip --in-object-path-encoding=big5" for a project whose tree object pathnames are in that encoding (and we always record paths in UTF-8 when writing zipfiles) should be the minimal that we need for now.

Optionally, with a configuration variable i18n.inObjectPathEncoding (as opposed to the eventual i18n.worktreePathEncoding) set to big5, users of such a project can say "git archive --format=zip" without the "--in-object-path-encoding" option.

Considering that zip is a format meant for exchange, I'd think we would be fine to always write in UTF-8 and leaving the readers responsible for converting the pathname while extracting. If a major zip extractor is incapable of handling UTF-8 (or even if capable it is cumbersome, for that matter), we may end up having to add "--in-archive-path-encoding=UTF-8" option to "git archive", with associated "zip.archivePathEncoding" variable, though.