{"thread":{"id":"19529","subject":"git-svn dcommits messages not in UTF-8 charset to mod_dav_svn?","startedAt":"2009-05-27T16:16:24Z","lastAt":"2009-05-30T00:14:47Z","messageCount":7,"participants":["Matthias Andree","Eric Wong","Junio C Hamano","Brandon Casey"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"114810","messageId":"op.uuljhmmg1e62zd@balu.cs.uni-paderborn.de","threadId":"19529","inReplyTo":null,"subject":"git-svn dcommits messages not in UTF-8 charset to mod_dav_svn?","fromName":"Matthias Andree","fromEmail":"matthias.andree@gmx.de","sentAt":"2009-05-27T16:16:24Z","receivedAt":"2009-05-27T16:16:24Z","isPatch":false,"sender":{"key":"matthias.andree@gmx.de","avatar":null},"body":"Greetings,\n\nI had tried to use git cvsimport and git svn to transfer a CVS repository \n(I have access to it) to an SVN repository (where I don't have access to \nthe repo, so I cannot use cvs2svn or similar).\n\nThe problem is that the CVS repo had non-UTF-8 commit log messages, and I \ndidn't bother to convert them to UTF-8. However, SVN insists on encoding \nfilenames and log messages in UTF-8.\n\n\"git svn dcommit\" (which uses the SVN Perl bindings under the hood) \nhappily committed such a non-UTF-8 message and br0ke the repo. The actual \nreason is that the SVN server (https://...) is now wedged, as in:\n\n$ svn log -r130\nsvn: REPORT of '/repos/!svn/bc/130': 200 OK (https://svn-serv...de)\n\n$ svn --xml log -r130 2>/dev/null\n[stdout:]\n\n\n[stderr:]\nsvn: REPORT of '/repos/!svn/bc/130': 200 OK (https://svn-serv...de)\n\n$ svn propget --revprop svn:log -r130 https://svn-serv...de/path/\naktuelle version (disclaimer)\nkopf und fu?\\223zeilen\netc.\n\n\nWhile mod_dav_svn arguably shouldn't accept b0rked messages, git-svn \nshouldn't attempt to commit them either. It seems that the svn command \nline utilities validate the message format by themselves, and apparently \nthe svn server module (likely mod_dav_svn - or are there others?) does not.\n\nSo, could\na) git-svn be modified to refuse dcommiting non-UTF-8 messages?\nb) mod_dav_svn be modified to refuse commits/propedits/propsets with \nnon-UTF-8 messages?\n\nI'm sorry to say I don't have information how the SVN server is configured \nand which version it's running.\n\nThanks.\n\n-- \nMatthias Andree\n"},{"id":"114902","messageId":"20090528080706.GA9239@dcvr.yhbt.net","threadId":"19529","inReplyTo":"op.uuljhmmg1e62zd@balu.cs.uni-paderborn.de","subject":"Re: git-svn dcommits messages not in UTF-8 charset to mod_dav_svn?","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-05-28T08:08:41Z","receivedAt":"2009-05-28T08:08:41Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Matthias Andree wrote:\n> Greetings,\n>\n> I had tried to use git cvsimport and git svn to transfer a CVS repository \n> (I have access to it) to an SVN repository (where I don't have access to \n> the repo, so I cannot use cvs2svn or similar).\n>\n> The problem is that the CVS repo had non-UTF-8 commit log messages, and I \n> didn't bother to convert them to UTF-8. However, SVN insists on encoding \n> filenames and log messages in UTF-8.\n>\n> \"git svn dcommit\" (which uses the SVN Perl bindings under the hood) \n> happily committed such a non-UTF-8 message and br0ke the repo. The actual \n> reason is that the SVN server (https://...) is now wedged, as in:\n\n> While mod_dav_svn arguably shouldn't accept b0rked messages, git-svn \n> shouldn't attempt to commit them either. It seems that the svn command \n> line utilities validate the message format by themselves, and apparently \n> the svn server module (likely mod_dav_svn - or are there others?) does \n> not.\n\nThis was partially fixed in commit\n16fc08e2d86dad152194829d21bc55b2ef0c8fb1. You just need to manually\nspecify the i18n.commitencoding in your .git/config\n\n> So, could\n> a) git-svn be modified to refuse dcommiting non-UTF-8 messages?\n\nOn the way is a patch that makes git-svn refuse to dcommit messages\nthat are malformed UTF-8 and tell the user about i18n.commitencoding.\n\nThanks for reminding me\n\n-- \nEric Wong\n"},{"id":"114903","messageId":"20090528081804.GB9239@dcvr.yhbt.net","threadId":"19529","inReplyTo":"20090528080706.GA9239@dcvr.yhbt.net","subject":"[PATCH] git-svn: refuse to dcommit non-UTF-8 messages","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-05-28T08:18:04Z","receivedAt":"2009-05-28T08:18:04Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"...without i18n.commitencoding set in the config.\n\nSVN tries to store all commit messages in UTF-8, however it is\nup to the job of the clients to enforce this rule. SVN servers\nthemselves do not always enforce this; allowing clients to\ncommit malformed UTF-8 messages and break repositories.\n\nSo git-svn will enforce this and tell the user to set\ni18n.commitencoding when a git commit is is not in UTF-8.\n\nSigned-off-by: Eric Wong \n---\n\n Also pushed to git://git.bogomips.org/git-svn.git\n\n git-svn.perl | 17 ++++++++--\n t/t9139-git-svn-non-utf8-commitencoding.sh | 47 ++++++++++++++++++++++++++++\n 2 files changed, 61 insertions(+), 3 deletions(-)\n create mode 100755 t/t9139-git-svn-non-utf8-commitencoding.sh\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex a70c7d7..3301797 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1178,16 +1178,27 @@ sub get_commit_entry {\n \t}\n \trename $commit_editmsg, $commit_msg or croak $!;\n \t{\n+\t\trequire Encode;\n \t\t# SVN requires messages to be UTF-8 when entering the repo\n \t\tlocal $/;\n \t\topen $log_fh, '<', $commit_msg or croak $!;\n \t\tbinmode $log_fh;\n \t\tchomp($log_entry{log} = <$log_fh>);\n \n-\t\tif (my $enc = Git::config('i18n.commitencoding')) {\n-\t\t\trequire Encode;\n-\t\t\tEncode::from_to($log_entry{log}, $enc, 'UTF-8');\n+\t\tmy $enc = Git::config('i18n.commitencoding') || 'UTF-8';\n+\t\tmy $msg = $log_entry{log};\n+\n+\t\teval { $msg = Encode::decode($enc, $msg, 1) };\n+\t\tif ($@) {\n+\t\t\tdie \"Could not decode as $enc:\\n\", $msg,\n+\t\t\t \"\\nPerhaps you need to set i18n.commitencoding\\n\";\n \t\t}\n+\n+\t\teval { $msg = Encode::encode('UTF-8', $msg, 1) };\n+\t\tdie \"Could not encode as UTF-8:\\n$msg\\n\" if $@;\n+\n+\t\t$log_entry{log} = $msg;\n+\n \t\tclose $log_fh or croak $!;\n \t}\n \tunlink $commit_msg;\ndiff --git a/t/t9139-git-svn-non-utf8-commitencoding.sh b/t/t9139-git-svn-non-utf8-commitencoding.sh\nnew file mode 100755\nindex 0000000..2b1db97\n--- /dev/null\n+++ b/t/t9139-git-svn-non-utf8-commitencoding.sh\n@@ -0,0 +1,47 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2009 Eric Wong\n+\n+test_description='git svn refuses to dcommit non-UTF8 messages'\n+\n+. ./lib-git-svn.sh\n+\n+# ISO-2022-JP can pass for valid UTF-8, so skipping that in this test\n+\n+for H in ISO-8859-1 EUCJP\n+do\n+\ttest_expect_success \"$H setup\" '\n+\t\tmkdir $H &&\n+\t\tsvn_cmd import -m \"$H test\" $H \"$svnrepo\"/$H &&\n+\t\tgit svn clone \"$svnrepo\"/$H $H\n+\t'\n+done\n+\n+for H in ISO-8859-1 EUCJP\n+do\n+\ttest_expect_success \"$H commit on git side\" '\n+\t(\n+\t\tcd $H &&\n+\t\tgit config i18n.commitencoding $H &&\n+\t\tgit checkout -b t refs/remotes/git-svn &&\n+\t\techo $H >F &&\n+\t\tgit add F &&\n+\t\tgit commit -a -F \"$TEST_DIRECTORY\"/t3900/$H.txt &&\n+\t\tE=$(git cat-file commit HEAD | sed -ne \"s/^encoding //p\") &&\n+\t\ttest \"z$E\" = \"z$H\"\n+\t)\n+\t'\n+done\n+\n+for H in ISO-8859-1 EUCJP\n+do\n+\ttest_expect_success \"$H dcommit to svn\" '\n+\t(\n+\t\tcd $H &&\n+\t\tgit config --unset i18n.commitencoding &&\n+\t\t! git svn dcommit\n+\t)\n+\t'\n+done\n+\n+test_done\n-- \nEric Wong\n"},{"id":"114984","messageId":"7vskioe66l.fsf@alter.siamese.dyndns.org","threadId":"19529","inReplyTo":"20090528081804.GB9239@dcvr.yhbt.net","subject":"Re: [PATCH] git-svn: refuse to dcommit non-UTF-8 messages","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-29T07:09:38Z","receivedAt":"2009-05-29T07:09:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.\n"},{"id":"114989","messageId":"7v3aaoe40j.fsf@alter.siamese.dyndns.org","threadId":"19529","inReplyTo":"20090528081804.GB9239@dcvr.yhbt.net","subject":"Re: [PATCH] git-svn: refuse to dcommit non-UTF-8 messages","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-29T07:56:28Z","receivedAt":"2009-05-29T07:56:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> t/t9139-git-svn-non-utf8-commitencoding.sh | 47 ++++++++++++++++++++++++++++\n\nHmm.\n\n> +# Copyright (c) 2009 Eric Wong\n> +\n> +test_description='git svn refuses to dcommit non-UTF8 messages'\n> +\n> +. ./lib-git-svn.sh\n\nThis passes when merged to 'master', but together with bc/old-iconv branch\ncooking in 'next' it breaks.\n"},{"id":"115013","messageId":"6I7Qq-BKaJmSe6FJkYgjMpNqo6RFAVB0STpL09QJO1skOCB2ezM6mA@cipher.nrlssc.navy.mil","threadId":"19529","inReplyTo":"7v3aaoe40j.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] git-svn: refuse to dcommit non-UTF-8 messages","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2009-05-29T14:11:22Z","receivedAt":"2009-05-29T14:11:22Z","isPatch":true,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Junio C Hamano wrote:\n> Eric Wong writes:\n> \n>> t/t9139-git-svn-non-utf8-commitencoding.sh | 47 ++++++++++++++++++++++++++++\n> \n> Hmm.\n> \n>> +# Copyright (c) 2009 Eric Wong\n>> +\n>> +test_description='git svn refuses to dcommit non-UTF8 messages'\n>> +\n>> +. ./lib-git-svn.sh\n> \n> This passes when merged to 'master', but together with bc/old-iconv branch\n> cooking in 'next' it breaks.\n\nYeah, it's the second for loop which accesses the files in t/t3900/.\nbc/old-iconv replaces each occurrence of ISO-8859-1 with ISO8859-1\nand EUCJP with eucJP since old Solaris didn't know both names and\nmodern platforms handle either name. The text files in t/t3900/\nwere renamed accordingly.\n\nIt would be nice to use these older names here too, even though I\nwon't be able to test it since svn is not installed on the older\nplatforms I have access to.\n\n-brandon\n"},{"id":"115059","messageId":"20090530001447.GA32188@dcvr.yhbt.net","threadId":"19529","inReplyTo":"6I7Qq-BKaJmSe6FJkYgjMpNqo6RFAVB0STpL09QJO1skOCB2ezM6mA@cipher.nrlssc.navy.mil","subject":"[PATCH] t9139 uses ancient, backwards-compatible iconv names","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-05-30T00:14:47Z","receivedAt":"2009-05-30T00:14:47Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This is needed to work with\n5ae93dfdccfe9457bdb1f54b33c76359f6c3b861:\n t3900: use ancient iconv names for backward compatibility\n\nSigned-off-by: Eric Wong \n---\n\n Brandon Casey wrote:\n > Junio C Hamano wrote:\n > > Eric Wong writes:\n > > \n > >> t/t9139-git-svn-non-utf8-commitencoding.sh | 47 ++++++++++++++++++++++++++++\n > > \n > > Hmm.\n > > \n > >> +# Copyright (c) 2009 Eric Wong\n > >> +\n > >> +test_description='git svn refuses to dcommit non-UTF8 messages'\n > >> +\n > >> +. ./lib-git-svn.sh\n > > \n > > This passes when merged to 'master', but together with bc/old-iconv branch\n > > cooking in 'next' it breaks.\n > \n > Yeah, it's the second for loop which accesses the files in t/t3900/.\n > bc/old-iconv replaces each occurrence of ISO-8859-1 with ISO8859-1\n > and EUCJP with eucJP since old Solaris didn't know both names and\n > modern platforms handle either name. The text files in t/t3900/\n > were renamed accordingly.\n > \n > It would be nice to use these older names here too, even though I\n > won't be able to test it since svn is not installed on the older\n > platforms I have access to.\n\n t/t9139-git-svn-non-utf8-commitencoding.sh | 6 +++---\n 1 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t9139-git-svn-non-utf8-commitencoding.sh b/t/t9139-git-svn-non-utf8-commitencoding.sh\nindex 2b1db97..f337959 100755\n--- a/t/t9139-git-svn-non-utf8-commitencoding.sh\n+++ b/t/t9139-git-svn-non-utf8-commitencoding.sh\n@@ -8,7 +8,7 @@ test_description='git svn refuses to dcommit non-UTF8 messages'\n \n # ISO-2022-JP can pass for valid UTF-8, so skipping that in this test\n \n-for H in ISO-8859-1 EUCJP\n+for H in ISO8859-1 eucJP\n do\n \ttest_expect_success \"$H setup\" '\n \t\tmkdir $H &&\n@@ -17,7 +17,7 @@ do\n \t'\n done\n \n-for H in ISO-8859-1 EUCJP\n+for H in ISO8859-1 eucJP\n do\n \ttest_expect_success \"$H commit on git side\" '\n \t(\n@@ -33,7 +33,7 @@ do\n \t'\n done\n \n-for H in ISO-8859-1 EUCJP\n+for H in ISO8859-1 eucJP\n do\n \ttest_expect_success \"$H dcommit to svn\" '\n \t(\n-- \nEric Wong\n"}]}