{"thread":{"id":"16076","subject":"Encoding problems using git-svn","startedAt":"2008-10-29T03:14:26Z","lastAt":"2008-11-02T13:45:36Z","messageCount":6,"participants":["James North","Eric Wong","Robin Rosenberg"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"94162","messageId":"8b168cfb0810282014r789ac01dnec51824de1078f0@mail.gmail.com","threadId":"16076","inReplyTo":null,"subject":"Encoding problems using git-svn","fromName":"James North","fromEmail":"tocapicha@gmail.com","sentAt":"2008-10-29T03:14:26Z","receivedAt":"2008-10-29T03:14:26Z","isPatch":false,"sender":{"key":"tocapicha@gmail.com","avatar":null},"body":"Hi,\n\nI'm using git-svn on a system with ISO-8859-1 encoding. The problem is\nwhen I try to use \"git svn dcommit\" to send changes to a remote svn\n(also ISO-8859-1).\n\nSeems like git-svn is sending commit messages with utf-8 (just a\nguessing...) and they look bad on the remote svn log. E.g. \"Ca?\\241a\nde cami?\\243n\"\n\nI have tried using i18n.commitencoding=ISO-8859-1 as suggested by the\nwarning when doing \"git svn dcommit\" but messages still are sent with\nwrong encoding.\n\nI'm mising something?\n\nThanks everyone\n"},{"id":"94260","messageId":"8b168cfb0810292028x225202b9hece862a350440472@mail.gmail.com","threadId":"16076","inReplyTo":"8b168cfb0810282014r789ac01dnec51824de1078f0@mail.gmail.com","subject":"Re: Encoding problems using git-svn","fromName":"James North","fromEmail":"tocapicha@gmail.com","sentAt":"2008-10-30T03:28:23Z","receivedAt":"2008-10-30T03:28:23Z","isPatch":false,"sender":{"key":"tocapicha@gmail.com","avatar":null},"body":"Ok, I made a quick change in git-svn script and seems like is working\nnow in my system with locale set to iso-8859-1.\n\nDunno if this is the right place to post this, but I hope someone\nknowledgeable see this and tells if this would work as a general fix.\n\nThis patch is against 1.6.0.2\n\n--- git-svn     2008-09-15 13:04:46.000000000 +0200\n+++ git-svn.mine        2008-10-30 04:21:09.000000000 +0100\n@@ -43,6 +43,7 @@\n use Getopt::Long qw/:config gnu_getopt no_ignore_case auto_abbrev/;\n use IPC::Open3;\n use Git;\n+use Encode;\n\n BEGIN {\n        # import functions from Git into our packages, en masse\n@@ -1061,6 +1062,7 @@\n                    && !$saw_from) {\n                        $msgbuf .= \"\\n\\nFrom: $author\";\n                }\n+        $msgbuf = encode(\"utf8\", $msgbuf);\n                print $log_fh $msgbuf or croak $!;\n                command_close_pipe($msg_fh, $ctx);\n        }\n\n\nOn Wed, Oct 29, 2008 at 4:14 AM, James North <tocapicha@gmail.com> wrote:\n> Hi,\n>\n> I'm using git-svn on a system with ISO-8859-1 encoding. The problem is\n> when I try to use \"git svn dcommit\" to send changes to a remote svn\n> (also ISO-8859-1).\n>\n> Seems like git-svn is sending commit messages with utf-8 (just a\n> guessing...) and they look bad on the remote svn log. E.g. \"Ca?\\241a\n> de cami?\\243n\"\n>\n> I have tried using i18n.commitencoding=ISO-8859-1 as suggested by the\n> warning when doing \"git svn dcommit\" but messages still are sent with\n> wrong encoding.\n>\n> I'm mising something?\n>\n> Thanks everyone\n>\n"},{"id":"94264","messageId":"20081030074114.GA26926@yp-box.dyndns.org","threadId":"16076","inReplyTo":"8b168cfb0810282014r789ac01dnec51824de1078f0@mail.gmail.com","subject":"Re: Encoding problems using git-svn","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-10-30T07:41:49Z","receivedAt":"2008-10-30T07:41:49Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hi James,\n\nI saw your other patch too late, I had already started working on my\npatch earlier today but got distracted by other things (being at\nGitTogether :) and lacked a stable Internet connection afterwards.\n\nAnyways, here's my version, it handles the case where the user specifies\nthe --edit option to interactively edit the commit message before\ncommitting; and also reencodes the messages when fetching from SVN.\n\nCan you let me know if it works for you?\n\nNote: I'll be in transit tomorrow and may not have time to follow\nup on this until Saturday.\n\n>From 84f003e0c39414ebf27a98de167643e95bed6abb Mon Sep 17 00:00:00 2001\nFrom: Eric Wong <normalperson@yhbt.net>\nDate: Wed, 29 Oct 2008 23:49:26 -0700\nSubject: [PATCH] git-svn: respect i18n.commitencoding config\n\nSVN itself always stores log messages in the repository as\nUTF-8.  git always stores/retrieves everything as raw binary\ndata with no transformations whatsoever.\n\nTo interact with SVN, we need to encode log messages as UTF-8\nbefore sending them to SVN, as SVN cannot do it for us.  When\nretrieving log messages from SVN, we also need to (attempt to)\nreencode the UTF-8 log message back to the user-specified commit\nencoding.\n\nNote, handling i18n.logoutputencoding for \"git svn log\" also\nneeds to be done in a future change.\n\nAlso, this change only deals with the encoding of commit\nmessages and nothing else (path names, blob content, ...).\n\nIn-Reply-To: <8b168cfb0810282014r789ac01dnec51824de1078f0@mail.gmail.com>\nJames North <tocapicha@gmail.com> wrote:\n> Hi,\n>\n> I'm using git-svn on a system with ISO-8859-1 encoding. The problem is\n> when I try to use \"git svn dcommit\" to send changes to a remote svn\n> (also ISO-8859-1).\n>\n> Seems like git-svn is sending commit messages with utf-8 (just a\n> guessing...) and they look bad on the remote svn log. E.g. \"Ca?\\241a\n> de cami?\\243n\"\n>\n> I have tried using i18n.commitencoding=ISO-8859-1 as suggested by the\n> warning when doing \"git svn dcommit\" but messages still are sent with\n> wrong encoding.\n\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n git-svn.perl                           |   24 ++++++++-\n t/t9129-git-svn-i18n-commitencoding.sh |   80 ++++++++++++++++++++++++++++++++\n 2 files changed, 101 insertions(+), 3 deletions(-)\n create mode 100755 t/t9129-git-svn-i18n-commitencoding.sh\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex f90ddac..f24559c 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -1136,9 +1136,19 @@ sub get_commit_entry {\n \t\tsystem($editor, $commit_editmsg);\n \t}\n \trename $commit_editmsg, $commit_msg or croak $!;\n-\topen $log_fh, '<', $commit_msg or croak $!;\n-\t{ local $/; chomp($log_entry{log} = <$log_fh>); }\n-\tclose $log_fh or croak $!;\n+\t{\n+\t\t# SVN requires messages to be UTF-8 when entering the repo\n+\t\tlocal $/;\n+\t\topen $log_fh, '<', $commit_msg or croak $!;\n+\t\tbinmode $log_fh;\n+\t\tchomp($log_entry{log} = <$log_fh>);\n+\n+\t\tif (my $enc = Git::config('i18n.commitencoding')) {\n+\t\t\trequire Encode;\n+\t\t\tEncode::from_to($log_entry{log}, $enc, 'UTF-8');\n+\t\t}\n+\t\tclose $log_fh or croak $!;\n+\t}\n \tunlink $commit_msg;\n \t\\%log_entry;\n }\n@@ -2273,6 +2283,14 @@ sub do_git_commit {\n \t}\n \tdefined(my $pid = open3(my $msg_fh, my $out_fh, '>&STDERR', @exec))\n \t                                                           or croak $!;\n+\tbinmode $msg_fh;\n+\n+\t# we always get UTF-8 from SVN, but we may want our commits in\n+\t# a different encoding.\n+\tif (my $enc = Git::config('i18n.commitencoding')) {\n+\t\trequire Encode;\n+\t\tEncode::from_to($log_entry->{log}, 'UTF-8', $enc);\n+\t}\n \tprint $msg_fh $log_entry->{log} or croak $!;\n \trestore_commit_header_env($old_env);\n \tunless ($self->no_metadata) {\ndiff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh\nnew file mode 100755\nindex 0000000..2848e46\n--- /dev/null\n+++ b/t/t9129-git-svn-i18n-commitencoding.sh\n@@ -0,0 +1,80 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2008 Eric Wong\n+\n+test_description='git svn honors i18n.commitEncoding in config'\n+\n+. ./lib-git-svn.sh\n+\n+compare_git_head_with () {\n+\tnr=`wc -l < \"$1\"`\n+\ta=7\n+\tb=$(($a + $nr - 1))\n+\tgit cat-file commit HEAD | sed -ne \"$a,${b}p\" >current &&\n+\ttest_cmp current \"$1\"\n+}\n+\n+compare_svn_head_with () {\n+\tLC_ALL=en_US.UTF-8 svn log --limit 1 `git svn info --url` | \\\n+\t\tsed -e 1,3d -e \"/^-\\+\\$/d\" >current &&\n+\ttest_cmp current \"$1\"\n+}\n+\n+for H in ISO-8859-1 EUCJP ISO-2022-JP\n+do\n+\ttest_expect_success \"$H setup\" '\n+\t\tmkdir $H &&\n+\t\tsvn import -m \"$H test\" $H \"$svnrepo\"/$H &&\n+\t\tgit svn clone \"$svnrepo\"/$H $H\n+\t'\n+done\n+\n+for H in ISO-8859-1 EUCJP ISO-2022-JP\n+do\n+\ttest_expect_success \"$H commit on git side\" '\n+\t(\n+\t\tcd $H &&\n+\t\tgit config i18n.commitencoding $H &&\n+\t\tgit checkout -b t refs/remotes/git-svn &&\n+\t\techo $H >F &&\n+\t\tgit add F &&\n+\t\tgit commit -a -F \"$TEST_DIRECTORY\"/t3900/$H.txt &&\n+\t\tE=$(git cat-file commit HEAD | sed -ne \"s/^encoding //p\") &&\n+\t\ttest \"z$E\" = \"z$H\"\n+\t\tcompare_git_head_with \"$TEST_DIRECTORY\"/t3900/$H.txt\n+\t)\n+\t'\n+done\n+\n+for H in ISO-8859-1 EUCJP ISO-2022-JP\n+do\n+\ttest_expect_success \"$H dcommit to svn\" '\n+\t(\n+\t\tcd $H &&\n+\t\tgit svn dcommit &&\n+\t\tgit cat-file commit HEAD | grep git-svn-id: &&\n+\t\tE=$(git cat-file commit HEAD | sed -ne \"s/^encoding //p\") &&\n+\t\ttest \"z$E\" = \"z$H\" &&\n+\t\tcompare_git_head_with \"$TEST_DIRECTORY\"/t3900/$H.txt\n+\t)\n+\t'\n+done\n+\n+test_expect_success 'ISO-8859-1 should match UTF-8 in svn' '\n+(\n+\tcd ISO-8859-1 &&\n+\tcompare_svn_head_with \"$TEST_DIRECTORY\"/t3900/1-UTF-8.txt\n+)\n+'\n+\n+for H in EUCJP ISO-2022-JP\n+do\n+\ttest_expect_success '$H should match UTF-8 in svn' '\n+\t(\n+\t\tcd $H &&\n+\t\tcompare_svn_head_with \"$TEST_DIRECTORY\"/t3900/2-UTF-8.txt\n+\t)\n+\t'\n+done\n+\n+test_done\n-- \nEric Wong\n"},{"id":"94302","messageId":"8b168cfb0810300814i53a264c2x997543e145d5e15a@mail.gmail.com","threadId":"16076","inReplyTo":"20081030074114.GA26926@yp-box.dyndns.org","subject":"Re: Encoding problems using git-svn","fromName":"James North","fromEmail":"tocapicha@gmail.com","sentAt":"2008-10-30T15:14:18Z","receivedAt":"2008-10-30T15:14:18Z","isPatch":false,"sender":{"key":"tocapicha@gmail.com","avatar":null},"body":"Hi Eric,\n\nDon't worry about not seeing the patch and thanks for the answer :)\n\nYour patch works great.\n\nMessages appear without problems on \"svn log\" and \"git log\", I haven't\nfound any gotcha that I know of.\n\nThe weird thing is that this problem was not found by anyone before, I\nguessed there should be some people with a setup similar to mine.\n\nThanks again.\n\nOn Thu, Oct 30, 2008 at 8:41 AM, Eric Wong <normalperson@yhbt.net> wrote:\n> Hi James,\n>\n> I saw your other patch too late, I had already started working on my\n> patch earlier today but got distracted by other things (being at\n> GitTogether :) and lacked a stable Internet connection afterwards.\n>\n> Anyways, here's my version, it handles the case where the user specifies\n> the --edit option to interactively edit the commit message before\n> committing; and also reencodes the messages when fetching from SVN.\n>\n> Can you let me know if it works for you?\n>\n> Note: I'll be in transit tomorrow and may not have time to follow\n> up on this until Saturday.\n>\n> From 84f003e0c39414ebf27a98de167643e95bed6abb Mon Sep 17 00:00:00 2001\n> From: Eric Wong <normalperson@yhbt.net>\n> Date: Wed, 29 Oct 2008 23:49:26 -0700\n> Subject: [PATCH] git-svn: respect i18n.commitencoding config\n>\n> SVN itself always stores log messages in the repository as\n> UTF-8.  git always stores/retrieves everything as raw binary\n> data with no transformations whatsoever.\n>\n> To interact with SVN, we need to encode log messages as UTF-8\n> before sending them to SVN, as SVN cannot do it for us.  When\n> retrieving log messages from SVN, we also need to (attempt to)\n> reencode the UTF-8 log message back to the user-specified commit\n> encoding.\n>\n> Note, handling i18n.logoutputencoding for \"git svn log\" also\n> needs to be done in a future change.\n>\n> Also, this change only deals with the encoding of commit\n> messages and nothing else (path names, blob content, ...).\n>\n> In-Reply-To: <8b168cfb0810282014r789ac01dnec51824de1078f0@mail.gmail.com>\n> James North <tocapicha@gmail.com> wrote:\n>> Hi,\n>>\n>> I'm using git-svn on a system with ISO-8859-1 encoding. The problem is\n>> when I try to use \"git svn dcommit\" to send changes to a remote svn\n>> (also ISO-8859-1).\n>>\n>> Seems like git-svn is sending commit messages with utf-8 (just a\n>> guessing...) and they look bad on the remote svn log. E.g. \"Ca?\\241a\n>> de cami?\\243n\"\n>>\n>> I have tried using i18n.commitencoding=ISO-8859-1 as suggested by the\n>> warning when doing \"git svn dcommit\" but messages still are sent with\n>> wrong encoding.\n>\n> Signed-off-by: Eric Wong <normalperson@yhbt.net>\n> ---\n>  git-svn.perl                           |   24 ++++++++-\n>  t/t9129-git-svn-i18n-commitencoding.sh |   80 ++++++++++++++++++++++++++++++++\n>  2 files changed, 101 insertions(+), 3 deletions(-)\n>  create mode 100755 t/t9129-git-svn-i18n-commitencoding.sh\n>\n> diff --git a/git-svn.perl b/git-svn.perl\n> index f90ddac..f24559c 100755\n> --- a/git-svn.perl\n> +++ b/git-svn.perl\n> @@ -1136,9 +1136,19 @@ sub get_commit_entry {\n>                system($editor, $commit_editmsg);\n>        }\n>        rename $commit_editmsg, $commit_msg or croak $!;\n> -       open $log_fh, '<', $commit_msg or croak $!;\n> -       { local $/; chomp($log_entry{log} = <$log_fh>); }\n> -       close $log_fh or croak $!;\n> +       {\n> +               # SVN requires messages to be UTF-8 when entering the repo\n> +               local $/;\n> +               open $log_fh, '<', $commit_msg or croak $!;\n> +               binmode $log_fh;\n> +               chomp($log_entry{log} = <$log_fh>);\n> +\n> +               if (my $enc = Git::config('i18n.commitencoding')) {\n> +                       require Encode;\n> +                       Encode::from_to($log_entry{log}, $enc, 'UTF-8');\n> +               }\n> +               close $log_fh or croak $!;\n> +       }\n>        unlink $commit_msg;\n>        \\%log_entry;\n>  }\n> @@ -2273,6 +2283,14 @@ sub do_git_commit {\n>        }\n>        defined(my $pid = open3(my $msg_fh, my $out_fh, '>&STDERR', @exec))\n>                                                                   or croak $!;\n> +       binmode $msg_fh;\n> +\n> +       # we always get UTF-8 from SVN, but we may want our commits in\n> +       # a different encoding.\n> +       if (my $enc = Git::config('i18n.commitencoding')) {\n> +               require Encode;\n> +               Encode::from_to($log_entry->{log}, 'UTF-8', $enc);\n> +       }\n>        print $msg_fh $log_entry->{log} or croak $!;\n>        restore_commit_header_env($old_env);\n>        unless ($self->no_metadata) {\n> diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh\n> new file mode 100755\n> index 0000000..2848e46\n> --- /dev/null\n> +++ b/t/t9129-git-svn-i18n-commitencoding.sh\n> @@ -0,0 +1,80 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2008 Eric Wong\n> +\n> +test_description='git svn honors i18n.commitEncoding in config'\n> +\n> +. ./lib-git-svn.sh\n> +\n> +compare_git_head_with () {\n> +       nr=`wc -l < \"$1\"`\n> +       a=7\n> +       b=$(($a + $nr - 1))\n> +       git cat-file commit HEAD | sed -ne \"$a,${b}p\" >current &&\n> +       test_cmp current \"$1\"\n> +}\n> +\n> +compare_svn_head_with () {\n> +       LC_ALL=en_US.UTF-8 svn log --limit 1 `git svn info --url` | \\\n> +               sed -e 1,3d -e \"/^-\\+\\$/d\" >current &&\n> +       test_cmp current \"$1\"\n> +}\n> +\n> +for H in ISO-8859-1 EUCJP ISO-2022-JP\n> +do\n> +       test_expect_success \"$H setup\" '\n> +               mkdir $H &&\n> +               svn import -m \"$H test\" $H \"$svnrepo\"/$H &&\n> +               git svn clone \"$svnrepo\"/$H $H\n> +       '\n> +done\n> +\n> +for H in ISO-8859-1 EUCJP ISO-2022-JP\n> +do\n> +       test_expect_success \"$H commit on git side\" '\n> +       (\n> +               cd $H &&\n> +               git config i18n.commitencoding $H &&\n> +               git checkout -b t refs/remotes/git-svn &&\n> +               echo $H >F &&\n> +               git add F &&\n> +               git commit -a -F \"$TEST_DIRECTORY\"/t3900/$H.txt &&\n> +               E=$(git cat-file commit HEAD | sed -ne \"s/^encoding //p\") &&\n> +               test \"z$E\" = \"z$H\"\n> +               compare_git_head_with \"$TEST_DIRECTORY\"/t3900/$H.txt\n> +       )\n> +       '\n> +done\n> +\n> +for H in ISO-8859-1 EUCJP ISO-2022-JP\n> +do\n> +       test_expect_success \"$H dcommit to svn\" '\n> +       (\n> +               cd $H &&\n> +               git svn dcommit &&\n> +               git cat-file commit HEAD | grep git-svn-id: &&\n> +               E=$(git cat-file commit HEAD | sed -ne \"s/^encoding //p\") &&\n> +               test \"z$E\" = \"z$H\" &&\n> +               compare_git_head_with \"$TEST_DIRECTORY\"/t3900/$H.txt\n> +       )\n> +       '\n> +done\n> +\n> +test_expect_success 'ISO-8859-1 should match UTF-8 in svn' '\n> +(\n> +       cd ISO-8859-1 &&\n> +       compare_svn_head_with \"$TEST_DIRECTORY\"/t3900/1-UTF-8.txt\n> +)\n> +'\n> +\n> +for H in EUCJP ISO-2022-JP\n> +do\n> +       test_expect_success '$H should match UTF-8 in svn' '\n> +       (\n> +               cd $H &&\n> +               compare_svn_head_with \"$TEST_DIRECTORY\"/t3900/2-UTF-8.txt\n> +       )\n> +       '\n> +done\n> +\n> +test_done\n> --\n> Eric Wong\n>\n"},{"id":"94651","messageId":"20081102094845.GB16003@untitled","threadId":"16076","inReplyTo":"8b168cfb0810300814i53a264c2x997543e145d5e15a@mail.gmail.com","subject":"Re: Encoding problems using git-svn","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-11-02T09:48:45Z","receivedAt":"2008-11-02T09:48:45Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"James North <tocapicha@gmail.com> wrote:\n> Hi Eric,\n> \n> Don't worry about not seeing the patch and thanks for the answer :)\n> \n> Your patch works great.\n> \n> Messages appear without problems on \"svn log\" and \"git log\", I haven't\n> found any gotcha that I know of.\n\nThanks for the confirmation.\n\n> The weird thing is that this problem was not found by anyone before, I\n> guessed there should be some people with a setup similar to mine.\n\nSqueaky wheel gets the grease :)\n\nHonestly, I think most folks have just moved onto UTF-8 entirely and\nleft legacy encodings behind.  Especially people using modern tools like\ngit (along with SVN enforcing UTF-8 at the repository/protocol level).\n\n\nJunio:\n\nI've pushed the following out to git://git.bogomips.org/git-svn.git:\n\nEric Wong (2):\n      git-svn: don't escape tilde ('~') for http(s) URLs\n      git-svn: respect i18n.commitencoding config\n\nI'll try to get around to the more robust escaping checks\nand splitting out the monolithic git-svn.perl source next\nweek.\n\n-- \nEric Wong\n"},{"id":"94658","messageId":"200811021445.37149.robin.rosenberg@dewire.com","threadId":"16076","inReplyTo":"20081102094845.GB16003@untitled","subject":"Re: Encoding problems using git-svn","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-11-02T13:45:36Z","receivedAt":"2008-11-02T13:45:36Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"On söndag 02 november 2008 10:48 Eric Wong wrote:\n> James North <tocapicha@gmail.com> wrote:\n> > Hi Eric,\n> >\n> > Don't worry about not seeing the patch and thanks for the answer :)\n> >\n> > Your patch works great.\n> >\n> > Messages appear without problems on \"svn log\" and \"git log\", I haven't\n> > found any gotcha that I know of.\n>\n> Thanks for the confirmation.\n>\n> > The weird thing is that this problem was not found by anyone before, I\n> > guessed there should be some people with a setup similar to mine.\n>\n> Squeaky wheel gets the grease :)\n>\n> Honestly, I think most folks have just moved onto UTF-8 entirely and\n> left legacy encodings behind.  Especially people using modern tools like\n> git (along with SVN enforcing UTF-8 at the repository/protocol level).\n\n\"Most\" people don't have a legacy encoding problem, but some of us do and \ntools that help with migration by enforcing UTF-8 internally help. SVN is such\nan example, though not very helpful as an SCM. That way we can still use \nlegacy encodings for old stupid tools until we can move to an all UTF-8 world. \nWe're not there yet, but in a few years hopefully. That's when it's sad that \nthe git command line for example still enforce the legacy encoding. Some \nGUI's, like git gui, jgit and probably a few others help by recoding when \nnecessary.\n\n-- robiin\n"}]}