{"thread":{"id":"12897","subject":"[ANNOUNCE] GIT 1.5.5-rc2","startedAt":"2008-03-28T06:30:27Z","lastAt":"2008-05-21T19:47:34Z","messageCount":32,"participants":["Junio C Hamano","Jeff King","Robin Rosenberg","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"73242","messageId":"7viqz72we4.fsf@gitster.siamese.dyndns.org","threadId":"12897","inReplyTo":null,"subject":"[ANNOUNCE] GIT 1.5.5-rc2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-03-28T06:30:27Z","receivedAt":"2008-03-28T06:30:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"GIT 1.5.5-rc2 was tagged tonight, and it is available from the usual\nplaces.\n\n http://www.kernel.org/pub/software/scm/git/\n\n git-1.5.5.rc2.tar.{gz,bz2}\t\t\t(tarball)\n git-htmldocs-1.5.5.rc2.tar.{gz,bz2}\t\t(preformatted docs)\n git-manpages-1.5.5.rc2.tar.{gz,bz2}\t\t(preformatted docs)\n testing/git-*-1.5.5.rc2-1.$arch.rpm (RPM)\n\nThe draft release notes as of tonight follows.\n\nGIT v1.5.5 Release Notes\n========================\n\nUpdates since v1.5.4\n--------------------\n\n(subsystems)\n\n * Comes with git-gui 0.9.3.\n\n(portability)\n\n * We shouldn't ask for BSD group ownership semantics by setting g+s bit\n on directories on older BSD systems that refuses chmod() by non root\n users. BSD semantics is the default there anyway.\n\n * Bunch of portability improvement patches coming from an effort to port\n to Solaris has been applied.\n\n(performance)\n\n * On platforms with suboptimal qsort(3) implementation, there\n is an option to use more reasonable substitute we ship with\n our software.\n\n * New configuration variable \"pack.packsizelimit\" can be used\n in place of command line option --max-pack-size.\n\n * \"git fetch\" over the native git protocol used to make a\n connection to find out the set of current remote refs and\n another to actually download the pack data. We now use only\n one connection for these tasks.\n\n * \"git commit\" does not run lstat(2) more than necessary\n anymore.\n\n(usability, bells and whistles)\n\n * Bash completion script (in contrib) are aware of more commands and\n options.\n\n * You can be warned when core.autocrlf conversion is applied in\n such a way that results in an irreversible conversion.\n\n * A catch-all \"color.ui\" configuration variable can be used to\n enable coloring of all color-capable commands, instead of\n individual ones such as \"color.status\" and \"color.branch\".\n\n * The commands refused to take absolute pathnames where they\n require pathnames relative to the work tree or the current\n subdirectory. They now can take absolute pathnames in such a\n case as long as the pathnames do not refer outside of the\n work tree. E.g. \"git add $(pwd)/foo\" now works.\n\n * Error messages used to be sent to stderr, only to get hidden,\n when $PAGER was in use. They now are sent to stdout along\n with the command output to be shown in the $PAGER.\n\n * A pattern \"foo/\" in .gitignore file now matches a directory\n \"foo\". Pattern \"foo\" also matches as before.\n\n * bash completion's prompt helper function can talk about\n operation in-progress (e.g. merge, rebase, etc.).\n\n * Configuration variables \"url..insteadof = \" can be\n used to tell \"git-fetch\" and \"git-push\" to use different URL than what\n is given from the command line.\n\n * \"git add -i\" behaves better even before you make an initial commit.\n\n * \"git am\" refused to run from a subdirectory without a good reason.\n\n * After \"git apply --whitespace=fix\" fixes whitespace errors in a patch,\n a line before the fix can appear as a context or preimage line in a\n later patch, causing the patch not to apply. The command now knows to\n see through whitespace fixes done to context lines to successfully\n apply such a patch series.\n\n * \"git branch\" (and \"git checkout -b\") to branch from a local branch can\n optionally set \"branch..merge\" to mark the new branch to build on\n the other local branch, when \"branch.autosetupmerge\" is set to\n \"always\", or when passing the command line option \"--track\" (this option\n was ignored when branching from local branches). By default, this does\n not happen when branching from a local branch.\n\n * \"git checkout\" to switch to a branch that has \"branch..merge\" set\n (i.e. marked to build on another branch) reports how much the branch\n and the other branch diverged.\n\n * When \"git checkout\" has to update a lot of paths, it used to be silent\n for 4 seconds before it showed any progress report. It is now a bit\n more impatient and starts showing progress report early.\n\n * \"git commit\" learned a new hook \"prepare-commit-msg\" that can\n inspect what is going to be committed and prepare the commit\n log message template to be edited.\n\n * \"git cvsimport\" can now take more than one -M options.\n\n * \"git describe\" learned to limit the tags to be used for\n naming with --match option.\n\n * \"git describe --contains\" now barfs when the named commit\n cannot be described.\n\n * \"git describe --exact-match\" describes only commits that are tagged.\n\n * \"git describe --long\" describes a tagged commit as $tag-0-$sha1,\n instead of just showing the exact tagname.\n\n * \"git describe\" warns when using a tag whose name and path contradict\n with each other.\n\n * \"git diff\" learned \"--relative\" option to limit and output paths\n relative to the current directory when working in a subdirectory.\n\n * \"git diff\" learned \"--dirstat\" option to show birds-eye-summary of\n changes more concisely than \"--diffstat\".\n\n * \"git format-patch\" learned --cover-letter option to generate a cover\n letter template.\n\n * \"git gc\" learned --quiet option.\n\n * \"git gc\" now automatically prunes unreachable objects that are two\n weeks old or older.\n\n * \"git gc --auto\" can be disabled more easily by just setting gc.auto\n to zero. It also tolerates more packfiles by default.\n\n * \"git grep\" now knows \"--name-only\" is a synonym for the \"-l\" option.\n\n * \"git help \" now reports \"'git ' is alias to \",\n instead of saying \"No manual entry for git-\".\n\n * \"git help\" can use different backends to show manual pages and this can\n be configured using \"man.viewer\" configuration.\n\n * \"gitk\" does not restore window position from $HOME/.gitk anymore (it\n still restores the size).\n\n * \"git log --grep=\" learned \"--fixed-strings\" option to look for\n without treating it as a regular expression.\n\n * \"git gui\" learned an auto-spell checking.\n\n * \"git push HEAD\" and \"git push +HEAD\" works as\n expected; they push the current branch (and only the current branch).\n In addition, HEAD can be written as the value of \"remote..push\"\n configuration variable.\n\n * When the configuration variable \"pack.threads\" is set to 0, \"git\n repack\" auto detects the number of CPUs and uses that many threads.\n\n * \"git send-email\" learned to prompt for passwords\n interactively.\n\n * \"git send-email\" learned an easier way to suppress CC\n recipients.\n\n * \"git stash\" learned \"pop\" command, that applies the latest stash and\n removes it from the stash, and \"drop\" command to discard the named\n stash entry.\n\n * \"git submodule\" learned a new subcommand \"summary\" to show the\n symmetric difference between the HEAD version and the work tree version\n of the submodule commits.\n\n * Various \"git cvsimport\", \"git cvsexportcommit\", \"git svn\" and\n \"git p4\" improvements.\n\n(internal)\n\n * Duplicated code between git-help and git-instaweb that\n launches user's preferred browser has been refactored.\n\n * It is now easier to write test scripts that records known\n breakages.\n\n * \"git checkout\" is rewritten in C.\n\n * \"git remote\" is rewritten in C.\n\n * Two conflict hunks that are separated by a very short span of common\n lines are now coalesced into one larger hunk, to make the result easier\n to read.\n\n * Run-command API's use of file descriptors is documented clearer and\n is more consistent now.\n\n * diff output can be sent to FILE * that is different from stdout. This\n will help reimplementing more things in C.\n\nFixes since v1.5.4\n------------------\n\nAll of the fixes in v1.5.4 maintenance series are included in\nthis release, unless otherwise noted.\n\n * \"git-http-push\" did not allow deletion of remote ref with the usual\n \"push :\" syntax.\n\n * \"git-rebase --abort\" did not go back to the right location if\n \"git-reset\" was run during the \"git-rebase\" session.\n\n * \"git imap-send\" without setting imap.host did not error out but\n segfaulted.\n"},{"id":"73285","messageId":"20080328181345.GB8299@coredump.intra.peff.net","threadId":"12897","inReplyTo":"7viqz72we4.fsf@gitster.siamese.dyndns.org","subject":"Re: [ANNOUNCE] GIT 1.5.5-rc2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-28T18:13:45Z","receivedAt":"2008-03-28T18:13:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 27, 2008 at 11:30:27PM -0700, Junio C Hamano wrote:\n\n> GIT 1.5.5-rc2 was tagged tonight, and it is available from the usual\n> places.\n\nI never got a response to my patches to fix encoding issues in\n\"send-email --compose\". It _is_ a bugfix, but I don't know if it is\n1.5.5-worthy. Forgotten (and I should resubmit now), or should I wait\nuntil after the release?\n\n-Peff\n"},{"id":"73289","messageId":"7vtziq1rwa.fsf@gitster.siamese.dyndns.org","threadId":"12897","inReplyTo":"20080328181345.GB8299@coredump.intra.peff.net","subject":"Re: [ANNOUNCE] GIT 1.5.5-rc2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-03-28T21:05:09Z","receivedAt":"2008-03-28T21:05:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Mar 27, 2008 at 11:30:27PM -0700, Junio C Hamano wrote:\n>\n>> GIT 1.5.5-rc2 was tagged tonight, and it is available from the usual\n>> places.\n>\n> I never got a response to my patches to fix encoding issues in\n> \"send-email --compose\". It _is_ a bugfix, but I don't know if it is\n> 1.5.5-worthy. Forgotten (and I should resubmit now), or should I wait\n> until after the release?\n\nI was getting the impression that it was still in \"ah, but this is\nbetter\", \"you are right, but how about doing this\", stage and was hoping\nthat \"ok, based on the discussion here is the final one\" will come soon.\n"},{"id":"73291","messageId":"20080328212340.GC9131@coredump.intra.peff.net","threadId":"12897","inReplyTo":"7vtziq1rwa.fsf@gitster.siamese.dyndns.org","subject":"Re: [ANNOUNCE] GIT 1.5.5-rc2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-28T21:23:40Z","receivedAt":"2008-03-28T21:23:40Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 28, 2008 at 02:05:09PM -0700, Junio C Hamano wrote:\n\n> > I never got a response to my patches to fix encoding issues in\n> > \"send-email --compose\". It _is_ a bugfix, but I don't know if it is\n> > 1.5.5-worthy. Forgotten (and I should resubmit now), or should I wait\n> > until after the release?\n> \n> I was getting the impression that it was still in \"ah, but this is\n> better\", \"you are right, but how about doing this\", stage and was hoping\n> that \"ok, based on the discussion here is the final one\" will come soon.\n\nAh. I think the current status is \"here are two patches that work, but\nwill always assume utf-8 encoding\" which I think is not unreasonable as\na bugfix. A nice feature would be to allow setting the encoding, but:\n\n - I think that is a feature, and one that nobody has expressed an\n interest in. In fact, the little rfc2047 encoding already being done\n in send-email blindly assumed utf-8.\n - if that feature is going to be done, I think some thought would have\n to go into how encodings should be specified so we don't end up with\n too many (or too few) places where you have to specify the encoding\n (IOW, I think that send-email.compose-encoding is probably too\n specific, but reusing an existing encoding variable is not quite\n right).\n - As a non-user of send-email, a rare user of encodings at all, and an\n always user of utf-8, I'm not too interested in such a feature, nor\n would I feel comfortable speaking on behalf of users who _would_ use\n such a feature.\n\nSo I think it is worth taking the patches for 1.5.5 as they are a strict\nimprovement over the old behavior (the only reason they would not be is\nif somebody used a mail pipeline that assumed non-MIME stuff was in some\nrandom charset instead of us-ascii, and by setting utf-8 we are now\nconfusing that pipeline; it seems unlikely to me, and it violates the\nstandards).\n\nOn top of which I think they are a fine stepping stone to selecting the\nencoding (IOW, if I _were_ going to do such a feature, I think I would\nstill submit those two patches as-is, and add configurability as a third\npatch anyway).\n\nMy only real concern is that they break something unrelated, as we are\nlate in the -rc cycle.\n\n-Peff\n"},{"id":"73292","messageId":"20080328212700.GA9529@coredump.intra.peff.net","threadId":"12897","inReplyTo":"20080328212340.GC9131@coredump.intra.peff.net","subject":"Re: [ANNOUNCE] GIT 1.5.5-rc2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-28T21:27:01Z","receivedAt":"2008-03-28T21:27:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 28, 2008 at 05:23:40PM -0400, Jeff King wrote:\n\n> > I was getting the impression that it was still in \"ah, but this is\n> > better\", \"you are right, but how about doing this\", stage and was hoping\n> > that \"ok, based on the discussion here is the final one\" will come soon.\n> \n> Ah. I think the current status is \"here are two patches that work, but\n> will always assume utf-8 encoding\" which I think is not unreasonable as\n> a bugfix. A nice feature would be to allow setting the encoding, but:\n\nAfter sending my lengthy response, I realized that you might have\nactually been talking about the minor fixups that happened, and missed\nthe \"correct\" 2/2 which was sent later in the thread. So following this\nare the most up-to-date versions. 1/1 is the same as the original, 2/2\ndoes Teemu's \"let user set subject in editor\" suggestion, plus the\nfollow-on syntax fixup.\n\n-Peff\n"},{"id":"73293","messageId":"20080328212833.GA9656@coredump.intra.peff.net","threadId":"12897","inReplyTo":"20080328212700.GA9529@coredump.intra.peff.net","subject":"[PATCH 1/2] send-email: specify content-type of --compose body","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-28T21:28:33Z","receivedAt":"2008-03-28T21:28:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"If the compose message contains non-ascii characters, then\nwe assume it is in utf-8 and include the appropriate MIME\nheaders. If the user has already included a MIME-Version\nheader, then we assume they know what they are doing and\ndon't add any headers.\n\nSigned-off-by: Jeff King \n---\n git-send-email.perl | 24 ++++++++++++++++++++++++\n t/t9001-send-email.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 68 insertions(+), 0 deletions(-)\n\ndiff --git a/git-send-email.perl b/git-send-email.perl\nindex 9e568bf..7c4f06c 100755\n--- a/git-send-email.perl\n+++ b/git-send-email.perl\n@@ -520,8 +520,22 @@ EOT\n \topen(C,\"<\",$compose_filename)\n \t\tor die \"Failed to open $compose_filename : \" . $!;\n \n+\tmy $need_8bit_cte = file_has_nonascii($compose_filename);\n+\tmy $in_body = 0;\n \twhile() {\n \t\tnext if m/^GIT: /;\n+\t\tif (!$in_body && /^\\n$/) {\n+\t\t\t$in_body = 1;\n+\t\t\tif ($need_8bit_cte) {\n+\t\t\t\tprint C2 \"MIME-Version: 1.0\\n\",\n+\t\t\t\t\t \"Content-Type: text/plain; \",\n+\t\t\t\t\t \"charset=utf-8\\n\",\n+\t\t\t\t\t \"Content-Transfer-Encoding: 8bit\\n\";\n+\t\t\t}\n+\t\t}\n+\t\tif (!$in_body && /^MIME-Version:/i) {\n+\t\t\t$need_8bit_cte = 0;\n+\t\t}\n \t\tprint C2 $_;\n \t}\n \tclose(C);\n@@ -958,3 +972,13 @@ sub validate_patch {\n \t}\n \treturn undef;\n }\n+\n+sub file_has_nonascii {\n+\tmy $fn = shift;\n+\topen(my $fh, '<', $fn)\n+\t\tor die \"unable to open $fn: $!\\n\";\n+\twhile (my $line = <$fh>) {\n+\t\treturn 1 if $line =~ /[^[:ascii:]]/;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh\nindex c0973b4..e222c49 100755\n--- a/t/t9001-send-email.sh\n+++ b/t/t9001-send-email.sh\n@@ -166,4 +166,48 @@ test_expect_success 'second message is patch' '\n \tgrep \"Subject:.*Second\" msgtxt2\n '\n \n+test_expect_success '--compose adds MIME for utf8 body' '\n+\tclean_fake_sendmail &&\n+\t(echo \"#!/bin/sh\" &&\n+\t echo \"echo utf8 body: àéìöú >>\\$1\"\n+\t) >fake-editor-utf8 &&\n+\tchmod +x fake-editor-utf8 &&\n+\techo y | \\\n+\t GIT_EDITOR=$(pwd)/fake-editor-utf8 \\\n+\t GIT_SEND_EMAIL_NOTTY=1 \\\n+\t git send-email \\\n+\t --compose --subject foo \\\n+\t --from=\"Example \" \\\n+\t --to=nobody@example.com \\\n+\t --smtp-server=\"$(pwd)/fake.sendmail\" \\\n+\t $patches &&\n+\tgrep \"^utf8 body\" msgtxt1 &&\n+\tgrep \"^Content-Type: text/plain; charset=utf-8\" msgtxt1\n+'\n+\n+test_expect_success '--compose respects user mime type' '\n+\tclean_fake_sendmail &&\n+\t(echo \"#!/bin/sh\" &&\n+\t echo \"(echo MIME-Version: 1.0\"\n+\t echo \" echo Content-Type: text/plain\\\\; charset=iso-8859-1\"\n+\t echo \" echo Content-Transfer-Encoding: 8bit\"\n+\t echo \" echo Subject: foo\"\n+\t echo \" echo \"\n+\t echo \" echo utf8 body: àéìöú) >\\$1\"\n+\t) >fake-editor-utf8-mime &&\n+\tchmod +x fake-editor-utf8-mime &&\n+\techo y | \\\n+\t GIT_EDITOR=$(pwd)/fake-editor-utf8-mime \\\n+\t GIT_SEND_EMAIL_NOTTY=1 \\\n+\t git send-email \\\n+\t --compose --subject foo \\\n+\t --from=\"Example \" \\\n+\t --to=nobody@example.com \\\n+\t --smtp-server=\"$(pwd)/fake.sendmail\" \\\n+\t $patches &&\n+\tgrep \"^utf8 body\" msgtxt1 &&\n+\tgrep \"^Content-Type: text/plain; charset=iso-8859-1\" msgtxt1 &&\n+\t! grep \"^Content-Type: text/plain; charset=utf-8\" msgtxt1\n+'\n+\n test_done\n-- \n1.5.5.rc1.141.g50ecd.dirty\n"},{"id":"73294","messageId":"20080328212900.GB9656@coredump.intra.peff.net","threadId":"12897","inReplyTo":"20080328212700.GA9529@coredump.intra.peff.net","subject":"[PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-28T21:29:01Z","receivedAt":"2008-03-28T21:29:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"We always use 'utf-8' as the encoding, since we currently\nhave no way of getting the information from the user.\n\nThis also refactors the quoting of recipient names, since\nboth processes can share the rfc2047 quoting code.\n\nSigned-off-by: Jeff King \n---\n git-send-email.perl | 19 +++++++++++++++++--\n t/t9001-send-email.sh | 15 +++++++++++++++\n 2 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/git-send-email.perl b/git-send-email.perl\nindex 7c4f06c..d0f9d4a 100755\n--- a/git-send-email.perl\n+++ b/git-send-email.perl\n@@ -536,6 +536,14 @@ EOT\n \t\tif (!$in_body && /^MIME-Version:/i) {\n \t\t\t$need_8bit_cte = 0;\n \t\t}\n+\t\tif (!$in_body && /^Subject: ?(.*)/i) {\n+\t\t\tmy $subject = $1;\n+\t\t\t$_ = \"Subject: \" .\n+\t\t\t\t($subject =~ /[^[:ascii:]]/ ?\n+\t\t\t\t quote_rfc2047($subject) :\n+\t\t\t\t $subject) .\n+\t\t\t\t\"\\n\";\n+\t\t}\n \t\tprint C2 $_;\n \t}\n \tclose(C);\n@@ -626,6 +634,14 @@ sub unquote_rfc2047 {\n \treturn wantarray ? ($_, $encoding) : $_;\n }\n \n+sub quote_rfc2047 {\n+\tlocal $_ = shift;\n+\tmy $encoding = shift || 'utf-8';\n+\ts/([^-a-zA-Z0-9!*+\\/])/sprintf(\"=%02X\", ord($1))/eg;\n+\ts/(.*)/=\\?$encoding\\?q\\?$1\\?=/;\n+\treturn $_;\n+}\n+\n # use the simplest quoting being able to handle the recipient\n sub sanitize_address\n {\n@@ -643,8 +659,7 @@ sub sanitize_address\n \n \t# rfc2047 is needed if a non-ascii char is included\n \tif ($recipient_name =~ /[^[:ascii:]]/) {\n-\t\t$recipient_name =~ s/([^-a-zA-Z0-9!*+\\/])/sprintf(\"=%02X\", ord($1))/eg;\n-\t\t$recipient_name =~ s/(.*)/=\\?utf-8\\?q\\?$1\\?=/;\n+\t\t$recipient_name = quote_rfc2047($recipient_name);\n \t}\n \n \t# double quotes are needed if specials or CTLs are included\ndiff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh\nindex e222c49..a4bcd28 100755\n--- a/t/t9001-send-email.sh\n+++ b/t/t9001-send-email.sh\n@@ -210,4 +210,19 @@ test_expect_success '--compose respects user mime type' '\n \t! grep \"^Content-Type: text/plain; charset=utf-8\" msgtxt1\n '\n \n+test_expect_success '--compose adds MIME for utf8 subject' '\n+\tclean_fake_sendmail &&\n+\techo y | \\\n+\t GIT_EDITOR=$(pwd)/fake-editor \\\n+\t GIT_SEND_EMAIL_NOTTY=1 \\\n+\t git send-email \\\n+\t --compose --subject utf8-sübjëct \\\n+\t --from=\"Example \" \\\n+\t --to=nobody@example.com \\\n+\t --smtp-server=\"$(pwd)/fake.sendmail\" \\\n+\t $patches &&\n+\tgrep \"^fake edit\" msgtxt1 &&\n+\tgrep \"^Subject: =?utf-8?q?utf8-s=C3=BCbj=C3=ABct?=\" msgtxt1\n+'\n+\n test_done\n-- \n1.5.5.rc1.141.g50ecd.dirty\n"},{"id":"73301","messageId":"200803290819.07280.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080328212900.GB9656@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T07:19:07Z","receivedAt":"2008-03-29T07:19:07Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Friday 28 March 2008 22.29.01 skrev Jeff King:\n> We always use 'utf-8' as the encoding, since we currently\n> have no way of getting the information from the user.\n\nDon't set encoding to UTF-8 unless it actually looks like UTF-8.\n\n-- robin\n"},{"id":"73302","messageId":"20080329072203.GA17625@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803290819.07280.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T07:22:03Z","receivedAt":"2008-03-29T07:22:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 08:19:07AM +0100, Robin Rosenberg wrote:\n\n> Den Friday 28 March 2008 22.29.01 skrev Jeff King:\n> > We always use 'utf-8' as the encoding, since we currently\n> > have no way of getting the information from the user.\n> \n> Don't set encoding to UTF-8 unless it actually looks like UTF-8.\n\nOK. Do you have an example function that guesses with high probability\nwhether a string is utf-8? If there are non-ascii characters but we\n_don't_ guess utf-8, what should we do?\n\n-Peff\n"},{"id":"73305","messageId":"200803290941.54091.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329072203.GA17625@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T08:41:53Z","receivedAt":"2008-03-29T08:41:53Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 08.22.03 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 08:19:07AM +0100, Robin Rosenberg wrote:\n> > Den Friday 28 March 2008 22.29.01 skrev Jeff King:\n> > > We always use 'utf-8' as the encoding, since we currently\n> > > have no way of getting the information from the user.\n> >\n> > Don't set encoding to UTF-8 unless it actually looks like UTF-8.\n>\n> OK. Do you have an example function that guesses with high probability\n> whether a string is utf-8? If there are non-ascii characters but we\n> _don't_ guess utf-8, what should we do?\n\nAny test for valid UTF-8 will do that with a very high probability. The\nperl UTF-8 \"api\" is a mess. I couldn't find such a routine!?. Calling \ndecode/encode and see if you get the original string works, but that is too\nclumsy, IMHO.\n\n-- robin\n"},{"id":"73306","messageId":"200803290944.55273.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329072203.GA17625@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T08:44:55Z","receivedAt":"2008-03-29T08:44:55Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 08.22.03 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 08:19:07AM +0100, Robin Rosenberg wrote:\n> > Den Friday 28 March 2008 22.29.01 skrev Jeff King:\n> > > We always use 'utf-8' as the encoding, since we currently\n> > > have no way of getting the information from the user.\n> >\n> > Don't set encoding to UTF-8 unless it actually looks like UTF-8.\n>\n> OK. Do you have an example function that guesses with high probability\n> whether a string is utf-8? If there are non-ascii characters but we\n> _don't_ guess utf-8, what should we do?\n\nI guess the best bet is to assume the locale. Btw, is the encoding header\nfrom the commit (when present) completely lost? (not that it can be trusted\nanyway).\n\n-- robin\n"},{"id":"73308","messageId":"20080329084947.GB19200@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803290941.54091.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T08:49:48Z","receivedAt":"2008-03-29T08:49:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 09:41:53AM +0100, Robin Rosenberg wrote:\n\n> > OK. Do you have an example function that guesses with high probability\n> > whether a string is utf-8? If there are non-ascii characters but we\n> > _don't_ guess utf-8, what should we do?\n> \n> Any test for valid UTF-8 will do that with a very high probability. The\n> perl UTF-8 \"api\" is a mess. I couldn't find such a routine!?. Calling \n> decode/encode and see if you get the original string works, but that is too\n> clumsy, IMHO.\n\nDoes that work? I would think you would have to compare the normalized\nversions of each string, since decode(encode($x)) is not, AIUI,\nguaranteed to produce $x.\n\n-Peff\n"},{"id":"73309","messageId":"20080329085304.GC19200@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803290944.55273.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T08:53:04Z","receivedAt":"2008-03-29T08:53:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 09:44:55AM +0100, Robin Rosenberg wrote:\n\n> > OK. Do you have an example function that guesses with high probability\n> > whether a string is utf-8? If there are non-ascii characters but we\n> > _don't_ guess utf-8, what should we do?\n> \n> I guess the best bet is to assume the locale. Btw, is the encoding header\n> from the commit (when present) completely lost? (not that it can be trusted\n> anyway).\n\nWhat do you mean by \"assume the locale\"? Is there a portable way to say\n\"this is the encoding of the locale the user has chosen?\" On my system I\nset LANG=en_US, and behind-the-scenes magic chooses utf-8 versus\niso8859-1.\n\nAnd there is no encoding header for the commit; the point of this patch\nis to handle the \"cover letter\" message created by \"send-email\n--compose\" (we should already be doing the right thing for the patch\nemails, since the commit encoding is output by format-patch in a\ncontent-type header before we even get to send-email).\n\n-Peff\n"},{"id":"73310","messageId":"200803291002.43768.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329084947.GB19200@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T09:02:43Z","receivedAt":"2008-03-29T09:02:43Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 09.49.48 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 09:41:53AM +0100, Robin Rosenberg wrote:\n> > > OK. Do you have an example function that guesses with high probability\n> > > whether a string is utf-8? If there are non-ascii characters but we\n> > > _don't_ guess utf-8, what should we do?\n> >\n> > Any test for valid UTF-8 will do that with a very high probability. The\n> > perl UTF-8 \"api\" is a mess. I couldn't find such a routine!?. Calling\n> > decode/encode and see if you get the original string works, but that is\n> > too clumsy, IMHO.\n>\n> Does that work? I would think you would have to compare the normalized\n> versions of each string, since decode(encode($x)) is not, AIUI,\n> guaranteed to produce $x.\n\nI don't claim to understand it either. Hopefully some perl guru will step \nforward and just explain how to do this in perl.\n\nMy proof is entirely empirical. What happens is that attempting to decode a \nnon-UTF-8 string will put a unicode surrogate pair into the (now Unicode) \nstring and encoding will just encode the surrogate pair into UTF-8 and not \nthe original. As a result, the encode(decode($x)) eq $x *only* if $x is a\nvalid UTF-8 octet sequence. Why would you not get the original back if\nyou start with valid UTF-8?\n\n-- robin\n"},{"id":"73311","messageId":"20080329091145.GA19501@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803291002.43768.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T09:11:45Z","receivedAt":"2008-03-29T09:11:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 10:02:43AM +0100, Robin Rosenberg wrote:\n\n> My proof is entirely empirical. What happens is that attempting to decode a \n> non-UTF-8 string will put a unicode surrogate pair into the (now Unicode) \n> string and encoding will just encode the surrogate pair into UTF-8 and not \n> the original. As a result, the encode(decode($x)) eq $x *only* if $x is a\n> valid UTF-8 octet sequence. Why would you not get the original back if\n> you start with valid UTF-8?\n\nBecause some UTF-8 sequences have multiple representations, and that\ninformation may be lost by whatever intermediate form is the result of\ndecode($x). In practice, I don't know if this happens or not.\n\nThough it looks like there is an Encode::is_utf8 function (which is also\nutf8::is_utf8, but only in perl >= 5.8.1). So we could use that, but it\nneeds the utf-8 flag turned on for the string. Maybe utf8::valid is\nactually what we want.\n\nBut there is still a larger question. You have some binary bytes that\nwill go in a subject header. There are non-ascii bytes. There are\nnon-utf8 sequences. What do you do?\n\n-Peff\n"},{"id":"73312","messageId":"200803291038.48847.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329085304.GC19200@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T09:38:48Z","receivedAt":"2008-03-29T09:38:48Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 09.53.04 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 09:44:55AM +0100, Robin Rosenberg wrote:\n> > > OK. Do you have an example function that guesses with high probability\n> > > whether a string is utf-8? If there are non-ascii characters but we\n> > > _don't_ guess utf-8, what should we do?\n> >\n> > I guess the best bet is to assume the locale. Btw, is the encoding header\n> > from the commit (when present) completely lost? (not that it can be\n> > trusted anyway).\n>\n> What do you mean by \"assume the locale\"? Is there a portable way to say\n> \"this is the encoding of the locale the user has chosen?\" On my system I\n> set LANG=en_US, and behind-the-scenes magic chooses utf-8 versus\n> iso8859-1.\n\nThe environment variables are only part of the story. There is a langinfo API \nfor this. See I18N::Langinfo(3pm) that knows about those and something else.\n\n# perl -e 'require I18N::Langinfo; I18N::Langinfo->import(qw(langinfo \nCODESET)); $codeset = langinfo(CODESET()); print \"My codeset=\".\n$codeset.\"\\n\";'\nMy codeset=ISO-8859-15\n\n-- robin\n"},{"id":"73313","messageId":"200803291039.43355.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329091145.GA19501@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T09:39:43Z","receivedAt":"2008-03-29T09:39:43Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 10.11.45 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 10:02:43AM +0100, Robin Rosenberg wrote:\n> > My proof is entirely empirical. What happens is that attempting to decode\n> > a non-UTF-8 string will put a unicode surrogate pair into the (now\n> > Unicode) string and encoding will just encode the surrogate pair into\n> > UTF-8 and not the original. As a result, the encode(decode($x)) eq $x\n> > *only* if $x is a valid UTF-8 octet sequence. Why would you not get the\n> > original back if you start with valid UTF-8?\n>\n> Because some UTF-8 sequences have multiple representations, and that\n\nCare to give an example?\n\n-- robon\n"},{"id":"73314","messageId":"20080329094322.GA21814@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803291039.43355.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T09:43:22Z","receivedAt":"2008-03-29T09:43:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 10:39:43AM +0100, Robin Rosenberg wrote:\n\n> > Because some UTF-8 sequences have multiple representations, and that\n> \n> Care to give an example?\n\nThere were several given in the \"OS X normalize your UTF-8 filenames\"\nthread a while back. They generally boil down to \"a\"\nversus \"\" both of which are valid UTF-8.\n\n-Peff\n"},{"id":"73315","messageId":"20080329095238.GB21814@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803291038.48847.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T09:52:38Z","receivedAt":"2008-03-29T09:52:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 10:38:48AM +0100, Robin Rosenberg wrote:\n\n> The environment variables are only part of the story. There is a langinfo API \n> for this. See I18N::Langinfo(3pm) that knows about those and something else.\n> \n> # perl -e 'require I18N::Langinfo; I18N::Langinfo->import(qw(langinfo \n> CODESET)); $codeset = langinfo(CODESET()); print \"My codeset=\".\n> $codeset.\"\\n\";'\n> My codeset=ISO-8859-15\n\nHmm, neat. So perhaps it would make sense to just use this value instead\nof utf-8, and not worry about examining the actual text (since any such\nexamination is at best a guess, anyway)?\n\nAny idea what version of perl started shipping I18N::Langinfo? I\ncouldn't see anything useful from grepping the Changes files.\n\n-Peff\n\nPS Your 'require' is more simply written as 'use I18N::Langinfo\nqw(langinfo CODESET)', or perhaps even simpler:\n\n perl -MI18N::Langinfo=langinfo,CODESET ...\n"},{"id":"73316","messageId":"200803291354.10368.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329095238.GB21814@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T12:54:10Z","receivedAt":"2008-03-29T12:54:10Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 10.52.38 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 10:38:48AM +0100, Robin Rosenberg wrote:\n> > The environment variables are only part of the story. There is a langinfo\n> > API for this. See I18N::Langinfo(3pm) that knows about those and\n> > something else.\n> >\n> > # perl -e 'require I18N::Langinfo; I18N::Langinfo->import(qw(langinfo\n> > CODESET)); $codeset = langinfo(CODESET()); print \"My codeset=\".\n> > $codeset.\"\\n\";'\n> > My codeset=ISO-8859-15\n>\n> Hmm, neat. So perhaps it would make sense to just use this value instead\n> of utf-8, and not worry about examining the actual text (since any such\n> examination is at best a guess, anyway)?\n\nI think you really should try the UTF-8 guess, since a file may well be UTF-8 \neven if the user locale is something else. Especially for XML files, UTF-8\nis common, but there are many more cases. Look into git-gui/po for more \nexamples. The probability of a UTF-8 test being wrong is just so unimaginable \nlow.\n\n> PS Your 'require' is more simply written as 'use I18N::Langinfo\n\n> qw(langinfo CODESET)', or perhaps even simpler:\n\nSee the man page, from which I stole it. It suggests you wrap it all inside \neval {}, just in case your perl does not have langinfo.\n\nAs for the is_utf8() i'm not sure what it does, but I can't make it work.\n\n-- robin\n"},{"id":"73317","messageId":"200803291354.47269.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329094322.GA21814@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T12:54:47Z","receivedAt":"2008-03-29T12:54:47Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 10.43.22 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 10:39:43AM +0100, Robin Rosenberg wrote:\n> > > Because some UTF-8 sequences have multiple representations, and that\n> >\n> > Care to give an example?\n>\n> There were several given in the \"OS X normalize your UTF-8 filenames\"\n> thread a while back. They generally boil down to \"a\"\n> versus \"\" both of which are valid UTF-8.\n\nThat is what /OS X/ does with file names. It changes one unicode code point\nto a sequence of other \"equivalent\" code points. I'm pretty sure perl does\nnot do that.\n\n-- robin\n"},{"id":"73335","messageId":"20080329211849.GA30851@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803291354.10368.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T21:18:49Z","receivedAt":"2008-03-29T21:18:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 01:54:10PM +0100, Robin Rosenberg wrote:\n\n> I think you really should try the UTF-8 guess, since a file may well be UTF-8 \n> even if the user locale is something else. Especially for XML files, UTF-8\n> is common, but there are many more cases. Look into git-gui/po for more \n> examples. The probability of a UTF-8 test being wrong is just so unimaginable \n> low.\n\nThinking about this more, I think it is only half the solution. If\nsomething is not valid utf-8, then we know it must be something else.\nBut if something is valid utf-8, is it necessarily utf-8? I think we are\ngoing to have a much higher probability of guessing wrong there.\n\nFor example, consider the bytes { 0xc3, 0xb6 }. In utf-8, they are 'ö'.\nBut in iso8859-1, they also have meaning (paragraph symbol followed by\nÃ). Now that is an unlikely combination to come up. And maybe for\nLatin-1, having two non-ascii characters next to each other is unlikely.\nBut over all commonly used encodings, what is the probability in an\naverage text of that encoding that it contains valid UTF-8?\nFor example, I have no idea what patterns can be found in EUCJP.\n\n> > PS Your 'require' is more simply written as 'use I18N::Langinfo\n> > qw(langinfo CODESET)', or perhaps even simpler:\n> \n> See the man page, from which I stole it. It suggests you wrap it all inside \n> eval {}, just in case your perl does not have langinfo.\n\nYes, that does make sense for a script (I just couldn't see it because\nthe entire toy example would be inside the eval).\n\n> As for the is_utf8() i'm not sure what it does, but I can't make it work.\n\nThere is some magic with how Perl marks strings as \"binary\" versus\n\"utf-8\" that I don't quite understand. And I think is_utf8 is really\nabout asking \"is the utf-8 flag set\".\n\nI think this discussion would benefit greatly from somebody who has more\nof a clue how perl i18n stuff works. Why don't you work up a patch that\nmakes sense for you, and then hopefully that will get some attention?\n\n-Peff\n"},{"id":"73336","messageId":"200803292243.40733.robin.rosenberg.lists@dewire.com","threadId":"12897","inReplyTo":"20080329211849.GA30851@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-03-29T21:43:40Z","receivedAt":"2008-03-29T21:43:40Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Den Saturday 29 March 2008 22.18.49 skrev Jeff King:\n> On Sat, Mar 29, 2008 at 01:54:10PM +0100, Robin Rosenberg wrote:\n> > I think you really should try the UTF-8 guess, since a file may well be\n> > UTF-8 even if the user locale is something else. Especially for XML\n> > files, UTF-8 is common, but there are many more cases. Look into\n> > git-gui/po for more examples. The probability of a UTF-8 test being wrong\n> > is just so unimaginable low.\n>\n> Thinking about this more, I think it is only half the solution. If\n> something is not valid utf-8, then we know it must be something else.\n> But if something is valid utf-8, is it necessarily utf-8? I think we are\n> going to have a much higher probability of guessing wrong there.\n>\n> For example, consider the bytes { 0xc3, 0xb6 }. In utf-8, they are 'ö'.\n> But in iso8859-1, they also have meaning (paragraph symbol followed by\n> Ã). Now that is an unlikely combination to come up. And maybe for\n> Latin-1, having two non-ascii characters next to each other is unlikely.\nFirst that is even by random an unlikely sequence. For any \"real\" is string\nit simply won't happen, even in this context. Try scanning everything you\ncan think of and see if you find such a sequence that is not actually UTF-8.\n\n> But over all commonly used encodings, what is the probability in an\n> average text of that encoding that it contains valid UTF-8?\n> For example, I have no idea what patterns can be found in EUCJP.\n\nSee here http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf\n\nNote that a random string is a randomly generated string. Not a random string\nfrom the set of actually existing strings.\n\n> There is some magic with how Perl marks strings as \"binary\" versus\n> \"utf-8\" that I don't quite understand. And I think is_utf8 is really\n> about asking \"is the utf-8 flag set\".\n>\n> I think this discussion would benefit greatly from somebody who has more\n> of a clue how perl i18n stuff works. Why don't you work up a patch that\n> makes sense for you, and then hopefully that will get some attention?\n\nThe only real question as I see it is whether perl has a builtin metod that \nworks better than the decode/encode. Anyone?\n\n-- robin\n"},{"id":"73337","messageId":"20080329214516.GB30851@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803291354.47269.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T21:45:16Z","receivedAt":"2008-03-29T21:45:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 01:54:47PM +0100, Robin Rosenberg wrote:\n\n> > There were several given in the \"OS X normalize your UTF-8 filenames\"\n> > thread a while back. They generally boil down to \"a\"\n> > versus \"\" both of which are valid UTF-8.\n> \n> That is what /OS X/ does with file names. It changes one unicode code point\n> to a sequence of other \"equivalent\" code points. I'm pretty sure perl does\n> not do that.\n\nMy point is that we don't _know_ what is happening in between the decode\nand encode. Does that intermediate form have the information required to\nconvert back to the exact same bytes as the original form? I don't think\nyou've provided any evidence that it does or does not.\n\nBut here is some evidence that it does work:\n\n$ cat test.pl\nsub is_valid {\n my $orig = shift;\n my $test = $orig;\n utf8::decode($test);\n utf8::encode($test);\n return $orig eq $test ? \"yes\" : \"no\";\n}\nprint \"utf-8: \", is_valid(\"\\xc3\\xb6\"), \"\\n\";\nprint \"latin-1: \", is_valid(\"\\xc3\"), \"\\n\";\nprint \"utf-8 w/ combining: \", is_valid(\"o\\xcc\\x88\"), \"\\n\";\n\n$ perl test.pl\nutf-8: yes\nlatin-1: no\nutf-8 w/ combining: yes\n\nBut it still feels a little wrong to test by converting. There must be\nsome way to ask \"is this valid utf-8\" (there are several candidate\nfunctions, but I don't think either of us quite knows the right way to\ninvoke them).\n\n-Peff\n"},{"id":"73338","messageId":"20080329220011.GC30851@coredump.intra.peff.net","threadId":"12897","inReplyTo":"200803292243.40733.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-29T22:00:11Z","receivedAt":"2008-03-29T22:00:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 29, 2008 at 10:43:40PM +0100, Robin Rosenberg wrote:\n\n> First that is even by random an unlikely sequence. For any \"real\" is string\n> it simply won't happen, even in this context. Try scanning everything you\n> can think of and see if you find such a sequence that is not actually UTF-8.\n\nThat's the problem I was mentioning: \"everything I can think of\" is\nbasically just us-ascii with a few accented characters. I don't know\nhow, e.g., Japanese texts will fare with such a test.\n\n> > But over all commonly used encodings, what is the probability in an\n> > average text of that encoding that it contains valid UTF-8?\n> > For example, I have no idea what patterns can be found in EUCJP.\n> \n> See here http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf\n\nThanks, that is an interesting read. And he seems to indicate that you\ncan guess with a reasonable degree of success. But a few points on that\nwork:\n\n - he has a specific methodology for guessing, which is more elaborate\n than what you proposed. So to get his results, you would need to\n implement his method. Hopefully if perl does have a \"guess if this\n looks like utf8\" method, it uses a similar scheme.\n\n - he does admit that some encodings have difficult to assess\n probabilities, and it will vary from language to language. See page\n 22:\n\n If a specific language does not use all three letters (a single\n letter on the left and the corresponding two letters on the\n right), then this combination presents no danger. Further checks\n can then be made with a dictionary, although there is the problem\n that a dictionary never contains all possible words, and that of\n course resource names don't necessarily have to be words.\n\n - he mentions Latin, Cyrillic, and Hebrew encodings. I note the\n conspicuous absence of any Asian languages.\n\n> Note that a random string is a randomly generated string. Not a random\n> string from the set of actually existing strings.\n\nSure. But looking at random strings isn't terribly useful; there is a\nnon-uniform distribution over the set of strings, dependent on the\n_actual_ encoding. So there are going to be \"good\" encodings that will\nguess well, and there will be \"bad\" encodings that might not (and by\n\"will\", I mean \"there may be\"; that is the very thing I am saying we\ndon't have good evidence for).\n\n-Peff\n"},{"id":"73347","messageId":"47EEF71E.7060805@vilain.net","threadId":"12897","inReplyTo":"20080329095238.GB21814@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-03-30T02:12:46Z","receivedAt":"2008-03-30T02:12:46Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Jeff King wrote:\n> Any idea what version of perl started shipping I18N::Langinfo? I\n> couldn't see anything useful from grepping the Changes files.\n\nModule::CoreList knows. See the man page for that.\n\nSam.\n"},{"id":"73349","messageId":"47EF0BC5.4040102@vilain.net","threadId":"12897","inReplyTo":"20080329214516.GB30851@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-03-30T03:40:53Z","receivedAt":"2008-03-30T03:40:53Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Jeff King wrote:\n> My point is that we don't _know_ what is happening in between the decode\n> and encode. Does that intermediate form have the information required to\n> convert back to the exact same bytes as the original form?\n\nNo, it doesn't. If you want that, save a copy of the string (it's a\nlazy copy anyway).\n\nThe module that will let you see into the strings to see what it\nhappening is Devel::Peek. Using that, you will see the state of the\nUTF8 scalar flag. For example;\n\n maia:~$ perl -Mutf8 -MDevel::Peek -le 'Dump \"Güt\"'\n SV = PV(0x605d08) at 0x62f230\n REFCNT = 1\n FLAGS = (PADBUSY,PADTMP,POK,READONLY,pPOK,UTF8)\n PV = 0x60cd20 \"G\\303\\274t\"\\0 [UTF8 \"G\\x{fc}t\"]\n CUR = 4\n LEN = 8\n\nBy default, all strings that are read from files will NOT have this flag\nset, unless the filehandle that was read from was marked as being utf-8\n(in order to preserve C semantics by default);\n\n maia:~$ echo \"Güt\" | perl -MDevel::Peek -nle 'Dump $_'\n SV = PV(0x6052d0) at 0x604220\n REFCNT = 1\n FLAGS = (POK,pPOK)\n PV = 0x62f0e0 \"G\\303\\274t\"\\0\n CUR = 4\n LEN = 80\n maia:~$ echo \"Güt\" | perl -MDevel::Peek -nle 'BEGIN { binmode STDIN,\n\":utf8\" } Dump $_'\n SV = PV(0x6052d0) at 0x604220\n REFCNT = 1\n FLAGS = (POK,pPOK,UTF8)\n PV = 0x62f100 \"G\\303\\274t\"\\0 [UTF8 \"G\\x{fc}t\"]\n CUR = 4\n LEN = 80\n\n> But it still feels a little wrong to test by converting.\n\nutf8::decode works in-place; it is essentially checking that the string\nis valid, and if so, marking it as UTF8.\n\n my ($encoding);\n if (utf8::decode($string)) {\n if (utf8::is_utf($string)) {\n $encoding = \"UTF-8\";\n }\n else {\n $encoding = \"US-ASCII\";\n }\n }\n else {\n $encoding = \"ISO8859-1\"\n }\n\nFor US-ASCII, you'll only have to encode if the string contains special\ncharacters (those below \\037) or any \"=\" characters.\n\nYou could try using langinfo CODESET instead of hardcoding ISO8859-1\nlike that, but at least on my system can return bizarre values like\nANSI_X3.4-1968, which may be in some contexts a \"correct\" description of\nthe encoding, but is unlikely to be understood by mail clients.\n\n> There must be\n> some way to ask \"is this valid utf-8\" (there are several candidate\n> functions, but I don't think either of us quite knows the right way to\n> invoke them).\n\nI think you were just reading the note on the utf8::valid function a\nlittle too strongly.\n\nYou could use this block;\n\n if ($string =~ m/[\\200-\\377]/) {\n Encode::_utf8_on($string);\n if (!utf8::valid($string)) {\n Encode::_utf8_off($string);\n }\n }\n\nAnyway, I guess all this rubbish is why people use CPAN modules, so that\nthey don't have to continually rediscover every single protocol quirk\nand reinvent the wheel.\n\nie, it would be much, much simpler to use MIME::Entity->build for all of\nthis, and remove the duplication of code.\n\nSam.\n"},{"id":"73350","messageId":"20080330043151.GA18603@sigill.intra.peff.net","threadId":"12897","inReplyTo":"47EEF71E.7060805@vilain.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-30T04:31:52Z","receivedAt":"2008-03-30T04:31:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Mar 30, 2008 at 03:12:46PM +1300, Sam Vilain wrote:\n\n> > Any idea what version of perl started shipping I18N::Langinfo? I\n> > couldn't see anything useful from grepping the Changes files.\n> Module::CoreList knows. See the man page for that.\n\nThanks, I didn't know about that (I foolishly assumed that such\ninformation would be, well, along with the core of perl).\n\nThe answer is: I18N::Langinfo started shipping with 5.007003. I think we\nhave pretty much given up on perl < 5.6 (at least from my experience\nwith 5.005 on Solaris), so it is probably safe to use.\n\n-Peff\n"},{"id":"73352","messageId":"20080330043920.GB18603@sigill.intra.peff.net","threadId":"12897","inReplyTo":"47EF0BC5.4040102@vilain.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-03-30T04:39:21Z","receivedAt":"2008-03-30T04:39:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Mar 30, 2008 at 04:40:53PM +1300, Sam Vilain wrote:\n\n> > My point is that we don't _know_ what is happening in between the decode\n> > and encode. Does that intermediate form have the information required to\n> > convert back to the exact same bytes as the original form?\n> No, it doesn't. If you want that, save a copy of the string (it's a\n> lazy copy anyway).\n\nWe do already save a copy. The question is that Robin is proposing\ndecode/encode to check for validity. It was not clear to me that such a\nprocess would always return the exact same bytes even for valid utf-8.\n\nBut it seems like you are saying below that it is really just the\n\"decode\" part of that which is interesting:\n\n> utf8::decode works in-place; it is essentially checking that the string\n> is valid, and if so, marking it as UTF8.\n> \n> my ($encoding);\n> if (utf8::decode($string)) {\n> if (utf8::is_utf($string)) {\n> $encoding = \"UTF-8\";\n> }\n> else {\n> $encoding = \"US-ASCII\";\n> }\n> }\n> else {\n> $encoding = \"ISO8859-1\"\n> }\n\nOK, that was the magic invocation we were looking for. Thank you.\n\n> For US-ASCII, you'll only have to encode if the string contains special\n> characters (those below \\037) or any \"=\" characters.\n\nAh, yeah. I think our tests are lacking in that they check for only\n[^[:ascii:]].\n\n> Anyway, I guess all this rubbish is why people use CPAN modules, so that\n> they don't have to continually rediscover every single protocol quirk\n> and reinvent the wheel.\n> \n> ie, it would be much, much simpler to use MIME::Entity->build for all of\n> this, and remove the duplication of code.\n\nYes, I actually made a similar comment recently. send-email could\nprobably be shorter, easier to read, and have fewer bugs if it used one\nof the many mail-handling CPAN modules. I think it would pretty much\ninvolve scrapping the current send-email and starting fresh, though.\n\nThanks for your input.\n\n-Peff\n"},{"id":"73382","messageId":"7vwsnjwz97.fsf@gitster.siamese.dyndns.org","threadId":"12897","inReplyTo":"200803290941.54091.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-03-30T23:47:16Z","receivedAt":"2008-03-30T23:47:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg writes:\n\n> Den Saturday 29 March 2008 08.22.03 skrev Jeff King:\n>> On Sat, Mar 29, 2008 at 08:19:07AM +0100, Robin Rosenberg wrote:\n>> > Den Friday 28 March 2008 22.29.01 skrev Jeff King:\n>> > > We always use 'utf-8' as the encoding, since we currently\n>> > > have no way of getting the information from the user.\n>> >\n>> > Don't set encoding to UTF-8 unless it actually looks like UTF-8.\n>>\n>> OK. Do you have an example function that guesses with high probability\n>> whether a string is utf-8? If there are non-ascii characters but we\n>> _don't_ guess utf-8, what should we do?\n>\n> Any test for valid UTF-8 will do that with a very high probability. The\n> perl UTF-8 \"api\" is a mess. I couldn't find such a routine!?. Calling \n> decode/encode and see if you get the original string works, but that is too\n> clumsy, IMHO.\n\nThe sequence to decode followed by encode will test if you have a valid\none and if it is canonically encoded, which is testing too much. You only\nwant to check if it is valid, and do not care about normalization.\n\nI see this in perluniintro.pod:\n\n =item *\n\n How Do I Detect Data That's Not Valid In a Particular Encoding?\n\n Use the C package to try converting it.\n For example,\n\n use Encode 'decode_utf8';\n if (decode_utf8($string_of_bytes_that_I_think_is_utf8)) {\n # valid\n } else {\n # invalid\n }\n\nFor commit log messages, we traditionally use similar idea to guess by\nchecking if it looks like an UTF-8 encoded string and otherwise assume\nLatin-1 (and I think we still do if the user does not tell us).\n\nIf this issue is only about the --compose part of send-email, perhaps you\ncan interactively ask instead of \"otherwise assume Latin-1\"?\n"},{"id":"77414","messageId":"7vve171m67.fsf@gitster.siamese.dyndns.org","threadId":"12897","inReplyTo":"20080328212900.GB9656@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-21T19:39:44Z","receivedAt":"2008-05-21T19:39:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Last night I was going through old mail-logs and found this and another\none that this is a follow-up to, which I think are still needed. Does\nanybody see anything wrong with them?\n\nJeff King writes:\n\n> We always use 'utf-8' as the encoding, since we currently\n> have no way of getting the information from the user.\n>\n> This also refactors the quoting of recipient names, since\n> both processes can share the rfc2047 quoting code.\n>\n> Signed-off-by: Jeff King \n> ---\n> git-send-email.perl | 19 +++++++++++++++++--\n> t/t9001-send-email.sh | 15 +++++++++++++++\n> 2 files changed, 32 insertions(+), 2 deletions(-)\n>\n> diff --git a/git-send-email.perl b/git-send-email.perl\n> index 7c4f06c..d0f9d4a 100755\n> --- a/git-send-email.perl\n> +++ b/git-send-email.perl\n> @@ -536,6 +536,14 @@ EOT\n> \t\tif (!$in_body && /^MIME-Version:/i) {\n> \t\t\t$need_8bit_cte = 0;\n> \t\t}\n> +\t\tif (!$in_body && /^Subject: ?(.*)/i) {\n> +\t\t\tmy $subject = $1;\n> +\t\t\t$_ = \"Subject: \" .\n> +\t\t\t\t($subject =~ /[^[:ascii:]]/ ?\n> +\t\t\t\t quote_rfc2047($subject) :\n> +\t\t\t\t $subject) .\n> +\t\t\t\t\"\\n\";\n> +\t\t}\n> \t\tprint C2 $_;\n> \t}\n> \tclose(C);\n> @@ -626,6 +634,14 @@ sub unquote_rfc2047 {\n> \treturn wantarray ? ($_, $encoding) : $_;\n> }\n> \n> +sub quote_rfc2047 {\n> +\tlocal $_ = shift;\n> +\tmy $encoding = shift || 'utf-8';\n> +\ts/([^-a-zA-Z0-9!*+\\/])/sprintf(\"=%02X\", ord($1))/eg;\n> +\ts/(.*)/=\\?$encoding\\?q\\?$1\\?=/;\n> +\treturn $_;\n> +}\n> +\n> # use the simplest quoting being able to handle the recipient\n> sub sanitize_address\n> {\n> @@ -643,8 +659,7 @@ sub sanitize_address\n> \n> \t# rfc2047 is needed if a non-ascii char is included\n> \tif ($recipient_name =~ /[^[:ascii:]]/) {\n> -\t\t$recipient_name =~ s/([^-a-zA-Z0-9!*+\\/])/sprintf(\"=%02X\", ord($1))/eg;\n> -\t\t$recipient_name =~ s/(.*)/=\\?utf-8\\?q\\?$1\\?=/;\n> +\t\t$recipient_name = quote_rfc2047($recipient_name);\n> \t}\n> \n> \t# double quotes are needed if specials or CTLs are included\n> diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh\n> index e222c49..a4bcd28 100755\n> --- a/t/t9001-send-email.sh\n> +++ b/t/t9001-send-email.sh\n> @@ -210,4 +210,19 @@ test_expect_success '--compose respects user mime type' '\n> \t! grep \"^Content-Type: text/plain; charset=utf-8\" msgtxt1\n> '\n> \n> +test_expect_success '--compose adds MIME for utf8 subject' '\n> +\tclean_fake_sendmail &&\n> +\techo y | \\\n> +\t GIT_EDITOR=$(pwd)/fake-editor \\\n> +\t GIT_SEND_EMAIL_NOTTY=1 \\\n> +\t git send-email \\\n> +\t --compose --subject utf8-sübjëct \\\n> +\t --from=\"Example \" \\\n> +\t --to=nobody@example.com \\\n> +\t --smtp-server=\"$(pwd)/fake.sendmail\" \\\n> +\t $patches &&\n> +\tgrep \"^fake edit\" msgtxt1 &&\n> +\tgrep \"^Subject: =?utf-8?q?utf8-s=C3=BCbj=C3=ABct?=\" msgtxt1\n> +'\n> +\n> test_done\n> -- \n> 1.5.5.rc1.141.g50ecd.dirty\n"},{"id":"77416","messageId":"20080521194734.GA17321@sigill.intra.peff.net","threadId":"12897","inReplyTo":"7vve171m67.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-05-21T19:47:34Z","receivedAt":"2008-05-21T19:47:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 21, 2008 at 12:39:44PM -0700, Junio C Hamano wrote:\n\n> Last night I was going through old mail-logs and found this and another\n> one that this is a follow-up to, which I think are still needed. Does\n> anybody see anything wrong with them?\n>\n> Jeff King writes:\n> \n> > We always use 'utf-8' as the encoding, since we currently\n> > have no way of getting the information from the user.\n\nAh, thanks for bringing this up. I noticed a few weeks ago that it\nhadn't been applied and meant to bring it up, but somehow I failed to\ndo so.\n\nObviously I'm in support of this one, but I also think Horst's patch\nlooks correct.\n\n-Peff\n"}]}