{"thread":{"id":"24806","subject":"git-filter-branch : LANG / LC_ALL = C breaks UTF-8 author names","startedAt":"2010-08-20T13:20:09Z","lastAt":"2010-09-01T01:08:55Z","messageCount":4,"participants":["Richard MICHAEL","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"148560","messageId":"4C6E8109.5030202@leadformance.com","threadId":"24806","inReplyTo":null,"subject":"git-filter-branch : LANG / LC_ALL = C breaks UTF-8 author names","fromName":"Richard MICHAEL","fromEmail":"rmichael@leadformance.com","sentAt":"2010-08-20T13:20:09Z","receivedAt":"2010-08-20T13:20:09Z","isPatch":false,"sender":{"key":"rmichael@leadformance.com","avatar":null},"body":"  Hello all,\n\nI am filtering our repo with git-filter-branch, but as the sed script \nruns with LANG=C LC_ALL=C (7 bit US ASCII), it dies on commits authored \nby our team members with accented names. Why are the locales \"C\"?  For \ncompatibility with older sed?  I've changed to LANG=en_US.UTF-8, will my \nchange will cause other git-breakage?\n\n\ngit-filter-branch\n\n95: LANG=C LC_ALL=C sed -ne \"$pick_id_script\"\n\n95: LANG=en_US.UTF-8 sed -ne \"$pick_id_script\"\n\n\nRegards,\nRichard\n"},{"id":"148562","messageId":"20100820133244.GB15736@burratino","threadId":"24806","inReplyTo":"4C6E8109.5030202@leadformance.com","subject":"Re: git-filter-branch : LANG / LC_ALL = C breaks UTF-8 author names","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-20T13:32:44Z","receivedAt":"2010-08-20T13:32:44Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Richard MICHAEL wrote:\n\n> I am filtering our repo with git-filter-branch, but as the sed\n> script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on\n> commits authored by our team members with accented names.\n\nYep, someone else recently sent a report about such a sed version,\ntoo.  It is breaking our fragile minds; we ought to find some way to\ndeal with it, but we haven't yet.\n\nJonathan\n"},{"id":"148565","messageId":"4C6E86AA.2020903@leadformance.com","threadId":"24806","inReplyTo":"20100820133244.GB15736@burratino","subject":"Re: git-filter-branch : LANG / LC_ALL = C breaks UTF-8 author names","fromName":"Richard MICHAEL","fromEmail":"rmichael@leadformance.com","sentAt":"2010-08-20T13:44:10Z","receivedAt":"2010-08-20T13:44:10Z","isPatch":false,"sender":{"key":"rmichael@leadformance.com","avatar":null},"body":"  On 10-08-20 3:32 PM, Jonathan Nieder wrote:\n> Richard MICHAEL wrote:\n>\n>> I am filtering our repo with git-filter-branch, but as the sed\n>> script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on\n>> commits authored by our team members with accented names.\n> Yep, someone else recently sent a report about such a sed version,\n> too.  It is breaking our fragile minds; we ought to find some way to\n> deal with it, but we haven't yet.\n>\n> Jonathan\n\nJonathan, thanks for your reply.\n\nWhat about special casing the bad sed (or whitelisting good sed)?  \nSurely a hack, but would those of us with GNU or BSD would be happy.  \nWhich was the troublesome sed?\n\nThat opposed to figuring out the problem, reading about unicode, and \nre-cloning and re-filtering 5,000 commits. :-)  Unfortunately, it \ndoesn't \"die\" well either; the 'export' shell var fails but it keeps \nprocessing commits.  (If I hadn't investigated and changed the LANG, \nwould I have lost those commits?)\n\nRegards,\nRichard\n"},{"id":"293747","messageId":"20100901010855.GD22968@burratino","threadId":"24806","inReplyTo":"4C6E86AA.2020903@leadformance.com","subject":"Re: git-filter-branch : LANG / LC_ALL = C breaks UTF-8 author names","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-09-01T01:08:55Z","receivedAt":"2010-09-01T01:08:55Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Richard,\n\nRichard MICHAEL wrote:\n>>Richard MICHAEL wrote:\n\n>>> I am filtering our repo with git-filter-branch, but as the sed\n>>> script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on\n>>> commits authored by our team members with accented names.\n[...]\n> What about special casing the bad sed (or whitelisting good sed)?\n> Surely a hack, but would those of us with GNU or BSD would be happy.\n> Which was the troublesome sed?\n\nSorry for the slow response.  The problematic sed is GNU sed from\nMacPorts (I think).  Even with LC_ALL=C, .* no longer matches\narbitrary sequences of bytes with such sed: you can check yours with\n\n $ echo 'étale' | LC_ALL=C sed 's/.*//'\n\nUnfortunately I have not been able to reproduce it on Linux.  Debian\nsed 4.2.1-7 and GNU sed v4.2.1-21-gc6d32f0 both produce the expected\nresult:\n\n $ echo 'étale' | LC_ALL=C sed 's/.*//'\n $\n\n> Unfortunately, it\n> doesn't \"die\" well either; the 'export' shell var fails but it keeps\n> processing commits.\n\nHmm, that sounds like a bug indeed.  Here is what the start to a fix\nmight look like, but I stopped early because it there's quite a lot of\nsed usage in git that expects to be able to process arbitrary data\nwith short, newline-terminated lines (regardless of encoding).\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 962a93b..34a5fa3 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -68,8 +68,8 @@ eval \"$functions\"\n # \"author\" or \"committer\n \n set_ident () {\n-\tlid=\"$(echo \"$1\" | tr \"[A-Z]\" \"[a-z]\")\"\n-\tuid=\"$(echo \"$1\" | tr \"[a-z]\" \"[A-Z]\")\"\n+\tlid=\"$(echo \"$1\" | tr \"[A-Z]\" \"[a-z]\")\" &&\n+\tuid=\"$(echo \"$1\" | tr \"[a-z]\" \"[A-Z]\")\" &&\n \tpick_id_script='\n \t\t/^'$lid' /{\n \t\t\ts/'\\''/'\\''\\\\'\\'\\''/g\n@@ -90,9 +90,9 @@ set_ident () {\n \n \t\t\tq\n \t\t}\n-\t'\n+\t' &&\n \n-\tLANG=C LC_ALL=C sed -ne \"$pick_id_script\"\n+\tLANG=C LC_ALL=C sed -ne \"$pick_id_script\" &&\n \t# Ensure non-empty id name.\n \techo \"case \\\"\\$GIT_${uid}_NAME\\\" in \\\"\\\") GIT_${uid}_NAME=\\\"\\${GIT_${uid}_EMAIL%%@*}\\\" && export GIT_${uid}_NAME;; esac\"\n }\n@@ -322,9 +322,11 @@ while read commit parents; do\n \tgit cat-file commit \"$commit\" >../commit ||\n \t\tdie \"Cannot read commit $commit\"\n \n-\teval \"$(set_ident AUTHOR <../commit)\" ||\n+\tset_author=$(set_ident AUTHOR <../commit) &&\n+\teval \"$set_author\" ||\n \t\tdie \"setting author failed for commit $commit\"\n-\teval \"$(set_ident COMMITTER <../commit)\" ||\n+\tset_committer=$(set_ident COMMITTER <../commit) &&\n+\teval \"$set_committer\" ||\n \t\tdie \"setting committer failed for commit $commit\"\n \teval \"$filter_env\" < /dev/null ||\n \t\tdie \"env filter failed: $filter_env\"\n"}]}