{"thread":{"id":"24895","subject":"Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","startedAt":"2010-08-28T21:17:00Z","lastAt":"2010-08-30T14:33:24Z","messageCount":20,"participants":["Ævar Arnfjörð Bjarmason","Jonathan Nieder","Marcin Cieslak"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"149177","messageId":"AANLkTi=cb5zyKyogdunB6NzWDk99V2hSg-c0vQpwKM-Z@mail.gmail.com","threadId":"24895","inReplyTo":null,"subject":"Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-28T21:17:00Z","receivedAt":"2010-08-28T21:17:00Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I'm having an odd encoding issue with gettext on my\ngettextize-git-mainporcelain branch that hadn't been turned up before\nbecause none of the existing messages used non-ASCII translations.\n\nWith this in is.po (full version at [is.po]):\n\n \"Content-Type: text/plain; charset=UTF-8\\n\"\n \"Content-Transfer-Encoding: 8bit\\n\n\nI do:\n\n $ msgfmt -o /opt/git/next-gettext/share/locale/is/LC_MESSAGES/git.mo is.po\n\nWhich, under an Icelandic locale gives me:\n\n $ rm -rf /tmp/meh; LANGUAGE= LC_ALL= LANG=is_IS.UTF-8 git init /tmp/meh\n Bj? til t?ma Git lind ? /tmp/meh/.git/\n\nThose \"?\" characters are actual ASCII question marks.\n\nBut if I don't specify an encoding msgfmt will complain:\n\n $ msgfmt -o /opt/git/next-gettext/share/locale/is/LC_MESSAGES/git.mo is.po\n is.po: warning: Charset missing in header.\n Message conversion to user's charset will not work.\n\nBut git will now emit the non-ASCII characters from its message\ncatalogue. Probably because some component now doesn't try to be smart\nabout encoding.\n\n $ rm -rf /tmp/meh; LANGUAGE= LC_ALL= LANG=is_IS.UTF-8 git init /tmp/meh\n Bjó til tóma Git lind í /tmp/meh/.git/\n\nThat'd probably break under a non-UTF-8 locale, like an ISO-8859-1 one\nthough.\n\nA `hexdump -C` of the two `.mo` files is exactly the same, aside from\nthe charset header. I.e. both contain valid UTF-8 sequences, so the\nissue is somewhere between the `*.mo` file being read and it being\nemitted by `libintl` and the `gettext` function.\n\nWe're not doing anything odd in our [gettext.c] that I can see that\ncould explain this.\n\nTo reproduce it, do:\n\n git clone --reference ~/g/git git://github.com/avar/git.git next-gettext\n cd next-gettext\n git checkout -t origin/gettextize-git-mainporcelain\n make -j 4 prefix=/tmp/git all install\n rm -rf /tmp/meh; LANGUAGE= LANG=is_IS.utf8 /tmp/git/bin/git init /tmp/meh\n\nWhich'll give (as mentioned above):\n\n Bj? til t?ma Git lind ? /tmp/meh/.git/\n\nBut editing out the Content-Type line gives:\n\n Bjó til tóma Git lind í /tmp/meh/.git/\n\n[gettextize-git-mainporcelain]:\nhttp://github.com/avar/git/tree/gettextize-git-mainporcelain]\n[is.po]: http://github.com/avar/git/blob/gettextize-git-mainporcelain/po/is.po\n[gettext.c]: http://github.com/avar/git/blob/gettextize-git-mainporcelain/gettext.c\n"},{"id":"149178","messageId":"AANLkTikHbxrmj3R1LDghVvMA1KNEdfeiXj44cq1KRN7M@mail.gmail.com","threadId":"24895","inReplyTo":"AANLkTi=cb5zyKyogdunB6NzWDk99V2hSg-c0vQpwKM-Z@mail.gmail.com","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-28T21:33:43Z","receivedAt":"2010-08-28T21:33:43Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Aug 28, 2010 at 21:17, Ævar Arnfjörð Bjarmason wrote:\n> I'm having an odd encoding issue with gettext on my\n> gettextize-git-mainporcelain branch that hadn't been turned up before\n> because none of the existing messages used non-ASCII translations.\n\nWell, this is fun. It turns out that reverting 107880a makes it work,\ni.e. you need to set LC_CTYPE since reading *.mo files in a\nlocale-awere manner involved character classification.\n\nBut as 107880a explains doing so broke other parts of Git.\n\nI'll have to think about how to solve that, one way obviously would be\nto fix up our vsnprintf() invocation, but there may be others like it\nthat I haven't spotted.\n"},{"id":"149179","messageId":"20100828214641.GA5515@burratino","threadId":"24895","inReplyTo":"AANLkTikHbxrmj3R1LDghVvMA1KNEdfeiXj44cq1KRN7M@mail.gmail.com","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-28T21:46:41Z","receivedAt":"2010-08-28T21:46:41Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n\n> Well, this is fun. It turns out that reverting 107880a makes it work,\n> i.e. you need to set LC_CTYPE since reading *.mo files in a\n> locale-awere manner involved character classification.\n> \n> But as 107880a explains doing so broke other parts of Git.\n> \n> I'll have to think about how to solve that, one way obviously would be\n> to fix up our vsnprintf() invocation, but there may be others like it\n> that I haven't spotted.\n\nIn case you remember: why did vsnprintf() fail in that example? If I\nunderstand what C99 says correctly (a big if), then\n\n printf(\"%s\\n\", some_nonsense_string);\n\nshould always just work.\n\nltrace indicates that something is wacky about the format string.\n\n vsnprintf(\"Author: \", 143, \"%s: %.*s%.*s\\n\", 0xbf8b5738) = -1\n\nThe regexes in http-backend are lc_collate- (but probably not lc_ctype-)\nsensitive. I am not sure how to go about exhaustively tracking down\nctype-dependencies.\n"},{"id":"149180","messageId":"20100828215956.GB5515@burratino","threadId":"24895","inReplyTo":"20100828214641.GA5515@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-28T21:59:56Z","receivedAt":"2010-08-28T21:59:56Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> printf(\"%s\\n\", some_nonsense_string);\n> \n> should always just work.\n\nOk, so apparently\n\n #include \n #include \n\n int main(void)\n {\n setlocale(LC_CTYPE, \"\");\n printf(\"%.11s\\n\", \"Author: \\277\");\n\treturn 0;\n }\n\ndoes not work. Even while\n\n\tprintf(\"%.1s\\n\", \"étale\");\n\ndoes print only one byte.\n\nIdeas?\n"},{"id":"149185","messageId":"alpine.BSF.2.00.1008282213420.67930@x.fncre.vasb","threadId":"24895","inReplyTo":"20100828215956.GB5515@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Marcin Cieslak","fromEmail":"saper@saper.info","sentAt":"2010-08-28T22:14:36Z","receivedAt":"2010-08-28T22:14:36Z","isPatch":false,"sender":{"key":"saper@saper.info","avatar":"https://gravatar.com/avatar/8313a958c439a2d52eb2ee4cfcfa64018cefce5400d70dba51f38867f9ce7d36?d=mp&s=160"},"body":"On Sat, 28 Aug 2010, Jonathan Nieder wrote:\n\n> Jonathan Nieder wrote:\n>\n>> printf(\"%s\\n\", some_nonsense_string);\n>>\n>> should always just work.\n>\n> Ok, so apparently\n>\n> #include \n> #include \n>\n> int main(void)\n> {\n> setlocale(LC_CTYPE, \"\");\n> printf(\"%.11s\\n\", \"Author: \\277\");\n> \treturn 0;\n> }\n\nOn my FreeBSD box (various locales tested) I get the following bytes\noutput:\n\n41 75 74 68 6f 72 3a 20 bf 0a\n\n> does not work.\n\nWhat's wrong?\n\n--Marcin\n"},{"id":"149181","messageId":"20100828221655.GB5777@burratino","threadId":"24895","inReplyTo":"alpine.BSF.2.00.1008282213420.67930@x.fncre.vasb","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-28T22:16:55Z","receivedAt":"2010-08-28T22:16:55Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Marcin Cieslak wrote:\n\n> What's wrong?\n\n$ /lib/libc.so.6 |head -1\nGNU C Library (Debian EGLIBC 2.11.2-2) stable release version 2.11.2, by Roland McGrath et al.\n$ cat test.c\n#include \n#include \n\nint main(void)\n{\n int n;\n\n setlocale(LC_CTYPE, \"\");\n n = printf(\"%.11s\\n\", \"Author: \\277\");\n perror(\"printf\");\n fprintf(stderr, \"return value: %d\\n\", n);\n return 0;\n}\n$ make test\ncc test.c -o test\n$ ./test\nprintf: Invalid or incomplete multibyte or wide character\nreturn value: -1\n\nglibc bug?\n"},{"id":"149183","messageId":"20100828222031.GC5777@burratino","threadId":"24895","inReplyTo":"alpine.BSF.2.00.1008282213420.67930@x.fncre.vasb","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-28T22:20:31Z","receivedAt":"2010-08-28T22:20:31Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Marcin Cieslak wrote:\n\n> What's wrong?\n\nhttp://sourceware.org/bugzilla/show_bug.cgi?id=6530\n"},{"id":"149184","messageId":"alpine.BSF.2.00.1008282229460.67930@x.fncre.vasb","threadId":"24895","inReplyTo":"20100828222031.GC5777@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Marcin Cieslak","fromEmail":"saper@saper.info","sentAt":"2010-08-28T22:30:14Z","receivedAt":"2010-08-28T22:30:14Z","isPatch":false,"sender":{"key":"saper@saper.info","avatar":"https://gravatar.com/avatar/8313a958c439a2d52eb2ee4cfcfa64018cefce5400d70dba51f38867f9ce7d36?d=mp&s=160"},"body":"On Sat, 28 Aug 2010, Jonathan Nieder wrote:\n\n> Marcin Cieslak wrote:\n>\n>> What's wrong?\n>\n> http://sourceware.org/bugzilla/show_bug.cgi?id=6530\n\nYes, a pretty old SPARC Solaris box gives me the same result\nas FreeBSD.\n\n--Marcin\n"},{"id":"149203","messageId":"AANLkTimKQO1MQASvQCz=gpz8FTs3h6FSvm12oFUe7WFD@mail.gmail.com","threadId":"24895","inReplyTo":"20100828221655.GB5777@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-29T07:36:31Z","receivedAt":"2010-08-29T07:36:31Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Aug 28, 2010 at 22:16, Jonathan Nieder wrote:\n> Marcin Cieslak wrote:\n>\n>> What's wrong?\n>\n> $ /lib/libc.so.6  |head -1\n> GNU C Library (Debian EGLIBC 2.11.2-2) stable release version 2.11.2, by Roland McGrath et al.\n> $ cat test.c\n> #include \n> #include \n>\n> int main(void)\n> {\n>        int n;\n>\n>        setlocale(LC_CTYPE, \"\");\n>        n = printf(\"%.11s\\n\", \"Author: \\277\");\n>        perror(\"printf\");\n>        fprintf(stderr, \"return value: %d\\n\", n);\n>        return 0;\n> }\n> $ make test\n> cc     test.c   -o test\n> $ ./test\n> printf: Invalid or incomplete multibyte or wide character\n> return value: -1\n>\n> glibc bug?\n\nIt would appear so. It seems my monkeypatch in 107880a was the wrong\nway to do it. We should be setting LC_CTYPE, and providing a fallback\nfor GNU's buggy sprintf().\n\nWe also have another bug, compiling git with\nSNPRINTF_RETURNS_BOGUS=YesGNuIsBuggy and running \"git show v0.99.6~1\"\non our own repository causes a segfault, presumably due to the same\nbug, but I didn't track it down further than this:\n\n (gdb) run show v0.99.6~1\n Starting program: /home/avar/g/git/git show v0.99.6~1\n [Thread debugging using libthread_db enabled]\n\n Program received signal SIGSEGV, Segmentation fault.\n __strnlen (str=0x8f
, maxlen=) at strnlen.c:47\n 47 strnlen.c: No such file or directory.\n in strnlen.c\n (gdb) bt\n #0 __strnlen (str=0x8f
,\nmaxlen=) at strnlen.c:47\n #1 0x00007ffff73318cd in __mbsnrtowcs (dst=0x7fffffffb960\nL\"\\xffffb9a0翿\\x404500\", src=0x7fffffffcf78, nmc=8557680, len=,\n ps=0x7fffffffcfb0) at mbsnrtowcs.c:66\n #2 0x00007ffff72ef932 in _IO_vfprintf_internal (s=0x7fffffffd010,\nformat=, ap=0x7fffffffd330) at vfprintf.c:1614\n #3 0x00007ffff7311432 in _IO_vsnprintf (\n string=0x829510\n\"H\\215C\\020H\\213\\\\$\\bH\\213l$\\020L\\213d$\\030L\\213l$\nL\\213t$(L\\213|$0H\\203\\304\\070\\303f\\017\\037D: \\177\", maxlen=,\n format=0x58aa28 \"%s: %.*s%.*s\\n\", args=0x7fffffffd330) at\nvsnprintf.c:120\n #4 0x00000000005569c6 in git_vsnprintf (\n str=0x829510\n\"H\\215C\\020H\\213\\\\$\\bH\\213l$\\020L\\213d$\\030L\\213l$\nL\\213t$(L\\213|$0H\\203\\304\\070\\303f\\017\\037D: \\177\", maxsize=572,\n format=0x58aa28 \"%s: %.*s%.*s\\n\", ap=0x7fffffffd330) at snprintf.c:45\n #5 0x000000000053877f in strbuf_addf (sb=0x7fffffffd640,\nfmt=0x58aa28 \"%s: %.*s%.*s\\n\") at strbuf.c:203\n #6 0x000000000050267d in pp_user_info (what=0x58aa4e \"Author\",\nfmt=CMIT_FMT_MEDIUM, sb=0x7fffffffd640,\n line=0x830675 \"David_K\\345gedal \n1126078160 +0200\\ncommitter Junio C Hamano 1126128590\n-0700\\n\\n[PATCH] Simplify git script\\n\\nThe code for listing the\navailable subcommands was unnec\"..., dmode=DATE_NORMAL,\nencoding=0x57025d \"UTF-8\") at pretty.c:283\n #7 0x0000000000503be9 in pp_header (fmt=CMIT_FMT_MEDIUM,\nabbrev=7, dmode=DATE_NORMAL, encoding=0x57025d \"UTF-8\",\ncommit=0x832cf8, msg_p=0x7fffffffd558,\n sb=0x7fffffffd640) at pretty.c:1077\n #8 0x0000000000503865 in pretty_print_commit\n(fmt=CMIT_FMT_MEDIUM, commit=0x832cf8, sb=0x7fffffffd640,\ncontext=0x7fffffffd5e8) at pretty.c:1219\n #9 0x00000000004ead8b in show_log (opt=0x7fffffffd858) at log-tree.c:508\n #10 0x00000000004eb1b2 in log_tree_diff_flush (opt=0x7fffffffd858)\nat log-tree.c:557\n #11 0x00000000004eb5e9 in log_tree_diff (opt=0x7fffffffd858,\ncommit=0x832cf8, log=0x7fffffffd728) at log-tree.c:635\n #12 0x00000000004eb322 in log_tree_commit (opt=0x7fffffffd858,\ncommit=0x832cf8) at log-tree.c:658\n #13 0x000000000044ff0b in cmd_log_walk (rev=0x7fffffffd858) at log.c:372\n #14 0x0000000000450555 in cmd_show (argc=2, argv=0x7fffffffe070,\nprefix=0x0) at log.c:561\n #15 0x00000000004058a7 in run_builtin (p=0x7bf708, argc=2,\nargv=0x7fffffffe070) at git.c:278\n #16 0x0000000000404a54 in handle_internal_command (argc=2,\nargv=0x7fffffffe070) at git.c:434\n #17 0x000000000040513e in run_argv (argcp=0x7fffffffdf78,\nargv=0x7fffffffdf70) at git.c:478\n #18 0x0000000000404875 in main (argc=2, argv=0x7fffffffe070) at git.c:553\n (gdb) frame 4\n #4 0x00000000005569c6 in git_vsnprintf (\n str=0x829510\n\"H\\215C\\020H\\213\\\\$\\bH\\213l$\\020L\\213d$\\030L\\213l$\nL\\213t$(L\\213|$0H\\203\\304\\070\\303f\\017\\037D: \\177\", maxsize=572,\n format=0x58aa28 \"%s: %.*s%.*s\\n\", ap=0x7fffffffd330) at snprintf.c:45\n 45 ret = vsnprintf(str,\nmaxsize-SNPRINTF_SIZE_CORR, format, ap);\n (gdb) p str\n $1 = 0x829510 \"H\\215C\\020H\\213\\\\$\\bH\\213l$\\020L\\213d$\\030L\\213l$\nL\\213t$(L\\213|$0H\\203\\304\\070\\303f\\017\\037D: \\177\"\n (gdb) p maxsize\n $2 = 572\n (gdb) p SNPRINTF_SIZE_CORR\n No symbol \"SNPRINTF_SIZE_CORR\" in current context.\n (gdb) p format\n $3 = 0x58aa28 \"%s: %.*s%.*s\\n\"\n (gdb) p ap\n $4 = (__va_list_tag *) 0x7fffffffd330\n (gdb) frame 6\n #6 0x000000000050267d in pp_user_info (what=0x58aa4e \"Author\",\nfmt=CMIT_FMT_MEDIUM, sb=0x7fffffffd640,\n line=0x830675 \"David_K\\345gedal \n1126078160 +0200\\ncommitter Junio C Hamano 1126128590\n-0700\\n\\n[PATCH] Simplify git script\\n\\nThe code for listing the\navailable subcommands was unnec\"..., dmode=DATE_NORMAL,\nencoding=0x57025d \"UTF-8\") at pretty.c:283\n 283 strbuf_addf(sb, \"%s: %.*s%.*s\\n\", what,\n"},{"id":"149204","messageId":"AANLkTinzXSSGHjauw7X9SOfNhUR75+rZ+H0EgLHARS0k@mail.gmail.com","threadId":"24895","inReplyTo":"AANLkTimKQO1MQASvQCz=gpz8FTs3h6FSvm12oFUe7WFD@mail.gmail.com","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-29T08:37:10Z","receivedAt":"2010-08-29T08:37:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Aug 29, 2010 at 07:36, Ævar Arnfjörð Bjarmason wrote:\n\n> We also have another bug, compiling git with\n> SNPRINTF_RETURNS_BOGUS=YesGNuIsBuggy and running \"git show v0.99.6~1\"\n> on our own repository causes a segfault, presumably due to the same\n> bug, but I didn't track it down further than this:\n\nI forgot to mention, compiling it with this partial revert of 107880a of course:\n\n diff --git a/gettext.c b/gettext.c\n index db99742..7ae5cae 100644\n --- a/gettext.c\n +++ b/gettext.c\n @@ -19,2 +19,3 @@ extern void git_setup_gettext(void) {\n (void)setlocale(LC_MESSAGES, \"\");\n + (void)setlocale(LC_CTYPE, \"\");\n (void)textdomain(\"git\");\n"},{"id":"149219","messageId":"AANLkTimn+-5Vys+jg=ryDfwdJ=WZfGwgZ+065M_=TF8r@mail.gmail.com","threadId":"24895","inReplyTo":"20100828221655.GB5777@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-29T18:12:05Z","receivedAt":"2010-08-29T18:12:05Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Aug 28, 2010 at 22:16, Jonathan Nieder wrote:\n\n> $ /lib/libc.so.6  |head -1\n> GNU C Library (Debian EGLIBC 2.11.2-2) stable release version 2.11.2, by Roland McGrath et al.\n> $ cat test.c\n> #include \n> #include \n>\n> int main(void)\n> {\n>        int n;\n>\n>        setlocale(LC_CTYPE, \"\");\n>        n = printf(\"%.11s\\n\", \"Author: \\277\");\n>        perror(\"printf\");\n>        fprintf(stderr, \"return value: %d\\n\", n);\n>        return 0;\n> }\n> $ make test\n> cc     test.c   -o test\n> $ ./test\n> printf: Invalid or incomplete multibyte or wide character\n> return value: -1\n\nSo, my plan of attack is:\n\n * Add compat/printf from Free, Open or NetBSD. Maybe make\n compat/snprintf.c use that while I'm at it.\n * Use that instead of the GNU libc printf on systems that have glibc.\n * Add a configure check for that.\n * Revert 107880a\n * Get gettext goodness with LC_CTYPE\n\nDoes anyone see a problem with that? The potential issue is that\nLC_CTYPE is for:\n\n \"regular expression matching, character classification,\n conversion, case-sensitive comparison, and wide character\n functions.\"\n\nSo it might have unintended side-effects. But the only other\nworkaround I can see is to decree that all consumers of the localized\nmessages must have a UTF-8 locale.\n"},{"id":"149237","messageId":"20100829204519.GB1890@burratino","threadId":"24895","inReplyTo":"AANLkTimn+-5Vys+jg=ryDfwdJ=WZfGwgZ+065M_=TF8r@mail.gmail.com","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-29T20:45:19Z","receivedAt":"2010-08-29T20:45:19Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n\n> So, my plan of attack is:\n> \n> * Add compat/printf from Free, Open or NetBSD. Maybe make\n> compat/snprintf.c use that while I'm at it.\n\nI would prefer to get this fixed in glibc, but of course that\nhas nothing to do with git.\n\n> * Use that instead of the GNU libc printf on systems that have glibc.\n> * Add a configure check for that.\n> * Revert 107880a\n> * Get gettext goodness with LC_CTYPE\n> \n> Does anyone see a problem with that? The potential issue is that\n> LC_CTYPE is for:\n> \n> \"regular expression matching,\n\nshould be okay, I think (unless http-backend is a problem)\n\n> character classification,\n\nworked around (see git-compat-util.h)\n\n> conversion,\n\nI don't know what this means; iconv() is not affected by LC_CTYPE,\nis it?\n\n> case-sensitive comparison,\n\nCould be a problem: we use strcasecmp() heavily.\n\n> and wide character\n> functions.\"\n\nno problem. :)\n\n> So it might have unintended side-effects. But the only other\n> workaround I can see is to decree that all consumers of the localized\n> messages must have a UTF-8 locale.\n\nAnd that is no workaround at all; the problem is still seen with UTF-8\nlocales, no?\n"},{"id":"149272","messageId":"20100830022225.GK2305@burratino","threadId":"24895","inReplyTo":"AANLkTimKQO1MQASvQCz=gpz8FTs3h6FSvm12oFUe7WFD@mail.gmail.com","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-30T02:22:25Z","receivedAt":"2010-08-30T02:22:25Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n\n> We also have another bug, compiling git with\n> SNPRINTF_RETURNS_BOGUS=YesGNuIsBuggy and running \"git show v0.99.6~1\"\n> on our own repository causes a segfault\n\nThat's because the glibc bug is not the bug SNPRINTF_RETURNS_BOGUS is\nmeant to guard against. Hopefully no printf implementation has both\nbugs. :)\n"},{"id":"149292","messageId":"AANLkTin4A6B9zqSYHsWQ+GLWRqLvzJWAQ0F2WC85zDqC@mail.gmail.com","threadId":"24895","inReplyTo":"20100829204519.GB1890@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-30T08:57:34Z","receivedAt":"2010-08-30T08:57:34Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Aug 29, 2010 at 20:45, Jonathan Nieder wrote:\n> Ævar Arnfjörð Bjarmason wrote:\n>\n>> So, my plan of attack is:\n>>\n>>  * Add compat/printf from Free, Open or NetBSD. Maybe make\n>>    compat/snprintf.c use that while I'm at it.\n>\n> I would prefer to get this fixed in glibc, but of course that\n> has nothing to do with git.\n\nYeah, but even if it's fixed there everyone's glibc won't be updated\nfor at least ten years as far as the glibc's we have to support go.\n\nSo even if the bug were fixed upstream today we'd still need a\nworkaround.\n\n>>  * Use that instead of the GNU libc printf on systems that have glibc.\n>>  * Add a configure check for that.\n>>  * Revert 107880a\n>>  * Get gettext goodness with LC_CTYPE\n>>\n>> Does anyone see a problem with that? The potential issue is that\n>> LC_CTYPE is for:\n>>\n>>     \"regular expression matching,\n>\n> should be okay, I think (unless http-backend is a problem)\n\nUser-level commands that take regexes would have different semantics\nbased on the locale though, e.g. git log --grep=.\n\n>> character classification,\n>\n> worked around (see git-compat-util.h)\n\nYay sane_istest!\n\n>>     conversion,\n>\n> I don't know what this means; iconv() is not affected by LC_CTYPE,\n> is it?\n\nI think it's only to do with functions like btowc, see:\nhttp://www.gnu.org/s/libc/manual/html_node/Restartable-multibyte-conversion.html#Restartable-multibyte-conversion\n\n>> case-sensitive comparison,\n>\n> Could be a problem: we use strcasecmp() heavily.\n\nYeah, strcasecmp is affected by LC_CTYPE.\n\n>> and wide character\n>>     functions.\"\n>\n> no problem. :)\n\nNope.\n\n>> So it might have unintended side-effects. But the only other\n>> workaround I can see is to decree that all consumers of the localized\n>> messages must have a UTF-8 locale.\n>\n> And that is no workaround at all; the problem is still seen with UTF-8\n> locales, no?\n\nNo, it'll be seen with all non-UTF-8 locales. Here's the issue:\n\nWhen we add non-ASCII to the po/*.po files we'll write it in UTF-8 as\na matter of policy, simply because that's all the rave these days.\n\nHowever, unless we put \"Content-Type: text/plain; charset=UTF-8\\n\" in\nthe file the gettext utilities won't *know* that it's in UTF-8, if\nit's not there then to them it'll just be a raw stream of bytes. So\nthey won't do the right conversion under non-UTF-8 locales.\n\nBut users using a gettext translation under a UTF-8 locale won't tell\nthe difference, since the *.po encoding and their expected encoding\ndon't differ they don't need any conversion anyway.\n\nWe can even keep the \"Content-Type: text/plain; charset=UTF-8\\n\" and\n*not* use LC_CTYPE if we add a bind_textdomain_codeset(\"git\", \"UTF-8\")\ncall to gettext.c. That call declares that the *.po file is in UTF-8\n(but without LC_CTYPE there still won't be any conversion), see\nhttp://www.gnu.org/s/libc/manual/html_node/Charset-conversion-in-gettext.html#Charset-conversion-in-gettext\n\nHere's a table explaining the various approaches:\n\n A: [correctness] LC_CTYPE + *.po charset=UTF-8\n B: [UTF-8-only hack] no LC_CTYPE + no *.po charset=UTF-8\n C: [UTF-8-only hack] no LC_CTYPE + A *.po charset=UTF-8 +\nbind_textdomain_codeset(\"git\", \"UTF-8\")\n\n | Approach | Correct *.po encoding header | GNU printf() issue |\nLANG=is_IS.utf8 OK | LANG=is_IS.iso88591 OK |\n |----------+------------------------------+--------------------+--------------------+-------------------------|\n | A | X | X | X\n | X |\n | B | No | No, no LC_CTYPE | X\n | No, still outputs UTF-8 |\n | C | X | No, no LC_CTYPE | X\n | No, still outputs UTF-8 |\n\nA would be preferred for correctness, and with a fallback BSD printf()\nwe can avoid the GNU libc bug, however that'll mean using LC_CTYPE,\nwhich'll have some small side-effects for the rest of the code.\n"},{"id":"149316","messageId":"20100830134136.GA2315@burratino","threadId":"24895","inReplyTo":"AANLkTin4A6B9zqSYHsWQ+GLWRqLvzJWAQ0F2WC85zDqC@mail.gmail.com","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-30T13:41:36Z","receivedAt":"2010-08-30T13:41:36Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n\n> We can even keep the \"Content-Type: text/plain; charset=UTF-8\\n\" and\n> *not* use LC_CTYPE if we add a bind_textdomain_codeset(\"git\", \"UTF-8\")\n> call to gettext.\n\nOh! I'd personally prefer to do that for now. :) (Not because of the\nknown printf problem but because I like to reduce possible unknowns.)\n"},{"id":"149319","messageId":"alpine.BSF.2.00.1008301353260.95618@x.fncre.vasb","threadId":"24895","inReplyTo":"20100830134136.GA2315@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Marcin Cieslak","fromEmail":"saper@saper.info","sentAt":"2010-08-30T14:00:11Z","receivedAt":"2010-08-30T14:00:11Z","isPatch":false,"sender":{"key":"saper@saper.info","avatar":"https://gravatar.com/avatar/8313a958c439a2d52eb2ee4cfcfa64018cefce5400d70dba51f38867f9ce7d36?d=mp&s=160"},"body":"On Mon, 30 Aug 2010, Ævar Arnfjörð Bjarmason wrote:\n\n> On Sun, Aug 29, 2010 at 20:45, Jonathan Nieder wrote:\n> A would be preferred for correctness, and with a fallback BSD printf()\n> we can avoid the GNU libc bug, however that'll mean using LC_CTYPE,\n> which'll have some small side-effects for the rest of the code.\n\nThe real problem is that you are probably using same functions\n(locale-enable) for the user-facing side as well as for the \nbackend (talking to repository). Some projects decided to use\nsome special encoding internally (like UCS-2 in case of Java\nand Python 2.x, Unicode ordinals in Python 3.x). Otherwise\nyou may end up in some incompatibilities in the on-disk on \non-network format. I don't think you want to keep telling all bug \nreporters for few years - \"Can you try that again with env LANG=C,\nplease?\" :)\n\nBringing Unicode onboard means that simple strlen() is no longer\nwhat you normally think it does.\n\nOn Mon, 30 Aug 2010, Jonathan Nieder wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>\n>> We can even keep the \"Content-Type: text/plain; charset=UTF-8\\n\" and\n>> *not* use LC_CTYPE if we add a bind_textdomain_codeset(\"git\", \"UTF-8\")\n>> call to gettext.\n>\n> Oh! I'd personally prefer to do that for now. :) (Not because of the\n> known printf problem but because I like to reduce possible unknowns.)\n\nWell, in this case everybody will be force to have UTF-8 in output\non-screen, not useful for people using ISO8859-*, KOI8-R and similar\nthings...\n\n--Marcin"},{"id":"149321","messageId":"AANLkTi=otZ-mxoaPJ3Mg9EFb5+jy1A=DS3w73weB5EJ=@mail.gmail.com","threadId":"24895","inReplyTo":"20100830134136.GA2315@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-30T14:04:10Z","receivedAt":"2010-08-30T14:04:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, Aug 30, 2010 at 13:41, Jonathan Nieder wrote:\n> Ævar Arnfjörð Bjarmason wrote:\n>\n>> We can even keep the \"Content-Type: text/plain; charset=UTF-8\\n\" and\n>> *not* use LC_CTYPE if we add a bind_textdomain_codeset(\"git\", \"UTF-8\")\n>> call to gettext.\n>\n> Oh!  I'd personally prefer to do that for now. :)  (Not because of the\n> known printf problem but because I like to reduce possible unknowns.)\n\nBy now I want to do that too. I've been experimenting with including\n*printf*.c from OpenBSD, NetBSD or FreeBSD and the uClibc and in all\nthose cases it's a major PITA to wade through the OS-specific code\nthat deep in the libc.\n\nEven if I could get that sorted it'll be non-trivial to audit all the\ncode whose semantics will change with LC_CTYPE, and there's a good\nchance I'll miss something and cause an embarrasing bug in some\nunrelated component.\n\nBetter to just document this limitation for now and decree that\ngettext users must have a UTF-8 locale.\n"},{"id":"149322","messageId":"20100830140958.GE2315@burratino","threadId":"24895","inReplyTo":"alpine.BSF.2.00.1008301353260.95618@x.fncre.vasb","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-30T14:09:58Z","receivedAt":"2010-08-30T14:09:58Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Marcin Cieslak wrote:\n\n> Well, in this case everybody will be force to have UTF-8 in output\n> on-screen, not useful for people using ISO8859-*, KOI8-R and similar\n> things...\n\nCan't we do:\n\n\tsetlocale(LC_CTYPE, \"\");\n\tcharset = nl_langinfo(CODESET);\n\tsetlocale(LC_CTYPE, \"C\");\n\nto allow an arbitrary character set?\n"},{"id":"149323","messageId":"AANLkTimKQ9a4GXA4u2hVW+ECAFRzd5NwuP+9p6htBU1y@mail.gmail.com","threadId":"24895","inReplyTo":"alpine.BSF.2.00.1008301353260.95618@x.fncre.vasb","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-30T14:13:21Z","receivedAt":"2010-08-30T14:13:21Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, Aug 30, 2010 at 14:00, Marcin Cieslak wrote:\n> On Mon, 30 Aug 2010, Ævar Arnfjörð Bjarmason wrote:\n>> On Sun, Aug 29, 2010 at 20:45, Jonathan Nieder wrote:\n>> A would be preferred for correctness, and with a fallback BSD printf()\n>> we can avoid the GNU libc bug, however that'll mean using LC_CTYPE,\n>> which'll have some small side-effects for the rest of the code.\n>\n> The real problem is that you are probably using same functions\n> (locale-enable) for the user-facing side as well as for the backend (talking\n> to repository). Some projects decided to use\n> some special encoding internally (like UCS-2 in case of Java\n> and Python 2.x, Unicode ordinals in Python 3.x). Otherwise\n> you may end up in some incompatibilities in the on-disk on on-network\n> format. I don't think you want to keep telling all bug reporters for few\n> years - \"Can you try that again with env LANG=C,\n> please?\" :)\n\nYeah, those programs can probably get away with it too because they\neither implement their own string functions, or don't use setlocale()\nat all for their localizations.\n\n> Bringing Unicode onboard means that simple strlen() is no longer\n> what you normally think it does.\n\nI'm pretty sure strlen() always gives you the number of\nnull-terminated bytes regardless of locale settings. wcslen is the\nwide-characted equivalent.\n"},{"id":"149329","messageId":"AANLkTik-7FCm-WLOKOKd5uA9astMXtdZZLTT=0euX5RL@mail.gmail.com","threadId":"24895","inReplyTo":"20100830140958.GE2315@burratino","subject":"Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-30T14:33:24Z","receivedAt":"2010-08-30T14:33:24Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, Aug 30, 2010 at 14:09, Jonathan Nieder wrote:\n> Marcin Cieslak wrote:\n>\n>> Well, in this case everybody will be force to have UTF-8 in output\n>> on-screen, not useful for people using ISO8859-*, KOI8-R and similar\n>> things...\n>\n> Can't we do:\n>\n>        setlocale(LC_CTYPE, \"\");\n>        charset = nl_langinfo(CODESET);\n>        setlocale(LC_CTYPE, \"C\");\n>\n> to allow an arbitrary character set?\n\nYes, it seems so! With this patch:\n\n --- a/gettext.c\n +++ b/gettext.c\n @@ -3,2 +3,3 @@\n #include \n +#include \n #include \n @@ -8,2 +9,3 @@ extern void git_setup_gettext(void) {\n char *envdir = getenv(\"GIT_TEXTDOMAINDIR\");\n + char *charset;\n\n @@ -19,2 +21,6 @@ extern void git_setup_gettext(void) {\n (void)setlocale(LC_MESSAGES, \"\");\n + (void)setlocale(LC_CTYPE, \"\");\n + charset = nl_langinfo(CODESET);\n + (void)bind_textdomain_codeset(\"git\", charset);\n + (void)setlocale(LC_CTYPE, \"C\");\n (void)textdomain(\"git\");\n\nThe sanity test still passes:\n\n ./t0201-gettext-fallbacks.sh ......... ok\n ./t0200-gettext-basic.sh ............. ok\n ./t0203-gettext-setlocale-sanity.sh .. ok\n ./t0202-gettext-perl.sh .............. ok\n All tests successful.\n\nAnd the resulting git binary can emit both UTF-8 and ISO-8859-1 text\nfrom its *.po files, respectively:\n\n Bjó til tóma Git lind í /tmp/meh/.git/\n Bj� til t�ma Git lind � /tmp/meh/.git/\n"}]}