Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Aug 29, 2010, 07:36 UTC
Message-ID: <AANLkTimKQO1MQASvQCz=gpz8FTs3h6FSvm12oFUe7WFD@mail.gmail.com>
In-Reply-To: <20100828221655.GB5777@burratino>

On Sat, Aug 28, 2010 at 22:16, Jonathan Nieder <jrnieder@gmail.com> wrote:

Show 27 quoted lines

> Marcin Cieslak wrote:
>
>> What's wrong?
>
> $ /lib/libc.so.6  |head -1
> GNU C Library (Debian EGLIBC 2.11.2-2) stable release version 2.11.2, by Roland McGrath et al.
> $ cat test.c
> #include <stdio.h>
> #include <locale.h>
>
> int main(void)
> {
>        int n;
>
>        setlocale(LC_CTYPE, "");
>        n = printf("%.11s\n", "Author: \277");
>        perror("printf");
>        fprintf(stderr, "return value: %d\n", n);
>        return 0;
> }
> $ make test
> cc     test.c   -o test
> $ ./test
> printf: Invalid or incomplete multibyte or wide character
> return value: -1
>
> glibc bug?

It would appear so. It seems my monkeypatch in 107880a was the wrong way to do it. We should be setting LC_CTYPE, and providing a fallback for GNU's buggy sprintf().

We also have another bug, compiling git with SNPRINTF_RETURNS_BOGUS=YesGNuIsBuggy and running "git show v0.99.6~1" on our own repository causes a segfault, presumably due to the same bug, but I didn't track it down further than this:

    (gdb) run show v0.99.6~1
    Starting program: /home/avar/g/git/git show v0.99.6~1
    [Thread debugging using libthread_db enabled]

    Program received signal SIGSEGV, Segmentation fault.
    __strnlen (str=0x8f <Address 0x8f out of bounds>, maxlen=<value
optimized out>) at strnlen.c:47
    47      strnlen.c: No such file or directory.
            in strnlen.c
    (gdb) bt
    #0  __strnlen (str=0x8f <Address 0x8f out of bounds>,
maxlen=<value optimized out>) at strnlen.c:47
    #1  0x00007ffff73318cd in __mbsnrtowcs (dst=0x7fffffffb960
L"\xffffb9a0翿\x404500", src=0x7fffffffcf78, nmc=8557680, len=<value
optimized out>,
        ps=0x7fffffffcfb0) at mbsnrtowcs.c:66
    #2  0x00007ffff72ef932 in _IO_vfprintf_internal (s=0x7fffffffd010,
format=<value optimized out>, ap=0x7fffffffd330) at vfprintf.c:1614
    #3  0x00007ffff7311432 in _IO_vsnprintf (
        string=0x829510
"H\215C\020H\213\\$\bH\213l$\020L\213d$\030L\213l$
L\213t$(L\213|$0H\203\304\070\303f\017\037D: \177", maxlen=<value
optimized out>,
        format=0x58aa28 "%s: %.*s%.*s\n", args=0x7fffffffd330) at
vsnprintf.c:120
    #4  0x00000000005569c6 in git_vsnprintf (
        str=0x829510
"H\215C\020H\213\\$\bH\213l$\020L\213d$\030L\213l$
L\213t$(L\213|$0H\203\304\070\303f\017\037D: \177", maxsize=572,
        format=0x58aa28 "%s: %.*s%.*s\n", ap=0x7fffffffd330) at snprintf.c:45
    #5  0x000000000053877f in strbuf_addf (sb=0x7fffffffd640,
fmt=0x58aa28 "%s: %.*s%.*s\n") at strbuf.c:203
    #6  0x000000000050267d in pp_user_info (what=0x58aa4e "Author",
fmt=CMIT_FMT_MEDIUM, sb=0x7fffffffd640,
        line=0x830675 "David_K\345gedal <davidk@lysator.liu.se>
1126078160 +0200\ncommitter Junio C Hamano <junkio@cox.net> 1126128590
-0700\n\n[PATCH] Simplify git script\n\nThe code for listing the
available subcommands was unnec"..., dmode=DATE_NORMAL,
encoding=0x57025d "UTF-8") at pretty.c:283
    #7  0x0000000000503be9 in pp_header (fmt=CMIT_FMT_MEDIUM,
abbrev=7, dmode=DATE_NORMAL, encoding=0x57025d "UTF-8",
commit=0x832cf8, msg_p=0x7fffffffd558,
        sb=0x7fffffffd640) at pretty.c:1077
    #8  0x0000000000503865 in pretty_print_commit
(fmt=CMIT_FMT_MEDIUM, commit=0x832cf8, sb=0x7fffffffd640,
context=0x7fffffffd5e8) at pretty.c:1219
    #9  0x00000000004ead8b in show_log (opt=0x7fffffffd858) at log-tree.c:508
    #10 0x00000000004eb1b2 in log_tree_diff_flush (opt=0x7fffffffd858)
at log-tree.c:557
    #11 0x00000000004eb5e9 in log_tree_diff (opt=0x7fffffffd858,
commit=0x832cf8, log=0x7fffffffd728) at log-tree.c:635
    #12 0x00000000004eb322 in log_tree_commit (opt=0x7fffffffd858,
commit=0x832cf8) at log-tree.c:658
    #13 0x000000000044ff0b in cmd_log_walk (rev=0x7fffffffd858) at log.c:372
    #14 0x0000000000450555 in cmd_show (argc=2, argv=0x7fffffffe070,
prefix=0x0) at log.c:561
    #15 0x00000000004058a7 in run_builtin (p=0x7bf708, argc=2,
argv=0x7fffffffe070) at git.c:278
    #16 0x0000000000404a54 in handle_internal_command (argc=2,
argv=0x7fffffffe070) at git.c:434
    #17 0x000000000040513e in run_argv (argcp=0x7fffffffdf78,
argv=0x7fffffffdf70) at git.c:478
    #18 0x0000000000404875 in main (argc=2, argv=0x7fffffffe070) at git.c:553
    (gdb) frame 4
    #4  0x00000000005569c6 in git_vsnprintf (
        str=0x829510
"H\215C\020H\213\\$\bH\213l$\020L\213d$\030L\213l$
L\213t$(L\213|$0H\203\304\070\303f\017\037D: \177", maxsize=572,
        format=0x58aa28 "%s: %.*s%.*s\n", ap=0x7fffffffd330) at snprintf.c:45
    45                      ret = vsnprintf(str,
maxsize-SNPRINTF_SIZE_CORR, format, ap);
    (gdb) p str
    $1 = 0x829510 "H\215C\020H\213\\$\bH\213l$\020L\213d$\030L\213l$
L\213t$(L\213|$0H\203\304\070\303f\017\037D: \177"
    (gdb) p maxsize
    $2 = 572
    (gdb) p SNPRINTF_SIZE_CORR
    No symbol "SNPRINTF_SIZE_CORR" in current context.
    (gdb) p format
    $3 = 0x58aa28 "%s: %.*s%.*s\n"
    (gdb) p ap
    $4 = (__va_list_tag *) 0x7fffffffd330
    (gdb) frame 6
    #6  0x000000000050267d in pp_user_info (what=0x58aa4e "Author",
fmt=CMIT_FMT_MEDIUM, sb=0x7fffffffd640,
        line=0x830675 "David_K\345gedal <davidk@lysator.liu.se>
1126078160 +0200\ncommitter Junio C Hamano <junkio@cox.net> 1126128590
-0700\n\n[PATCH] Simplify git script\n\nThe code for listing the
available subcommands was unnec"..., dmode=DATE_NORMAL,
encoding=0x57025d "UTF-8") at pretty.c:283
    283                     strbuf_addf(sb, "%s: %.*s%.*s\n", what,