git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII

From
Marcin Cieslak <saper@saper.info>
Date
Aug 30, 2010, 14:00 UTC
Message-ID
<alpine.BSF.2.00.1008301353260.95618@x.fncre.vasb>
In-Reply-To
<20100830134136.GA2315@burratino>
On Mon, 30 Aug 2010, Ævar Arnfjörð Bjarmason wrote:
> On Sun, Aug 29, 2010 at 20:45, Jonathan Nieder <jrnieder@gmail.com> wrote:
> A would be preferred for correctness, and with a fallback BSD printf()
> we can avoid the GNU libc bug, however that'll mean using LC_CTYPE,
> which'll have some small side-effects for the rest of the code.

The real problem is that you are probably using same functions (locale-enable) for the user-facing side as well as for the backend (talking to repository). Some projects decided to use some special encoding internally (like UCS-2 in case of Java and Python 2.x, Unicode ordinals in Python 3.x). Otherwise you may end up in some incompatibilities in the on-disk on on-network format. I don't think you want to keep telling all bug reporters for few years - "Can you try that again with env LANG=C, please?" :)

Bringing Unicode onboard means that simple strlen() is no longer what you normally think it does.

On Mon, 30 Aug 2010, Jonathan Nieder wrote:
Show 8 quoted lines
> Ævar Arnfjörð Bjarmason wrote:
>
>> We can even keep the "Content-Type: text/plain; charset=UTF-8\n" and
>> *not* use LC_CTYPE if we add a bind_textdomain_codeset("git", "UTF-8")
>> call to gettext.
>
> Oh!  I'd personally prefer to do that for now. :)  (Not because of the
> known printf problem but because I like to reduce possible unknowns.)

Well, in this case everybody will be force to have UTF-8 in output on-screen, not useful for people using ISO8859-*, KOI8-R and similar things...

--Marcin
Previous: Jonathan NiederNext: Jonathan Nieder
Message 14 of 20 in “Odd encoding issue with UTF-8 + gettext yields ? on non-ASCII”
  1. Ævar Arnfjörð BjarmasonAug 28, 2010
  2. Ævar Arnfjörð BjarmasonAug 28, 2010
  3. Jonathan NiederAug 28, 2010
  4. Jonathan NiederAug 28, 2010
  5. Marcin CieslakAug 28, 2010
  6. Jonathan NiederAug 28, 2010
  7. Ævar Arnfjörð BjarmasonAug 29, 2010
  8. Ævar Arnfjörð BjarmasonAug 29, 2010
  9. Jonathan NiederAug 30, 2010
  10. Ævar Arnfjörð BjarmasonAug 29, 2010
  11. Jonathan NiederAug 29, 2010
  12. Ævar Arnfjörð BjarmasonAug 30, 2010
  13. Jonathan NiederAug 30, 2010
  14. Marcin CieslakAug 30, 2010
  15. Jonathan NiederAug 30, 2010
  16. Ævar Arnfjörð BjarmasonAug 30, 2010
  17. Ævar Arnfjörð BjarmasonAug 30, 2010
  18. Ævar Arnfjörð BjarmasonAug 30, 2010
  19. Jonathan NiederAug 28, 2010
  20. Marcin CieslakAug 28, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.