git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 1/1] gettext: always use UTF-8 on native Windows

From
Karsten Blees via GitGitGadget <gitgitgadget@gmail.com>
Date
Jul 3, 2019, 20:46 UTC
Message-ID
<2d2253faef14e5157f8aac4534d9ac9640f3d5fa.1562186762.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.217.v2.git.gitgitgadget@gmail.com>
From: Karsten Blees <blees@dcon.de>

On native Windows, Git exclusively uses UTF-8 for console output (both with MinTTY and native Win32 Console). Gettext uses `setlocale()` to determine the output encoding for translated text, however, MSVCRT's `setlocale()` does not support UTF-8. As a result, translated text is encoded in system encoding (as per `GetAPC()`), and non-ASCII chars are mangled in console output.

Side note: There is actually a code page for UTF-8: 65001. In practice, it does not work as expected at least on Windows 7, though, so we cannot use it in Git. Besides, if we overrode the code page, any process spawned from Git would inherit that code page (as opposed to the code page configured for the current user), which would quite possibly break e.g. diff or merge helpers. So we really cannot override the code page.

In `init_gettext_charset()`, Git calls gettext's `bind_textdomain_codeset()` with the character set obtained via `locale_charset()`; Let's override that latter function to force the encoding to UTF-8 on native Windows.

In Git for Windows' SDK, there is a `libcharset.h` and therefore we define `HAVE_LIBCHARSET_H` in the MINGW-specific section in `config.mak.uname`, therefore we need to add the override before that conditionally-compiled code block.

Rather than simply defining `locale_charset()` to return the string `"UTF-8"`, though, we are careful not to break `LC_ALL=C`: the `ab/no-kwset` patch series, for example, needs to have a way to prevent Git from expecting UTF-8-encoded input.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 gettext.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/gettext.c b/gettext.c
index d4021d690c..3f2aca5c3b 100644
--- a/gettext.c
+++ b/gettext.c
@@ -12,7 +12,25 @@
 #ifndef NO_GETTEXT
 #	include <locale.h>
 #	include <libintl.h>
-#	ifdef HAVE_LIBCHARSET_H
+#	ifdef GIT_WINDOWS_NATIVE
+
+static const char *locale_charset(void)
+{
+	const char *env = getenv("LC_ALL"), *dot;
+
+	if (!env || !*env)
+		env = getenv("LC_CTYPE");
+	if (!env || !*env)
+		env = getenv("LANG");
+
+	if (!env)
+		return "UTF-8";
+
+	dot = strchr(env, '.');
+	return !dot ? env : dot + 1;
+}
+
+#	elif defined HAVE_LIBCHARSET_H
 #		include <libcharset.h>
 #	else
 #		include <langinfo.h>
-- 
gitgitgadget
Previous: Johannes Schindelin via GitGitGadgetNext: Ævar Arnfjörð Bjarmason
Message 6 of 9 in “gettext(windows): always use UTF-8”
  1. 0/1 gettext(windows): always use UTF-8Johannes Schindelin via GitGitGadget, Jun 27, 2019
  2. 1/1 gettext: always use UTF-8 on native WindowsKarsten Blees via GitGitGadget, Jun 27, 2019
  3. Johannes SchindelinJul 3, 2019
  4. Junio C HamanoJul 3, 2019
  5. 0/1 gettext(windows): always use UTF-8Johannes Schindelin via GitGitGadget, Jul 3, 2019
  6. 1/1 gettext: always use UTF-8 on native WindowsKarsten Blees via GitGitGadget, Jul 3, 2019
  7. Ævar Arnfjörð BjarmasonJul 4, 2019
  8. Johannes SchindelinJul 8, 2019
  9. Junio C HamanoJul 8, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.