{"thread":{"id":"36854","subject":"[PATCH 1/5] Support Unicode console output on Windows","startedAt":"2014-06-06T13:42:48Z","lastAt":"2014-06-13T22:09:06Z","messageCount":22,"participants":["Stepan Kasal","Karsten Blees","Peter Krefting","Johannes Sixt"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"243458","messageId":"1402062173-9602-1-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":null,"subject":"[PATCH 0/5] First part of Unicode console support for msysgit","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T13:42:48Z","receivedAt":"2014-06-06T13:42:48Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"Hello,\n\nthis is first part of the unicode support pathes from msysgit.\n\nThe first three patches originate in Jun 2010, though some fixups from 2012\nhave been squashed in.\nThe fourth one is just a trivial prerequisite for\nthe last one, that was written in Jan 2012, with a fixup from Mar 2012.\n\nRegards,\n\tStepan\n\nKarsten Blees (5):\n Support Unicode console output on Windows\n Detect console streams more reliably on Windows\n Warn if the Windows console font doesn't support Unicode\n Win32: move main macro to a function\n Win32: Thread-safe windows console output\n\n compat/mingw.c | 24 ++-\n compat/mingw.h | 24 +--\n compat/winansi.c | 446 ++++++++++++++++++++++++++++++++++++++++---------------\n 3 files changed, 356 insertions(+), 138 deletions(-)\n\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243457","messageId":"1402062173-9602-2-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402062173-9602-1-git-send-email-kasal@ucw.cz","subject":"[PATCH 1/5] Support Unicode console output on Windows","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T13:42:49Z","receivedAt":"2014-06-06T13:42:49Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \n\nWriteConsoleW seems to be the only way to reliably print unicode to the\nconsole (without weird code page conversions).\n\nAlso redirects vfprintf to the winansi.c version.\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.h | 2 ++\n compat/winansi.c | 26 ++++++++++++++++++++------\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 3eaf822..a465d1e 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -320,9 +320,11 @@ int mingw_raise(int sig);\n int winansi_fputs(const char *str, FILE *stream);\n int winansi_printf(const char *format, ...) __attribute__((format (printf, 1, 2)));\n int winansi_fprintf(FILE *stream, const char *format, ...) __attribute__((format (printf, 2, 3)));\n+int winansi_vfprintf(FILE *stream, const char *format, va_list list);\n #define fputs winansi_fputs\n #define printf(...) winansi_printf(__VA_ARGS__)\n #define fprintf(...) winansi_fprintf(__VA_ARGS__)\n+#define vfprintf winansi_vfprintf\n \n /*\n * git specific compatibility\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex dedce21..abe0fea 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -3,6 +3,7 @@\n */\n \n #include \"../git-compat-util.h\"\n+#include \n \n /*\n Functions to be wrapped:\n@@ -10,6 +11,7 @@\n #undef printf\n #undef fprintf\n #undef fputs\n+#undef vfprintf\n /* TODO: write */\n \n /*\n@@ -46,6 +48,18 @@ static void init(void)\n \tinitialized = 1;\n }\n \n+static int write_console(const char *str, size_t len)\n+{\n+\t/* convert utf-8 to utf-16, write directly to console */\n+\tint wlen = MultiByteToWideChar(CP_UTF8, 0, str, len, NULL, 0);\n+\twchar_t *wbuf = (wchar_t *) alloca(wlen * sizeof(wchar_t));\n+\tMultiByteToWideChar(CP_UTF8, 0, str, len, wbuf, wlen);\n+\n+\tWriteConsoleW(console, wbuf, wlen, NULL, NULL);\n+\n+\t/* return original (utf-8 encoded) length */\n+\treturn len;\n+}\n \n #define FOREGROUND_ALL (FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE)\n #define BACKGROUND_ALL (BACKGROUND_RED | BACKGROUND_GREEN | BACKGROUND_BLUE)\n@@ -245,13 +259,15 @@ static int ansi_emulate(const char *str, FILE *stream)\n \tint rv = 0;\n \tconst char *pos = str;\n \n+\tfflush(stream);\n+\n \twhile (*pos) {\n \t\tpos = strstr(str, \"\\033[\");\n \t\tif (pos) {\n \t\t\tsize_t len = pos - str;\n \n \t\t\tif (len) {\n-\t\t\t\tsize_t out_len = fwrite(str, 1, len, stream);\n+\t\t\t\tsize_t out_len = write_console(str, len);\n \t\t\t\trv += out_len;\n \t\t\t\tif (out_len < len)\n \t\t\t\t\treturn rv;\n@@ -260,14 +276,12 @@ static int ansi_emulate(const char *str, FILE *stream)\n \t\t\tstr = pos + 2;\n \t\t\trv += 2;\n \n-\t\t\tfflush(stream);\n-\n \t\t\tpos = set_attr(str);\n \t\t\trv += pos - str;\n \t\t\tstr = pos;\n \t\t} else {\n-\t\t\trv += strlen(str);\n-\t\t\tfputs(str, stream);\n+\t\t\tsize_t len = strlen(str);\n+\t\t\trv += write_console(str, len);\n \t\t\treturn rv;\n \t\t}\n \t}\n@@ -294,7 +308,7 @@ int winansi_fputs(const char *str, FILE *stream)\n \t\treturn EOF;\n }\n \n-static int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n+int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n {\n \tint len, rv;\n \tchar small_buf[256];\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243462","messageId":"1402062173-9602-3-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402062173-9602-1-git-send-email-kasal@ucw.cz","subject":"[PATCH 2/5] Detect console streams more reliably on Windows","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T13:42:50Z","receivedAt":"2014-06-06T13:42:50Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \n\nGetStdHandle(STD_OUTPUT_HANDLE) doesn't work for stderr if stdout is\nredirected. Use _get_osfhandle of the FILE* instead.\n\n_isatty() is true for all character devices (including parallel and serial\nports). Check return value of GetConsoleScreenBufferInfo instead to\nreliably detect console handles (also don't initialize internal state from\nan uninitialized CONSOLE_SCREEN_BUFFER_INFO structure if the function\nfails).\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Stepan Kasal \n---\n compat/winansi.c | 50 ++++++++++++++++++++++++++------------------------\n 1 file changed, 26 insertions(+), 24 deletions(-)\n\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex abe0fea..c4be401 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -25,27 +25,39 @@ static HANDLE console;\n static WORD plain_attr;\n static WORD attr;\n static int negative;\n+static FILE *last_stream = NULL;\n \n-static void init(void)\n+static int is_console(FILE *stream)\n {\n \tCONSOLE_SCREEN_BUFFER_INFO sbi;\n+\tHANDLE hcon;\n \n \tstatic int initialized = 0;\n-\tif (initialized)\n-\t\treturn;\n \n-\tconsole = GetStdHandle(STD_OUTPUT_HANDLE);\n-\tif (console == INVALID_HANDLE_VALUE)\n-\t\tconsole = NULL;\n+\t/* use cached value if stream hasn't changed */\n+\tif (stream == last_stream)\n+\t\treturn console != NULL;\n \n-\tif (!console)\n-\t\treturn;\n+\tlast_stream = stream;\n+\tconsole = NULL;\n \n-\tGetConsoleScreenBufferInfo(console, &sbi);\n-\tattr = plain_attr = sbi.wAttributes;\n-\tnegative = 0;\n+\t/* get OS handle of the stream */\n+\thcon = (HANDLE) _get_osfhandle(_fileno(stream));\n+\tif (hcon == INVALID_HANDLE_VALUE)\n+\t\treturn 0;\n+\n+\t/* check if its a handle to a console output screen buffer */\n+\tif (!GetConsoleScreenBufferInfo(hcon, &sbi))\n+\t\treturn 0;\n+\n+\tif (!initialized) {\n+\t\tattr = plain_attr = sbi.wAttributes;\n+\t\tnegative = 0;\n+\t\tinitialized = 1;\n+\t}\n \n-\tinitialized = 1;\n+\tconsole = hcon;\n+\treturn 1;\n }\n \n static int write_console(const char *str, size_t len)\n@@ -292,12 +304,7 @@ int winansi_fputs(const char *str, FILE *stream)\n {\n \tint rv;\n \n-\tif (!isatty(fileno(stream)))\n-\t\treturn fputs(str, stream);\n-\n-\tinit();\n-\n-\tif (!console)\n+\tif (!is_console(stream))\n \t\treturn fputs(str, stream);\n \n \trv = ansi_emulate(str, stream);\n@@ -315,12 +322,7 @@ int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n \tchar *buf = small_buf;\n \tva_list cp;\n \n-\tif (!isatty(fileno(stream)))\n-\t\tgoto abort;\n-\n-\tinit();\n-\n-\tif (!console)\n+\tif (!is_console(stream))\n \t\tgoto abort;\n \n \tva_copy(cp, list);\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243461","messageId":"1402062173-9602-4-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402062173-9602-1-git-send-email-kasal@ucw.cz","subject":"[PATCH 3/5] Warn if the Windows console font doesn't support Unicode","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T13:42:51Z","receivedAt":"2014-06-06T13:42:51Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \n\nUnicode console output won't display correctly with default settings\nbecause the default console font (\"Terminal\") only supports the system's\nOEM charset. Unfortunately, this is a user specific setting, so it cannot\nbe easily fixed by e.g. some registry tricks in the setup program.\n\nThis change prints a warning on exit if console output contained non-ascii\ncharacters and the console font is supposedly not a TrueType font (which\nusually have decent Unicode support).\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Stepan Kasal \n---\n compat/winansi.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 66 insertions(+)\n\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex c4be401..bec6713 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -2,8 +2,11 @@\n * Copyright 2008 Peter Harris \n */\n \n+#undef NOGDI\n #include \"../git-compat-util.h\"\n #include \n+#include \n+#include \n \n /*\n Functions to be wrapped:\n@@ -27,6 +30,62 @@ static WORD attr;\n static int negative;\n static FILE *last_stream = NULL;\n \n+#ifdef __MINGW32__\n+typedef struct _CONSOLE_FONT_INFOEX {\n+\tULONG cbSize;\n+\tDWORD nFont;\n+\tCOORD dwFontSize;\n+\tUINT FontFamily;\n+\tUINT FontWeight;\n+\tWCHAR FaceName[LF_FACESIZE];\n+} CONSOLE_FONT_INFOEX, *PCONSOLE_FONT_INFOEX;\n+#endif\n+\n+typedef BOOL (WINAPI *PGETCURRENTCONSOLEFONTEX)(HANDLE, BOOL,\n+\t\tPCONSOLE_FONT_INFOEX);\n+\n+static void print_font_warning(void)\n+{\n+\twarning(\"Your console font probably doesn\\'t support Unicode. If \"\n+\t\t\"you experience strange characters in the output, consider \"\n+\t\t\"switching to a TrueType font such as Lucida Console!\");\n+}\n+\n+static void check_truetype_font(void)\n+{\n+\tstatic int truetype_font_checked;\n+\tDWORD fontFamily = 0;\n+\tPGETCURRENTCONSOLEFONTEX pGetCurrentConsoleFontEx;\n+\n+\t/* don't do this twice */\n+\tif (truetype_font_checked)\n+\t\treturn;\n+\ttruetype_font_checked = 1;\n+\n+\t/* GetCurrentConsoleFontEx is available since Vista */\n+\tpGetCurrentConsoleFontEx = (PGETCURRENTCONSOLEFONTEX) GetProcAddress(\n+\t\t\tGetModuleHandle(\"kernel32.dll\"), \"GetCurrentConsoleFontEx\");\n+\tif (pGetCurrentConsoleFontEx) {\n+\t\tCONSOLE_FONT_INFOEX cfi;\n+\t\tcfi.cbSize = sizeof(cfi);\n+\t\tif (pGetCurrentConsoleFontEx(console, 0, &cfi))\n+\t\t\tfontFamily = cfi.FontFamily;\n+\t} else {\n+\t\t/* pre-Vista: check default console font in registry */\n+\t\tHKEY hkey;\n+\t\tif (ERROR_SUCCESS == RegOpenKeyExA(HKEY_CURRENT_USER, \"Console\", 0,\n+\t\t\t\tKEY_READ, &hkey)) {\n+\t\t\tDWORD size = sizeof(fontFamily);\n+\t\t\tRegQueryValueExA(hkey, \"FontFamily\", NULL, NULL,\n+\t\t\t\t\t(LPVOID) &fontFamily, &size);\n+\t\t\tRegCloseKey(hkey);\n+\t\t}\n+\t}\n+\n+\tif (!(fontFamily & TMPF_TRUETYPE))\n+\t\tatexit(print_font_warning);\n+}\n+\n static int is_console(FILE *stream)\n {\n \tCONSOLE_SCREEN_BUFFER_INFO sbi;\n@@ -69,6 +128,13 @@ static int write_console(const char *str, size_t len)\n \n \tWriteConsoleW(console, wbuf, wlen, NULL, NULL);\n \n+\t/*\n+\t * if non-ascii characters are printed, check that the current console\n+\t * font supports this\n+\t */\n+\tif (wlen != len)\n+\t\tcheck_truetype_font();\n+\n \t/* return original (utf-8 encoded) length */\n \treturn len;\n }\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243460","messageId":"1402062173-9602-5-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402062173-9602-1-git-send-email-kasal@ucw.cz","subject":"[PATCH 4/5] Win32: move main macro to a function","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T13:42:52Z","receivedAt":"2014-06-06T13:42:52Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \n\nThe code in the MinGW main macro is getting more and more complex, move to\na separate initialization function for readabiliy and extensibility.\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Erik Faye-Lund \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.c | 15 +++++++++++++++\n compat/mingw.h | 14 ++++----------\n 2 files changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex a0e13bc..c03bafa 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -1847,3 +1847,18 @@ int mingw_offset_1st_component(const char *path)\n \n \treturn offset + is_dir_sep(path[offset]);\n }\n+\n+void mingw_startup()\n+{\n+\t/* copy executable name to argv[0] */\n+\t__argv[0] = xstrdup(_pgmptr);\n+\n+\t/* initialize critical section for waitpid pinfo_t list */\n+\tInitializeCriticalSection(&pinfo_cs);\n+\n+\t/* set up default file mode and file modes for stdin/out/err */\n+\t_fmode = _O_BINARY;\n+\t_setmode(_fileno(stdin), _O_BINARY);\n+\t_setmode(_fileno(stdout), _O_BINARY);\n+\t_setmode(_fileno(stderr), _O_BINARY);\n+}\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex a465d1e..96d15ca 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -365,22 +365,16 @@ void free_environ(char **env);\n extern CRITICAL_SECTION pinfo_cs;\n \n /*\n- * A replacement of main() that ensures that argv[0] has a path\n- * and that default fmode and std(in|out|err) are in binary mode\n+ * A replacement of main() that adds win32 specific initialization.\n */\n \n+void mingw_startup();\n #define main(c,v) dummy_decl_mingw_main(); \\\n static int mingw_main(c,v); \\\n int main(int argc, char **argv) \\\n { \\\n-\textern CRITICAL_SECTION pinfo_cs; \\\n-\t_fmode = _O_BINARY; \\\n-\t_setmode(_fileno(stdin), _O_BINARY); \\\n-\t_setmode(_fileno(stdout), _O_BINARY); \\\n-\t_setmode(_fileno(stderr), _O_BINARY); \\\n-\targv[0] = xstrdup(_pgmptr); \\\n-\tInitializeCriticalSection(&pinfo_cs); \\\n-\treturn mingw_main(argc, argv); \\\n+\tmingw_startup(); \\\n+\treturn mingw_main(__argc, __argv); \\\n } \\\n static int mingw_main(c,v)\n \n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243459","messageId":"1402062173-9602-6-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402062173-9602-1-git-send-email-kasal@ucw.cz","subject":"[PATCH 5/5] Win32: Thread-safe windows console output","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T13:42:53Z","receivedAt":"2014-06-06T13:42:53Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \n\nWinansi.c has many static variables that are accessed and modified from\nthe [v][f]printf / fputs functions overridden in the file. This may cause\nmulti threaded git commands that print to the console to produce corrupted\noutput or even crash.\n\nAdditionally, winansi.c doesn't override all functions that can be used to\nprint to the console (e.g. fwrite, write, fputc are missing), so that ANSI\nescapes don't work properly for some git commands (e.g. git-grep).\n\nInstead of doing ANSI emulation in just a few wrapped functions on top of\nthe IO API, let's plug into the IO system and take advantage of the thread\nsafety inherent to the IO system.\n\nRedirect stdout and stderr to a pipe if they point to the console. A\nbackground thread reads from the pipe, handles ANSI escape sequences and\nUTF-8 to UTF-16 conversion, then writes to the console.\n\nThe pipe-based stdout and stderr replacements must be set to unbuffered, as\nMSVCRT doesn't support line buffering and fully buffered streams are\ninappropriate for console output.\n\nDue to the byte-oriented pipe, ANSI escape sequences and multi-byte UTF-8\nsequences can no longer be expected to arrive in one piece. Replace the\nstring-based ansi_emulate() with a simple stateful parser (this also fixes\ncolored diff hunk headers, which were broken as of commit 2efcc977).\n\nOverride isatty to return true for the pipes redirecting to the console.\n\nExec/spawn obtain the original console handle to pass to the next process\nvia winansi_get_osfhandle().\n\nAll other overrides are gone, the default stdio implementations work as\nexpected with the piped stdout/stderr descriptors.\n\nGlobal variables are either initialized on startup (single threaded) or\nexclusively modified by the background thread. Threads communicate through\nthe pipe, no further synchronization is necessary.\n\nThe background thread is terminated by disonnecting the pipe after flushing\nthe stdio and pipe buffers. This doesn't work for anonymous pipes (created\nvia CreatePipe), as DisconnectNamedPipe only works on the read end, which\ndiscards remaining data. Thus we have to setup the pipe manually, with the\nwrite end beeing the server (opened with CreateNamedPipe) and the read end\nthe client (opened with CreateFile).\n\nLimitations: doesn't track reopened or duped file descriptors, i.e.:\n- fdopen(1/2) returns fully buffered streams\n- dup(1/2), dup2(1/2) returns normal pipe descriptors (i.e. isatty() =\n false, winansi_get_osfhandle won't return the original console handle)\n\nCurrently, only the git-format-patch command uses xfdopen(xdup(1)) (see\n\"realstdout\" in builtin/log.c), but works well with these limitations.\n\nMany thanks to Atsushi Nakagawa for suggesting and\nreviewing the thread-exit-mechanism.\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.c | 9 +-\n compat/mingw.h | 12 +-\n compat/winansi.c | 402 ++++++++++++++++++++++++++++++++++++-------------------\n 3 files changed, 274 insertions(+), 149 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex c03bafa..831043e 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -865,9 +865,9 @@ static pid_t mingw_spawnve_fd(const char *cmd, const char **argv, char **env,\n \tmemset(&si, 0, sizeof(si));\n \tsi.cb = sizeof(si);\n \tsi.dwFlags = STARTF_USESTDHANDLES;\n-\tsi.hStdInput = (HANDLE) _get_osfhandle(fhin);\n-\tsi.hStdOutput = (HANDLE) _get_osfhandle(fhout);\n-\tsi.hStdError = (HANDLE) _get_osfhandle(fherr);\n+\tsi.hStdInput = winansi_get_osfhandle(fhin);\n+\tsi.hStdOutput = winansi_get_osfhandle(fhout);\n+\tsi.hStdError = winansi_get_osfhandle(fherr);\n \n \t/* concatenate argv, quoting args as we go */\n \tstrbuf_init(&args, 0);\n@@ -1861,4 +1861,7 @@ void mingw_startup()\n \t_setmode(_fileno(stdin), _O_BINARY);\n \t_setmode(_fileno(stdout), _O_BINARY);\n \t_setmode(_fileno(stderr), _O_BINARY);\n+\n+\t/* initialize Unicode console */\n+\twinansi_init();\n }\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 96d15ca..82e75d3 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -317,14 +317,10 @@ int mingw_raise(int sig);\n * ANSI emulation wrappers\n */\n \n-int winansi_fputs(const char *str, FILE *stream);\n-int winansi_printf(const char *format, ...) __attribute__((format (printf, 1, 2)));\n-int winansi_fprintf(FILE *stream, const char *format, ...) __attribute__((format (printf, 2, 3)));\n-int winansi_vfprintf(FILE *stream, const char *format, va_list list);\n-#define fputs winansi_fputs\n-#define printf(...) winansi_printf(__VA_ARGS__)\n-#define fprintf(...) winansi_fprintf(__VA_ARGS__)\n-#define vfprintf winansi_vfprintf\n+void winansi_init(void);\n+int winansi_isatty(int fd);\n+HANDLE winansi_get_osfhandle(int fd);\n+#define isatty winansi_isatty\n \n /*\n * git specific compatibility\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex bec6713..598fa1a 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -4,18 +4,13 @@\n \n #undef NOGDI\n #include \"../git-compat-util.h\"\n-#include \n #include \n #include \n \n /*\n Functions to be wrapped:\n */\n-#undef printf\n-#undef fprintf\n-#undef fputs\n-#undef vfprintf\n-/* TODO: write */\n+#undef isatty\n \n /*\n ANSI codes used by git: m, K\n@@ -28,7 +23,10 @@ static HANDLE console;\n static WORD plain_attr;\n static WORD attr;\n static int negative;\n-static FILE *last_stream = NULL;\n+static int non_ascii_used = 0;\n+static HANDLE hthread, hread, hwrite;\n+static HANDLE hwrite1 = INVALID_HANDLE_VALUE, hwrite2 = INVALID_HANDLE_VALUE;\n+static HANDLE hconsole1, hconsole2;\n \n #ifdef __MINGW32__\n typedef struct _CONSOLE_FONT_INFOEX {\n@@ -44,27 +42,19 @@ typedef struct _CONSOLE_FONT_INFOEX {\n typedef BOOL (WINAPI *PGETCURRENTCONSOLEFONTEX)(HANDLE, BOOL,\n \t\tPCONSOLE_FONT_INFOEX);\n \n-static void print_font_warning(void)\n+static void warn_if_raster_font(void)\n {\n-\twarning(\"Your console font probably doesn\\'t support Unicode. If \"\n-\t\t\"you experience strange characters in the output, consider \"\n-\t\t\"switching to a TrueType font such as Lucida Console!\");\n-}\n-\n-static void check_truetype_font(void)\n-{\n-\tstatic int truetype_font_checked;\n \tDWORD fontFamily = 0;\n \tPGETCURRENTCONSOLEFONTEX pGetCurrentConsoleFontEx;\n \n-\t/* don't do this twice */\n-\tif (truetype_font_checked)\n+\t/* don't bother if output was ascii only */\n+\tif (!non_ascii_used)\n \t\treturn;\n-\ttruetype_font_checked = 1;\n \n \t/* GetCurrentConsoleFontEx is available since Vista */\n \tpGetCurrentConsoleFontEx = (PGETCURRENTCONSOLEFONTEX) GetProcAddress(\n-\t\t\tGetModuleHandle(\"kernel32.dll\"), \"GetCurrentConsoleFontEx\");\n+\t\t\tGetModuleHandle(\"kernel32.dll\"),\n+\t\t\t\"GetCurrentConsoleFontEx\");\n \tif (pGetCurrentConsoleFontEx) {\n \t\tCONSOLE_FONT_INFOEX cfi;\n \t\tcfi.cbSize = sizeof(cfi);\n@@ -73,8 +63,8 @@ static void check_truetype_font(void)\n \t} else {\n \t\t/* pre-Vista: check default console font in registry */\n \t\tHKEY hkey;\n-\t\tif (ERROR_SUCCESS == RegOpenKeyExA(HKEY_CURRENT_USER, \"Console\", 0,\n-\t\t\t\tKEY_READ, &hkey)) {\n+\t\tif (ERROR_SUCCESS == RegOpenKeyExA(HKEY_CURRENT_USER, \"Console\",\n+\t\t\t\t0, KEY_READ, &hkey)) {\n \t\t\tDWORD size = sizeof(fontFamily);\n \t\t\tRegQueryValueExA(hkey, \"FontFamily\", NULL, NULL,\n \t\t\t\t\t(LPVOID) &fontFamily, &size);\n@@ -82,61 +72,65 @@ static void check_truetype_font(void)\n \t\t}\n \t}\n \n-\tif (!(fontFamily & TMPF_TRUETYPE))\n-\t\tatexit(print_font_warning);\n+\tif (!(fontFamily & TMPF_TRUETYPE)) {\n+\t\tconst wchar_t *msg = L\"\\nWarning: Your console font probably \"\n+\t\t\tL\"doesn\\'t support Unicode. If you experience strange \"\n+\t\t\tL\"characters in the output, consider switching to a \"\n+\t\t\tL\"TrueType font such as Lucida Console!\\n\";\n+\t\tDWORD dummy;\n+\t\tWriteConsoleW(console, msg, wcslen(msg), &dummy, NULL);\n+\t}\n }\n \n-static int is_console(FILE *stream)\n+static int is_console(int fd)\n {\n \tCONSOLE_SCREEN_BUFFER_INFO sbi;\n \tHANDLE hcon;\n \n \tstatic int initialized = 0;\n \n-\t/* use cached value if stream hasn't changed */\n-\tif (stream == last_stream)\n-\t\treturn console != NULL;\n-\n-\tlast_stream = stream;\n-\tconsole = NULL;\n-\n-\t/* get OS handle of the stream */\n-\thcon = (HANDLE) _get_osfhandle(_fileno(stream));\n+\t/* get OS handle of the file descriptor */\n+\thcon = (HANDLE) _get_osfhandle(fd);\n \tif (hcon == INVALID_HANDLE_VALUE)\n \t\treturn 0;\n \n+\t/* check if its a device (i.e. console, printer, serial port) */\n+\tif (GetFileType(hcon) != FILE_TYPE_CHAR)\n+\t\treturn 0;\n+\n \t/* check if its a handle to a console output screen buffer */\n \tif (!GetConsoleScreenBufferInfo(hcon, &sbi))\n \t\treturn 0;\n \n+\t/* initialize attributes */\n \tif (!initialized) {\n \t\tattr = plain_attr = sbi.wAttributes;\n \t\tnegative = 0;\n \t\tinitialized = 1;\n \t}\n \n-\tconsole = hcon;\n \treturn 1;\n }\n \n-static int write_console(const char *str, size_t len)\n+#define BUFFER_SIZE 4096\n+#define MAX_PARAMS 16\n+\n+static void write_console(unsigned char *str, size_t len)\n {\n-\t/* convert utf-8 to utf-16, write directly to console */\n-\tint wlen = MultiByteToWideChar(CP_UTF8, 0, str, len, NULL, 0);\n-\twchar_t *wbuf = (wchar_t *) alloca(wlen * sizeof(wchar_t));\n-\tMultiByteToWideChar(CP_UTF8, 0, str, len, wbuf, wlen);\n+\t/* only called from console_thread, so a static buffer will do */\n+\tstatic wchar_t wbuf[2 * BUFFER_SIZE + 1];\n+\tDWORD dummy;\n \n-\tWriteConsoleW(console, wbuf, wlen, NULL, NULL);\n+\t/* convert utf-8 to utf-16 */\n+\tint wlen = MultiByteToWideChar(CP_UTF8, 0, (char*) str, len, wbuf,\n+\t\t\tARRAY_SIZE(wbuf));\n \n-\t/*\n-\t * if non-ascii characters are printed, check that the current console\n-\t * font supports this\n-\t */\n-\tif (wlen != len)\n-\t\tcheck_truetype_font();\n+\t/* write directly to console */\n+\tWriteConsoleW(console, wbuf, wlen, &dummy, NULL);\n \n-\t/* return original (utf-8 encoded) length */\n-\treturn len;\n+\t/* remember if non-ascii characters are printed */\n+\tif (wlen != len)\n+\t\tnon_ascii_used = 1;\n }\n \n #define FOREGROUND_ALL (FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE)\n@@ -182,18 +176,13 @@ static void erase_in_line(void)\n \t\t&dummy);\n }\n \n-\n-static const char *set_attr(const char *str)\n+static void set_attr(char func, const int *params, int paramlen)\n {\n-\tconst char *func;\n-\tsize_t len = strspn(str, \"0123456789;\");\n-\tfunc = str + len;\n-\n-\tswitch (*func) {\n+\tint i;\n+\tswitch (func) {\n \tcase 'm':\n-\t\tdo {\n-\t\t\tlong val = strtol(str, (char **)&str, 10);\n-\t\t\tswitch (val) {\n+\t\tfor (i = 0; i < paramlen; i++) {\n+\t\t\tswitch (params[i]) {\n \t\t\tcase 0: /* reset */\n \t\t\t\tattr = plain_attr;\n \t\t\t\tnegative = 0;\n@@ -316,9 +305,7 @@ static const char *set_attr(const char *str)\n \t\t\t\t/* Unsupported code */\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\tstr++;\n-\t\t} while (*(str-1) == ';');\n-\n+\t\t}\n \t\tset_console_attr();\n \t\tbreak;\n \tcase 'K':\n@@ -328,112 +315,251 @@ static const char *set_attr(const char *str)\n \t\t/* Unsupported code */\n \t\tbreak;\n \t}\n-\n-\treturn func + 1;\n }\n \n-static int ansi_emulate(const char *str, FILE *stream)\n-{\n-\tint rv = 0;\n-\tconst char *pos = str;\n+enum {\n+\tTEXT = 0, ESCAPE = 033, BRACKET = '['\n+};\n \n-\tfflush(stream);\n+static DWORD WINAPI console_thread(LPVOID unused)\n+{\n+\tunsigned char buffer[BUFFER_SIZE];\n+\tDWORD bytes;\n+\tint start, end = 0, c, parampos = 0, state = TEXT;\n+\tint params[MAX_PARAMS];\n+\n+\twhile (1) {\n+\t\t/* read next chunk of bytes from the pipe */\n+\t\tif (!ReadFile(hread, buffer + end, BUFFER_SIZE - end, &bytes,\n+\t\t\t\tNULL)) {\n+\t\t\t/* exit if pipe has been closed or disconnected */\n+\t\t\tif (GetLastError() == ERROR_PIPE_NOT_CONNECTED ||\n+\t\t\t\t\tGetLastError() == ERROR_BROKEN_PIPE)\n+\t\t\t\tbreak;\n+\t\t\t/* ignore other errors */\n+\t\t\tcontinue;\n+\t\t}\n \n-\twhile (*pos) {\n-\t\tpos = strstr(str, \"\\033[\");\n-\t\tif (pos) {\n-\t\t\tsize_t len = pos - str;\n+\t\t/* scan the bytes and handle ANSI control codes */\n+\t\tbytes += end;\n+\t\tstart = end = 0;\n+\t\twhile (end < bytes) {\n+\t\t\tc = buffer[end++];\n+\t\t\tswitch (state) {\n+\t\t\tcase TEXT:\n+\t\t\t\tif (c == ESCAPE) {\n+\t\t\t\t\t/* print text seen so far */\n+\t\t\t\t\tif (end - 1 > start)\n+\t\t\t\t\t\twrite_console(buffer + start,\n+\t\t\t\t\t\t\tend - 1 - start);\n+\n+\t\t\t\t\t/* then start parsing escape sequence */\n+\t\t\t\t\tstart = end - 1;\n+\t\t\t\t\tmemset(params, 0, sizeof(params));\n+\t\t\t\t\tparampos = 0;\n+\t\t\t\t\tstate = ESCAPE;\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\n+\t\t\tcase ESCAPE:\n+\t\t\t\t/* continue if \"\\033[\", otherwise bail out */\n+\t\t\t\tstate = (c == BRACKET) ? BRACKET : TEXT;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase BRACKET:\n+\t\t\t\t/* parse [0-9;]* into array of parameters */\n+\t\t\t\tif (c >= '0' && c <= '9') {\n+\t\t\t\t\tparams[parampos] *= 10;\n+\t\t\t\t\tparams[parampos] += c - '0';\n+\t\t\t\t} else if (c == ';') {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * next parameter, bail out if out of\n+\t\t\t\t\t * bounds\n+\t\t\t\t\t */\n+\t\t\t\t\tparampos++;\n+\t\t\t\t\tif (parampos >= MAX_PARAMS)\n+\t\t\t\t\t\tstate = TEXT;\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * end of escape sequence, change\n+\t\t\t\t\t * console attributes\n+\t\t\t\t\t */\n+\t\t\t\t\tset_attr(c, params, parampos + 1);\n+\t\t\t\t\tstart = end;\n+\t\t\t\t\tstate = TEXT;\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n \n-\t\t\tif (len) {\n-\t\t\t\tsize_t out_len = write_console(str, len);\n-\t\t\t\trv += out_len;\n-\t\t\t\tif (out_len < len)\n-\t\t\t\t\treturn rv;\n+\t\t/* print remaining text unless parsing an escape sequence */\n+\t\tif (state == TEXT && end > start) {\n+\t\t\t/* check for incomplete UTF-8 sequences and fix end */\n+\t\t\tif (buffer[end - 1] >= 0x80) {\n+\t\t\t\tif (buffer[end -1] >= 0xc0)\n+\t\t\t\t\tend--;\n+\t\t\t\telse if (end - 1 > start &&\n+\t\t\t\t\t\tbuffer[end - 2] >= 0xe0)\n+\t\t\t\t\tend -= 2;\n+\t\t\t\telse if (end - 2 > start &&\n+\t\t\t\t\t\tbuffer[end - 3] >= 0xf0)\n+\t\t\t\t\tend -= 3;\n \t\t\t}\n \n-\t\t\tstr = pos + 2;\n-\t\t\trv += 2;\n+\t\t\t/* print remaining complete UTF-8 sequences */\n+\t\t\tif (end > start)\n+\t\t\t\twrite_console(buffer + start, end - start);\n \n-\t\t\tpos = set_attr(str);\n-\t\t\trv += pos - str;\n-\t\t\tstr = pos;\n+\t\t\t/* move remaining bytes to the front */\n+\t\t\tif (end < bytes)\n+\t\t\t\tmemmove(buffer, buffer + end, bytes - end);\n+\t\t\tend = bytes - end;\n \t\t} else {\n-\t\t\tsize_t len = strlen(str);\n-\t\t\trv += write_console(str, len);\n-\t\t\treturn rv;\n+\t\t\t/* all data has been consumed, mark buffer empty */\n+\t\t\tend = 0;\n \t\t}\n \t}\n-\treturn rv;\n-}\n-\n-int winansi_fputs(const char *str, FILE *stream)\n-{\n-\tint rv;\n-\n-\tif (!is_console(stream))\n-\t\treturn fputs(str, stream);\n \n-\trv = ansi_emulate(str, stream);\n+\t/* check if the console font supports unicode */\n+\twarn_if_raster_font();\n \n-\tif (rv >= 0)\n-\t\treturn 0;\n-\telse\n-\t\treturn EOF;\n+\tCloseHandle(hread);\n+\treturn 0;\n }\n \n-int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n+static void winansi_exit(void)\n {\n-\tint len, rv;\n-\tchar small_buf[256];\n-\tchar *buf = small_buf;\n-\tva_list cp;\n-\n-\tif (!is_console(stream))\n-\t\tgoto abort;\n+\t/* flush all streams */\n+\t_flushall();\n+\n+\t/* signal console thread to exit */\n+\tFlushFileBuffers(hwrite);\n+\tDisconnectNamedPipe(hwrite);\n+\n+\t/* wait for console thread to copy remaining data */\n+\tWaitForSingleObject(hthread, INFINITE);\n+\n+\t/* cleanup handles... */\n+\tif (hwrite1 != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(hwrite1);\n+\tif (hwrite2 != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(hwrite2);\n+\tCloseHandle(hwrite);\n+\tCloseHandle(hthread);\n+}\n \n-\tva_copy(cp, list);\n-\tlen = vsnprintf(small_buf, sizeof(small_buf), format, cp);\n-\tva_end(cp);\n+static void die_lasterr(const char *fmt, ...)\n+{\n+\tva_list params;\n+\tva_start(params, fmt);\n+\terrno = err_win_to_posix(GetLastError());\n+\tdie_errno(fmt, params);\n+\tva_end(params);\n+}\n \n-\tif (len > sizeof(small_buf) - 1) {\n-\t\tbuf = malloc(len + 1);\n-\t\tif (!buf)\n-\t\t\tgoto abort;\n+static HANDLE duplicate_handle(HANDLE hnd)\n+{\n+\tHANDLE hresult, hproc = GetCurrentProcess();\n+\tif (!DuplicateHandle(hproc, hnd, hproc, &hresult, 0, TRUE,\n+\t\t\tDUPLICATE_SAME_ACCESS))\n+\t\tdie_lasterr(\"DuplicateHandle(%li) failed\", (long) hnd);\n+\treturn hresult;\n+}\n \n-\t\tlen = vsnprintf(buf, len + 1, format, list);\n-\t}\n+static HANDLE redirect_console(FILE *stream, HANDLE *phcon, int new_fd)\n+{\n+\t/* get original console handle */\n+\tint fd = _fileno(stream);\n+\tHANDLE hcon = (HANDLE) _get_osfhandle(fd);\n+\tif (hcon == INVALID_HANDLE_VALUE)\n+\t\tdie_errno(\"_get_osfhandle(%i) failed\", fd);\n \n-\trv = ansi_emulate(buf, stream);\n+\t/* save a copy to phcon and console (used by the background thread) */\n+\tconsole = *phcon = duplicate_handle(hcon);\n \n-\tif (buf != small_buf)\n-\t\tfree(buf);\n-\treturn rv;\n+\t/* duplicate new_fd over fd (closes fd and associated handle (hcon)) */\n+\tif (_dup2(new_fd, fd))\n+\t\tdie_errno(\"_dup2(%i, %i) failed\", new_fd, fd);\n \n-abort:\n-\trv = vfprintf(stream, format, list);\n-\treturn rv;\n+\t/* no buffering, or stdout / stderr will be out of sync */\n+\tsetbuf(stream, NULL);\n+\treturn (HANDLE) _get_osfhandle(fd);\n }\n \n-int winansi_fprintf(FILE *stream, const char *format, ...)\n+void winansi_init(void)\n {\n-\tva_list list;\n-\tint rv;\n+\tint con1, con2, hwrite_fd;\n+\tchar name[32];\n \n-\tva_start(list, format);\n-\trv = winansi_vfprintf(stream, format, list);\n-\tva_end(list);\n+\t/* check if either stdout or stderr is a console output screen buffer */\n+\tcon1 = is_console(1);\n+\tcon2 = is_console(2);\n+\tif (!con1 && !con2)\n+\t\treturn;\n \n-\treturn rv;\n+\t/* create a named pipe to communicate with the console thread */\n+\tsprintf(name, \"\\\\\\\\.\\\\pipe\\\\winansi%lu\", GetCurrentProcessId());\n+\thwrite = CreateNamedPipe(name, PIPE_ACCESS_OUTBOUND,\n+\t\tPIPE_TYPE_BYTE | PIPE_WAIT, 1, BUFFER_SIZE, 0, 0, NULL);\n+\tif (hwrite == INVALID_HANDLE_VALUE)\n+\t\tdie_lasterr(\"CreateNamedPipe failed\");\n+\n+\thread = CreateFile(name, GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);\n+\tif (hread == INVALID_HANDLE_VALUE)\n+\t\tdie_lasterr(\"CreateFile for named pipe failed\");\n+\n+\t/* start console spool thread on the pipe's read end */\n+\ththread = CreateThread(NULL, 0, console_thread, NULL, 0, NULL);\n+\tif (hthread == INVALID_HANDLE_VALUE)\n+\t\tdie_lasterr(\"CreateThread(console_thread) failed\");\n+\n+\t/* schedule cleanup routine */\n+\tif (atexit(winansi_exit))\n+\t\tdie_errno(\"atexit(winansi_exit) failed\");\n+\n+\t/* create a file descriptor for the write end of the pipe */\n+\thwrite_fd = _open_osfhandle((long) duplicate_handle(hwrite), _O_BINARY);\n+\tif (hwrite_fd == -1)\n+\t\tdie_errno(\"_open_osfhandle(%li) failed\", (long) hwrite);\n+\n+\t/* redirect stdout / stderr to the pipe */\n+\tif (con1)\n+\t\thwrite1 = redirect_console(stdout, &hconsole1, hwrite_fd);\n+\tif (con2)\n+\t\thwrite2 = redirect_console(stderr, &hconsole2, hwrite_fd);\n+\n+\t/* close pipe file descriptor (also closes the duped hwrite) */\n+\tclose(hwrite_fd);\n }\n \n-int winansi_printf(const char *format, ...)\n+static int is_same_handle(HANDLE hnd, int fd)\n {\n-\tva_list list;\n-\tint rv;\n+\treturn hnd != INVALID_HANDLE_VALUE && hnd == (HANDLE) _get_osfhandle(fd);\n+}\n \n-\tva_start(list, format);\n-\trv = winansi_vfprintf(stdout, format, list);\n-\tva_end(list);\n+/*\n+ * Return true if stdout / stderr is a pipe redirecting to the console.\n+ */\n+int winansi_isatty(int fd)\n+{\n+\tif (fd == 1 && is_same_handle(hwrite1, 1))\n+\t\treturn 1;\n+\telse if (fd == 2 && is_same_handle(hwrite2, 2))\n+\t\treturn 1;\n+\telse\n+\t\treturn isatty(fd);\n+}\n \n-\treturn rv;\n+/*\n+ * Returns the real console handle if stdout / stderr is a pipe redirecting\n+ * to the console. Allows spawn / exec to pass the console to the next process.\n+ */\n+HANDLE winansi_get_osfhandle(int fd)\n+{\n+\tif (fd == 1 && is_same_handle(hwrite1, 1))\n+\t\treturn hconsole1;\n+\telse if (fd == 2 && is_same_handle(hwrite2, 2))\n+\t\treturn hconsole2;\n+\telse\n+\t\treturn (HANDLE) _get_osfhandle(fd);\n }\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243507","messageId":"5391FE01.9040501@gmail.com","threadId":"36854","inReplyTo":"1402062173-9602-1-git-send-email-kasal@ucw.cz","subject":"Re: [PATCH 0/5] First part of Unicode console support for msysgit","fromName":"Karsten Blees","fromEmail":"karsten.blees@gmail.com","sentAt":"2014-06-06T17:44:33Z","receivedAt":"2014-06-06T17:44:33Z","isPatch":true,"sender":{"key":"karsten.blees@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1111200?v=4"},"body":"Am 06.06.2014 15:42, schrieb Stepan Kasal:\n> Hello,\n> \n> this is first part of the unicode support pathes from msysgit.\n> \n\nNicely done, thanks!\n\nI think its important to reiterate that these patches were written several years apart, so there are some inconsistencies and back-and-forth changes (e.g. [5/5] fixes line break errors introduced in [3/5]). I'm OK with merging this as is, if there are no objections from the list, simply because it represents the battle tested history we have in Git for Windows.\n\nThe only real complaint I have is that I'm missing [6/5] \"Win32: fix broken pipe detection\" [1], which leaves this series in a slightly broken state (terminating the pager will not terminate the calling git process).\n\n\nNitpicking follows...\n\n> The first three patches originate in Jun 2010, though some fixups from 2012\n> have been squashed in.\n> The fourth one is just a trivial prerequisite for\n> the last one, that was written in Jan 2012, with a fixup from Mar 2012.\n> \n\nThe dates are missing from the patches.\nIt would also have been nice to name (or link to) the patches you sqashed.\n\n> Regards,\n> \tStepan\n> \n> Karsten Blees (5):\n> Support Unicode console output on Windows\n\nThis introduces WriteConsoleW, so you could have squashed half of \"Win32: fix segfault in WriteConsoleW when debugging in gdb\" [2] (second half in [5/5]).\n\n> Detect console streams more reliably on Windows\n> Warn if the Windows console font doesn't support Unicode\n\nI think this one includes \"MSVC: fix winansi.c compile errors \" [3] and \"Unicode console: fix font warning on Vista and Win7\" [4] (which is partly reverted by [5/5], which also fixes the formatting).\n\n> Win32: move main macro to a function\n\nNote: this one was submitted seperately on May 29 and May 1 (can't find it in the gmane archive, though).\n\n> Win32: Thread-safe windows console output\n> \n> compat/mingw.c | 24 ++-\n> compat/mingw.h | 24 +--\n> compat/winansi.c | 446 ++++++++++++++++++++++++++++++++++++++++---------------\n> 3 files changed, 356 insertions(+), 138 deletions(-)\n> \n\n[1] https://github.com/msysgit/git/commit/67934f93\n[2] https://github.com/msysgit/git/commit/cd0792af\n[3] https://github.com/msysgit/git/commit/3abcb04d\n[4] https://github.com/msysgit/git/commit/981aa538\n\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243510","messageId":"20140606183935.GA4197@camelia.ucw.cz","threadId":"36854","inReplyTo":"5391FE01.9040501@gmail.com","subject":"Re: Re: [PATCH 0/5] First part of Unicode console support for msysgit","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T18:39:35Z","receivedAt":"2014-06-06T18:39:35Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"Hello Karsten,\n\nOn Fri, Jun 06, 2014 at 07:44:33PM +0200, Karsten Blees wrote:\n> Nicely done, thanks!\n\nthank you for your kind words.\n\nPlease hold back, I will re-submit in a few days.\n\n> Note: this one was submitted seperately on May 29 and May 1 (can't\n> find it in the gmane archive, though).\n\nIt was cc'ed to msysgit as usual and gmane selected it as the primary\nplace to store it:\nhttp://thread.gmane.org/gmane.comp.version-control.msysgit/20324\n\nregards,\n\tStepan\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243520","messageId":"20140606204830.GA4671@camelia.ucw.cz","threadId":"36854","inReplyTo":"5391FE01.9040501@gmail.com","subject":"Re: Re: [PATCH 0/5] First part of Unicode console support for msysgit","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-06T20:48:30Z","receivedAt":"2014-06-06T20:48:30Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"Hello,\n\nOn Fri, Jun 06, 2014 at 07:44:33PM +0200, Karsten Blees wrote:\n> > Karsten Blees (5):\n> > Support Unicode console output on Windows\n> \n> [..] you could have squashed half of\n> \"Win32: fix segfault in WriteConsoleW when debugging in gdb\" [2] (second half in [5/5]).\n> \n> > Detect console streams more reliably on Windows\n> > Warn if the Windows console font doesn't support Unicode\n> \n> I think this one includes\n> \"MSVC: fix winansi.c compile errors \" [3] and\n> \"Unicode console: fix font warning on Vista and Win7\" [4]\n[...]\n> > Win32: move main macro to a function\n> > Win32: Thread-safe windows console output\n\n> [2] https://github.com/msysgit/git/commit/cd0792af\n> [3] https://github.com/msysgit/git/commit/3abcb04d\n> [4] https://github.com/msysgit/git/commit/981aa538\n\nIndeed, you identified them correctly. And [2] is actually squashed\nin [5/5]; I think I can keep it that way.\n\nI'll add the missing fix, take care about original dates, improve the\ncover letter (the above links), and resubmit.\n\nThanks for review this batch of your patches.\n\nRegards,\n\tStepan\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243524","messageId":"alpine.DEB.2.00.1406062211360.17391@ds9.cixit.se","threadId":"36854","inReplyTo":"1402062173-9602-4-git-send-email-kasal@ucw.cz","subject":"Re: [PATCH 3/5] Warn if the Windows console font doesn't support Unicode","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2014-06-06T21:18:43Z","receivedAt":"2014-06-06T21:18:43Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Stepan Kasal:\n\n> +\twarning(\"Your console font probably doesn\\'t support Unicode. If \"\n> +\t\t\"you experience strange characters in the output, consider \"\n> +\t\t\"switching to a TrueType font such as Lucida Console!\");\n\nAs you mention this is an old patch series, but I would recommend \nmodernizing the suggestion here to recomment \"Consolas\". It is \navailable in all current versions of Windows (Vista and later), and \nseem to have better Unicode support according to \n vs \n\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"243525","messageId":"alpine.DEB.2.00.1406062224530.17391@ds9.cixit.se","threadId":"36854","inReplyTo":"1402062173-9602-6-git-send-email-kasal@ucw.cz","subject":"Re: [PATCH 5/5] Win32: Thread-safe windows console output","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2014-06-06T21:29:12Z","receivedAt":"2014-06-06T21:29:12Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Stepan Kasal:\n\n> +\t/* only called from console_thread, so a static buffer will do */\n> +\tstatic wchar_t wbuf[2 * BUFFER_SIZE + 1];\n\nWouldn't BUFFER_SIZE + 1 (or even BUFFER_SIZE) do here? If you convert \nfrom up to BUFFER_SIZE octets of UTF-8 input, you should never get \nback more than BUFFER_SIZE code units of UTF-16 output. Worst case \nwould be ASCII, which is one UTF-16 code unit per UTF-8 octet, \neverything else is less (non-BMP is four UTF-8 octets mapping to two \nUTF-16 code units).\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"243526","messageId":"53923AC8.2030309@gmail.com","threadId":"36854","inReplyTo":"alpine.DEB.2.00.1406062224530.17391@ds9.cixit.se","subject":"Re: [PATCH 5/5] Win32: Thread-safe windows console output","fromName":"Karsten Blees","fromEmail":"karsten.blees@gmail.com","sentAt":"2014-06-06T22:03:52Z","receivedAt":"2014-06-06T22:03:52Z","isPatch":true,"sender":{"key":"karsten.blees@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1111200?v=4"},"body":"Am 06.06.2014 23:29, schrieb Peter Krefting:\n> Stepan Kasal:\n> \n>> + /* only called from console_thread, so a static buffer will do */\n>> + static wchar_t wbuf[2 * BUFFER_SIZE + 1];\n> \n> Wouldn't BUFFER_SIZE + 1 (or even BUFFER_SIZE) do here? If you convert from up to BUFFER_SIZE octets of UTF-8 input, you should never get back more than BUFFER_SIZE code units of UTF-16 output. Worst case would be ASCII, which is one UTF-16 code unit per UTF-8 octet, everything else is less (non-BMP is four UTF-8 octets mapping to two UTF-16 code units).\n> \n\nYou're right for MultiByteToWideChar. However, the next patch series will introduce another conversion function that converts invalid UTF-8 to hex code, i.e. two wide chars per invalid UTF-8 char, +1 for L'\\0' (see [1] mingw.h:365ff for space requirement rationale). And yet another patch will replace this patch's MultiByteToWideChar for consistentcy [2].\n\n[1] https://github.com/msysgit/git/commit/018c94a8\n[2] https://github.com/msysgit/git/commit/45e28a4d\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243581","messageId":"20140607070208.GA8400@camelia.ucw.cz","threadId":"36854","inReplyTo":"alpine.DEB.2.00.1406062211360.17391@ds9.cixit.se","subject":"Re: Re: [PATCH 3/5] Warn if the Windows console font doesn't support Unicode","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:02:08Z","receivedAt":"2014-06-07T07:02:08Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"Hi,\n\nOn Fri, Jun 06, 2014 at 10:18:43PM +0100, Peter Krefting wrote:\n> Stepan Kasal:\n>> +\t\t\"switching to a TrueType font such as Lucida Console!\");\n[...]\n> modernizing the suggestion here to recomment \"Consolas\". It is available \n\nIndeed.\n\nSo, I'll keep this patch as it is, for the records, and apply your\nsuggestion to the subsequent\n\t[PATCH 5/5] Win32: Thread-safe windows console output\n\nStepan\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243586","messageId":"1402127845-4862-1-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"20140606183935.GA4197@camelia.ucw.cz","subject":"[PATCH v2 0/6] First part of Unicode console support for msysgit","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:19Z","receivedAt":"2014-06-07T07:57:19Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"Hello,\nthis patch series is to be applied on top of \"move main() macro to a function\",\ndiscussed in another thread.\n\nI added the two patches Karsten mentioned:\n Win32: add Unicode conversion functions\n Win32: fix broken pipe detection\n\nI also copied the links for the fixups incuded.\n\nRegards,\n\tStepan\n\nKarsten Blees (6):\n Support Unicode console output on Windows\n Detect console streams more reliably on Windows\n Warn if the Windows console font doesn't support Unicode\n includes fixups:\n - https://github.com/msysgit/git/commit/3abcb04d\n - https://github.com/msysgit/git/commit/981aa538\n Win32: add Unicode conversion functions\n Win32: Thread-safe windows console output\n includes fixups:\n - https://github.com/msysgit/git/commit/cd0792af\n - https://github.com/msysgit/git/commit/45e28a4d\n Win32: fix broken pipe detection\n\n compat/mingw.c | 94 ++++++++++-\n compat/mingw.h | 112 +++++++++++++-\n compat/winansi.c | 465 +++++++++++++++++++++++++++++++++++++++++--------------\n 3 files changed, 546 insertions(+), 125 deletions(-)\n\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243590","messageId":"1402127845-4862-2-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402127845-4862-1-git-send-email-kasal@ucw.cz","subject":"[PATCH v2 1/6] Support Unicode console output on Windows","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:20Z","receivedAt":"2014-06-07T07:57:20Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \nDate: Sat, 31 Jul 2010 00:04:01 +0000\n\nWriteConsoleW seems to be the only way to reliably print unicode to the\nconsole (without weird code page conversions).\n\nAlso redirects vfprintf to the winansi.c version.\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.h | 2 ++\n compat/winansi.c | 26 ++++++++++++++++++++------\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 6dc8b1a..d3cffb7 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -320,9 +320,11 @@ int mingw_raise(int sig);\n int winansi_fputs(const char *str, FILE *stream);\n int winansi_printf(const char *format, ...) __attribute__((format (printf, 1, 2)));\n int winansi_fprintf(FILE *stream, const char *format, ...) __attribute__((format (printf, 2, 3)));\n+int winansi_vfprintf(FILE *stream, const char *format, va_list list);\n #define fputs winansi_fputs\n #define printf(...) winansi_printf(__VA_ARGS__)\n #define fprintf(...) winansi_fprintf(__VA_ARGS__)\n+#define vfprintf winansi_vfprintf\n \n /*\n * git specific compatibility\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex dedce21..abe0fea 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -3,6 +3,7 @@\n */\n \n #include \"../git-compat-util.h\"\n+#include \n \n /*\n Functions to be wrapped:\n@@ -10,6 +11,7 @@\n #undef printf\n #undef fprintf\n #undef fputs\n+#undef vfprintf\n /* TODO: write */\n \n /*\n@@ -46,6 +48,18 @@ static void init(void)\n \tinitialized = 1;\n }\n \n+static int write_console(const char *str, size_t len)\n+{\n+\t/* convert utf-8 to utf-16, write directly to console */\n+\tint wlen = MultiByteToWideChar(CP_UTF8, 0, str, len, NULL, 0);\n+\twchar_t *wbuf = (wchar_t *) alloca(wlen * sizeof(wchar_t));\n+\tMultiByteToWideChar(CP_UTF8, 0, str, len, wbuf, wlen);\n+\n+\tWriteConsoleW(console, wbuf, wlen, NULL, NULL);\n+\n+\t/* return original (utf-8 encoded) length */\n+\treturn len;\n+}\n \n #define FOREGROUND_ALL (FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE)\n #define BACKGROUND_ALL (BACKGROUND_RED | BACKGROUND_GREEN | BACKGROUND_BLUE)\n@@ -245,13 +259,15 @@ static int ansi_emulate(const char *str, FILE *stream)\n \tint rv = 0;\n \tconst char *pos = str;\n \n+\tfflush(stream);\n+\n \twhile (*pos) {\n \t\tpos = strstr(str, \"\\033[\");\n \t\tif (pos) {\n \t\t\tsize_t len = pos - str;\n \n \t\t\tif (len) {\n-\t\t\t\tsize_t out_len = fwrite(str, 1, len, stream);\n+\t\t\t\tsize_t out_len = write_console(str, len);\n \t\t\t\trv += out_len;\n \t\t\t\tif (out_len < len)\n \t\t\t\t\treturn rv;\n@@ -260,14 +276,12 @@ static int ansi_emulate(const char *str, FILE *stream)\n \t\t\tstr = pos + 2;\n \t\t\trv += 2;\n \n-\t\t\tfflush(stream);\n-\n \t\t\tpos = set_attr(str);\n \t\t\trv += pos - str;\n \t\t\tstr = pos;\n \t\t} else {\n-\t\t\trv += strlen(str);\n-\t\t\tfputs(str, stream);\n+\t\t\tsize_t len = strlen(str);\n+\t\t\trv += write_console(str, len);\n \t\t\treturn rv;\n \t\t}\n \t}\n@@ -294,7 +308,7 @@ int winansi_fputs(const char *str, FILE *stream)\n \t\treturn EOF;\n }\n \n-static int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n+int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n {\n \tint len, rv;\n \tchar small_buf[256];\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243591","messageId":"1402127845-4862-3-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402127845-4862-1-git-send-email-kasal@ucw.cz","subject":"[PATCH v2 2/6] Detect console streams more reliably on Windows","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:21Z","receivedAt":"2014-06-07T07:57:21Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \nDate: Sat, 31 Jul 2010 00:04:02 +0000\n\nGetStdHandle(STD_OUTPUT_HANDLE) doesn't work for stderr if stdout is\nredirected. Use _get_osfhandle of the FILE* instead.\n\n_isatty() is true for all character devices (including parallel and serial\nports). Check return value of GetConsoleScreenBufferInfo instead to\nreliably detect console handles (also don't initialize internal state from\nan uninitialized CONSOLE_SCREEN_BUFFER_INFO structure if the function\nfails).\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Stepan Kasal \n---\n compat/winansi.c | 50 ++++++++++++++++++++++++++------------------------\n 1 file changed, 26 insertions(+), 24 deletions(-)\n\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex abe0fea..c4be401 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -25,27 +25,39 @@ static HANDLE console;\n static WORD plain_attr;\n static WORD attr;\n static int negative;\n+static FILE *last_stream = NULL;\n \n-static void init(void)\n+static int is_console(FILE *stream)\n {\n \tCONSOLE_SCREEN_BUFFER_INFO sbi;\n+\tHANDLE hcon;\n \n \tstatic int initialized = 0;\n-\tif (initialized)\n-\t\treturn;\n \n-\tconsole = GetStdHandle(STD_OUTPUT_HANDLE);\n-\tif (console == INVALID_HANDLE_VALUE)\n-\t\tconsole = NULL;\n+\t/* use cached value if stream hasn't changed */\n+\tif (stream == last_stream)\n+\t\treturn console != NULL;\n \n-\tif (!console)\n-\t\treturn;\n+\tlast_stream = stream;\n+\tconsole = NULL;\n \n-\tGetConsoleScreenBufferInfo(console, &sbi);\n-\tattr = plain_attr = sbi.wAttributes;\n-\tnegative = 0;\n+\t/* get OS handle of the stream */\n+\thcon = (HANDLE) _get_osfhandle(_fileno(stream));\n+\tif (hcon == INVALID_HANDLE_VALUE)\n+\t\treturn 0;\n+\n+\t/* check if its a handle to a console output screen buffer */\n+\tif (!GetConsoleScreenBufferInfo(hcon, &sbi))\n+\t\treturn 0;\n+\n+\tif (!initialized) {\n+\t\tattr = plain_attr = sbi.wAttributes;\n+\t\tnegative = 0;\n+\t\tinitialized = 1;\n+\t}\n \n-\tinitialized = 1;\n+\tconsole = hcon;\n+\treturn 1;\n }\n \n static int write_console(const char *str, size_t len)\n@@ -292,12 +304,7 @@ int winansi_fputs(const char *str, FILE *stream)\n {\n \tint rv;\n \n-\tif (!isatty(fileno(stream)))\n-\t\treturn fputs(str, stream);\n-\n-\tinit();\n-\n-\tif (!console)\n+\tif (!is_console(stream))\n \t\treturn fputs(str, stream);\n \n \trv = ansi_emulate(str, stream);\n@@ -315,12 +322,7 @@ int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n \tchar *buf = small_buf;\n \tva_list cp;\n \n-\tif (!isatty(fileno(stream)))\n-\t\tgoto abort;\n-\n-\tinit();\n-\n-\tif (!console)\n+\tif (!is_console(stream))\n \t\tgoto abort;\n \n \tva_copy(cp, list);\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243589","messageId":"1402127845-4862-4-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402127845-4862-1-git-send-email-kasal@ucw.cz","subject":"[PATCH v2 3/6] Warn if the Windows console font doesn't support Unicode","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:22Z","receivedAt":"2014-06-07T07:57:22Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \nDate: Sat, 31 Jul 2010 00:04:03 +0000\n\nUnicode console output won't display correctly with default settings\nbecause the default console font (\"Terminal\") only supports the system's\nOEM charset. Unfortunately, this is a user specific setting, so it cannot\nbe easily fixed by e.g. some registry tricks in the setup program.\n\nThis change prints a warning on exit if console output contained non-ascii\ncharacters and the console font is supposedly not a TrueType font (which\nusually have decent Unicode support).\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Stepan Kasal \n---\n compat/winansi.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 66 insertions(+)\n\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex c4be401..bec6713 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -2,8 +2,11 @@\n * Copyright 2008 Peter Harris \n */\n \n+#undef NOGDI\n #include \"../git-compat-util.h\"\n #include \n+#include \n+#include \n \n /*\n Functions to be wrapped:\n@@ -27,6 +30,62 @@ static WORD attr;\n static int negative;\n static FILE *last_stream = NULL;\n \n+#ifdef __MINGW32__\n+typedef struct _CONSOLE_FONT_INFOEX {\n+\tULONG cbSize;\n+\tDWORD nFont;\n+\tCOORD dwFontSize;\n+\tUINT FontFamily;\n+\tUINT FontWeight;\n+\tWCHAR FaceName[LF_FACESIZE];\n+} CONSOLE_FONT_INFOEX, *PCONSOLE_FONT_INFOEX;\n+#endif\n+\n+typedef BOOL (WINAPI *PGETCURRENTCONSOLEFONTEX)(HANDLE, BOOL,\n+\t\tPCONSOLE_FONT_INFOEX);\n+\n+static void print_font_warning(void)\n+{\n+\twarning(\"Your console font probably doesn\\'t support Unicode. If \"\n+\t\t\"you experience strange characters in the output, consider \"\n+\t\t\"switching to a TrueType font such as Lucida Console!\");\n+}\n+\n+static void check_truetype_font(void)\n+{\n+\tstatic int truetype_font_checked;\n+\tDWORD fontFamily = 0;\n+\tPGETCURRENTCONSOLEFONTEX pGetCurrentConsoleFontEx;\n+\n+\t/* don't do this twice */\n+\tif (truetype_font_checked)\n+\t\treturn;\n+\ttruetype_font_checked = 1;\n+\n+\t/* GetCurrentConsoleFontEx is available since Vista */\n+\tpGetCurrentConsoleFontEx = (PGETCURRENTCONSOLEFONTEX) GetProcAddress(\n+\t\t\tGetModuleHandle(\"kernel32.dll\"), \"GetCurrentConsoleFontEx\");\n+\tif (pGetCurrentConsoleFontEx) {\n+\t\tCONSOLE_FONT_INFOEX cfi;\n+\t\tcfi.cbSize = sizeof(cfi);\n+\t\tif (pGetCurrentConsoleFontEx(console, 0, &cfi))\n+\t\t\tfontFamily = cfi.FontFamily;\n+\t} else {\n+\t\t/* pre-Vista: check default console font in registry */\n+\t\tHKEY hkey;\n+\t\tif (ERROR_SUCCESS == RegOpenKeyExA(HKEY_CURRENT_USER, \"Console\", 0,\n+\t\t\t\tKEY_READ, &hkey)) {\n+\t\t\tDWORD size = sizeof(fontFamily);\n+\t\t\tRegQueryValueExA(hkey, \"FontFamily\", NULL, NULL,\n+\t\t\t\t\t(LPVOID) &fontFamily, &size);\n+\t\t\tRegCloseKey(hkey);\n+\t\t}\n+\t}\n+\n+\tif (!(fontFamily & TMPF_TRUETYPE))\n+\t\tatexit(print_font_warning);\n+}\n+\n static int is_console(FILE *stream)\n {\n \tCONSOLE_SCREEN_BUFFER_INFO sbi;\n@@ -69,6 +128,13 @@ static int write_console(const char *str, size_t len)\n \n \tWriteConsoleW(console, wbuf, wlen, NULL, NULL);\n \n+\t/*\n+\t * if non-ascii characters are printed, check that the current console\n+\t * font supports this\n+\t */\n+\tif (wlen != len)\n+\t\tcheck_truetype_font();\n+\n \t/* return original (utf-8 encoded) length */\n \treturn len;\n }\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243588","messageId":"1402127845-4862-5-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402127845-4862-1-git-send-email-kasal@ucw.cz","subject":"[PATCH v2 4/6] Win32: add Unicode conversion functions","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:23Z","receivedAt":"2014-06-07T07:57:23Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \nDate: Fri, 25 Nov 2011 21:05:06 +0100\n\nAdd Unicode conversion functions to convert between Windows native UTF-16LE\nencoding to UTF-8 and back.\n\nTo support repositories with legacy-encoded file names, the UTF-8 to UTF-16\nconversion function tries to create valid, unique file names even for\ninvalid UTF-8 byte sequences, so that these repositories can be checked out\nwithout error.\n\nThe current implementation leaves invalid UTF-8 bytes in range 0xa0 - 0xff\nas is (producing printable Unicode chars \\u00a0 - \\u00ff, equivalent to\nISO-8859-1), and converts 0x80 - 0x9f to hex-code (\\u0080 - \\u009f are\ncontrol chars).\n\nThe Windows MultiByteToWideChar API was not used as it either drops invalid\nUTF-8 sequences (on Win2k/XP; producing non-unique or even empty file\nnames) or converts them to the replacement char \\ufffd (Vista/7; causing\nERROR_INVALID_NAME in subsequent calls to file system APIs).\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++\n compat/mingw.h | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 189 insertions(+)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex c03bafa..6f1fb10 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -1848,6 +1848,91 @@ int mingw_offset_1st_component(const char *path)\n \treturn offset + is_dir_sep(path[offset]);\n }\n \n+int xutftowcsn(wchar_t *wcs, const char *utfs, size_t wcslen, int utflen)\n+{\n+\tint upos = 0, wpos = 0;\n+\tconst unsigned char *utf = (const unsigned char*) utfs;\n+\tif (!utf || !wcs || wcslen < 1) {\n+\t\terrno = EINVAL;\n+\t\treturn -1;\n+\t}\n+\t/* reserve space for \\0 */\n+\twcslen--;\n+\tif (utflen < 0)\n+\t\tutflen = INT_MAX;\n+\n+\twhile (upos < utflen) {\n+\t\tint c = utf[upos++] & 0xff;\n+\t\tif (utflen == INT_MAX && c == 0)\n+\t\t\tbreak;\n+\n+\t\tif (wpos >= wcslen) {\n+\t\t\twcs[wpos] = 0;\n+\t\t\terrno = ERANGE;\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (c < 0x80) {\n+\t\t\t/* ASCII */\n+\t\t\twcs[wpos++] = c;\n+\t\t} else if (c >= 0xc2 && c < 0xe0 && upos < utflen &&\n+\t\t\t\t(utf[upos] & 0xc0) == 0x80) {\n+\t\t\t/* 2-byte utf-8 */\n+\t\t\tc = ((c & 0x1f) << 6);\n+\t\t\tc |= (utf[upos++] & 0x3f);\n+\t\t\twcs[wpos++] = c;\n+\t\t} else if (c >= 0xe0 && c < 0xf0 && upos + 1 < utflen &&\n+\t\t\t\t!(c == 0xe0 && utf[upos] < 0xa0) && /* over-long encoding */\n+\t\t\t\t(utf[upos] & 0xc0) == 0x80 &&\n+\t\t\t\t(utf[upos + 1] & 0xc0) == 0x80) {\n+\t\t\t/* 3-byte utf-8 */\n+\t\t\tc = ((c & 0x0f) << 12);\n+\t\t\tc |= ((utf[upos++] & 0x3f) << 6);\n+\t\t\tc |= (utf[upos++] & 0x3f);\n+\t\t\twcs[wpos++] = c;\n+\t\t} else if (c >= 0xf0 && c < 0xf5 && upos + 2 < utflen &&\n+\t\t\t\twpos + 1 < wcslen &&\n+\t\t\t\t!(c == 0xf0 && utf[upos] < 0x90) && /* over-long encoding */\n+\t\t\t\t!(c == 0xf4 && utf[upos] >= 0x90) && /* > \\u10ffff */\n+\t\t\t\t(utf[upos] & 0xc0) == 0x80 &&\n+\t\t\t\t(utf[upos + 1] & 0xc0) == 0x80 &&\n+\t\t\t\t(utf[upos + 2] & 0xc0) == 0x80) {\n+\t\t\t/* 4-byte utf-8: convert to \\ud8xx \\udcxx surrogate pair */\n+\t\t\tc = ((c & 0x07) << 18);\n+\t\t\tc |= ((utf[upos++] & 0x3f) << 12);\n+\t\t\tc |= ((utf[upos++] & 0x3f) << 6);\n+\t\t\tc |= (utf[upos++] & 0x3f);\n+\t\t\tc -= 0x10000;\n+\t\t\twcs[wpos++] = 0xd800 | (c >> 10);\n+\t\t\twcs[wpos++] = 0xdc00 | (c & 0x3ff);\n+\t\t} else if (c >= 0xa0) {\n+\t\t\t/* invalid utf-8 byte, printable unicode char: convert 1:1 */\n+\t\t\twcs[wpos++] = c;\n+\t\t} else {\n+\t\t\t/* invalid utf-8 byte, non-printable unicode: convert to hex */\n+\t\t\tstatic const char *hex = \"0123456789abcdef\";\n+\t\t\twcs[wpos++] = hex[c >> 4];\n+\t\t\tif (wpos < wcslen)\n+\t\t\t\twcs[wpos++] = hex[c & 0x0f];\n+\t\t}\n+\t}\n+\twcs[wpos] = 0;\n+\treturn wpos;\n+}\n+\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen)\n+{\n+\tif (!wcs || !utf || utflen < 1) {\n+\t\terrno = EINVAL;\n+\t\treturn -1;\n+\t}\n+\tutflen = WideCharToMultiByte(CP_UTF8, 0, wcs, -1, utf, utflen, NULL, NULL);\n+\tif (utflen)\n+\t\treturn utflen - 1;\n+\terrno = ERANGE;\n+\treturn -1;\n+}\n+\n void mingw_startup()\n {\n \t/* copy executable name to argv[0] */\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex d3cffb7..921ba08 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -357,6 +357,110 @@ void mingw_open_html(const char *path);\n char **make_augmented_environ(const char *const *vars);\n void free_environ(char **env);\n \n+/**\n+ * Converts UTF-8 encoded string to UTF-16LE.\n+ *\n+ * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n+ * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n+ * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n+ * Unicode) are converted to hex-code.\n+ *\n+ * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n+ * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is two wide chars per UTF-8\n+ * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n+ *\n+ * | | UTF-8 | UTF-16 |\n+ * Code point | UTF-8 sequence | bytes | words | ratio\n+ * --------------+-------------------+-------+--------+-------\n+ * 000000-00007f | 0-7f | 1 | 1 | 1\n+ * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n+ * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n+ * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n+ * invalid | 80-9f | 1 | 2 (b) | 2\n+ * invalid | a0-ff | 1 | 1 | 1\n+ *\n+ * (a) encoded as UTF-16 surrogate pair\n+ * (b) encoded as two hex digits\n+ *\n+ * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n+ * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n+ * encodes as only 4 UTF-8 bytes.\n+ *\n+ * Parameters:\n+ * wcs: wide char target buffer\n+ * utf: string to convert\n+ * wcslen: size of target buffer (in wchar_t's)\n+ * utflen: size of string to convert, or -1 if 0-terminated\n+ *\n+ * Returns:\n+ * length of converted string (_wcslen(wcs)), or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n+\n+/**\n+ * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n+ */\n+static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n+{\n+\treturn xutftowcsn(wcs, utf, wcslen, -1);\n+}\n+\n+/**\n+ * Simplified file system specific variant of xutftowcsn, assumes output\n+ * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n+ * fails with ENAMETOOLONG if input string is too long.\n+ */\n+static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n+{\n+\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n+\tif (result < 0 && errno == ERANGE)\n+\t\terrno = ENAMETOOLONG;\n+\treturn result;\n+}\n+\n+/**\n+ * Converts UTF-16LE encoded string to UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is three UTF-8 chars per\n+ * wide char ((_wcslen(wcs) * 3) + 1).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n+ * modulo surrogate pairs), as per the following table:\n+ *\n+ * | | UTF-16 | UTF-8 |\n+ * Code point | UTF-16 sequence | words | bytes | ratio\n+ * --------------+-----------------------+--------+-------+-------\n+ * 000000-00007f | 0000-007f | 1 | 1 | 1\n+ * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n+ * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n+ * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n+ *\n+ * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n+ *\n+ * Parameters:\n+ * utf: target buffer\n+ * wcs: wide string to convert\n+ * utflen: size of target buffer\n+ *\n+ * Returns:\n+ * length of converted string, or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n+\n /*\n * A critical section used in the implementation of the spawn\n * functions (mingw_spawnv[p]e()) and waitpid(). Intialised in\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243585","messageId":"1402127845-4862-6-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402127845-4862-1-git-send-email-kasal@ucw.cz","subject":"[PATCH v2 5/6] Win32: Thread-safe windows console output","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:24Z","receivedAt":"2014-06-07T07:57:24Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \nDate: Sat, 14 Jan 2012 22:24:19 +0100\n\nWinansi.c has many static variables that are accessed and modified from\nthe [v][f]printf / fputs functions overridden in the file. This may cause\nmulti threaded git commands that print to the console to produce corrupted\noutput or even crash.\n\nAdditionally, winansi.c doesn't override all functions that can be used to\nprint to the console (e.g. fwrite, write, fputc are missing), so that ANSI\nescapes don't work properly for some git commands (e.g. git-grep).\n\nInstead of doing ANSI emulation in just a few wrapped functions on top of\nthe IO API, let's plug into the IO system and take advantage of the thread\nsafety inherent to the IO system.\n\nRedirect stdout and stderr to a pipe if they point to the console. A\nbackground thread reads from the pipe, handles ANSI escape sequences and\nUTF-8 to UTF-16 conversion, then writes to the console.\n\nThe pipe-based stdout and stderr replacements must be set to unbuffered, as\nMSVCRT doesn't support line buffering and fully buffered streams are\ninappropriate for console output.\n\nDue to the byte-oriented pipe, ANSI escape sequences and multi-byte UTF-8\nsequences can no longer be expected to arrive in one piece. Replace the\nstring-based ansi_emulate() with a simple stateful parser (this also fixes\ncolored diff hunk headers, which were broken as of commit 2efcc977).\n\nOverride isatty to return true for the pipes redirecting to the console.\n\nExec/spawn obtain the original console handle to pass to the next process\nvia winansi_get_osfhandle().\n\nAll other overrides are gone, the default stdio implementations work as\nexpected with the piped stdout/stderr descriptors.\n\nGlobal variables are either initialized on startup (single threaded) or\nexclusively modified by the background thread. Threads communicate through\nthe pipe, no further synchronization is necessary.\n\nThe background thread is terminated by disonnecting the pipe after flushing\nthe stdio and pipe buffers. This doesn't work for anonymous pipes (created\nvia CreatePipe), as DisconnectNamedPipe only works on the read end, which\ndiscards remaining data. Thus we have to setup the pipe manually, with the\nwrite end beeing the server (opened with CreateNamedPipe) and the read end\nthe client (opened with CreateFile).\n\nLimitations: doesn't track reopened or duped file descriptors, i.e.:\n- fdopen(1/2) returns fully buffered streams\n- dup(1/2), dup2(1/2) returns normal pipe descriptors (i.e. isatty() =\n false, winansi_get_osfhandle won't return the original console handle)\n\nCurrently, only the git-format-patch command uses xfdopen(xdup(1)) (see\n\"realstdout\" in builtin/log.c), but works well with these limitations.\n\nMany thanks to Atsushi Nakagawa for suggesting and\nreviewing the thread-exit-mechanism.\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.c | 9 +-\n compat/mingw.h | 12 +-\n compat/winansi.c | 401 ++++++++++++++++++++++++++++++++++++-------------------\n 3 files changed, 273 insertions(+), 149 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 6f1fb10..d242557 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -865,9 +865,9 @@ static pid_t mingw_spawnve_fd(const char *cmd, const char **argv, char **env,\n \tmemset(&si, 0, sizeof(si));\n \tsi.cb = sizeof(si);\n \tsi.dwFlags = STARTF_USESTDHANDLES;\n-\tsi.hStdInput = (HANDLE) _get_osfhandle(fhin);\n-\tsi.hStdOutput = (HANDLE) _get_osfhandle(fhout);\n-\tsi.hStdError = (HANDLE) _get_osfhandle(fherr);\n+\tsi.hStdInput = winansi_get_osfhandle(fhin);\n+\tsi.hStdOutput = winansi_get_osfhandle(fhout);\n+\tsi.hStdError = winansi_get_osfhandle(fherr);\n \n \t/* concatenate argv, quoting args as we go */\n \tstrbuf_init(&args, 0);\n@@ -1946,4 +1946,7 @@ void mingw_startup()\n \t_setmode(_fileno(stdin), _O_BINARY);\n \t_setmode(_fileno(stdout), _O_BINARY);\n \t_setmode(_fileno(stderr), _O_BINARY);\n+\n+\t/* initialize Unicode console */\n+\twinansi_init();\n }\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 921ba08..4b638d8 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -317,14 +317,10 @@ int mingw_raise(int sig);\n * ANSI emulation wrappers\n */\n \n-int winansi_fputs(const char *str, FILE *stream);\n-int winansi_printf(const char *format, ...) __attribute__((format (printf, 1, 2)));\n-int winansi_fprintf(FILE *stream, const char *format, ...) __attribute__((format (printf, 2, 3)));\n-int winansi_vfprintf(FILE *stream, const char *format, va_list list);\n-#define fputs winansi_fputs\n-#define printf(...) winansi_printf(__VA_ARGS__)\n-#define fprintf(...) winansi_fprintf(__VA_ARGS__)\n-#define vfprintf winansi_vfprintf\n+void winansi_init(void);\n+int winansi_isatty(int fd);\n+HANDLE winansi_get_osfhandle(int fd);\n+#define isatty winansi_isatty\n \n /*\n * git specific compatibility\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex bec6713..fcdd6dc 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -4,18 +4,13 @@\n \n #undef NOGDI\n #include \"../git-compat-util.h\"\n-#include \n #include \n #include \n \n /*\n Functions to be wrapped:\n */\n-#undef printf\n-#undef fprintf\n-#undef fputs\n-#undef vfprintf\n-/* TODO: write */\n+#undef isatty\n \n /*\n ANSI codes used by git: m, K\n@@ -28,7 +23,10 @@ static HANDLE console;\n static WORD plain_attr;\n static WORD attr;\n static int negative;\n-static FILE *last_stream = NULL;\n+static int non_ascii_used = 0;\n+static HANDLE hthread, hread, hwrite;\n+static HANDLE hwrite1 = INVALID_HANDLE_VALUE, hwrite2 = INVALID_HANDLE_VALUE;\n+static HANDLE hconsole1, hconsole2;\n \n #ifdef __MINGW32__\n typedef struct _CONSOLE_FONT_INFOEX {\n@@ -44,27 +42,19 @@ typedef struct _CONSOLE_FONT_INFOEX {\n typedef BOOL (WINAPI *PGETCURRENTCONSOLEFONTEX)(HANDLE, BOOL,\n \t\tPCONSOLE_FONT_INFOEX);\n \n-static void print_font_warning(void)\n+static void warn_if_raster_font(void)\n {\n-\twarning(\"Your console font probably doesn\\'t support Unicode. If \"\n-\t\t\"you experience strange characters in the output, consider \"\n-\t\t\"switching to a TrueType font such as Lucida Console!\");\n-}\n-\n-static void check_truetype_font(void)\n-{\n-\tstatic int truetype_font_checked;\n \tDWORD fontFamily = 0;\n \tPGETCURRENTCONSOLEFONTEX pGetCurrentConsoleFontEx;\n \n-\t/* don't do this twice */\n-\tif (truetype_font_checked)\n+\t/* don't bother if output was ascii only */\n+\tif (!non_ascii_used)\n \t\treturn;\n-\ttruetype_font_checked = 1;\n \n \t/* GetCurrentConsoleFontEx is available since Vista */\n \tpGetCurrentConsoleFontEx = (PGETCURRENTCONSOLEFONTEX) GetProcAddress(\n-\t\t\tGetModuleHandle(\"kernel32.dll\"), \"GetCurrentConsoleFontEx\");\n+\t\t\tGetModuleHandle(\"kernel32.dll\"),\n+\t\t\t\"GetCurrentConsoleFontEx\");\n \tif (pGetCurrentConsoleFontEx) {\n \t\tCONSOLE_FONT_INFOEX cfi;\n \t\tcfi.cbSize = sizeof(cfi);\n@@ -73,8 +63,8 @@ static void check_truetype_font(void)\n \t} else {\n \t\t/* pre-Vista: check default console font in registry */\n \t\tHKEY hkey;\n-\t\tif (ERROR_SUCCESS == RegOpenKeyExA(HKEY_CURRENT_USER, \"Console\", 0,\n-\t\t\t\tKEY_READ, &hkey)) {\n+\t\tif (ERROR_SUCCESS == RegOpenKeyExA(HKEY_CURRENT_USER, \"Console\",\n+\t\t\t\t0, KEY_READ, &hkey)) {\n \t\t\tDWORD size = sizeof(fontFamily);\n \t\t\tRegQueryValueExA(hkey, \"FontFamily\", NULL, NULL,\n \t\t\t\t\t(LPVOID) &fontFamily, &size);\n@@ -82,61 +72,64 @@ static void check_truetype_font(void)\n \t\t}\n \t}\n \n-\tif (!(fontFamily & TMPF_TRUETYPE))\n-\t\tatexit(print_font_warning);\n+\tif (!(fontFamily & TMPF_TRUETYPE)) {\n+\t\tconst wchar_t *msg = L\"\\nWarning: Your console font probably \"\n+\t\t\tL\"doesn\\'t support Unicode. If you experience strange \"\n+\t\t\tL\"characters in the output, consider switching to a \"\n+\t\t\tL\"TrueType font such as Consolas!\\n\";\n+\t\tDWORD dummy;\n+\t\tWriteConsoleW(console, msg, wcslen(msg), &dummy, NULL);\n+\t}\n }\n \n-static int is_console(FILE *stream)\n+static int is_console(int fd)\n {\n \tCONSOLE_SCREEN_BUFFER_INFO sbi;\n \tHANDLE hcon;\n \n \tstatic int initialized = 0;\n \n-\t/* use cached value if stream hasn't changed */\n-\tif (stream == last_stream)\n-\t\treturn console != NULL;\n-\n-\tlast_stream = stream;\n-\tconsole = NULL;\n-\n-\t/* get OS handle of the stream */\n-\thcon = (HANDLE) _get_osfhandle(_fileno(stream));\n+\t/* get OS handle of the file descriptor */\n+\thcon = (HANDLE) _get_osfhandle(fd);\n \tif (hcon == INVALID_HANDLE_VALUE)\n \t\treturn 0;\n \n+\t/* check if its a device (i.e. console, printer, serial port) */\n+\tif (GetFileType(hcon) != FILE_TYPE_CHAR)\n+\t\treturn 0;\n+\n \t/* check if its a handle to a console output screen buffer */\n \tif (!GetConsoleScreenBufferInfo(hcon, &sbi))\n \t\treturn 0;\n \n+\t/* initialize attributes */\n \tif (!initialized) {\n \t\tattr = plain_attr = sbi.wAttributes;\n \t\tnegative = 0;\n \t\tinitialized = 1;\n \t}\n \n-\tconsole = hcon;\n \treturn 1;\n }\n \n-static int write_console(const char *str, size_t len)\n+#define BUFFER_SIZE 4096\n+#define MAX_PARAMS 16\n+\n+static void write_console(unsigned char *str, size_t len)\n {\n-\t/* convert utf-8 to utf-16, write directly to console */\n-\tint wlen = MultiByteToWideChar(CP_UTF8, 0, str, len, NULL, 0);\n-\twchar_t *wbuf = (wchar_t *) alloca(wlen * sizeof(wchar_t));\n-\tMultiByteToWideChar(CP_UTF8, 0, str, len, wbuf, wlen);\n+\t/* only called from console_thread, so a static buffer will do */\n+\tstatic wchar_t wbuf[2 * BUFFER_SIZE + 1];\n+\tDWORD dummy;\n \n-\tWriteConsoleW(console, wbuf, wlen, NULL, NULL);\n+\t/* convert utf-8 to utf-16 */\n+\tint wlen = xutftowcsn(wbuf, (char*) str, ARRAY_SIZE(wbuf), len);\n \n-\t/*\n-\t * if non-ascii characters are printed, check that the current console\n-\t * font supports this\n-\t */\n-\tif (wlen != len)\n-\t\tcheck_truetype_font();\n+\t/* write directly to console */\n+\tWriteConsoleW(console, wbuf, wlen, &dummy, NULL);\n \n-\t/* return original (utf-8 encoded) length */\n-\treturn len;\n+\t/* remember if non-ascii characters are printed */\n+\tif (wlen != len)\n+\t\tnon_ascii_used = 1;\n }\n \n #define FOREGROUND_ALL (FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_BLUE)\n@@ -182,18 +175,13 @@ static void erase_in_line(void)\n \t\t&dummy);\n }\n \n-\n-static const char *set_attr(const char *str)\n+static void set_attr(char func, const int *params, int paramlen)\n {\n-\tconst char *func;\n-\tsize_t len = strspn(str, \"0123456789;\");\n-\tfunc = str + len;\n-\n-\tswitch (*func) {\n+\tint i;\n+\tswitch (func) {\n \tcase 'm':\n-\t\tdo {\n-\t\t\tlong val = strtol(str, (char **)&str, 10);\n-\t\t\tswitch (val) {\n+\t\tfor (i = 0; i < paramlen; i++) {\n+\t\t\tswitch (params[i]) {\n \t\t\tcase 0: /* reset */\n \t\t\t\tattr = plain_attr;\n \t\t\t\tnegative = 0;\n@@ -316,9 +304,7 @@ static const char *set_attr(const char *str)\n \t\t\t\t/* Unsupported code */\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\tstr++;\n-\t\t} while (*(str-1) == ';');\n-\n+\t\t}\n \t\tset_console_attr();\n \t\tbreak;\n \tcase 'K':\n@@ -328,112 +314,251 @@ static const char *set_attr(const char *str)\n \t\t/* Unsupported code */\n \t\tbreak;\n \t}\n-\n-\treturn func + 1;\n }\n \n-static int ansi_emulate(const char *str, FILE *stream)\n-{\n-\tint rv = 0;\n-\tconst char *pos = str;\n+enum {\n+\tTEXT = 0, ESCAPE = 033, BRACKET = '['\n+};\n \n-\tfflush(stream);\n+static DWORD WINAPI console_thread(LPVOID unused)\n+{\n+\tunsigned char buffer[BUFFER_SIZE];\n+\tDWORD bytes;\n+\tint start, end = 0, c, parampos = 0, state = TEXT;\n+\tint params[MAX_PARAMS];\n+\n+\twhile (1) {\n+\t\t/* read next chunk of bytes from the pipe */\n+\t\tif (!ReadFile(hread, buffer + end, BUFFER_SIZE - end, &bytes,\n+\t\t\t\tNULL)) {\n+\t\t\t/* exit if pipe has been closed or disconnected */\n+\t\t\tif (GetLastError() == ERROR_PIPE_NOT_CONNECTED ||\n+\t\t\t\t\tGetLastError() == ERROR_BROKEN_PIPE)\n+\t\t\t\tbreak;\n+\t\t\t/* ignore other errors */\n+\t\t\tcontinue;\n+\t\t}\n \n-\twhile (*pos) {\n-\t\tpos = strstr(str, \"\\033[\");\n-\t\tif (pos) {\n-\t\t\tsize_t len = pos - str;\n+\t\t/* scan the bytes and handle ANSI control codes */\n+\t\tbytes += end;\n+\t\tstart = end = 0;\n+\t\twhile (end < bytes) {\n+\t\t\tc = buffer[end++];\n+\t\t\tswitch (state) {\n+\t\t\tcase TEXT:\n+\t\t\t\tif (c == ESCAPE) {\n+\t\t\t\t\t/* print text seen so far */\n+\t\t\t\t\tif (end - 1 > start)\n+\t\t\t\t\t\twrite_console(buffer + start,\n+\t\t\t\t\t\t\tend - 1 - start);\n+\n+\t\t\t\t\t/* then start parsing escape sequence */\n+\t\t\t\t\tstart = end - 1;\n+\t\t\t\t\tmemset(params, 0, sizeof(params));\n+\t\t\t\t\tparampos = 0;\n+\t\t\t\t\tstate = ESCAPE;\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\n+\t\t\tcase ESCAPE:\n+\t\t\t\t/* continue if \"\\033[\", otherwise bail out */\n+\t\t\t\tstate = (c == BRACKET) ? BRACKET : TEXT;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase BRACKET:\n+\t\t\t\t/* parse [0-9;]* into array of parameters */\n+\t\t\t\tif (c >= '0' && c <= '9') {\n+\t\t\t\t\tparams[parampos] *= 10;\n+\t\t\t\t\tparams[parampos] += c - '0';\n+\t\t\t\t} else if (c == ';') {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * next parameter, bail out if out of\n+\t\t\t\t\t * bounds\n+\t\t\t\t\t */\n+\t\t\t\t\tparampos++;\n+\t\t\t\t\tif (parampos >= MAX_PARAMS)\n+\t\t\t\t\t\tstate = TEXT;\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * end of escape sequence, change\n+\t\t\t\t\t * console attributes\n+\t\t\t\t\t */\n+\t\t\t\t\tset_attr(c, params, parampos + 1);\n+\t\t\t\t\tstart = end;\n+\t\t\t\t\tstate = TEXT;\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n \n-\t\t\tif (len) {\n-\t\t\t\tsize_t out_len = write_console(str, len);\n-\t\t\t\trv += out_len;\n-\t\t\t\tif (out_len < len)\n-\t\t\t\t\treturn rv;\n+\t\t/* print remaining text unless parsing an escape sequence */\n+\t\tif (state == TEXT && end > start) {\n+\t\t\t/* check for incomplete UTF-8 sequences and fix end */\n+\t\t\tif (buffer[end - 1] >= 0x80) {\n+\t\t\t\tif (buffer[end -1] >= 0xc0)\n+\t\t\t\t\tend--;\n+\t\t\t\telse if (end - 1 > start &&\n+\t\t\t\t\t\tbuffer[end - 2] >= 0xe0)\n+\t\t\t\t\tend -= 2;\n+\t\t\t\telse if (end - 2 > start &&\n+\t\t\t\t\t\tbuffer[end - 3] >= 0xf0)\n+\t\t\t\t\tend -= 3;\n \t\t\t}\n \n-\t\t\tstr = pos + 2;\n-\t\t\trv += 2;\n+\t\t\t/* print remaining complete UTF-8 sequences */\n+\t\t\tif (end > start)\n+\t\t\t\twrite_console(buffer + start, end - start);\n \n-\t\t\tpos = set_attr(str);\n-\t\t\trv += pos - str;\n-\t\t\tstr = pos;\n+\t\t\t/* move remaining bytes to the front */\n+\t\t\tif (end < bytes)\n+\t\t\t\tmemmove(buffer, buffer + end, bytes - end);\n+\t\t\tend = bytes - end;\n \t\t} else {\n-\t\t\tsize_t len = strlen(str);\n-\t\t\trv += write_console(str, len);\n-\t\t\treturn rv;\n+\t\t\t/* all data has been consumed, mark buffer empty */\n+\t\t\tend = 0;\n \t\t}\n \t}\n-\treturn rv;\n-}\n-\n-int winansi_fputs(const char *str, FILE *stream)\n-{\n-\tint rv;\n-\n-\tif (!is_console(stream))\n-\t\treturn fputs(str, stream);\n \n-\trv = ansi_emulate(str, stream);\n+\t/* check if the console font supports unicode */\n+\twarn_if_raster_font();\n \n-\tif (rv >= 0)\n-\t\treturn 0;\n-\telse\n-\t\treturn EOF;\n+\tCloseHandle(hread);\n+\treturn 0;\n }\n \n-int winansi_vfprintf(FILE *stream, const char *format, va_list list)\n+static void winansi_exit(void)\n {\n-\tint len, rv;\n-\tchar small_buf[256];\n-\tchar *buf = small_buf;\n-\tva_list cp;\n-\n-\tif (!is_console(stream))\n-\t\tgoto abort;\n+\t/* flush all streams */\n+\t_flushall();\n+\n+\t/* signal console thread to exit */\n+\tFlushFileBuffers(hwrite);\n+\tDisconnectNamedPipe(hwrite);\n+\n+\t/* wait for console thread to copy remaining data */\n+\tWaitForSingleObject(hthread, INFINITE);\n+\n+\t/* cleanup handles... */\n+\tif (hwrite1 != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(hwrite1);\n+\tif (hwrite2 != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(hwrite2);\n+\tCloseHandle(hwrite);\n+\tCloseHandle(hthread);\n+}\n \n-\tva_copy(cp, list);\n-\tlen = vsnprintf(small_buf, sizeof(small_buf), format, cp);\n-\tva_end(cp);\n+static void die_lasterr(const char *fmt, ...)\n+{\n+\tva_list params;\n+\tva_start(params, fmt);\n+\terrno = err_win_to_posix(GetLastError());\n+\tdie_errno(fmt, params);\n+\tva_end(params);\n+}\n \n-\tif (len > sizeof(small_buf) - 1) {\n-\t\tbuf = malloc(len + 1);\n-\t\tif (!buf)\n-\t\t\tgoto abort;\n+static HANDLE duplicate_handle(HANDLE hnd)\n+{\n+\tHANDLE hresult, hproc = GetCurrentProcess();\n+\tif (!DuplicateHandle(hproc, hnd, hproc, &hresult, 0, TRUE,\n+\t\t\tDUPLICATE_SAME_ACCESS))\n+\t\tdie_lasterr(\"DuplicateHandle(%li) failed\", (long) hnd);\n+\treturn hresult;\n+}\n \n-\t\tlen = vsnprintf(buf, len + 1, format, list);\n-\t}\n+static HANDLE redirect_console(FILE *stream, HANDLE *phcon, int new_fd)\n+{\n+\t/* get original console handle */\n+\tint fd = _fileno(stream);\n+\tHANDLE hcon = (HANDLE) _get_osfhandle(fd);\n+\tif (hcon == INVALID_HANDLE_VALUE)\n+\t\tdie_errno(\"_get_osfhandle(%i) failed\", fd);\n \n-\trv = ansi_emulate(buf, stream);\n+\t/* save a copy to phcon and console (used by the background thread) */\n+\tconsole = *phcon = duplicate_handle(hcon);\n \n-\tif (buf != small_buf)\n-\t\tfree(buf);\n-\treturn rv;\n+\t/* duplicate new_fd over fd (closes fd and associated handle (hcon)) */\n+\tif (_dup2(new_fd, fd))\n+\t\tdie_errno(\"_dup2(%i, %i) failed\", new_fd, fd);\n \n-abort:\n-\trv = vfprintf(stream, format, list);\n-\treturn rv;\n+\t/* no buffering, or stdout / stderr will be out of sync */\n+\tsetbuf(stream, NULL);\n+\treturn (HANDLE) _get_osfhandle(fd);\n }\n \n-int winansi_fprintf(FILE *stream, const char *format, ...)\n+void winansi_init(void)\n {\n-\tva_list list;\n-\tint rv;\n+\tint con1, con2, hwrite_fd;\n+\tchar name[32];\n \n-\tva_start(list, format);\n-\trv = winansi_vfprintf(stream, format, list);\n-\tva_end(list);\n+\t/* check if either stdout or stderr is a console output screen buffer */\n+\tcon1 = is_console(1);\n+\tcon2 = is_console(2);\n+\tif (!con1 && !con2)\n+\t\treturn;\n \n-\treturn rv;\n+\t/* create a named pipe to communicate with the console thread */\n+\tsprintf(name, \"\\\\\\\\.\\\\pipe\\\\winansi%lu\", GetCurrentProcessId());\n+\thwrite = CreateNamedPipe(name, PIPE_ACCESS_OUTBOUND,\n+\t\tPIPE_TYPE_BYTE | PIPE_WAIT, 1, BUFFER_SIZE, 0, 0, NULL);\n+\tif (hwrite == INVALID_HANDLE_VALUE)\n+\t\tdie_lasterr(\"CreateNamedPipe failed\");\n+\n+\thread = CreateFile(name, GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);\n+\tif (hread == INVALID_HANDLE_VALUE)\n+\t\tdie_lasterr(\"CreateFile for named pipe failed\");\n+\n+\t/* start console spool thread on the pipe's read end */\n+\ththread = CreateThread(NULL, 0, console_thread, NULL, 0, NULL);\n+\tif (hthread == INVALID_HANDLE_VALUE)\n+\t\tdie_lasterr(\"CreateThread(console_thread) failed\");\n+\n+\t/* schedule cleanup routine */\n+\tif (atexit(winansi_exit))\n+\t\tdie_errno(\"atexit(winansi_exit) failed\");\n+\n+\t/* create a file descriptor for the write end of the pipe */\n+\thwrite_fd = _open_osfhandle((long) duplicate_handle(hwrite), _O_BINARY);\n+\tif (hwrite_fd == -1)\n+\t\tdie_errno(\"_open_osfhandle(%li) failed\", (long) hwrite);\n+\n+\t/* redirect stdout / stderr to the pipe */\n+\tif (con1)\n+\t\thwrite1 = redirect_console(stdout, &hconsole1, hwrite_fd);\n+\tif (con2)\n+\t\thwrite2 = redirect_console(stderr, &hconsole2, hwrite_fd);\n+\n+\t/* close pipe file descriptor (also closes the duped hwrite) */\n+\tclose(hwrite_fd);\n }\n \n-int winansi_printf(const char *format, ...)\n+static int is_same_handle(HANDLE hnd, int fd)\n {\n-\tva_list list;\n-\tint rv;\n+\treturn hnd != INVALID_HANDLE_VALUE && hnd == (HANDLE) _get_osfhandle(fd);\n+}\n \n-\tva_start(list, format);\n-\trv = winansi_vfprintf(stdout, format, list);\n-\tva_end(list);\n+/*\n+ * Return true if stdout / stderr is a pipe redirecting to the console.\n+ */\n+int winansi_isatty(int fd)\n+{\n+\tif (fd == 1 && is_same_handle(hwrite1, 1))\n+\t\treturn 1;\n+\telse if (fd == 2 && is_same_handle(hwrite2, 2))\n+\t\treturn 1;\n+\telse\n+\t\treturn isatty(fd);\n+}\n \n-\treturn rv;\n+/*\n+ * Returns the real console handle if stdout / stderr is a pipe redirecting\n+ * to the console. Allows spawn / exec to pass the console to the next process.\n+ */\n+HANDLE winansi_get_osfhandle(int fd)\n+{\n+\tif (fd == 1 && is_same_handle(hwrite1, 1))\n+\t\treturn hconsole1;\n+\telse if (fd == 2 && is_same_handle(hwrite2, 2))\n+\t\treturn hconsole2;\n+\telse\n+\t\treturn (HANDLE) _get_osfhandle(fd);\n }\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"243587","messageId":"1402127845-4862-7-git-send-email-kasal@ucw.cz","threadId":"36854","inReplyTo":"1402127845-4862-1-git-send-email-kasal@ucw.cz","subject":"[PATCH v2 6/6] Win32: fix broken pipe detection","fromName":"Stepan Kasal","fromEmail":"kasal@ucw.cz","sentAt":"2014-06-07T07:57:25Z","receivedAt":"2014-06-07T07:57:25Z","isPatch":true,"sender":{"key":"kasal@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/1481596?v=4"},"body":"From: Karsten Blees \nDate: Thu, 1 Mar 2012 21:53:54 +0100\n\nAs of \"Win32: Thread-safe windows console output\", git-log no longer\nterminates when the pager process dies. This is due to disabling buffering\nfor the replaced stdout / stderr streams. Git-log will periodically fflush\nstdout (see write_or_die.c/mayble_flush_or_die()), but with no buffering,\nthis is a NOP that always succeeds (so we never detect the EPIPE error).\n\nExchange the original console handles with our console thread pipe handles\nby accessing the internal MSVCRT data structures directly (which are\nexposed via __pioinfo for some reason).\n\nImplement this with minimal assumptions about the actual data structure to\nmake it work with different (hopefully even future) MSVCRT versions.\n\nWhile messing with internal data structures is ugly, this patch solves the\nproblem at the source instead of adding more workarounds. We no longer need\nthe special winansi_isatty override, and the limitations documented in\n\"Win32: Thread-safe windows console output\" are gone (i.e. fdopen(1/2)\nreturns unbuffered streams now, and isatty() for duped console file\ndescriptors works as expected).\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Stepan Kasal \n---\n compat/mingw.h | 2 -\n compat/winansi.c | 114 ++++++++++++++++++++++++++++++++++---------------------\n 2 files changed, 70 insertions(+), 46 deletions(-)\n\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 4b638d8..8dac6f9 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -318,9 +318,7 @@ int mingw_raise(int sig);\n */\n \n void winansi_init(void);\n-int winansi_isatty(int fd);\n HANDLE winansi_get_osfhandle(int fd);\n-#define isatty winansi_isatty\n \n /*\n * git specific compatibility\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex fcdd6dc..f96d5c2 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -8,11 +8,6 @@\n #include \n \n /*\n- Functions to be wrapped:\n-*/\n-#undef isatty\n-\n-/*\n ANSI codes used by git: m, K\n \n This file is git-specific. Therefore, this file does not attempt\n@@ -104,6 +99,7 @@ static int is_console(int fd)\n \n \t/* initialize attributes */\n \tif (!initialized) {\n+\t\tconsole = hcon;\n \t\tattr = plain_attr = sbi.wAttributes;\n \t\tnegative = 0;\n \t\tinitialized = 1;\n@@ -465,29 +461,80 @@ static HANDLE duplicate_handle(HANDLE hnd)\n \treturn hresult;\n }\n \n-static HANDLE redirect_console(FILE *stream, HANDLE *phcon, int new_fd)\n-{\n-\t/* get original console handle */\n-\tint fd = _fileno(stream);\n-\tHANDLE hcon = (HANDLE) _get_osfhandle(fd);\n-\tif (hcon == INVALID_HANDLE_VALUE)\n-\t\tdie_errno(\"_get_osfhandle(%i) failed\", fd);\n \n-\t/* save a copy to phcon and console (used by the background thread) */\n-\tconsole = *phcon = duplicate_handle(hcon);\n+/*\n+ * Make MSVCRT's internal file descriptor control structure accessible\n+ * so that we can tweak OS handles and flags directly (we need MSVCRT\n+ * to treat our pipe handle as if it were a console).\n+ *\n+ * We assume that the ioinfo structure (exposed by MSVCRT.dll via\n+ * __pioinfo) starts with the OS handle and the flags. The exact size\n+ * varies between MSVCRT versions, so we try different sizes until\n+ * toggling the FDEV bit of _pioinfo(1)->osflags is reflected in\n+ * isatty(1).\n+ */\n+typedef struct {\n+\tHANDLE osfhnd;\n+\tchar osflags;\n+} ioinfo;\n+\n+extern __declspec(dllimport) ioinfo *__pioinfo[];\n \n-\t/* duplicate new_fd over fd (closes fd and associated handle (hcon)) */\n-\tif (_dup2(new_fd, fd))\n-\t\tdie_errno(\"_dup2(%i, %i) failed\", new_fd, fd);\n+static size_t sizeof_ioinfo = 0;\n \n-\t/* no buffering, or stdout / stderr will be out of sync */\n-\tsetbuf(stream, NULL);\n-\treturn (HANDLE) _get_osfhandle(fd);\n+#define IOINFO_L2E 5\n+#define IOINFO_ARRAY_ELTS (1 << IOINFO_L2E)\n+\n+#define FDEV 0x40\n+\n+static inline ioinfo* _pioinfo(int fd)\n+{\n+\treturn (ioinfo*)((char*)__pioinfo[fd >> IOINFO_L2E] +\n+\t\t\t(fd & (IOINFO_ARRAY_ELTS - 1)) * sizeof_ioinfo);\n+}\n+\n+static int init_sizeof_ioinfo()\n+{\n+\tint istty, wastty;\n+\t/* don't init twice */\n+\tif (sizeof_ioinfo)\n+\t\treturn sizeof_ioinfo >= 256;\n+\n+\tsizeof_ioinfo = sizeof(ioinfo);\n+\twastty = isatty(1);\n+\twhile (sizeof_ioinfo < 256) {\n+\t\t/* toggle FDEV flag, check isatty, then toggle back */\n+\t\t_pioinfo(1)->osflags ^= FDEV;\n+\t\tistty = isatty(1);\n+\t\t_pioinfo(1)->osflags ^= FDEV;\n+\t\t/* return if we found the correct size */\n+\t\tif (istty != wastty)\n+\t\t\treturn 0;\n+\t\tsizeof_ioinfo += sizeof(void*);\n+\t}\n+\terror(\"Tweaking file descriptors doesn't work with this MSVCRT.dll\");\n+\treturn 1;\n+}\n+\n+static HANDLE swap_osfhnd(int fd, HANDLE new_handle)\n+{\n+\tioinfo *pioinfo;\n+\tHANDLE old_handle;\n+\n+\t/* init ioinfo size if we haven't done so */\n+\tif (init_sizeof_ioinfo())\n+\t\treturn INVALID_HANDLE_VALUE;\n+\n+\t/* get ioinfo pointer and change the handles */\n+\tpioinfo = _pioinfo(fd);\n+\told_handle = pioinfo->osfhnd;\n+\tpioinfo->osfhnd = new_handle;\n+\treturn old_handle;\n }\n \n void winansi_init(void)\n {\n-\tint con1, con2, hwrite_fd;\n+\tint con1, con2;\n \tchar name[32];\n \n \t/* check if either stdout or stderr is a console output screen buffer */\n@@ -516,19 +563,11 @@ void winansi_init(void)\n \tif (atexit(winansi_exit))\n \t\tdie_errno(\"atexit(winansi_exit) failed\");\n \n-\t/* create a file descriptor for the write end of the pipe */\n-\thwrite_fd = _open_osfhandle((long) duplicate_handle(hwrite), _O_BINARY);\n-\tif (hwrite_fd == -1)\n-\t\tdie_errno(\"_open_osfhandle(%li) failed\", (long) hwrite);\n-\n \t/* redirect stdout / stderr to the pipe */\n \tif (con1)\n-\t\thwrite1 = redirect_console(stdout, &hconsole1, hwrite_fd);\n+\t\thconsole1 = swap_osfhnd(1, hwrite1 = duplicate_handle(hwrite));\n \tif (con2)\n-\t\thwrite2 = redirect_console(stderr, &hconsole2, hwrite_fd);\n-\n-\t/* close pipe file descriptor (also closes the duped hwrite) */\n-\tclose(hwrite_fd);\n+\t\thconsole2 = swap_osfhnd(2, hwrite2 = duplicate_handle(hwrite));\n }\n \n static int is_same_handle(HANDLE hnd, int fd)\n@@ -537,19 +576,6 @@ static int is_same_handle(HANDLE hnd, int fd)\n }\n \n /*\n- * Return true if stdout / stderr is a pipe redirecting to the console.\n- */\n-int winansi_isatty(int fd)\n-{\n-\tif (fd == 1 && is_same_handle(hwrite1, 1))\n-\t\treturn 1;\n-\telse if (fd == 2 && is_same_handle(hwrite2, 2))\n-\t\treturn 1;\n-\telse\n-\t\treturn isatty(fd);\n-}\n-\n-/*\n * Returns the real console handle if stdout / stderr is a pipe redirecting\n * to the console. Allows spawn / exec to pass the console to the next process.\n */\n-- \n2.0.0.9635.g0be03cb\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"244054","messageId":"539A95F1.9030900@kdbg.org","threadId":"36854","inReplyTo":"1402127845-4862-6-git-send-email-kasal@ucw.cz","subject":"Re: [PATCH v2 5/6] Win32: Thread-safe windows console output","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2014-06-13T06:10:57Z","receivedAt":"2014-06-13T06:10:57Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 07.06.2014 09:57, schrieb Stepan Kasal:\n> From: Karsten Blees \n> Date: Sat, 14 Jan 2012 22:24:19 +0100\n>\n> Winansi.c has many static variables that are accessed and modified from\n> the [v][f]printf / fputs functions overridden in the file. This may cause\n> multi threaded git commands that print to the console to produce corrupted\n> output or even crash.\n>\n> Additionally, winansi.c doesn't override all functions that can be used to\n> print to the console (e.g. fwrite, write, fputc are missing), so that ANSI\n> escapes don't work properly for some git commands (e.g. git-grep).\n>\n> Instead of doing ANSI emulation in just a few wrapped functions on top of\n> the IO API, let's plug into the IO system and take advantage of the thread\n> safety inherent to the IO system.\n>\n> Redirect stdout and stderr to a pipe if they point to the console. A\n> background thread reads from the pipe, handles ANSI escape sequences and\n> UTF-8 to UTF-16 conversion, then writes to the console.\n\nThere's something fishy with this patch. Please checkout and build \neac14f8909d9. Then run t5000-tar-tree.sh like so from a CMD prompt:\n\n sh t5000-tar-tree.sh -v -i\n\nNotice that in test 36 (invoke tar filter by extension) the tar file is \nwritten to the console instead of the file. Hit Ctrl-C to interrupt the \ntest; do not remove the trash directory. You can verify the incorrect \nbehavior like this:\n\n cd \"t\\trash directory.t5000-tar-tree\"\n ..\\..\\git archive -o config-implicit.tar.foo HEAD\n\nIt writes the tar file to the console. When you build the parent commit, \nand repeat these two commands, there is no unexpected console output.\n\nThe patch fcd428f4a952 (Win32: fix broken pipe detection) does not fix the \nincorrect behavior.\n\nI haven't dug (and won't dig) deeper than that.\n\n-- Hannes\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/d/optout.\n"},{"id":"244179","messageId":"539B7682.7070308@gmail.com","threadId":"36854","inReplyTo":"539A95F1.9030900@kdbg.org","subject":"[PATCH 7/6] Win32: reliably detect console pipe handles","fromName":"Karsten Blees","fromEmail":"karsten.blees@gmail.com","sentAt":"2014-06-13T22:09:06Z","receivedAt":"2014-06-13T22:09:06Z","isPatch":true,"sender":{"key":"karsten.blees@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1111200?v=4"},"body":"As of \"Win32: Thread-safe windows console output\", child processes may\nprint to the console even if stdout has been redirected to a file. E.g.:\n\n git config tar.cat.command \"cat\"\n git archive -o test.cat HEAD\n\nDetecting whether stdout / stderr point to our console pipe is currently\nbased on the assumption that OS HANDLE values are never reused. This is\napparently not true if stdout / stderr is replaced via dup2() (as in\nbuiltin/archive.c:17).\n\nInstead of comparing handle values, check if the file descriptor isatty()\nbacked by a pipe OS handle. This is only possible by swapping the handles\nin MSVCRT's internal data structures, as we do in winansi_init().\n\nReported-by: Johannes Sixt \nSigned-off-by: Karsten Blees \n---\n\nThanks for reporting this.\n\nThe fix applies on top of [6/6] Win32: fix broken pipe detection (should\nprobably not be squashed, as its obviously not as well tested as the\nrest of the series).\n\nCheers,\nKarsten\n\n\n compat/winansi.c | 25 +++++++------------------\n 1 file changed, 7 insertions(+), 18 deletions(-)\n\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex f96d5c2..efc5bb3 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -20,7 +20,6 @@ static WORD attr;\n static int negative;\n static int non_ascii_used = 0;\n static HANDLE hthread, hread, hwrite;\n-static HANDLE hwrite1 = INVALID_HANDLE_VALUE, hwrite2 = INVALID_HANDLE_VALUE;\n static HANDLE hconsole1, hconsole2;\n \n #ifdef __MINGW32__\n@@ -435,10 +434,6 @@ static void winansi_exit(void)\n \tWaitForSingleObject(hthread, INFINITE);\n \n \t/* cleanup handles... */\n-\tif (hwrite1 != INVALID_HANDLE_VALUE)\n-\t\tCloseHandle(hwrite1);\n-\tif (hwrite2 != INVALID_HANDLE_VALUE)\n-\t\tCloseHandle(hwrite2);\n \tCloseHandle(hwrite);\n \tCloseHandle(hthread);\n }\n@@ -565,14 +560,9 @@ void winansi_init(void)\n \n \t/* redirect stdout / stderr to the pipe */\n \tif (con1)\n-\t\thconsole1 = swap_osfhnd(1, hwrite1 = duplicate_handle(hwrite));\n+\t\thconsole1 = swap_osfhnd(1, duplicate_handle(hwrite));\n \tif (con2)\n-\t\thconsole2 = swap_osfhnd(2, hwrite2 = duplicate_handle(hwrite));\n-}\n-\n-static int is_same_handle(HANDLE hnd, int fd)\n-{\n-\treturn hnd != INVALID_HANDLE_VALUE && hnd == (HANDLE) _get_osfhandle(fd);\n+\t\thconsole2 = swap_osfhnd(2, duplicate_handle(hwrite));\n }\n \n /*\n@@ -581,10 +571,9 @@ static int is_same_handle(HANDLE hnd, int fd)\n */\n HANDLE winansi_get_osfhandle(int fd)\n {\n-\tif (fd == 1 && is_same_handle(hwrite1, 1))\n-\t\treturn hconsole1;\n-\telse if (fd == 2 && is_same_handle(hwrite2, 2))\n-\t\treturn hconsole2;\n-\telse\n-\t\treturn (HANDLE) _get_osfhandle(fd);\n+\tHANDLE hnd = (HANDLE) _get_osfhandle(fd);\n+\tif ((fd == 1 || fd == 2) && isatty(fd)\n+\t && GetFileType(hnd) == FILE_TYPE_PIPE)\n+\t\treturn (fd == 1) ? hconsole1 : hconsole2;\n+\treturn hnd;\n }\n-- \n1.9.3.10.ge3256ea\n"}]}