{"thread":{"id":"30977","subject":"[PATCH v9] git on Mac OS and precomposed unicode","startedAt":"2012-07-08T13:50:25Z","lastAt":"2012-08-20T22:02:43Z","messageCount":11,"participants":["Torsten Bögershausen","Robin Rosenberg","Junio C Hamano"],"isPatch":true,"patchVersion":9,"patchTotal":null},"messages":[{"id":"194786","messageId":"201207081550.25850.tboegi@web.de","threadId":"30977","inReplyTo":null,"subject":"[PATCH v9] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-07-08T13:50:25Z","receivedAt":"2012-07-08T13:50:25Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Mac OS X mangles file names containing unicode on file systems HFS+,\nVFAT or SAMBA.  When a file using unicode code points outside ASCII\nis created on a HFS+ drive, the file name is converted into\ndecomposed unicode and written to disk. No conversion is done if\nthe file name is already decomposed unicode.\n\nCalling open(\"\\xc3\\x84\", ...) with a precomposed \"Ä\" yields the same\nresult as open(\"\\x41\\xcc\\x88\",...) with a decomposed \"Ä\".\n\nAs a consequence, readdir() returns the file names in decomposed\nunicode, even if the user expects precomposed unicode.  Unlike on\nHFS+, Mac OS X stores files on a VFAT drive (e.g. an USB drive) in\nprecomposed unicode, but readdir() still returns file names in\ndecomposed unicode.  When a git repository is stored on a network\nshare using SAMBA, file names are send over the wire and written to\ndisk on the remote system in precomposed unicode, but Mac OS X\nreaddir() returns decomposed unicode to be compatible with its\nbehaviour on HFS+ and VFAT.\n\nThe unicode decomposition causes many problems:\n\n- The names \"git add\" and other commands get from the end user may\n  often be precomposed form (the decomposed form is not easily input\n  from the keyboard), but when the commands read from the filesystem\n  to see what it is going to update the index with already is on the\n  filesystem, readdir() will give decomposed form, which is different.\n\n- Similarly \"git log\", \"git mv\" and all other commands that need to\n  compare pathnames found on the command line (often but not always\n  precomposed form; a command line input resulting from globbing may\n  be in decomposed) with pathnames found in the tree objects (should\n  be precomposed form to be compatible with other systems and for\n  consistency in general).\n\n- The same for names stored in the index, which should be\n  precomposed, that may need to be compared with the names read from\n  readdir().\n\nNFS mounted from Linux is fully transparent and does not suffer from\nthe above.\n\nAs Mac OS X treats precomposed and decomposed file names as equal,\nwe can\n\n - wrap readdir() on Mac OS X to return the precomposed form, and\n\n - normalize decomposed form given from the command line also to the\n   precomposed form,\n\nto ensure that all pathnames used in Git are always in the\nprecomposed form.  This behaviour can be requested by setting\n\"core.precomposedunicode\" configuration variable to true.\n\nThe code in compat/precomposed_utf8.c implements basically 4 new\nfunctions: precomposed_utf8_opendir(), precomposed_utf8_readdir(),\nprecomposed_utf8_closedir() and precompose_argv().  The first three\nare to wrap opendir(3), readdir(3), and closedir(3) functions.\n\nThe argv[] conversion allows to use the TAB filename completion done\nby the shell on command line.  It tolerates other tools which use\nreaddir() to feed decomposed file names into git.\n\nWhen creating a new git repository with \"git init\" or \"git clone\",\n\"core.precomposedunicode\" will be set \"false\".\n\nThe user needs to activate this feature manually.  She typically\nsets core.precomposedunicode to \"true\" on HFS and VFAT, or file\nsystems mounted via SAMBA.\n\nHelped-by: Junio C Hamano <gitster@pobox.com>\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\n\nThanks to Junio & Andreas for the comments\n\nChanges since 7v:\n Code is 3% easier to read:\n   s/precomposed/precompomse/\n   s/__PRECOMPOSED_UNICODE_[H]C__/PRECOMPOSE_UNICODE_[CH]/\n\n  Documentation/config.txt:\n   clarified what happens when core.precomposeunicode == false:\n    -\tWhen false, file names are handled fully transparent by git, which means\n    -\tthat file names are stored as decomposed unicode in the repository.\n    +\tWhen false, file names are handled fully transparent by git,\n    +\twhich is backward compatible with older versions of git.\n   s/Git for Windows/msysGit 1.7.10/Git for Windows 1.7.10 or higher/\n\n Documentation/config.txt     |   9 ++\n Makefile                     |   3 +\n builtin/init-db.c            |   1 +\n cache.h                      |   1 +\n compat/precompose_utf8.c     | 190 +++++++++++++++++++++++++++++++++++++++++++\n compat/precompose_utf8.h     |  45 ++++++++++\n config.c                     |   5 ++\n environment.c                |   1 +\n git-compat-util.h            |   9 ++\n parse-options.c              |   1 +\n t/t3910-mac-os-precompose.sh | 164 +++++++++++++++++++++++++++++++++++++\n utf8.c                       |  26 +++---\n utf8.h                       |   1 +\n 13 files changed, 446 insertions(+), 10 deletions(-)\n create mode 100644 compat/precompose_utf8.c\n create mode 100644 compat/precompose_utf8.h\n create mode 100755 t/t3910-mac-os-precompose.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0bcea8a..e6a50fa 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -211,6 +211,15 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignorecase true if appropriate when the repository\n is created.\n \n+core.precomposeunicode::\n+\tThis option is only used by Mac OS implementation of git.\n+\tWhen core.precomposeunicode=true, git reverts the unicode decomposition\n+\tof filenames done by Mac OS. This is useful when sharing a repository\n+\tbetween Mac OS and Linux or Windows.\n+\t(Git for Windows 1.7.10 or higher is needed, or git under cygwin 1.7).\n+\tWhen false, file names are handled fully transparent by git,\n+\twhich is backward compatible with older versions of git.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Makefile b/Makefile\nindex cba9f77..d55484f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -605,6 +605,7 @@ LIB_H += compat/bswap.h\n LIB_H += compat/cygwin.h\n LIB_H += compat/mingw.h\n LIB_H += compat/obstack.h\n+LIB_H += compat/precompose_utf8.h\n LIB_H += compat/terminal.h\n LIB_H += compat/win32/dirent.h\n LIB_H += compat/win32/poll.h\n@@ -998,6 +999,8 @@ ifeq ($(uname_S),Darwin)\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tCOMPAT_OBJS += compat/precompose_utf8.o\n+\tBASIC_CFLAGS += -DPRECOMPOSE_UNICODE\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 0dacb8b..244fb7f 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -290,6 +290,7 @@ static int create_default_files(const char *template_path)\n \t\tstrcpy(path + len, \"CoNfIg\");\n \t\tif (!access(path, F_OK))\n \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n+\t\tprobe_utf8_pathname_composition(path, len);\n \t}\n \n \treturn reinit;\ndiff --git a/cache.h b/cache.h\nindex 506d157..697f0d1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -562,6 +562,7 @@ extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern int precomposed_unicode;\n \n enum branch_track {\n \tBRANCH_TRACK_UNSPECIFIED = -1,\ndiff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c\nnew file mode 100644\nindex 0000000..d40d1b3\n--- /dev/null\n+++ b/compat/precompose_utf8.c\n@@ -0,0 +1,190 @@\n+/*\n+ * Converts filenames from decomposed unicode into precomposed unicode.\n+ * Used on MacOS X.\n+*/\n+\n+\n+#define PRECOMPOSE_UNICODE_C\n+\n+#include \"cache.h\"\n+#include \"utf8.h\"\n+#include \"precompose_utf8.h\"\n+\n+typedef char *iconv_ibp;\n+const static char *repo_encoding = \"UTF-8\";\n+const static char *path_encoding = \"UTF-8-MAC\";\n+\n+\n+static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n+{\n+\tconst uint8_t *utf8p = (const uint8_t*) s;\n+\tsize_t strlen_chars = 0;\n+\tsize_t ret = 0;\n+\n+\tif ((!utf8p) || (!*utf8p)) {\n+\t\treturn 0;\n+\t}\n+\n+\twhile((*utf8p) && maxlen) {\n+\t\tif (*utf8p & 0x80)\n+\t\t\tret++;\n+\t\tstrlen_chars++;\n+\t\tutf8p++;\n+\t\tmaxlen--;\n+\t}\n+\tif (strlen_c)\n+\t\t*strlen_c = strlen_chars;\n+\n+\treturn ret;\n+}\n+\n+\n+void probe_utf8_pathname_composition(char *path, int len)\n+{\n+\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n+\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n+\tint output_fd;\n+\tif (precomposed_unicode != -1)\n+\t\treturn; /* We found it defined in the global config, respect it */\n+\tpath[len] = 0;\n+\tstrcpy(path + len, auml_nfc);\n+\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n+\tif (output_fd >=0) {\n+\t\tclose(output_fd);\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfd);\n+\t\t/* Indicate to the user, that we can configure it to true */\n+\t\tif (0 == access(path, R_OK))\n+\t\t\tgit_config_set(\"core.precomposeunicode\", \"false\");\n+\t\t\t/* To be backward compatible, set precomposed_unicode to 0 */\n+\t\tprecomposed_unicode = 0;\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfc);\n+\t\tunlink(path);\n+\t}\n+}\n+\n+\n+void precompose_argv(int argc, const char **argv)\n+{\n+\tint i = 0;\n+\tconst char *oldarg;\n+\tchar *newarg;\n+\ticonv_t ic_precompose;\n+\n+\tif (precomposed_unicode != 1)\n+\t\treturn;\n+\n+\tic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (ic_precompose == (iconv_t) -1)\n+\t\treturn;\n+\n+\twhile (i < argc) {\n+\t\tsize_t namelen;\n+\t\toldarg = argv[i];\n+\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n+\t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n+\t\t\tif (newarg)\n+\t\t\t\targv[i] = newarg;\n+\t\t}\n+\t\ti++;\n+\t}\n+\ticonv_close(ic_precompose);\n+}\n+\n+\n+PREC_DIR *precompose_utf8_opendir(const char *dirname)\n+{\n+\tPREC_DIR *prec_dir = xmalloc(sizeof(PREC_DIR));\n+\tprec_dir->dirent_nfc = xmalloc(sizeof(dirent_prec_psx));\n+\tprec_dir->dirent_nfc->max_name_len = sizeof(prec_dir->dirent_nfc->d_name);\n+\n+\tprec_dir->dirp = opendir(dirname);\n+\tif (!prec_dir->dirp) {\n+\t\tfree(prec_dir->dirent_nfc);\n+\t\tfree(prec_dir);\n+\t\treturn NULL;\n+\t} else {\n+\t\tint ret_errno = errno;\n+\t\tprec_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);\n+\t\t/* if iconv_open() fails, die() in readdir() if needed */\n+\t\terrno = ret_errno;\n+\t}\n+\n+\treturn prec_dir;\n+}\n+\n+struct dirent_prec_psx *precompose_utf8_readdir(PREC_DIR *prec_dir)\n+{\n+\tstruct dirent *res;\n+\tres = readdir(prec_dir->dirp);\n+\tif (res) {\n+\t\tsize_t namelenz = strlen(res->d_name) + 1; /* \\0 */\n+\t\tsize_t new_maxlen = namelenz;\n+\n+\t\tint ret_errno = errno;\n+\n+\t\tif (new_maxlen > prec_dir->dirent_nfc->max_name_len) {\n+\t\t\tsize_t new_len = sizeof(dirent_prec_psx) + new_maxlen -\n+\t\t\t\tsizeof(prec_dir->dirent_nfc->d_name);\n+\n+\t\t\tprec_dir->dirent_nfc = xrealloc(prec_dir->dirent_nfc, new_len);\n+\t\t\tprec_dir->dirent_nfc->max_name_len = new_maxlen;\n+\t\t}\n+\n+\t\tprec_dir->dirent_nfc->d_ino  = res->d_ino;\n+\t\tprec_dir->dirent_nfc->d_type = res->d_type;\n+\n+\t\tif ((precomposed_unicode == 1) && has_utf8(res->d_name, (size_t)-1, NULL)) {\n+\t\t\tif (prec_dir->ic_precompose == (iconv_t)-1) {\n+\t\t\t\tdie(\"iconv_open(%s,%s) failed, but needed:\\n\"\n+\t\t\t\t\t\t\"    precomposed unicode is not supported.\\n\"\n+\t\t\t\t\t\t\"    If you wnat to use decomposed unicode, run\\n\"\n+\t\t\t\t\t\t\"    \\\"git config core.precomposeunicode false\\\"\\n\",\n+\t\t\t\t\t\trepo_encoding, path_encoding);\n+\t\t\t} else {\n+\t\t\t\ticonv_ibp\tcp = (iconv_ibp)res->d_name;\n+\t\t\t\tsize_t inleft = namelenz;\n+\t\t\t\tchar *outpos = &prec_dir->dirent_nfc->d_name[0];\n+\t\t\t\tsize_t outsz = prec_dir->dirent_nfc->max_name_len;\n+\t\t\t\tsize_t cnt;\n+\t\t\t\terrno = 0;\n+\t\t\t\tcnt = iconv(prec_dir->ic_precompose, &cp, &inleft, &outpos, &outsz);\n+\t\t\t\tif (errno || inleft) {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * iconv() failed and errno could be E2BIG, EILSEQ, EINVAL, EBADF\n+\t\t\t\t\t * MacOS X avoids illegal byte sequemces.\n+\t\t\t\t\t * If they occur on a mounted drive (e.g. NFS) it is not worth to\n+\t\t\t\t\t * die() for that, but rather let the user see the original name\n+\t\t\t\t\t*/\n+\t\t\t\t\tnamelenz = 0; /* trigger strlcpy */\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t\telse\n+\t\t\tnamelenz = 0;\n+\n+\t\tif (!namelenz)\n+\t\t\tstrlcpy(prec_dir->dirent_nfc->d_name, res->d_name,\n+\t\t\t\t\t\t\tprec_dir->dirent_nfc->max_name_len);\n+\n+\t\terrno = ret_errno;\n+\t\treturn prec_dir->dirent_nfc;\n+\t}\n+\treturn NULL;\n+}\n+\n+\n+int precompose_utf8_closedir(PREC_DIR *prec_dir)\n+{\n+\tint ret_value;\n+\tint ret_errno;\n+\tret_value = closedir(prec_dir->dirp);\n+\tret_errno = errno;\n+\tif (prec_dir->ic_precompose != (iconv_t)-1)\n+\t\ticonv_close(prec_dir->ic_precompose);\n+\tfree(prec_dir->dirent_nfc);\n+\tfree(prec_dir);\n+\terrno = ret_errno;\n+\treturn ret_value;\n+}\ndiff --git a/compat/precompose_utf8.h b/compat/precompose_utf8.h\nnew file mode 100644\nindex 0000000..3b73585\n--- /dev/null\n+++ b/compat/precompose_utf8.h\n@@ -0,0 +1,45 @@\n+#ifndef PRECOMPOSE_UNICODE_H\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <dirent.h>\n+#include <iconv.h>\n+\n+\n+typedef struct dirent_prec_psx {\n+\tino_t d_ino;            /* Posix */\n+\tsize_t max_name_len;    /* See below */\n+\tunsigned char d_type;   /* available on all systems git runs on */\n+\n+\t/*\n+\t * See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html\n+\t * NAME_MAX + 1 should be enough, but some systems have\n+\t * NAME_MAX=255 and strlen(d_name) may return 508 or 510\n+\t * Solution: allocate more when needed, see precompose_utf8_readdir()\n+\t */\n+\tchar   d_name[NAME_MAX+1];\n+} dirent_prec_psx;\n+\n+\n+typedef struct {\n+\ticonv_t ic_precompose;\n+\tDIR *dirp;\n+\tstruct dirent_prec_psx *dirent_nfc;\n+} PREC_DIR;\n+\n+void precompose_argv(int argc, const char **argv);\n+void probe_utf8_pathname_composition(char *, int);\n+\n+PREC_DIR *precompose_utf8_opendir(const char *dirname);\n+struct dirent_prec_psx *precompose_utf8_readdir(PREC_DIR *dirp);\n+int precompose_utf8_closedir(PREC_DIR *dirp);\n+\n+#ifndef PRECOMPOSE_UNICODE_C\n+#define dirent dirent_prec_psx\n+#define opendir(n) precompose_utf8_opendir(n)\n+#define readdir(d) precompose_utf8_readdir(d)\n+#define closedir(d) precompose_utf8_closedir(d)\n+#define DIR PREC_DIR\n+#endif /* PRECOMPOSE_UNICODE_C */\n+\n+#define  PRECOMPOSE_UNICODE_H\n+#endif /* PRECOMPOSE_UNICODE_H */\ndiff --git a/config.c b/config.c\nindex 71ef171..eaef3b6 100644\n--- a/config.c\n+++ b/config.c\n@@ -758,6 +758,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.precomposeunicode\")) {\n+\t\tprecomposed_unicode = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 669e498..85edd7f 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -58,6 +58,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n+int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 5bd9ad7..35b095e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -153,6 +153,15 @@\n #endif\n #endif\n \n+/* used on Mac OS X */\n+#ifdef PRECOMPOSE_UNICODE\n+#include \"compat/precompose_utf8.h\"\n+#else\n+#define precompose_str(in,i_nfd2nfc)\n+#define precompose_argv(c,v)\n+#define probe_utf8_pathname_composition(a,b)\n+#endif\n+\n #ifndef NO_LIBGEN_H\n #include <libgen.h>\n #else\ndiff --git a/parse-options.c b/parse-options.c\nindex ab70c29..c1c66bd 100644\n--- a/parse-options.c\n+++ b/parse-options.c\n@@ -476,6 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n \t\tusage_with_options(usagestr, options);\n \t}\n \n+\tprecompose_argv(argc, argv);\n \treturn parse_options_end(&ctx);\n }\n \ndiff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh\nnew file mode 100755\nindex 0000000..88b7a20\n--- /dev/null\n+++ b/t/t3910-mac-os-precompose.sh\n@@ -0,0 +1,164 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2012 Torsten Bögershausen\n+#\n+\n+test_description='utf-8 decomposed (nfd) converted to precomposed (nfc)'\n+\n+. ./test-lib.sh\n+\n+Adiarnfc=`printf '\\303\\204'`\n+Adiarnfd=`printf 'A\\314\\210'`\n+\n+# check if the feature is compiled in\n+mkdir junk &&\n+>junk/\"$Adiarnfc\" &&\n+case \"$(cd junk && echo *)\" in\n+\t\"$Adiarnfd\")\n+\ttest_nfd=1\n+\t;;\n+\t*)\t;;\n+esac\n+rm -rf junk\n+\n+\n+if test \"$test_nfd\"\n+then\n+\t# create more utf-8 variables\n+\tOdiarnfc=`printf '\\303\\226'`\n+\tOdiarnfd=`printf 'O\\314\\210'`\n+\tAEligatu=`printf '\\303\\206'`\n+\tInvalidu=`printf '\\303\\377'`\n+\n+\n+\t#Create a string with 255 bytes (decomposed)\n+\tAlongd=$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd #21 Byte\n+\tAlongd=$Alongd$Alongd$Alongd                                           #63 Byte\n+\tAlongd=$Alongd$Alongd$Alongd$Alongd$Adiarnfd                           #255 Byte\n+\n+\t#Create a string with 254 bytes (precomposed)\n+\tAlongc=$AEligatu$AEligatu$AEligatu$AEligatu$AEligatu #10 Byte\n+\tAlongc=$Alongc$Alongc$Alongc$Alongc$Alongc           #50 Byte\n+\tAlongc=$Alongc$Alongc$Alongc$Alongc$Alongc           #250 Byte\n+\tAlongc=$Alongc$AEligatu$AEligatu                     #254 Byte\n+\n+\ttest_expect_success \"detect if nfd needed\" '\n+\t\tprecomposeunicode=`git config core.precomposeunicode` &&\n+\t\ttest \"$precomposeunicode\" = false &&\n+\t\tgit config core.precomposeunicode true\n+\t'\n+\ttest_expect_success \"setup\" '\n+\t\t>x &&\n+\t\tgit add x &&\n+\t\tgit commit -m \"1st commit\" &&\n+\t\tgit rm x &&\n+\t\tgit commit -m \"rm x\"\n+\t'\n+\ttest_expect_success \"setup case mac\" '\n+\t\tgit checkout -b mac_os\n+\t'\n+\t# This will test nfd2nfc in readdir()\n+\ttest_expect_success \"add file Adiarnfc\" '\n+\t\techo f.Adiarnfc >f.$Adiarnfc &&\n+\t\tgit add f.$Adiarnfc &&\n+\t\tgit commit -m \"add f.$Adiarnfc\"\n+\t'\n+\t# This will test nfd2nfc in git stage()\n+\ttest_expect_success \"stage file d.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d.$Adiarnfd &&\n+\t\techo d.$Adiarnfd/f.$Adiarnfd >d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit stage d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d.$Adiarnfd/f.$Adiarnfd\"\n+\t'\n+\ttest_expect_success \"add link Adiarnfc\" '\n+\t\tln -s d.$Adiarnfd/f.$Adiarnfd l.$Adiarnfc &&\n+\t\tgit add l.$Adiarnfc &&\n+\t\tgit commit -m \"add l.Adiarnfc\"\n+\t'\n+\t# This will test git log\n+\ttest_expect_success \"git log f.Adiar\" '\n+\t\tgit log f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit log f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git ls-files\n+\ttest_expect_success \"git lsfiles f.Adiar\" '\n+\t\tgit ls-files f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit ls-files f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git mv\n+\ttest_expect_success \"git mv\" '\n+\t\tgit mv f.$Adiarnfd f.$Odiarnfc &&\n+\t\tgit mv d.$Adiarnfd d.$Odiarnfc &&\n+\t\tgit mv l.$Adiarnfd l.$Odiarnfc &&\n+\t\tgit commit -m \"mv Adiarnfd Odiarnfc\"\n+\t'\n+\t# Files can be checked out as nfc\n+\t# And the link has been corrected from nfd to nfc\n+\ttest_expect_success \"git checkout nfc\" '\n+\t\trm f.$Odiarnfc &&\n+\t\tgit checkout f.$Odiarnfc\n+\t'\n+\t# Make it possible to checkout files with their NFD names\n+\ttest_expect_success \"git checkout file nfd\" '\n+\t\trm -f f.* &&\n+\t\tgit checkout f.$Odiarnfd\n+\t'\n+\t# Make it possible to checkout links with their NFD names\n+\ttest_expect_success \"git checkout link nfd\" '\n+\t\trm l.* &&\n+\t\tgit checkout l.$Odiarnfd\n+\t'\n+\ttest_expect_success \"setup case mac2\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_2\n+\t'\n+\t# This will test nfd2nfc in git commit\n+\ttest_expect_success \"commit file d2.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d2.$Adiarnfd &&\n+\t\techo d2.$Adiarnfd/f.$Adiarnfd >d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit add d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d2.$Adiarnfd/f.$Adiarnfd\" -- d2.$Adiarnfd/f.$Adiarnfd\n+\t'\n+\ttest_expect_success \"setup for long decomposed filename\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_long_nfd_fn\n+\t'\n+\ttest_expect_success \"Add long decomposed filename\" '\n+\t\techo longd >$Alongd &&\n+\t\tgit add * &&\n+\t\tgit commit -m \"Long filename\"\n+\t'\n+\ttest_expect_success \"setup for long precomposed filename\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_long_nfc_fn\n+\t'\n+\ttest_expect_success \"Add long precomposed filename\" '\n+\t\techo longc >$Alongc &&\n+\t\tgit add * &&\n+\t\tgit commit -m \"Long filename\"\n+\t'\n+\t# Test if the global core.precomposeunicode stops autosensing\n+\t# Must be the last test case\n+\ttest_expect_success \"respect git config --global core.precomposeunicode\" '\n+\t\tgit config --global core.precomposeunicode true &&\n+\t\trm -rf .git &&\n+\t\tgit init &&\n+\t\tprecomposeunicode=`git config core.precomposeunicode` &&\n+\t\ttest \"$precomposeunicode\" = \"true\"\n+\t'\n+else\n+\t say \"Skipping nfc/nfd tests\"\n+fi\n+\n+test_done\ndiff --git a/utf8.c b/utf8.c\nindex 8acbc66..a544f15 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -433,19 +433,12 @@ int is_encoding_utf8(const char *name)\n #else\n \ttypedef char * iconv_ibp;\n #endif\n-char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding)\n+char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)\n {\n-\ticonv_t conv;\n-\tsize_t insz, outsz, outalloc;\n+\tsize_t outsz, outalloc;\n \tchar *out, *outpos;\n \ticonv_ibp cp;\n \n-\tif (!in_encoding)\n-\t\treturn NULL;\n-\tconv = iconv_open(out_encoding, in_encoding);\n-\tif (conv == (iconv_t) -1)\n-\t\treturn NULL;\n-\tinsz = strlen(in);\n \toutsz = insz;\n \toutalloc = outsz + 1; /* for terminating NUL */\n \tout = xmalloc(outalloc);\n@@ -459,7 +452,6 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n \t\t\tsize_t sofar;\n \t\t\tif (errno != E2BIG) {\n \t\t\t\tfree(out);\n-\t\t\t\ticonv_close(conv);\n \t\t\t\treturn NULL;\n \t\t\t}\n \t\t\t/* insz has remaining number of bytes.\n@@ -478,6 +470,20 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n \t\t\tbreak;\n \t\t}\n \t}\n+\treturn out;\n+}\n+\n+char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding)\n+{\n+\ticonv_t conv;\n+\tchar *out;\n+\n+\tif (!in_encoding)\n+\t\treturn NULL;\n+\tconv = iconv_open(out_encoding, in_encoding);\n+\tif (conv == (iconv_t) -1)\n+\t\treturn NULL;\n+\tout = reencode_string_iconv(in, strlen(in), conv);\n \ticonv_close(conv);\n \treturn out;\n }\ndiff --git a/utf8.h b/utf8.h\nindex 81f2c82..3c0ae76 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -14,6 +14,7 @@ int strbuf_add_wrapped_bytes(struct strbuf *buf, const char *data, int len,\n \t\t\t     int indent, int indent2, int width);\n \n #ifndef NO_ICONV\n+char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv);\n char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding);\n #else\n #define reencode_string(a,b,c) NULL\n-- \n1.7.11.1.47.ge34cb30\n"},{"id":"195898","messageId":"5011AD4A.8060901@dewire.com","threadId":"30977","inReplyTo":"201207081550.25850.tboegi@web.de","subject":"Re: [PATCH v9] git on Mac OS and precomposed unicode","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2012-07-26T20:49:14Z","receivedAt":"2012-07-26T20:49:14Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Just a couple of nitpicks.\n\nTorsten Bögershausen skrev 2012-07-08 15.50:\n> diff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c\n[...]\n> +static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n> +{\n> +\tconst uint8_t *utf8p = (const uint8_t*) s;\n> +\tsize_t strlen_chars = 0;\n> +\tsize_t ret = 0;\n> +\n> +\tif ((!utf8p) || (!*utf8p)) {\nStyle: Drop the extra parentheses\n> +\t\treturn 0;\n> +\t}\n> +\n> +\twhile((*utf8p) && maxlen) {\nStyle: Drop the extra parentheses\n[...]\n\n> +void probe_utf8_pathname_composition(char *path, int len)\n> +{\n> +\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n> +\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n> +\tint output_fd;\n> +\tif (precomposed_unicode != -1)\n> +\t\treturn; /* We found it defined in the global config, respect it */\n> +\tpath[len] = 0;\nNot needed, will be overwritten by strcpy\n\n> +\tstrcpy(path + len, auml_nfc);\n> +\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n> +\tif (output_fd >=0) {\n> +\t\tclose(output_fd);\n> +\t\tpath[len] = 0;\nNot needed, will be overwritten by strcpy\n\n> +\t\tstrcpy(path + len, auml_nfd);\n> +\t\t/* Indicate to the user, that we can configure it to true */\n> +\t\tif (0 == access(path, R_OK))\n> +\t\t\tgit_config_set(\"core.precomposeunicode\", \"false\");\n> +\t\t\t/* To be backward compatible, set precomposed_unicode to 0 */\n> +\t\tprecomposed_unicode = 0;\n> +\t\tpath[len] = 0;\nNot needed, will be overwritten by strcpy\n\n> +\t\tstrcpy(path + len, auml_nfc);\n> +\t\tunlink(path);\nErr out if path cannot be deleted?\n\n-- robin\n"},{"id":"195901","messageId":"7vhasu5dld.fsf@alter.siamese.dyndns.org","threadId":"30977","inReplyTo":"5011AD4A.8060901@dewire.com","subject":"Re: [PATCH v9] git on Mac OS and precomposed unicode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-26T23:04:14Z","receivedAt":"2012-07-26T23:04:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg <robin.rosenberg@dewire.com> writes:\n\n> Just a couple of nitpicks.\n\nPolishing is always good and better late than never, but for a topic\nthat has long been graduated to 'master' already, it would be easier\nto review and discuss if it came as a patch form relative to the\ncodebase _after_ the topic has been applied.\n\nThanks.\n"},{"id":"197158","messageId":"7v393mxkpk.fsf@alter.siamese.dyndns.org","threadId":"30977","inReplyTo":"7vhasu5dld.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v9] git on Mac OS and precomposed unicode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-17T03:26:31Z","receivedAt":"2012-08-17T03:26:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Robin Rosenberg <robin.rosenberg@dewire.com> writes:\n>\n>> Just a couple of nitpicks.\n>\n> Polishing is always good and better late than never, but for a topic\n> that has long been graduated to 'master' already, it would be easier\n> to review and discuss if it came as a patch form relative to the\n> codebase _after_ the topic has been applied.\n\nMild ping for a possible follow-ups that didn't seem to have\nhappened...\n"},{"id":"197183","messageId":"1345215190-95976-1-git-send-email-robin.rosenberg@dewire.com","threadId":"30977","inReplyTo":"7v393mxkpk.fsf@alter.siamese.dyndns.org","subject":"[PATCH] cleanup precompose_utf8","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2012-08-17T14:53:10Z","receivedAt":"2012-08-17T14:53:10Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Remove extraneous parentheses and braces\nRemove redundant NUL-termination\nCheck result of unlink when probing for decomposed file names\n\nSigned-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n---\n compat/precompose_utf8.c | 11 ++++-------\n 1 fil ändrad, 4 tillägg(+), 7 borttagningar(-)\n\ndiff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c\nindex d40d1b3..9563760 100644\n--- a/compat/precompose_utf8.c\n+++ b/compat/precompose_utf8.c\n@@ -21,11 +21,10 @@ static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n \tsize_t strlen_chars = 0;\n \tsize_t ret = 0;\n \n-\tif ((!utf8p) || (!*utf8p)) {\n+\tif (!utf8p || !*utf8p)\n \t\treturn 0;\n-\t}\n \n-\twhile((*utf8p) && maxlen) {\n+\twhile(*utf8p && maxlen) {\n \t\tif (*utf8p & 0x80)\n \t\t\tret++;\n \t\tstrlen_chars++;\n@@ -46,21 +45,19 @@ void probe_utf8_pathname_composition(char *path, int len)\n \tint output_fd;\n \tif (precomposed_unicode != -1)\n \t\treturn; /* We found it defined in the global config, respect it */\n-\tpath[len] = 0;\n \tstrcpy(path + len, auml_nfc);\n \toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n \tif (output_fd >=0) {\n \t\tclose(output_fd);\n-\t\tpath[len] = 0;\n \t\tstrcpy(path + len, auml_nfd);\n \t\t/* Indicate to the user, that we can configure it to true */\n \t\tif (0 == access(path, R_OK))\n \t\t\tgit_config_set(\"core.precomposeunicode\", \"false\");\n \t\t\t/* To be backward compatible, set precomposed_unicode to 0 */\n \t\tprecomposed_unicode = 0;\n-\t\tpath[len] = 0;\n \t\tstrcpy(path + len, auml_nfc);\n-\t\tunlink(path);\n+\t\tif (unlink(path))\n+\t\t\tdie_errno(_(\"failed to unlink '%s'\"), path);\n \t}\n }\n \n-- \n1.7.12.rc3.4.ga867a5c\n"},{"id":"197194","messageId":"7vsjblv3j2.fsf@alter.siamese.dyndns.org","threadId":"30977","inReplyTo":"1345215190-95976-1-git-send-email-robin.rosenberg@dewire.com","subject":"Re: [PATCH] cleanup precompose_utf8","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-17T17:20:33Z","receivedAt":"2012-08-17T17:20:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.\n"},{"id":"197195","messageId":"7vobm9v32e.fsf@alter.siamese.dyndns.org","threadId":"30977","inReplyTo":"1345215190-95976-1-git-send-email-robin.rosenberg@dewire.com","subject":"Re: [PATCH] cleanup precompose_utf8","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-17T17:30:33Z","receivedAt":"2012-08-17T17:30:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg <robin.rosenberg@dewire.com> writes:\n\n> Remove extraneous parentheses and braces\n> Remove redundant NUL-termination\n> Check result of unlink when probing for decomposed file names\n>\n> Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n> ---\n\nThanks.  I've found and fixed a bit more style violations while we\nare at it.\n"},{"id":"197290","messageId":"50311DEB.4050700@web.de","threadId":"30977","inReplyTo":"7vobm9v32e.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] cleanup precompose_utf8","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-08-19T17:10:03Z","receivedAt":"2012-08-19T17:10:03Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 17.08.12 19:30, Junio C Hamano wrote:\n> Robin Rosenberg <robin.rosenberg@dewire.com> writes:\n>\n>> Remove extraneous parentheses and braces\n>> Remove redundant NUL-termination\n>> Check result of unlink when probing for decomposed file names\n>>\n>> Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n>> ---\n> Thanks.  I've found and fixed a bit more style violations while we\n> are at it.\n\n(I was offline for a couple of days)\n\nThanks to all, and ACK from my side.\n/Torsten\n"},{"id":"197429","messageId":"7vobm5jusz.fsf_-_@alter.siamese.dyndns.org","threadId":"30977","inReplyTo":"50311DEB.4050700@web.de","subject":"[PATCH] precompose-utf8: do not call checks for non-ascii \"utf8\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-20T18:13:32Z","receivedAt":"2012-08-20T18:13:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"As suggested by Linus, this function is not checking UTF-8-ness of the\nstring; it only is seeing if it is pure US-ASCII.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n\n * Just for completeness, this on top.\n\n compat/precompose_utf8.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c\nindex 3190d50..8cf5955 100644\n--- a/compat/precompose_utf8.c\n+++ b/compat/precompose_utf8.c\n@@ -13,20 +13,20 @@ typedef char *iconv_ibp;\n static const char *repo_encoding = \"UTF-8\";\n static const char *path_encoding = \"UTF-8-MAC\";\n \n-static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n+static size_t has_non_ascii(const char *s, size_t maxlen, size_t *strlen_c)\n {\n-\tconst uint8_t *utf8p = (const uint8_t *)s;\n+\tconst uint8_t *ptr = (const uint8_t *)s;\n \tsize_t strlen_chars = 0;\n \tsize_t ret = 0;\n \n-\tif (!utf8p || !*utf8p)\n+\tif (!ptr || !*ptr)\n \t\treturn 0;\n \n-\twhile (*utf8p && maxlen) {\n-\t\tif (*utf8p & 0x80)\n+\twhile (*ptr && maxlen) {\n+\t\tif (*ptr & 0x80)\n \t\t\tret++;\n \t\tstrlen_chars++;\n-\t\tutf8p++;\n+\t\tptr++;\n \t\tmaxlen--;\n \t}\n \tif (strlen_c)\n@@ -77,7 +77,7 @@ void precompose_argv(int argc, const char **argv)\n \twhile (i < argc) {\n \t\tsize_t namelen;\n \t\toldarg = argv[i];\n-\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n+\t\tif (has_non_ascii(oldarg, (size_t)-1, &namelen)) {\n \t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n \t\t\tif (newarg)\n \t\t\t\targv[i] = newarg;\n@@ -130,7 +130,7 @@ struct dirent_prec_psx *precompose_utf8_readdir(PREC_DIR *prec_dir)\n \t\tprec_dir->dirent_nfc->d_ino  = res->d_ino;\n \t\tprec_dir->dirent_nfc->d_type = res->d_type;\n \n-\t\tif ((precomposed_unicode == 1) && has_utf8(res->d_name, (size_t)-1, NULL)) {\n+\t\tif ((precomposed_unicode == 1) && has_non_ascii(res->d_name, (size_t)-1, NULL)) {\n \t\t\tif (prec_dir->ic_precompose == (iconv_t)-1) {\n \t\t\t\tdie(\"iconv_open(%s,%s) failed, but needed:\\n\"\n \t\t\t\t\t\t\"    precomposed unicode is not supported.\\n\"\n-- \n1.7.12.129.g11de995\n"},{"id":"197435","messageId":"5032940A.4040005@web.de","threadId":"30977","inReplyTo":"7vobm5jusz.fsf_-_@alter.siamese.dyndns.org","subject":"Re: [PATCH] precompose-utf8: do not call checks for non-ascii \"utf8\"","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-08-20T19:46:18Z","receivedAt":"2012-08-20T19:46:18Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 08/20/2012 08:13 PM, Junio C Hamano wrote:\n> As suggested by Linus, this function is not checking UTF-8-ness of the\n> string; it only is seeing if it is pure US-ASCII.\n>\n> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> ---\n>\n>   * Just for completeness, this on top.\n>\n>   compat/precompose_utf8.c | 16 ++++++++--------\n>   1 file changed, 8 insertions(+), 8 deletions(-)\n>\n> diff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c\n> index 3190d50..8cf5955 100644\n> --- a/compat/precompose_utf8.c\n> +++ b/compat/precompose_utf8.c\n> @@ -13,20 +13,20 @@ typedef char *iconv_ibp;\n>   static const char *repo_encoding = \"UTF-8\";\n>   static const char *path_encoding = \"UTF-8-MAC\";\n>\n> -static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n> +static size_t has_non_ascii(const char *s, size_t maxlen, size_t *strlen_c)\n>   {\n> -\tconst uint8_t *utf8p = (const uint8_t *)s;\n> +\tconst uint8_t *ptr = (const uint8_t *)s;\n>   \tsize_t strlen_chars = 0;\n>   \tsize_t ret = 0;\n>\n> -\tif (!utf8p || !*utf8p)\n> +\tif (!ptr || !*ptr)\n>   \t\treturn 0;\n>\n> -\twhile (*utf8p && maxlen) {\n> -\t\tif (*utf8p & 0x80)\n> +\twhile (*ptr && maxlen) {\n> +\t\tif (*ptr & 0x80)\n>   \t\t\tret++;\n>   \t\tstrlen_chars++;\n> -\t\tutf8p++;\n> +\t\tptr++;\n>   \t\tmaxlen--;\n>   \t}\n>   \tif (strlen_c)\n> @@ -77,7 +77,7 @@ void precompose_argv(int argc, const char **argv)\n>   \twhile (i < argc) {\n>   \t\tsize_t namelen;\n>   \t\toldarg = argv[i];\n> -\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n> +\t\tif (has_non_ascii(oldarg, (size_t)-1, &namelen)) {\n>   \t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n>   \t\t\tif (newarg)\n>   \t\t\t\targv[i] = newarg;\n> @@ -130,7 +130,7 @@ struct dirent_prec_psx *precompose_utf8_readdir(PREC_DIR *prec_dir)\n>   \t\tprec_dir->dirent_nfc->d_ino  = res->d_ino;\n>   \t\tprec_dir->dirent_nfc->d_type = res->d_type;\n>\n> -\t\tif ((precomposed_unicode == 1) && has_utf8(res->d_name, (size_t)-1, NULL)) {\n> +\t\tif ((precomposed_unicode == 1) && has_non_ascii(res->d_name, (size_t)-1, NULL)) {\n>   \t\t\tif (prec_dir->ic_precompose == (iconv_t)-1) {\n>   \t\t\t\tdie(\"iconv_open(%s,%s) failed, but needed:\\n\"\n>   \t\t\t\t\t\t\"    precomposed unicode is not supported.\\n\"\n>\nThanks Junio,\nthat partly obsoletes the patch I'm working on.\nAnd as I didn't manage to catch up, may I send the result of my review?\n\na) in readdir we die() when iconv_open() fails, and we could/should do \nthat in void precompose_argv() as well?\n\nb) Should die(\"txt\") be converted into die(_(\"txt\")) to be able to \ntranslate the message?\n\nc) We can remove the save/restore of errno at one place, similar to this:\n\n\n@@ -106,11 +106,8 @@ PREC_DIR *precompose_utf8_opendir(const char *dirname)\n                 free(prec_dir->dirent_nfc);\n                 free(prec_dir);\n                 return NULL;\n-       } else {\n-               int ret_errno = errno;\n+       } else\n                 prec_dir->ic_precompose = (iconv_t)-1;\n-               errno = ret_errno;\n-       }\n"},{"id":"197463","messageId":"7v4nnxjk70.fsf@alter.siamese.dyndns.org","threadId":"30977","inReplyTo":"5032940A.4040005@web.de","subject":"Re: [PATCH] precompose-utf8: do not call checks for non-ascii \"utf8\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-20T22:02:43Z","receivedAt":"2012-08-20T22:02:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> a) in readdir we die() when iconv_open() fails, and we could/should do\n> that in void precompose_argv() as well?\n\nProbably.\n\n> b) Should die(\"txt\") be converted into die(_(\"txt\")) to be able to\n> translate the message?\n\nI do not even know what die(\"txt\") is trying to tell to\nEnglish-speaking users, so I dunno ;-).  The message needs to be\nupdated to human readable form and then annotated with _(), I guess.\n"}]}