{"thread":{"id":"30937","subject":"[PATCH v7] git on Mac OS and precomposed unicode","startedAt":"2012-07-02T20:32:50Z","lastAt":"2012-07-02T22:21:39Z","messageCount":2,"participants":["Torsten Bögershausen","Junio C Hamano"],"isPatch":true,"patchVersion":7,"patchTotal":null},"messages":[{"id":"194473","messageId":"201207022232.51737.tboegi@web.de","threadId":"30937","inReplyTo":null,"subject":"[PATCH v7] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-07-02T20:32:50Z","receivedAt":"2012-07-02T20:32:50Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Mac OS X mangles file names containing unicode on file systems HFS+,\nVFAT or SAMBA.  When a file using unicode code points outside ASCII\nis created on a HFS+ drive, the file name is converted into\ndecomposed unicode and written to disk. No conversion is done if\nthe file name is already decomposed unicode.\n\nCalling open(\"\\xc3\\x84\", ...) with a precomposed \"Ä\" yields the same\nresult as open(\"\\x41\\xcc\\x88\",...) with a decomposed \"Ä\".\n\nAs a consequence, readdir() returns the file names in decomposed\nunicode, even if the user expects precomposed unicode.  Unlike on\nHFS+, Mac OS X files on a VFAT drive (e.g. an USB drive), in\nprecomposed unicode, but readdir() still returns file names in\ndecomposed unicode.  When a git repository is stored on a network\nshare using SAMBA, file names are send over the wire and written to\ndisk on the remote system in precomposed unicode, but Mac OS X\nreaddir() returns decomposed unicode to be compatible with its\nbehaviour on HFS+ and VFAT.\n\nThe unicode decomposition causes many problems:\n\n- The names \"git add\" and other commands get from the end user may\n  often be precomposed form (the decomposed form is not easily input\n  from the keyboard), but when the commands read from the filesystem\n  to see what it is going to update the index with already is on the\n  filesystem, readdir() will give decomposed form, which is different.\n\n- Similarly \"git log\", \"git mv\" and all other commands that need to\n  compare pathnames found on the command line (often but not always\n  precomposed form; a command line input resulting from globbing may\n  be in decomposed) with pathnames found in the tree objects (should\n  be precomposed form to be compatible with other systems and for\n  consistency in general).\n\n- The same for names stored in the index, which should be\n  precomposed, that may need to be compared with the names read from\n  readdir().\n\nNFS mounted from Linux is fully transparent and does not suffer from\nthe above.\n\nAs Mac OS X treats precomposed and decomposed file names as equal,\nwe can\n\n - wrap readdir() on Mac OS X to return the precomposed form, and\n\n - normalize decomposed form given from the command line also to the\n   precomposed form,\n\nto ensure that all pathnames used in Git are always in the\nprecomposed form.  This behaviour can be requested by setting\n\"core.precomposedunicode\" configuration variable to true.\n\nThe code in compat/precomposed_utf8.c implements basically 4 new\nfunctions: precomposed_utf8_opendir(), precomposed_utf8_readdir(),\nprecomposed_utf8_closedir() and precompose_argv().  The first three\nare to wrap opendir(3), readdir(3), and closedir(3) functions.\n\nThe argv[] conversion allows to use the TAB filename completion done\nby the shell on command line.  It tolerates other tools which use\nreaddir() to feed decomposed file names into git.\n\nWhen creating a new git repository with \"git init\" or \"git clone\",\n\"core.precomposedunicode\" will be set \"false\".\n\nThe user needs to activate this feature manually.  She typically\nsets core.precomposedunicode to \"true\" on HFS and VFAT, or file\nsystems mounted via SAMBA.\n\nHelped-by: Junio C Hamano <gitster@pobox.com>\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\n  Thanks to Junio for intensive rewiew\n\n  Changes since v6:\n  Use Junios commit msg from pu; removed \"mounted onto..\"\n\n  s/mac_os_precomposed_unicode/precomposed_unicode\n\n  compat/precomposed_utf8.h:\n  - Introduced a more Posix/Unix like struct dirent_prec_psx\n    which has d_ino, max_name_len, d_type, d_name\n\n  compat/precomposed_utf8.c:\n  - Dynamically realloc with larger dirent_prec_psx.d_name,\n    if d_name should become too small\n    (Side note: my Mac OS X 10.6 defines d_name[1024], but readdir()\n    doesn't handle length >= 255 correct)\n  - When iconv_open() fails: let opendir() return, but die() when\n    readdir() detects UTF-8 which needs the conversion.\n  - When iconv() fails in precomposed_utf8_readdir(), return original string\n\n  utf8.[ch]:\n  - Moved reencode_string_iconv() into utf8.c\n\n  t/t3910:\n  - Added test case to test \"long\" (<255 bytes) filenames \n\n  Other small changes indicated by code review.\n\n Documentation/config.txt     |   9 +++\n Makefile                     |   3 +\n builtin/init-db.c            |   1 +\n cache.h                      |   1 +\n compat/precomposed_utf8.c    | 189 +++++++++++++++++++++++++++++++++++++++++++\n compat/precomposed_utf8.h    |  43 ++++++++++\n config.c                     |   5 ++\n environment.c                |   1 +\n git-compat-util.h            |   9 +++\n parse-options.c              |   1 +\n t/t3910-mac-os-precompose.sh | 164 +++++++++++++++++++++++++++++++++++++\n utf8.c                       |  26 +++---\n utf8.h                       |   1 +\n 13 files changed, 443 insertions(+), 10 deletions(-)\n create mode 100644 compat/precomposed_utf8.c\n create mode 100644 compat/precomposed_utf8.h\n create mode 100755 t/t3910-mac-os-precompose.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 915cb5a..4162cab 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -210,6 +210,15 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignorecase true if appropriate when the repository\n is created.\n \n+core.precomposedunicode::\n+\tThis option is only used by Mac OS implementation of git.\n+\tWhen core.precomposedunicode=true, git reverts the unicode decomposition\n+\tof filenames done by Mac OS. This is useful when sharing a repository\n+\tbetween Mac OS and Linux or Windows.\n+\t(Git for Windows/msysGit 1.7.10 is needed, or git under cygwin 1.7).\n+\tWhen false, file names are handled fully transparent by git, which means\n+\tthat file names are stored as decomposed unicode in the repository.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Makefile b/Makefile\nindex f62ca2a..55ceb10 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -607,6 +607,7 @@ LIB_H += compat/bswap.h\n LIB_H += compat/cygwin.h\n LIB_H += compat/mingw.h\n LIB_H += compat/obstack.h\n+LIB_H += compat/precomposed_utf8.h\n LIB_H += compat/terminal.h\n LIB_H += compat/win32/dirent.h\n LIB_H += compat/win32/poll.h\n@@ -1000,6 +1001,8 @@ ifeq ($(uname_S),Darwin)\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tCOMPAT_OBJS += compat/precomposed_utf8.o\n+\tBASIC_CFLAGS += -DPRECOMPOSED_UNICODE\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 0dacb8b..244fb7f 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -290,6 +290,7 @@ static int create_default_files(const char *template_path)\n \t\tstrcpy(path + len, \"CoNfIg\");\n \t\tif (!access(path, F_OK))\n \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n+\t\tprobe_utf8_pathname_composition(path, len);\n \t}\n \n \treturn reinit;\ndiff --git a/cache.h b/cache.h\nindex 06413e1..5e551d3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -560,6 +560,7 @@ extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern int precomposed_unicode;\n \n enum branch_track {\n \tBRANCH_TRACK_UNSPECIFIED = -1,\ndiff --git a/compat/precomposed_utf8.c b/compat/precomposed_utf8.c\nnew file mode 100644\nindex 0000000..14bb0ce\n--- /dev/null\n+++ b/compat/precomposed_utf8.c\n@@ -0,0 +1,189 @@\n+/* Converts filenames from decomposed unicode into precomposed unicode.\n+   Used on MacOS X.\n+*/\n+\n+\n+#define __PRECOMPOSED_UNICODE_C__\n+\n+#include \"cache.h\"\n+#include \"utf8.h\"\n+#include \"precomposed_utf8.h\"\n+#include \"stdio.h\"\n+\n+typedef char *iconv_ibp;\n+const static char *repo_encoding = \"UTF-8\";\n+const static char *path_encoding = \"UTF-8-MAC\";\n+\n+\n+static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n+{\n+\tconst uint8_t *utf8p = (const uint8_t*) s;\n+\tsize_t strlen_chars = 0;\n+\tsize_t ret = 0;\n+\n+\tif ((!utf8p) || (!*utf8p)) {\n+\t\treturn 0;\n+\t}\n+\n+\twhile((*utf8p) && maxlen) {\n+\t\tif (*utf8p & 0x80)\n+\t\t\tret++;\n+\t\tstrlen_chars++;\n+\t\tutf8p++;\n+\t\tmaxlen--;\n+\t}\n+\tif (strlen_c)\n+\t\t*strlen_c = strlen_chars;\n+\n+\treturn ret;\n+}\n+\n+\n+void probe_utf8_pathname_composition(char *path, int len)\n+{\n+\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n+\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n+\tint output_fd;\n+\tif (precomposed_unicode != -1)\n+\t\treturn; /* We found it defined in the global config, respect it */\n+\tpath[len] = 0;\n+\tstrcpy(path + len, auml_nfc);\n+\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n+\tif (output_fd >=0) {\n+\t\tclose(output_fd);\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfd);\n+\t\t/* Indicate the the user, that we can configure it to true */\n+\t\tif (0 == access(path, R_OK))\n+\t\t\tgit_config_set(\"core.precomposedunicode\", \"false\");\n+\t\t/* To be backward compatible, set precomposed_unicode to 0\n+\t\t   This may change in the future */\n+\t\tprecomposed_unicode = 0;\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfc);\n+\t\tunlink(path);\n+\t}\n+}\n+\n+\n+void precompose_argv(int argc, const char **argv)\n+{\n+\tint i = 0;\n+\tconst char *oldarg;\n+\tchar *newarg;\n+\ticonv_t ic_precompose;\n+\n+\tif (precomposed_unicode != 1)\n+\t\treturn;\n+\n+\tic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (ic_precompose == (iconv_t) -1)\n+\t\treturn;\n+\n+\twhile (i < argc) {\n+\t\tsize_t namelen;\n+\t\toldarg = argv[i];\n+\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n+\t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n+\t\t\tif (newarg)\n+\t\t\t\targv[i] = newarg;\n+\t\t}\n+\t\ti++;\n+\t}\n+\ticonv_close(ic_precompose);\n+}\n+\n+\n+PREC_DIR *precomposed_utf8_opendir(const char *dirname)\n+{\n+\tPREC_DIR *prec_dir = xmalloc(sizeof(PREC_DIR));\n+\tprec_dir->dirent_nfc = xmalloc(sizeof(dirent_prec_psx));\n+\tprec_dir->dirent_nfc->max_name_len = sizeof(prec_dir->dirent_nfc->d_name);\n+\n+\tprec_dir->dirp = opendir(dirname);\n+\tif (!prec_dir->dirp) {\n+\t\tfree(prec_dir->dirent_nfc);\n+\t\tfree(prec_dir);\n+\t\treturn NULL;\n+\t} else {\n+\t\tint ret_errno = errno;\n+\t\tprec_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);\n+\t\t/* if iconv_open() fails, die() in readdir() if needed */\n+\t\terrno = ret_errno;\n+\t}\n+\n+\treturn prec_dir;\n+}\n+\n+struct dirent_prec_psx *precomposed_utf8_readdir(PREC_DIR *prec_dir)\n+{\n+\tstruct dirent *res;\n+\tres = readdir(prec_dir->dirp);\n+\tif (res) {\n+\t\tsize_t namelenz = strlen(res->d_name) + 1; /* \\0 */\n+\t\tsize_t new_maxlen = namelenz;\n+\n+\t\tint ret_errno = errno;\n+\n+\t\tif (new_maxlen > prec_dir->dirent_nfc->max_name_len) {\n+\t\t\tsize_t new_len = sizeof(dirent_prec_psx) + new_maxlen -\n+\t\t\t\tsizeof(prec_dir->dirent_nfc->d_name);\n+\n+\t\t\tprec_dir->dirent_nfc = xrealloc(prec_dir->dirent_nfc, new_len);\n+\t\t\tprec_dir->dirent_nfc->max_name_len = new_maxlen;\n+\t\t}\n+\n+\t\tprec_dir->dirent_nfc->d_ino  = res->d_ino;\n+\t\tprec_dir->dirent_nfc->d_type = res->d_type;\n+\n+\t\tif ((precomposed_unicode == 1) && has_utf8(res->d_name, (size_t)-1, NULL)) {\n+\t\t\tif (prec_dir->ic_precompose == (iconv_t)-1) {\n+\t\t\t\tdie(\"iconv_open(%s,%s) failed, but needed:\\n\"\n+\t\t\t\t\t\t\"    precomposed unicode is not supported.\\n\"\n+\t\t\t\t\t\t\"    If you wnat to use decomposed unicode, run\\n\"\n+\t\t\t\t\t\t\"    \\\"git config core.precomposedunicode false\\\"\\n\",\n+\t\t\t\t\t\trepo_encoding, path_encoding);\n+\t\t\t} else {\n+\t\t\t\ticonv_ibp\tcp = (iconv_ibp)res->d_name;\n+\t\t\t\tsize_t inleft = namelenz;\n+\t\t\t\tchar *outpos = &prec_dir->dirent_nfc->d_name[0];\n+\t\t\t\tsize_t outsz = prec_dir->dirent_nfc->max_name_len;\n+\t\t\t\tsize_t cnt;\n+\t\t\t\terrno = 0;\n+\t\t\t\tcnt = iconv(prec_dir->ic_precompose, &cp, &inleft, &outpos, &outsz);\n+\t\t\t\tif (errno || inleft) {\n+\t\t\t\t\t/* iconv() failed and errno could be E2BIG, EILSEQ, EINVAL, EBADF\n+\t\t\t\t\t   MacOS X avoids illegal byte sequemces.\n+\t\t\t\t\t   If they occur on a mounted drive (e.g. NFS) it is not worth to\n+\t\t\t\t\t   die() for that, but rather let the user see the original name */\n+\t\t\t\t\tnamelenz = 0; /* trigger strlcpy */\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t\telse\n+\t\t\tnamelenz = 0;\n+\n+\t\tif (!namelenz)\n+\t\t\tstrlcpy(prec_dir->dirent_nfc->d_name, res->d_name,\n+\t\t\t\t\t\t\tprec_dir->dirent_nfc->max_name_len);\n+\n+\t\terrno = ret_errno;\n+\t\treturn prec_dir->dirent_nfc;\n+\t}\n+\treturn NULL;\n+}\n+\n+\n+int precomposed_utf8_closedir(PREC_DIR *prec_dir)\n+{\n+\tint ret_value;\n+\tint ret_errno;\n+\tret_value = closedir(prec_dir->dirp);\n+\tret_errno = errno;\n+\tif (prec_dir->ic_precompose != (iconv_t)-1)\n+\t\ticonv_close(prec_dir->ic_precompose);\n+\tfree(prec_dir->dirent_nfc);\n+\tfree(prec_dir);\n+\terrno = ret_errno;\n+\treturn ret_value;\n+}\ndiff --git a/compat/precomposed_utf8.h b/compat/precomposed_utf8.h\nnew file mode 100644\nindex 0000000..708a1c6\n--- /dev/null\n+++ b/compat/precomposed_utf8.h\n@@ -0,0 +1,43 @@\n+#ifndef __PRECOMPOSED_UNICODE_H__\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <dirent.h>\n+#include <iconv.h>\n+\n+\n+typedef struct dirent_prec_psx {\n+\tino_t d_ino;            /* Posix */\n+\tsize_t max_name_len;    /* See below */\n+\tunsigned char d_type;   /* available on all systems git runs on */\n+\n+\t/* See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html\n+\t\t NAME_MAX + 1 should be enough, but some systems have\n+\t\t NAME_MAX=255 and strlen(d_name) may return 508 or 510\n+\t   Solution: allocate more when needed, see precomposed_utf8_readdir()*/\n+\tchar   d_name[NAME_MAX+1];\n+} dirent_prec_psx;\n+\n+\n+typedef struct {\n+\ticonv_t ic_precompose;\n+\tDIR *dirp;\n+\tstruct dirent_prec_psx *dirent_nfc;\n+} PREC_DIR;\n+\n+void precompose_argv(int argc, const char **argv);\n+void probe_utf8_pathname_composition(char *, int);\n+\n+PREC_DIR *precomposed_utf8_opendir(const char *dirname);\n+struct dirent_prec_psx *precomposed_utf8_readdir(PREC_DIR *dirp);\n+int precomposed_utf8_closedir(PREC_DIR *dirp);\n+\n+#ifndef __PRECOMPOSED_UNICODE_C__\n+#define dirent dirent_prec_psx\n+#define opendir(n) precomposed_utf8_opendir(n)\n+#define readdir(d) precomposed_utf8_readdir(d)\n+#define closedir(d) precomposed_utf8_closedir(d)\n+#define DIR PREC_DIR\n+#endif /* __PRECOMPOSED_UNICODE_C__ */\n+\n+#define  __PRECOMPOSED_UNICODE_H__\n+#endif /* __PRECOMPOSED_UNICODE_H__ */\ndiff --git a/config.c b/config.c\nindex 71ef171..918f426 100644\n--- a/config.c\n+++ b/config.c\n@@ -758,6 +758,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.precomposedunicode\")) {\n+\t\tprecomposed_unicode = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 669e498..85edd7f 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -58,6 +58,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n+int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 5bd9ad7..600fa9e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -153,6 +153,15 @@\n #endif\n #endif\n \n+/* used on Mac OS X */\n+#ifdef PRECOMPOSED_UNICODE\n+#include \"compat/precomposed_utf8.h\"\n+#else\n+#define precompose_str(in,i_nfd2nfc)\n+#define precompose_argv(c,v)\n+#define probe_utf8_pathname_composition(a,b)\n+#endif\n+\n #ifndef NO_LIBGEN_H\n #include <libgen.h>\n #else\ndiff --git a/parse-options.c b/parse-options.c\nindex ab70c29..c1c66bd 100644\n--- a/parse-options.c\n+++ b/parse-options.c\n@@ -476,6 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n \t\tusage_with_options(usagestr, options);\n \t}\n \n+\tprecompose_argv(argc, argv);\n \treturn parse_options_end(&ctx);\n }\n \ndiff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh\nnew file mode 100755\nindex 0000000..df6f78d\n--- /dev/null\n+++ b/t/t3910-mac-os-precompose.sh\n@@ -0,0 +1,164 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2012 Torsten Bögershausen\n+#\n+\n+test_description='utf-8 decomposed (nfd) converted to precomposed (nfc)'\n+\n+. ./test-lib.sh\n+\n+Adiarnfc=`printf '\\303\\204'`\n+Adiarnfd=`printf 'A\\314\\210'`\n+\n+# check if the feature is compiled in\n+mkdir junk &&\n+>junk/\"$Adiarnfc\" &&\n+case \"$(cd junk && echo *)\" in\n+\t\"$Adiarnfd\")\n+\ttest_nfd=1\n+\t;;\n+\t*)\t;;\n+esac\n+rm -rf junk\n+\n+\n+if test \"$test_nfd\"\n+then\n+\t# create more utf-8 variables\n+\tOdiarnfc=`printf '\\303\\226'`\n+\tOdiarnfd=`printf 'O\\314\\210'`\n+\tAEligatu=`printf '\\303\\206'`\n+\tInvalidu=`printf '\\303\\377'`\n+\n+\n+\t#Create a string with 255 bytes (decomposed)\n+\tAlongd=$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd$Adiarnfd #21 Byte\n+\tAlongd=$Alongd$Alongd$Alongd                                           #63 Byte\n+\tAlongd=$Alongd$Alongd$Alongd$Alongd$Adiarnfd                           #255 Byte\n+\n+\t#Create a string with 254 bytes (precomposed)\n+\tAlongc=$AEligatu$AEligatu$AEligatu$AEligatu$AEligatu #10 Byte\n+\tAlongc=$Alongc$Alongc$Alongc$Alongc$Alongc           #50 Byte\n+\tAlongc=$Alongc$Alongc$Alongc$Alongc$Alongc           #250 Byte\n+\tAlongc=$Alongc$AEligatu$AEligatu                     #254 Byte\n+\n+\ttest_expect_success \"detect if nfd needed\" '\n+\t\tprecomposedunicode=`git config core.precomposedunicode` &&\n+\t\ttest \"$precomposedunicode\" = false &&\n+\t\tgit config core.precomposedunicode true\n+\t'\n+\ttest_expect_success \"setup\" '\n+\t\t>x &&\n+\t\tgit add x &&\n+\t\tgit commit -m \"1st commit\" &&\n+\t\tgit rm x &&\n+\t\tgit commit -m \"rm x\"\n+\t'\n+\ttest_expect_success \"setup case mac\" '\n+\t\tgit checkout -b mac_os\n+\t'\n+\t# This will test nfd2nfc in readdir()\n+\ttest_expect_success \"add file Adiarnfc\" '\n+\t\techo f.Adiarnfc >f.$Adiarnfc &&\n+\t\tgit add f.$Adiarnfc &&\n+\t\tgit commit -m \"add f.$Adiarnfc\"\n+\t'\n+\t# This will test nfd2nfc in git stage()\n+\ttest_expect_success \"stage file d.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d.$Adiarnfd &&\n+\t\techo d.$Adiarnfd/f.$Adiarnfd >d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit stage d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d.$Adiarnfd/f.$Adiarnfd\"\n+\t'\n+\ttest_expect_success \"add link Adiarnfc\" '\n+\t\tln -s d.$Adiarnfd/f.$Adiarnfd l.$Adiarnfc &&\n+\t\tgit add l.$Adiarnfc &&\n+\t\tgit commit -m \"add l.Adiarnfc\"\n+\t'\n+\t# This will test git log\n+\ttest_expect_success \"git log f.Adiar\" '\n+\t\tgit log f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit log f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git ls-files\n+\ttest_expect_success \"git lsfiles f.Adiar\" '\n+\t\tgit ls-files f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit ls-files f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git mv\n+\ttest_expect_success \"git mv\" '\n+\t\tgit mv f.$Adiarnfd f.$Odiarnfc &&\n+\t\tgit mv d.$Adiarnfd d.$Odiarnfc &&\n+\t\tgit mv l.$Adiarnfd l.$Odiarnfc &&\n+\t\tgit commit -m \"mv Adiarnfd Odiarnfc\"\n+\t'\n+\t# Files can be checked out as nfc\n+\t# And the link has been corrected from nfd to nfc\n+\ttest_expect_success \"git checkout nfc\" '\n+\t\trm f.$Odiarnfc &&\n+\t\tgit checkout f.$Odiarnfc\n+\t'\n+\t# Make it possible to checkout files with their NFD names\n+\ttest_expect_success \"git checkout file nfd\" '\n+\t\trm -f f.* &&\n+\t\tgit checkout f.$Odiarnfd\n+\t'\n+\t# Make it possible to checkout links with their NFD names\n+\ttest_expect_success \"git checkout link nfd\" '\n+\t\trm l.* &&\n+\t\tgit checkout l.$Odiarnfd\n+\t'\n+\ttest_expect_success \"setup case mac2\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_2\n+\t'\n+\t# This will test nfd2nfc in git commit\n+\ttest_expect_success \"commit file d2.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d2.$Adiarnfd &&\n+\t\techo d2.$Adiarnfd/f.$Adiarnfd >d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit add d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d2.$Adiarnfd/f.$Adiarnfd\" -- d2.$Adiarnfd/f.$Adiarnfd\n+\t'\n+\ttest_expect_success \"setup for long decomposed filename\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_long_nfd_fn\n+\t'\n+\ttest_expect_success \"Add long decomposed filename\" '\n+\t\techo longd >$Alongd &&\n+\t\tgit add * &&\n+\t\tgit commit -m \"Long filename\"\n+\t'\n+\ttest_expect_success \"setup for long precomposed filename\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_long_nfc_fn\n+\t'\n+\ttest_expect_success \"Add long precomposed filename\" '\n+\t\techo longc >$Alongc &&\n+\t\tgit add * &&\n+\t\tgit commit -m \"Long filename\"\n+\t'\n+\t# Test if the global core.precomposedunicode stops autosensing\n+\t# Must be the last test case\n+\ttest_expect_success \"respect git config --global core.precomposedunicode\" '\n+\t\tgit config --global core.precomposedunicode true &&\n+\t\trm -rf .git &&\n+\t\tgit init &&\n+\t\tprecomposedunicode=`git config core.precomposedunicode` &&\n+\t\ttest \"$precomposedunicode\" = \"true\"\n+\t'\n+else\n+\t say \"Skipping nfc/nfd tests\"\n+fi\n+\n+test_done\ndiff --git a/utf8.c b/utf8.c\nindex 8acbc66..a544f15 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -433,19 +433,12 @@ int is_encoding_utf8(const char *name)\n #else\n \ttypedef char * iconv_ibp;\n #endif\n-char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding)\n+char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)\n {\n-\ticonv_t conv;\n-\tsize_t insz, outsz, outalloc;\n+\tsize_t outsz, outalloc;\n \tchar *out, *outpos;\n \ticonv_ibp cp;\n \n-\tif (!in_encoding)\n-\t\treturn NULL;\n-\tconv = iconv_open(out_encoding, in_encoding);\n-\tif (conv == (iconv_t) -1)\n-\t\treturn NULL;\n-\tinsz = strlen(in);\n \toutsz = insz;\n \toutalloc = outsz + 1; /* for terminating NUL */\n \tout = xmalloc(outalloc);\n@@ -459,7 +452,6 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n \t\t\tsize_t sofar;\n \t\t\tif (errno != E2BIG) {\n \t\t\t\tfree(out);\n-\t\t\t\ticonv_close(conv);\n \t\t\t\treturn NULL;\n \t\t\t}\n \t\t\t/* insz has remaining number of bytes.\n@@ -478,6 +470,20 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n \t\t\tbreak;\n \t\t}\n \t}\n+\treturn out;\n+}\n+\n+char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding)\n+{\n+\ticonv_t conv;\n+\tchar *out;\n+\n+\tif (!in_encoding)\n+\t\treturn NULL;\n+\tconv = iconv_open(out_encoding, in_encoding);\n+\tif (conv == (iconv_t) -1)\n+\t\treturn NULL;\n+\tout = reencode_string_iconv(in, strlen(in), conv);\n \ticonv_close(conv);\n \treturn out;\n }\ndiff --git a/utf8.h b/utf8.h\nindex 81f2c82..3c0ae76 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -14,6 +14,7 @@ int strbuf_add_wrapped_bytes(struct strbuf *buf, const char *data, int len,\n \t\t\t     int indent, int indent2, int width);\n \n #ifndef NO_ICONV\n+char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv);\n char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding);\n #else\n #define reencode_string(a,b,c) NULL\n-- \n1.7.11.1.47.ge34cb30\n"},{"id":"194480","messageId":"7vr4styfbg.fsf@alter.siamese.dyndns.org","threadId":"30937","inReplyTo":"201207022232.51737.tboegi@web.de","subject":"Re: [PATCH v7] git on Mac OS and precomposed unicode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-02T22:21:39Z","receivedAt":"2012-07-02T22:21:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> +core.precomposedunicode::\n> +\t...\n> +\tWhen false, file names are handled fully transparent by git, which means\n> +\tthat file names are stored as decomposed unicode in the repository.\n\nI do not think it means any such thing.\n\nWe just take whatever the platform throws at us and shove that in\nthe repository.  On MacOS X with HFS+, it may be decomposed UTF-8,\nbut we do not even try to ensure everything (like the path added by\nsomebody else on a BSD system in a commit that you fetched) is in a\nparticular encoding.\n\n> diff --git a/Makefile b/Makefile\n> index f62ca2a..55ceb10 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -607,6 +607,7 @@ LIB_H += compat/bswap.h\n>  LIB_H += compat/cygwin.h\n>  LIB_H += compat/mingw.h\n>  LIB_H += compat/obstack.h\n> +LIB_H += compat/precomposed_utf8.h\n\nMicronit.  Shouldn't these all be called \"precompose_utf8\"\nthroughout the patch?\n\nWe are asking Git \"please normalize by precompose any UTF-8\npathnames\" when we give the -DPRECOMPOSE_UNICODE C-preprocessor\nmacro, and compat/precompose_utf8.[ch] are to implement the\nmachinery to do so.\n\n> diff --git a/compat/precomposed_utf8.c b/compat/precomposed_utf8.c\n> new file mode 100644\n> index 0000000..14bb0ce\n> --- /dev/null\n> +++ b/compat/precomposed_utf8.c\n> @@ -0,0 +1,189 @@\n> +/* Converts filenames from decomposed unicode into precomposed unicode.\n> +   Used on MacOS X.\n> +*/\n\nMicronit.\n\n\t/*\n         * Multi-line comments begin by slash asterisk newline.\n         * and ends with a run of SP to align asterisk, asterisk\n         * and then newline, like this.\n         */\n> +#define __PRECOMPOSED_UNICODE_C__\n> +\n> +#include \"cache.h\"\n> +#include \"utf8.h\"\n> +#include \"precomposed_utf8.h\"\n\n\n> +#include \"stdio.h\"\n\nYou shouldn't need \"stdio.h\" as you are including \"git-compat-util.h\"\nvia \"cache.h\".\n\n> diff --git a/compat/precomposed_utf8.h b/compat/precomposed_utf8.h\n> new file mode 100644\n> index 0000000..708a1c6\n> --- /dev/null\n> +++ b/compat/precomposed_utf8.h\n> ...\n> +#ifndef __PRECOMPOSED_UNICODE_C__\n> +#define dirent dirent_prec_psx\n> +#define opendir(n) precomposed_utf8_opendir(n)\n> +#define readdir(d) precomposed_utf8_readdir(d)\n> +#define closedir(d) precomposed_utf8_closedir(d)\n> +#define DIR PREC_DIR\n> +#endif /* __PRECOMPOSED_UNICODE_C__ */\n\nHrm, this is not wrong per-se, but looks somewhat unwieldy.\n\n> +#define  __PRECOMPOSED_UNICODE_H__\n> +#endif /* __PRECOMPOSED_UNICODE_H__ */\n\n> diff --git a/utf8.c b/utf8.c\n> index 8acbc66..a544f15 100644\n> --- a/utf8.c\n> +++ b/utf8.c\n> @@ -433,19 +433,12 @@ int is_encoding_utf8(const char *name)\n> ...\n> @@ -478,6 +470,20 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n>  \t\t\tbreak;\n>  \t\t}\n>  \t}\n> +\treturn out;\n> +}\n> +\n> +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding)\n> +{\n> +\ticonv_t conv;\n> +\tchar *out;\n> +\n> +\tif (!in_encoding)\n> +\t\treturn NULL;\n> +\tconv = iconv_open(out_encoding, in_encoding);\n> +\tif (conv == (iconv_t) -1)\n> +\t\treturn NULL;\n> +\tout = reencode_string_iconv(in, strlen(in), conv);\n>  \ticonv_close(conv);\n>  \treturn out;\n>  }\n\nMuch nicer ;-).\n"}]}