{"thread":{"id":"30888","subject":"[RFC/PATCH v5] git on Mac OS and precomposed unicode","startedAt":"2012-06-24T15:47:05Z","lastAt":"2012-06-26T06:27:58Z","messageCount":4,"participants":["Torsten Bögershausen","Junio C Hamano"],"isPatch":true,"patchVersion":5,"patchTotal":null},"messages":[{"id":"194166","messageId":"201206241747.06525.tboegi@web.de","threadId":"30888","inReplyTo":null,"subject":"[RFC/PATCH v5] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-06-24T15:47:05Z","receivedAt":"2012-06-24T15:47:05Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"The problem:\nMac OS X may manipulate file names containing unicode on file systems\nHFS+, VFAT or SAMBA.\n\nWhen a file using unicode code points outside ASCII is created on a HFS+ drive,\nthe file name is converted into decomposed unicode and written to disk.\nNo conversion is done if the file name is already decomposed unicode.\n\nCalling open(\"\\xc3\\x84\", ...) whith a precomposed \"Ä\" yields the same\nresult as open(\"\\x41\\xcc\\x88\",...) with a decomposed \"Ä\".\n\nThis precomposition is done for all file system calls like\ncreat(), open(), fopen(), stat/lstat() etc.\n\nThe result is that readdir() will return the file names in decomposed unicode,\neven if the user expects precomposed unicode.\n\nIn opposite to HFS+ stores Mac OS X  files on a VFAT drive (e.g. an USB drive),\nin precomposed unicode, but readdir() returns file names in decomposed unicode.\n\nWhen a git repository is stored on a network share using SAMBA, file names are\nsend over the wire and written to disk on the remote system in precomposed\nunicode.\nTo be compatible with HFS+ and VFAT returns Mac OS X readdir()\ndecomposed unicode.\n\nNFS mounted from Linux is fully transparent and does not need all this.\n\nSummary:\nfilenames which are unicode equivalent in their precomposed or decomposed\nform are folded onto the same file system object.\nCalling readdir() returns always the decomposed form.\n\nThe unicode decomposition creates some problems:\n- \"git add\" needs the decomposed form on the command line,\n  so that the file name is picked up when readdir() is called\n  to build a list of files on disk.\n- The decomposed form is not (easily) available on the keyboard.\n  To work around this, a wildcard could be used in \"git add\":\n  Instead of using \"git add Märchen.txt\" the user needs to enter\n  \"git add M*rchen\", \"git add M<TAB>\" or \"git add *\".\n- \"git log\", \"git mv\" and all other commands needs the decomposed form\n  to find the file name which is stored as decomposed in the index.\n- The file names are stored in decomposed unicode in the index, but\n  precomposed on disk.\n    This makes it impossible to use this repository under e.g.\n    Linux or Windows:\n    All files appear to be deleted in the decomposed form and\n    untracked in the precomposed form.\n\nKnowing that Mac OS X writes file names as precomposed to disk,\nand treats precomposed and decomposed file names as equal, git under Mac OS X\ncan be improved to revert the unicode decomposition of file names.\n\nA new confguration variable is added: \"core.precomposedunicode\"\n\nIf set to false, git behaves exactly as older versions of git.\nWhen a new git version is installed and there is a repository\nwhere the configuration \"core.precomposedunicode\" is not present,\nthe new git is backward compatible.\n\nIf set to true, git allows forces the index to only use precomposed unicode.\n\nThe code in compat/precomposed_utf8.c implements basically 4 new functions:\nprecomposed_utf8_opendir(), precomposed_utf8_readdir(),\nprecomposed_utf8_closedir() precompose_argv()\n\nAs theere are only precomposed unicode file names in the index,\nand precomposed and decomposed are treated as equal by the OS,\nall arguments into git (argv[1]..argv[n]) are converted into\nprecomposed unicode.\nThis is done in parse-options.c by calling precompose_argv().\nThis function is actually a #define, and it is only defined under Mac OS.\nNothing is converted on any other OS.\n\nThe argv[] conversion allows to use the TAB filename completion done by the\nshell on command line.\nIt tolerates other tools which use readdir() to feed decomposed file names\ninto git.\n\nAuto sensing:\nWhen creating a new git repository with \"git init\" or \"git clone\",\n\"core.precomposedunicode\" will be set \"false\".\n\nThe user needs to activate this feature manually.\nShe typically sets core.precomposedunicode to \"true\" on HFS and VFAT,\nor file systems mounted via SAMBA onto a Linux box.\n\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\n Documentation/config.txt     |   9 ++\n Makefile                     |   3 +\n builtin/init-db.c            |   2 +\n cache.h                      |   1 +\n compat/precomposed_utf8.c    | 200 +++++++++++++++++++++++++++++++++++++++++++\n compat/precomposed_utf8.h    |  30 +++++++\n config.c                     |   5 ++\n environment.c                |   1 +\n git-compat-util.h            |   9 ++\n parse-options.c              |   1 +\n t/t3910-mac-os-precompose.sh | 128 +++++++++++++++++++++++++++\n 11 files changed, 389 insertions(+)\n create mode 100644 compat/precomposed_utf8.c\n create mode 100644 compat/precomposed_utf8.h\n create mode 100755 t/t3910-mac-os-precompose.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 915cb5a..8e4a6a8 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -210,6 +210,15 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignorecase true if appropriate when the repository\n is created.\n \n+core.precomposedunicode::\n+\tThis option is only used by Mac OS implementation of git.\n+\tWhen core.precomposedunicode=true, git reverts the unicode decomposition\n+\tof filenames done by Mac OS. This is useful when sharing a repository\n+\tbetween Mac OS and Linux or\tWindows.\n+\t(Git for Windows/msysGit 1.7.10 is needed, or git under cygwin 1.7).\n+\tWhen false, file names are handled fully transparent by git, which means\n+\tthat file names are stored as decomposed unicode in the\trepository.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Makefile b/Makefile\nindex f62ca2a..55ceb10 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -607,6 +607,7 @@ LIB_H += compat/bswap.h\n LIB_H += compat/cygwin.h\n LIB_H += compat/mingw.h\n LIB_H += compat/obstack.h\n+LIB_H += compat/precomposed_utf8.h\n LIB_H += compat/terminal.h\n LIB_H += compat/win32/dirent.h\n LIB_H += compat/win32/poll.h\n@@ -1000,6 +1001,8 @@ ifeq ($(uname_S),Darwin)\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tCOMPAT_OBJS += compat/precomposed_utf8.o\n+\tBASIC_CFLAGS += -DPRECOMPOSED_UNICODE\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 0dacb8b..06953df 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -290,6 +290,8 @@ static int create_default_files(const char *template_path)\n \t\tstrcpy(path + len, \"CoNfIg\");\n \t\tif (!access(path, F_OK))\n \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n+\n+\t\tprobe_utf8_pathname_composition(path, len);\n \t}\n \n \treturn reinit;\ndiff --git a/cache.h b/cache.h\nindex 06413e1..bc69166 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -560,6 +560,7 @@ extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern int mac_os_precomposed_unicode;\n \n enum branch_track {\n \tBRANCH_TRACK_UNSPECIFIED = -1,\ndiff --git a/compat/precomposed_utf8.c b/compat/precomposed_utf8.c\nnew file mode 100644\nindex 0000000..f510f21\n--- /dev/null\n+++ b/compat/precomposed_utf8.c\n@@ -0,0 +1,200 @@\n+#define __PRECOMPOSED_UNICODE_C__\n+\n+#include \"../cache.h\"\n+#include \"../utf8.h\"\n+#include <stdlib.h>\n+#include <string.h>\n+#include <stdio.h>\n+#include <stdint.h>\n+#include \"precomposed_utf8.h\"\n+\n+const static char *repo_encoding = \"UTF-8\";\n+const static char *path_encoding = \"UTF-8-MAC\";\n+\n+\n+/* Code borrowed from utf8.c */\n+#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))\n+\ttypedef const char * iconv_ibp;\n+#else\n+\ttypedef char * iconv_ibp;\n+#endif\n+\n+static char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)\n+{\n+\tsize_t outsz, outalloc;\n+\tchar *out, *outpos;\n+\ticonv_ibp cp;\n+\n+\toutsz = insz;\n+\toutalloc = outsz + 1; /* for terminating NUL */\n+\tout = xmalloc(outalloc);\n+\toutpos = out;\n+\tcp = (iconv_ibp)in;\n+\n+\twhile (1) {\n+\t\tsize_t cnt = iconv(conv, &cp, &insz, &outpos, &outsz);\n+\n+\t\tif (cnt == -1) {\n+\t\t\tsize_t sofar;\n+\t\t\tif (errno != E2BIG) {\n+\t\t\t\tfree(out);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t\t/* insz has remaining number of bytes.\n+\t\t\t * since we started outsz the same as insz,\n+\t\t\t * it is likely that insz is not enough for\n+\t\t\t * converting the rest.\n+\t\t\t */\n+\t\t\tsofar = outpos - out;\n+\t\t\toutalloc = sofar + insz * 2 + 32;\n+\t\t\tout = xrealloc(out, outalloc);\n+\t\t\toutpos = out + sofar;\n+\t\t\toutsz = outalloc - sofar - 1;\n+\t\t}\n+\t\telse {\n+\t\t\t*outpos = '\\0';\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn out;\n+}\n+\n+static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n+{\n+\tconst uint8_t *utf8p = (const uint8_t*) s;\n+\tsize_t strlen_chars = 0;\n+\tsize_t ret = 0;\n+\n+\tif ((!utf8p) || (!*utf8p))\n+\t\treturn 0;\n+\n+\twhile((*utf8p) && maxlen) {\n+\t\tif (*utf8p & 0x80)\n+\t\t\tret++;\n+\t\tstrlen_chars++;\n+\t\tutf8p++;\n+\t\tmaxlen--;\n+\t}\n+\tif (strlen_c)\n+\t\t*strlen_c = strlen_chars;\n+\n+\treturn ret;\n+}\n+\n+\n+void probe_utf8_pathname_composition(char *path, int len)\n+{\n+\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n+\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n+\tint output_fd;\n+\tif (mac_os_precomposed_unicode != -1)\n+\t\treturn; /* We found it defined in the global config, respect it */\n+\tpath[len] = 0;\n+\tstrcpy(path + len, auml_nfc);\n+\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n+\tif (output_fd >=0) {\n+\t\tclose(output_fd);\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfd);\n+\t\t/* Indicate the the user, that we can configure it to true */\n+\t\tif (0 == access(path, R_OK))\n+\t\t\tgit_config_set(\"core.precomposedunicode\", \"false\");\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfc);\n+\t\tunlink(path);\n+\t}\n+}\n+\n+\n+void precompose_argv(int argc, const char **argv)\n+{\n+\tint i = 0;\n+\tconst char *oldarg;\n+\tchar *newarg;\n+\ticonv_t ic_precompose;\n+\n+\tif (mac_os_precomposed_unicode != 1)\n+\t\treturn;\n+\n+\tic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (ic_precompose == (iconv_t) -1)\n+\t\treturn;\n+\n+\twhile (i < argc) {\n+\t\tsize_t namelen;\n+\t\toldarg = argv[i];\n+\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n+\t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n+\t\t\tif (newarg)\n+\t\t\t\targv[i] = newarg;\n+\t\t}\n+\t\ti++;\n+\t}\n+\ticonv_close(ic_precompose);\n+}\n+\n+\n+PRECOMPOSED_UTF_DIR * precomposed_utf8_opendir(const char *dirname)\n+{\n+\tPRECOMPOSED_UTF_DIR *precomposed_utf8_dir;\n+\tprecomposed_utf8_dir = xmalloc(sizeof(PRECOMPOSED_UTF_DIR));\n+\n+\tprecomposed_utf8_dir->dirp = opendir(dirname);\n+\tif (!precomposed_utf8_dir->dirp) {\n+\t\tfree(precomposed_utf8_dir);\n+\t\treturn NULL;\n+\t}\n+\tprecomposed_utf8_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (precomposed_utf8_dir->ic_precompose == (iconv_t) -1) {\n+\t\tclosedir(precomposed_utf8_dir->dirp);\n+\t\tfree(precomposed_utf8_dir);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn precomposed_utf8_dir;\n+}\n+\n+struct dirent * precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)\n+{\n+\tstruct dirent *res;\n+\tsize_t namelen = 0;\n+\n+\tres = readdir(precomposed_utf8_dirp->dirp);\n+\tif (res && (mac_os_precomposed_unicode == 1) && has_utf8(res->d_name, (size_t)-1, &namelen)) {\n+\t\tint ret_errno = errno;\n+\t\tsize_t outsz = sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) - 1; /* one for \\0 */\n+\t\tchar *outpos = precomposed_utf8_dirp->dirent_nfc.d_name;\n+\t\ticonv_ibp cp;\n+\t\tsize_t cnt;\n+\t\tsize_t insz = namelen;\n+\t\tcp = (iconv_ibp)res->d_name;\n+\n+\t\t/* Copy all data except the name */\n+\t\tmemcpy(&precomposed_utf8_dirp->dirent_nfc, res,\n+\t\t       sizeof(precomposed_utf8_dirp->dirent_nfc)-sizeof(precomposed_utf8_dirp->dirent_nfc.d_name));\n+\t\terrno = 0;\n+\n+\t\tcnt = iconv(precomposed_utf8_dirp->ic_precompose, &cp, &insz, &outpos, &outsz);\n+\t\tif (cnt < sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) -1) {\n+\t\t\t*outpos = 0;\n+\t\t\terrno = ret_errno;\n+\t\t\treturn &precomposed_utf8_dirp->dirent_nfc;\n+\t\t}\n+\t\terrno = ret_errno;\n+\t}\n+\treturn res;\n+}\n+\n+\n+int precomposed_utf8_closedir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)\n+{\n+\tint ret_value;\n+\tint ret_errno;\n+\tret_value = closedir(precomposed_utf8_dirp->dirp);\n+\tret_errno = errno;\n+\tif (precomposed_utf8_dirp->ic_precompose != (iconv_t)-1)\n+\t\ticonv_close(precomposed_utf8_dirp->ic_precompose);\n+\tfree(precomposed_utf8_dirp);\n+\terrno = ret_errno;\n+\treturn ret_value;\n+}\ndiff --git a/compat/precomposed_utf8.h b/compat/precomposed_utf8.h\nnew file mode 100644\nindex 0000000..79e65e7\n--- /dev/null\n+++ b/compat/precomposed_utf8.h\n@@ -0,0 +1,30 @@\n+#ifndef __PRECOMPOSED_UNICODE_H__\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <dirent.h>\n+#include <iconv.h>\n+\n+\n+typedef struct {\n+\ticonv_t ic_precompose;\n+\tDIR *dirp;\n+\tstruct dirent dirent_nfc;\n+} PRECOMPOSED_UTF_DIR;\n+\n+char *precompose_str(const char *in, iconv_t ic_precompose);\n+void precompose_argv(int argc, const char **argv);\n+void probe_utf8_pathname_composition(char *, int);\n+\n+PRECOMPOSED_UTF_DIR *precomposed_utf8_opendir(const char *dirname);\n+struct dirent *precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *dirp);\n+int precomposed_utf8_closedir(PRECOMPOSED_UTF_DIR *dirp);\n+\n+#ifndef __PRECOMPOSED_UNICODE_C__\n+#define opendir(n) precomposed_utf8_opendir(n)\n+#define readdir(d) precomposed_utf8_readdir(d)\n+#define closedir(d) precomposed_utf8_closedir(d)\n+#define DIR PRECOMPOSED_UTF_DIR\n+#endif /* __PRECOMPOSED_UNICODE_C__ */\n+\n+#define  __PRECOMPOSED_UNICODE_H__\n+#endif /* __PRECOMPOSED_UNICODE_H__ */\ndiff --git a/config.c b/config.c\nindex 71ef171..50d4c0e 100644\n--- a/config.c\n+++ b/config.c\n@@ -758,6 +758,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.precomposedunicode\")) {\n+\t\tmac_os_precomposed_unicode = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 669e498..200a04f 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -58,6 +58,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n+int mac_os_precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 5bd9ad7..600fa9e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -153,6 +153,15 @@\n #endif\n #endif\n \n+/* used on Mac OS X */\n+#ifdef PRECOMPOSED_UNICODE\n+#include \"compat/precomposed_utf8.h\"\n+#else\n+#define precompose_str(in,i_nfd2nfc)\n+#define precompose_argv(c,v)\n+#define probe_utf8_pathname_composition(a,b)\n+#endif\n+\n #ifndef NO_LIBGEN_H\n #include <libgen.h>\n #else\ndiff --git a/parse-options.c b/parse-options.c\nindex ab70c29..c1c66bd 100644\n--- a/parse-options.c\n+++ b/parse-options.c\n@@ -476,6 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n \t\tusage_with_options(usagestr, options);\n \t}\n \n+\tprecompose_argv(argc, argv);\n \treturn parse_options_end(&ctx);\n }\n \ndiff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh\nnew file mode 100755\nindex 0000000..b40f1c1\n--- /dev/null\n+++ b/t/t3910-mac-os-precompose.sh\n@@ -0,0 +1,128 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2012 Torsten Bögershausen\n+#\n+\n+test_description='utf-8 decomposed (nfd) converted to precomposed (nfc)'\n+\n+. ./test-lib.sh\n+\n+Adiarnfc=`printf '\\303\\204'`\n+Odiarnfc=`printf '\\303\\226'`\n+Adiarnfd=`printf 'A\\314\\210'`\n+Odiarnfd=`printf 'O\\314\\210'`\n+\n+# check if the feature is compiled in\n+mkdir junk &&\n+>junk/\"$Adiarnfc\" &&\n+case \"$(cd junk && echo *)\" in\n+\t\"$Adiarnfd\")\n+\ttest_nfd=1\n+\t;;\n+\t*)\t;;\n+esac\n+rm -rf junk\n+\n+\n+if test \"$test_nfd\"\n+then\n+\ttest_expect_success \"detect if nfd needed\" '\n+\t\tprecomposedunicode=`git config core.precomposedunicode` &&\n+\t\ttest \"$precomposedunicode\" = false &&\n+\t\tgit config core.precomposedunicode true\n+\t'\n+\ttest_expect_success \"setup\" '\n+\t\t>x &&\n+\t\tgit add x &&\n+\t\tgit commit -m \"1st commit\" &&\n+\t\tgit rm x &&\n+\t\tgit commit -m \"rm x\"\n+\t'\n+\ttest_expect_success \"setup case mac\" '\n+\t\tgit checkout -b mac_os\n+\t'\n+\t# This will test nfd2nfc in readdir()\n+\ttest_expect_success \"add file Adiarnfc\" '\n+\t\techo f.Adiarnfc >f.$Adiarnfc &&\n+\t\tgit add f.$Adiarnfc &&\n+\t\tgit commit -m \"add f.$Adiarnfc\"\n+\t'\n+\t# This will test nfd2nfc in git stage()\n+\ttest_expect_success \"stage file d.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d.$Adiarnfd &&\n+\t\techo d.$Adiarnfd/f.$Adiarnfd >d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit stage d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d.$Adiarnfd/f.$Adiarnfd\"\n+\t'\n+\ttest_expect_success \"add link Adiarnfc\" '\n+\t\tln -s d.$Adiarnfd/f.$Adiarnfd l.$Adiarnfc &&\n+\t\tgit add l.$Adiarnfc &&\n+\t\tgit commit -m \"add l.Adiarnfc\"\n+\t'\n+\t# This will test git log\n+\ttest_expect_success \"git log f.Adiar\" '\n+\t\tgit log f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit log f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git ls-files\n+\ttest_expect_success \"git lsfiles f.Adiar\" '\n+\t\tgit ls-files f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit ls-files f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git mv\n+\ttest_expect_success \"git mv\" '\n+\t\tgit mv f.$Adiarnfd f.$Odiarnfc &&\n+\t\tgit mv d.$Adiarnfd d.$Odiarnfc &&\n+\t\tgit mv l.$Adiarnfd l.$Odiarnfc &&\n+\t\tgit commit -m \"mv Adiarnfd Odiarnfc\"\n+\t'\n+\t# Files can be checked out as nfc\n+\t# And the link has been corrected from nfd to nfc\n+\ttest_expect_success \"git checkout nfc\" '\n+\t\trm f.$Odiarnfc &&\n+\t\tgit checkout f.$Odiarnfc\n+\t'\n+\t# Make it possible to checkout files with their NFD names\n+\ttest_expect_success \"git checkout file nfd\" '\n+\t\trm -f f.* &&\n+\t\tgit checkout f.$Odiarnfd\n+\t'\n+\t# Make it possible to checkout links with their NFD names\n+\ttest_expect_success \"git checkout link nfd\" '\n+\t\trm l.* &&\n+\t\tgit checkout l.$Odiarnfd\n+\t'\n+\ttest_expect_success \"setup case mac2\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_2\n+\t'\n+\t# This will test nfd2nfc in git commit\n+\ttest_expect_success \"commit file d2.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d2.$Adiarnfd &&\n+\t\techo d2.$Adiarnfd/f.$Adiarnfd >d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit add d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d2.$Adiarnfd/f.$Adiarnfd\" -- d2.$Adiarnfd/f.$Adiarnfd\n+\t'\n+\t# Test if the global core.precomposedunicode stops autosensing\n+\t# Must be the last test case\n+\ttest_expect_success \"respect git config --global core.precomposedunicode\" '\n+\t\tgit config --global core.precomposedunicode true &&\n+\t\trm -rf .git &&\n+\t\tgit init &&\n+\t\tprecomposedunicode=`git config core.precomposedunicode` &&\n+\t\ttest \"$precomposedunicode\" = \"true\"\n+\t'\n+else\n+\t say \"Skipping nfc/nfd tests\"\n+fi\n+\n+test_done\n-- \n1.7.11.1.30.g00c4fee\n"},{"id":"194231","messageId":"7vehp3f5ki.fsf@alter.siamese.dyndns.org","threadId":"30888","inReplyTo":"201206241747.06525.tboegi@web.de","subject":"Re: [RFC/PATCH v5] git on Mac OS and precomposed unicode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-06-25T21:24:29Z","receivedAt":"2012-06-25T21:24:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> Knowing that Mac OS X writes file names as precomposed to disk,\n> and treats precomposed and decomposed file names as equal, git under Mac OS X\n> can be improved to revert the unicode decomposition of file names.\n\nThe basic idea looks really good, but there are a few nits.\n\nThanks.\n\n> The user needs to activate this feature manually.\n> She typically sets core.precomposedunicode to \"true\" on HFS and VFAT,\n> or file systems mounted via SAMBA onto a Linux box.\n\nWhat do you mean by the last part?  A Linux box that mounts a\nfilesystem from MacOS X box via cifs?  Or MacOS X mounts a\nfilesystem from a Linux box over cifs?\n\n> +core.precomposedunicode::\n> +\tThis option is only used by Mac OS implementation of git.\n> +\tWhen core.precomposedunicode=true, git reverts the unicode decomposition\n> +\tof filenames done by Mac OS. This is useful when sharing a repository\n> +\tbetween Mac OS and Linux or\tWindows.\n\nWhy this funny inter-word spacing?\n\n> +\t(Git for Windows/msysGit 1.7.10 is needed, or git under cygwin 1.7).\n> +\tWhen false, file names are handled fully transparent by git, which means\n> +\tthat file names are stored as decomposed unicode in the\trepository.\n\nI am assuming, after reading from this section, that the answer to\nmy earlier question is \"MacOS X that mounts over cifs from whatever\nfile server\" (i.e. the latter).  \n\nIt often is a good idea, after writing \"X , which means that Y\", if\nyou can just drop X and go strait to Y.  The result often becomes\nmuch more clear.\n\nAs this section is clearly labeled as \"Mac OS only\", I think in this\ncase it is OK to say \"when false, file names are always stored as\ndecomposed unicode...\" for now.\n\nGiven that your compat/precomposed_utf8.[ch] looks like it does not\nuse anything MacOS X specific (other than \"UTF-8-MAC\" given to\niconv(3) API), do you foresee that this might become useful on non\nMac build of Git in the future?\n\n> diff --git a/builtin/init-db.c b/builtin/init-db.c\n> index 0dacb8b..06953df 100644\n> --- a/builtin/init-db.c\n> +++ b/builtin/init-db.c\n> @@ -290,6 +290,8 @@ static int create_default_files(const char *template_path)\n>  \t\tstrcpy(path + len, \"CoNfIg\");\n>  \t\tif (!access(path, F_OK))\n>  \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n> +\n> +\t\tprobe_utf8_pathname_composition(path, len);\n>  \t}\n\nProbing for case-insensitiveness and probing for UTF-8 mangling are\nlogically related and do not deserve a separation with the extra\nblank line.\n\n> diff --git a/compat/precomposed_utf8.c b/compat/precomposed_utf8.c\n> new file mode 100644\n> index 0000000..f510f21\n> --- /dev/null\n> +++ b/compat/precomposed_utf8.c\n> @@ -0,0 +1,200 @@\n> +#define __PRECOMPOSED_UNICODE_C__\n> +\n> +#include \"../cache.h\"\n> +#include \"../utf8.h\"\n\nDo you really need \"../\"?  I thought we compiled with -I.. from the\nMakefile.\n\n> +#include <stdlib.h>\n> +#include <string.h>\n> +#include <stdio.h>\n> +#include <stdint.h>\n\nAnd as you are including \"cache.h\" (which in turn includes\n\"git-compat-util.h\"), I doubt you would need to include these\nyourself, and if you do not need them, I would prefer not to see\nthem.\n\n> +#include \"precomposed_utf8.h\"\n> +\n> +const static char *repo_encoding = \"UTF-8\";\n> +const static char *path_encoding = \"UTF-8-MAC\";\n> +\n> +\n> +/* Code borrowed from utf8.c */\n> +#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))\n> +\ttypedef const char * iconv_ibp;\n> +#else\n> +\ttypedef char * iconv_ibp;\n> +#endif\n\nSeeing \"defined(__sun__)\" here, is this not for Mac OS X-only after\nall?  If so, I'd have to ask you to come up with a better name than\n\"mac_os_precomposed_unicode\". It is primarily about sanitizing\ndecomposed utf-8 and it only is a happenstance that Mac OS X is the\nmost likely platform people may need this feature on (iow, \"mac_os\"\nis not the fundamental part of this issue; having to sanitize paths\nof decomposed UTF-8 is).\n\n> +static char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)\n> +{\n> +\tsize_t outsz, outalloc;\n> +\tchar *out, *outpos;\n> +\ticonv_ibp cp;\n> +\n> +\toutsz = insz;\n> +\toutalloc = outsz + 1; /* for terminating NUL */\n> +\tout = xmalloc(outalloc);\n> +\toutpos = out;\n> +\tcp = (iconv_ibp)in;\n> +\n> +\twhile (1) {\n> +\t...\n> +\t}\n> +\treturn out;\n> +}\n\nShouldn't this part be using a new helper function refactored out of\nthe utf8.c::reencode_string() function, instead of cutting and\npasting?\n\n> +static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n> +{\n> +\tconst uint8_t *utf8p = (const uint8_t*) s;\n> +\tsize_t strlen_chars = 0;\n> +\tsize_t ret = 0;\n> +\n> +\tif ((!utf8p) || (!*utf8p))\n> +\t\treturn 0;\n> +\n> +\twhile((*utf8p) && maxlen) {\n> +\t\tif (*utf8p & 0x80)\n> +\t\t\tret++;\n> +\t\tstrlen_chars++;\n> +\t\tutf8p++;\n> +\t\tmaxlen--;\n> +\t}\n> +\tif (strlen_c)\n> +\t\t*strlen_c = strlen_chars;\n> +\n> +\treturn ret;\n> +}\n> +\n> +\n> +void probe_utf8_pathname_composition(char *path, int len)\n> +{\n> +\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n> +\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n> +\tint output_fd;\n> +\tif (mac_os_precomposed_unicode != -1)\n> +\t\treturn; /* We found it defined in the global config, respect it */\n> +\tpath[len] = 0;\n> +\tstrcpy(path + len, auml_nfc);\n> +\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n> +\tif (output_fd >=0) {\n> +\t\tclose(output_fd);\n> +\t\tpath[len] = 0;\n> +\t\tstrcpy(path + len, auml_nfd);\n> +\t\t/* Indicate the the user, that we can configure it to true */\n> +\t\tif (0 == access(path, R_OK))\n> +\t\t\tgit_config_set(\"core.precomposedunicode\", \"false\");\n> +\t\tpath[len] = 0;\n> +\t\tstrcpy(path + len, auml_nfc);\n> +\t\tunlink(path);\n> +\t}\n\nNow this function has figured out if the filesystem mangles composed\nUTF-8 pathnames, shouldn't it flip mac_os_precomposed_unicode to\neither 0 or 1 before it returns?\n\n> +}\n> +\n> +\n> +void precompose_argv(int argc, const char **argv)\n> +{\n> +\tint i = 0;\n> +\tconst char *oldarg;\n> +\tchar *newarg;\n> +\ticonv_t ic_precompose;\n> +\n> +\tif (mac_os_precomposed_unicode != 1)\n> +\t\treturn;\n\nCould the mac_os_precomposed_unicode flag be -1 (unknown) here?\nOtherwise just write\n\n\tif (!mac_os_precomposed_unicode)\n        \treturn;\n\n> +PRECOMPOSED_UTF_DIR * precomposed_utf8_opendir(const char *dirname)\n\nAsterisk sticks to the identifier, i.e.\n\n    PRECOMPOSED_UTF_DIR *precomposed_utf8_opendir(const char *dirname)\n\n> +{\n> +\tPRECOMPOSED_UTF_DIR *precomposed_utf8_dir;\n> +\tprecomposed_utf8_dir = xmalloc(sizeof(PRECOMPOSED_UTF_DIR));\n> +\n> +\tprecomposed_utf8_dir->dirp = opendir(dirname);\n> +\tif (!precomposed_utf8_dir->dirp) {\n> +\t\tfree(precomposed_utf8_dir);\n> +\t\treturn NULL;\n> +\t}\n> +\tprecomposed_utf8_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);\n> +\tif (precomposed_utf8_dir->ic_precompose == (iconv_t) -1) {\n> +\t\tclosedir(precomposed_utf8_dir->dirp);\n> +\t\tfree(precomposed_utf8_dir);\n> +\t\treturn NULL;\n\nHrm, I wonder what value the \"errno\" variable should be set to when\nthis happens.\n\n> +\t}\n> +\n> +\treturn precomposed_utf8_dir;\n> +}\n> +\n> +struct dirent * precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)\n\nLikewise.\n\n> +{\n> +\tstruct dirent *res;\n> +\tsize_t namelen = 0;\n> +\n> +\tres = readdir(precomposed_utf8_dirp->dirp);\n> +\tif (res && (mac_os_precomposed_unicode == 1) && has_utf8(res->d_name, (size_t)-1, &namelen)) {\n\nLikewise.\n\n> +\t\tint ret_errno = errno;\n> +\t\tsize_t outsz = sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) - 1; /* one for \\0 */\n\nA few issues:\n\n - Why (-1)?  Don't you need terminating NUL at the end of the\n   output anyway?\n\n - sizeof(d_name) is most likely incorrect (Cf.\n   http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html)\n\n - Is the reason why it is appropriate to update \"res\" in-place\n   because turning a decomposed name into precomposed form will\n   always yield a shorter result?  Is that guaranteed?\n\n\n> +\t\t/* Copy all data except the name */\n> +\t\tmemcpy(&precomposed_utf8_dirp->dirent_nfc, res,\n> +\t\t       sizeof(precomposed_utf8_dirp->dirent_nfc)-sizeof(precomposed_utf8_dirp->dirent_nfc.d_name));\n> +\t\terrno = 0;\n> +\n> +\t\tcnt = iconv(precomposed_utf8_dirp->ic_precompose, &cp, &insz, &outpos, &outsz);\n> +\t\tif (cnt < sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) -1) {\n\ns/-1)/- 1)/;\n\nCan't this iconv() fail and return -1 here, which would be smaller\nthan the size of the structure minus one?\n\n> +\t\t\t*outpos = 0;\n> +\t\t\terrno = ret_errno;\n> +\t\t\treturn &precomposed_utf8_dirp->dirent_nfc;\n> +\t\t}\n> +\t\terrno = ret_errno;\n> +\t}\n> +\treturn res;\n> +}\n\n> diff --git a/compat/precomposed_utf8.h b/compat/precomposed_utf8.h\n> new file mode 100644\n> index 0000000..79e65e7\n> --- /dev/null\n> +++ b/compat/precomposed_utf8.h\n> @@ -0,0 +1,30 @@\n> +#ifndef __PRECOMPOSED_UNICODE_H__\n> +#include <sys/stat.h>\n> +#include <sys/types.h>\n> +#include <dirent.h>\n> +#include <iconv.h>\n\nThe same comments for #include applies here.\n\n> +typedef struct {\n> +\ticonv_t ic_precompose;\n> +\tDIR *dirp;\n> +\tstruct dirent dirent_nfc;\n> +} PRECOMPOSED_UTF_DIR;\n\nNote that \"struct dirent\" can be defined with d_name[1]; aren't you\nrisking the memory after this structure to be overwritten with a\nlonger name?\n"},{"id":"194232","messageId":"7vwr2vdnsz.fsf@alter.siamese.dyndns.org","threadId":"30888","inReplyTo":"201206241747.06525.tboegi@web.de","subject":"Re: [RFC/PATCH v5] git on Mac OS and precomposed unicode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-06-25T22:33:32Z","receivedAt":"2012-06-25T22:33:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> The problem:\n\nAs to the log message, I've rewritten it a bit by reordering\nparagraphs and cutting redundant sentences. For exact wording nits,\nplease check 'pu' when I finish today's integration cycle and push\nthe results out, but I'll justify the reasoning behind my rewrite\nhere.\n\n> Mac OS X may manipulate file names containing unicode on file systems\n> HFS+, VFAT or SAMBA.\n>\n> When a file using unicode code points outside ASCII is created on a HFS+ drive,\n> the file name is converted into decomposed unicode and written to disk.\n> No conversion is done if the file name is already decomposed unicode.\n\nI do not think it matters very much if it is written decomposed\n(HFS+) or precomposed (VFAT). The important glitch that affects us\nis that readdir(3) on Mac OS X gives the readers decomposed form,\nunless over NFS, and the important saving grace that your patch\nexploits is that stat/open/etc. will take either form and name the\nsame file.  So I tried to minimize the description on how it is\nwritten to disk in my rewrite.\n\n> The unicode decomposition creates some problems:\n> - \"git add\" needs the decomposed form on the command line,\n>   so that the file name is picked up when readdir() is called\n>   to build a list of files on disk.\n> - The decomposed form is not (easily) available on the keyboard.\n>   To work around this, a wildcard could be used in \"git add\":\n>   Instead of using \"git add Märchen.txt\" the user needs to enter\n>   \"git add M*rchen\", \"git add M<TAB>\" or \"git add *\".\n> - \"git log\", \"git mv\" and all other commands needs the decomposed form\n>   to find the file name which is stored as decomposed in the index.\n> - The file names are stored in decomposed unicode in the index, but\n>   precomposed on disk.\n>     This makes it impossible to use this repository under e.g.\n>     Linux or Windows:\n>     All files appear to be deleted in the decomposed form and\n>     untracked in the precomposed form.\n\nI do not think \"workaround\" deserves a mention; presense of mixture\nof precomposed and decomposed forms is the root cause of the\nproblem, and even if we prefer to use precomposed form (for\ninteroperability if nothing else), the \"workaround\" to force more\ndecomposed input will make the problem worse, not better.\n\n> Knowing that Mac OS X writes file names as precomposed to disk,\n\nAgain, how it writes is not important; readdir(3) giving us what is\ndifferent from what we used for creat(2) is.\n\n> The argv[] conversion allows to use the TAB filename completion done by the\n> shell on command line.\n\nYes, this is exactly why \"workaround\" is not a workaround, but is\nyet another problem.\n\n> When creating a new git repository with \"git init\" or \"git clone\",\n> \"core.precomposedunicode\" will be set \"false\".\n>\n> The user needs to activate this feature manually.\n> She typically sets core.precomposedunicode to \"true\" on HFS and VFAT,\n> or file systems mounted via SAMBA onto a Linux box.\n\nThis we might want to change it in a couple of cycles after this\nfeature hits 'next' and people gain experience with it.\n\nI think the reason to choose the safer \"false\" default is to keep\nthe behaviours between an old repository on Mac OS X and a new\nrepository cloned from it also on Mac OS X the same, but if we can\ndetect that the filesystem is broken, and have a code to work around\nthe breakage, I think the longer term direction would be to set it\nto ensure that the resulting history records paths consistently in\nprecomposed form (another choice might be to normalize to decomposed\nform, but my understanding is that it would not help anybody, as\nnobody other than Mac OS X uses it).\n\nThanks.\n"},{"id":"194265","messageId":"4FE9566E.1090208@web.de","threadId":"30888","inReplyTo":"7vwr2vdnsz.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH v5] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-06-26T06:27:58Z","receivedAt":"2012-06-26T06:27:58Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":">On 26.06.12 00:33, Junio C Hamano wrote:\n[...]\nThanks Junio for the long replies.\nI'll come back with better suggestion (within days, weeks or months)\n \n"}]}