{"thread":{"id":"29422","subject":"[PATCH V4] git on Mac OS and precomposed unicode","startedAt":"2012-01-21T19:36:56Z","lastAt":"2012-07-25T20:45:31Z","messageCount":10,"participants":["Torsten Bögershausen","Carlos Martín Nieto","Junio C Hamano","Nguyen Thai Ngoc Duy","Erik Faye-Lund","Robin Rosenberg"],"isPatch":true,"patchVersion":4,"patchTotal":null},"messages":[{"id":"182904","messageId":"201201212036.57632.tboegi@web.de","threadId":"29422","inReplyTo":null,"subject":"[PATCH V4] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-01-21T19:36:56Z","receivedAt":"2012-01-21T19:36:56Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Allow git on Mac OS to store file names in the index in precomposed unicode,\nwhile the file system uses decomposed unicode.\n\nThe problem:\nWhen a file called \"LATIN CAPITAL LETTER A WITH DIAERESIS\"\n(in utf-8 encoded as 0xc3 0x84) is created, the Mac OS filesystem\nconverts \"precomposed unicode\" into \"decomposed unicode\".\nThis means that readdir() will return 0x41 0xcc 0x88.\n\nGit under Mac OS reverts the unicode decomposition of filenames.\n\nThis is useful when pulling/pushing from repositories containing utf-8\nencoded filenames using precomposed utf-8 like Linux or Windows (*).\n\nIt allows sharing git repositories stored on a VFAT file system\n(e.g. a USB stick), and mounted network share using samba.\n\n* (Not all Windows versions support UTF-8 yet:\n   Msysgit needs the unicode branch, cygwin supports UTF-8 since 1.7)\n\nA new confguration variable is added: \"core.precomposedunicode\"\n\nIf set to false, git behaves exactly as older versions of git.\nWhen a new git version is installed and there is a repository\nwhere the configuration \"core.precomposedunicode\" is not present,\nthe new git is backward compatible.\n\nThe code in compat/precomposed_utf8.c implements basically 4 new functions:\nprecomposed_utf8_opendir(), precomposed_utf8_readdir(),\nprecomposed_utf8_closedir() precompose_argv()\n\nIn order to prevent that ever a file name in decomposed unicode is entering\nthe index, a \"brute force\" attempt is taken:\nall arguments into git (argv[1]..argv[n]) are converted into\nprecomposed unicode.\nThis is done in git.c by calling precompose_argv().\nThis function is actually a #define, and it is only defined under Mac OS.\nNothing is converted on any other OS.\n\nAuto sensing:\nWhen creating a new git repository with \"git init\" or \"git clone\",\n\"core.precomposedunicode\" will be set \"false\".\n\nThe user needs to activate this feature manually.\nShe typically sets core.precomposedunicode to \"true\" on HFS and VFAT,\nor file systems mounted via SAMBA onto a Linux box.\n\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\n Documentation/config.txt     |    9 ++\n Makefile                     |    3 +\n builtin/init-db.c            |    2 +\n compat/precomposed_utf8.c    |  208 ++++++++++++++++++++++++++++++++++++++++++\n compat/precomposed_utf8.h    |   30 ++++++\n git-compat-util.h            |    9 ++\n git.c                        |    1 +\n t/t3910-mac-os-precompose.sh |  117 +++++++++++++++++++++++\n 8 files changed, 379 insertions(+), 0 deletions(-)\n create mode 100644 compat/precomposed_utf8.c\n create mode 100644 compat/precomposed_utf8.h\n create mode 100755 t/t3910-mac-os-precompose.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2959390..29ba4b0 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -175,6 +175,15 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignorecase true if appropriate when the repository\n is created.\n \n+core.precomposedunicode::\n+\tThis option is only used by Mac OS implementation of git.\n+\tWhen core.precomposedunicode=true,\n+\tgit reverts the unicode decomposition of filenames done by Mac OS.\n+\tThis is useful when pulling/pushing from repositories containing utf-8\n+\tencoded filenames using precomposed unicode (like Linux).\n+\tWhen false, file names are handled fully transparent by git.\n+\tIf in doubt, keep core.precomposedunicode=false.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Makefile b/Makefile\nindex b21d2f1..a912b45 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -519,6 +519,7 @@ LIB_H += compat/bswap.h\n LIB_H += compat/cygwin.h\n LIB_H += compat/mingw.h\n LIB_H += compat/obstack.h\n+LIB_H += compat/precomposed_utf8.h\n LIB_H += compat/win32/pthread.h\n LIB_H += compat/win32/syslog.h\n LIB_H += compat/win32/poll.h\n@@ -884,6 +885,8 @@ ifeq ($(uname_S),Darwin)\n \tendif\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\n+\tCOMPAT_OBJS += compat/precomposed_utf8.o\n+\tBASIC_CFLAGS += -DPRECOMPOSED_UNICODE\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 0dacb8b..06953df 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -290,6 +290,8 @@ static int create_default_files(const char *template_path)\n \t\tstrcpy(path + len, \"CoNfIg\");\n \t\tif (!access(path, F_OK))\n \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n+\n+\t\tprobe_utf8_pathname_composition(path, len);\n \t}\n \n \treturn reinit;\ndiff --git a/compat/precomposed_utf8.c b/compat/precomposed_utf8.c\nnew file mode 100644\nindex 0000000..285fb45\n--- /dev/null\n+++ b/compat/precomposed_utf8.c\n@@ -0,0 +1,208 @@\n+#define __PRECOMPOSED_UNICODE_C__\n+\n+#include \"../cache.h\"\n+#include \"../utf8.h\"\n+#include <stdlib.h>\n+#include <string.h>\n+#include <stdio.h>\n+#include <stdint.h>\n+#include \"precomposed_utf8.h\"\n+\n+static int mac_os_precomposed_unicode;\n+const static char *repo_encoding = \"UTF-8\";\n+const static char *path_encoding = \"UTF-8-MAC\";\n+\n+\n+/* Code borrowed from utf8.c */\n+#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))\n+\ttypedef const char * iconv_ibp;\n+#else\n+\ttypedef char * iconv_ibp;\n+#endif\n+\n+static char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)\n+{\n+\tsize_t outsz, outalloc;\n+\tchar *out, *outpos;\n+\ticonv_ibp cp;\n+\n+\toutsz = insz;\n+\toutalloc = outsz + 1; /* for terminating NUL */\n+\tout = xmalloc(outalloc);\n+\toutpos = out;\n+\tcp = (iconv_ibp)in;\n+\n+\twhile (1) {\n+\t\tsize_t cnt = iconv(conv, &cp, &insz, &outpos, &outsz);\n+\n+\t\tif (cnt == -1) {\n+\t\t\tsize_t sofar;\n+\t\t\tif (errno != E2BIG) {\n+\t\t\t\tfree(out);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t\t/* insz has remaining number of bytes.\n+\t\t\t * since we started outsz the same as insz,\n+\t\t\t * it is likely that insz is not enough for\n+\t\t\t * converting the rest.\n+\t\t\t */\n+\t\t\tsofar = outpos - out;\n+\t\t\toutalloc = sofar + insz * 2 + 32;\n+\t\t\tout = xrealloc(out, outalloc);\n+\t\t\toutpos = out + sofar;\n+\t\t\toutsz = outalloc - sofar - 1;\n+\t\t}\n+\t\telse {\n+\t\t\t*outpos = '\\0';\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn out;\n+}\n+\n+static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n+{\n+\tconst uint8_t *utf8p = (const uint8_t*) s;\n+\tsize_t strlen_chars = 0;\n+\tsize_t ret = 0;\n+\n+\tif ((!utf8p) || (!*utf8p))\n+\t\treturn 0;\n+\n+\twhile((*utf8p) && maxlen) {\n+\t\tif (*utf8p & 0x80)\n+\t\t\tret++;\n+\t\tstrlen_chars++;\n+\t\tutf8p++;\n+\t\tmaxlen--;\n+\t}\n+\tif (strlen_c)\n+\t\t*strlen_c = strlen_chars;\n+\n+\treturn ret;\n+}\n+\n+\n+void probe_utf8_pathname_composition(char *path, int len)\n+{\n+\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n+\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n+\tint output_fd;\n+\tpath[len] = 0;\n+\tstrcpy(path + len, auml_nfc);\n+\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n+\tif (output_fd >=0) {\n+\t\tclose(output_fd);\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfd);\n+\t\tif (0 == access(path, R_OK))\n+\t\t\tgit_config_set(\"core.precomposedunicode\", \"false\");\n+\t\tpath[len] = 0;\n+\t\tstrcpy(path + len, auml_nfc);\n+\t\tunlink(path);\n+\t}\n+}\n+\n+\n+static int precomposed_unicode_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcasecmp(var, \"core.precomposedunicode\")) {\n+\t\tmac_os_precomposed_unicode = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+void precompose_argv(int argc, const char **argv)\n+{\n+\tint i = 0;\n+\tconst char *oldarg;\n+\tchar *newarg;\n+\ticonv_t ic_precompose;\n+\n+\tgit_config(precomposed_unicode_config, NULL);\n+\tif (!mac_os_precomposed_unicode)\n+\t\treturn;\n+\n+\tic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (ic_precompose == (iconv_t) -1)\n+\t\treturn;\n+\n+\twhile (i < argc) {\n+\t\tsize_t namelen;\n+\t\toldarg = argv[i];\n+\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n+\t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n+\t\t\tif (newarg)\n+\t\t\t\targv[i] = newarg;\n+\t\t}\n+\t\ti++;\n+\t}\n+\ticonv_close(ic_precompose);\n+}\n+\n+\n+PRECOMPOSED_UTF_DIR * precomposed_utf8_opendir(const char *dirname)\n+{\n+\tPRECOMPOSED_UTF_DIR *precomposed_utf8_dir;\n+\tprecomposed_utf8_dir = xmalloc(sizeof(PRECOMPOSED_UTF_DIR));\n+\n+\tprecomposed_utf8_dir->dirp = opendir(dirname);\n+\tif (!precomposed_utf8_dir->dirp) {\n+\t\tfree(precomposed_utf8_dir);\n+\t\treturn NULL;\n+\t}\n+\tprecomposed_utf8_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (precomposed_utf8_dir->ic_precompose == (iconv_t) -1) {\n+\t\tclosedir(precomposed_utf8_dir->dirp);\n+\t\tfree(precomposed_utf8_dir);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn precomposed_utf8_dir;\n+}\n+\n+struct dirent * precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)\n+{\n+\tstruct dirent *res;\n+\tsize_t namelen = 0;\n+\n+\tres = readdir(precomposed_utf8_dirp->dirp);\n+\tif (res && mac_os_precomposed_unicode && has_utf8(res->d_name, (size_t)-1, &namelen)) {\n+\t\tint ret_errno = errno;\n+\t\tsize_t outsz = sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) - 1; /* one for \\0 */\n+\t\tchar *outpos = precomposed_utf8_dirp->dirent_nfc.d_name;\n+\t\ticonv_ibp cp;\n+\t\tsize_t cnt;\n+\t\tsize_t insz = namelen;\n+\t\tcp = (iconv_ibp)res->d_name;\n+\n+\t\t/* Copy all data except the name */\n+\t\tmemcpy(&precomposed_utf8_dirp->dirent_nfc, res,\n+\t\t       sizeof(precomposed_utf8_dirp->dirent_nfc)-sizeof(precomposed_utf8_dirp->dirent_nfc.d_name));\n+\t\terrno = 0;\n+\n+\t\tcnt = iconv(precomposed_utf8_dirp->ic_precompose, &cp, &insz, &outpos, &outsz);\n+\t\tif (cnt < sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) -1) {\n+\t\t\t*outpos = 0;\n+\t\t\terrno = ret_errno;\n+\t\t\treturn &precomposed_utf8_dirp->dirent_nfc;\n+\t\t}\n+\t\terrno = ret_errno;\n+\t}\n+\treturn res;\n+}\n+\n+\n+int precomposed_utf8_closedir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)\n+{\n+\tint ret_value;\n+\tint ret_errno;\n+\tret_value = closedir(precomposed_utf8_dirp->dirp);\n+\tret_errno = errno;\n+\tif (precomposed_utf8_dirp->ic_precompose != (iconv_t)-1)\n+\t\ticonv_close(precomposed_utf8_dirp->ic_precompose);\n+\tfree(precomposed_utf8_dirp);\n+\terrno = ret_errno;\n+\treturn ret_value;\n+}\ndiff --git a/compat/precomposed_utf8.h b/compat/precomposed_utf8.h\nnew file mode 100644\nindex 0000000..79e65e7\n--- /dev/null\n+++ b/compat/precomposed_utf8.h\n@@ -0,0 +1,30 @@\n+#ifndef __PRECOMPOSED_UNICODE_H__\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <dirent.h>\n+#include <iconv.h>\n+\n+\n+typedef struct {\n+\ticonv_t ic_precompose;\n+\tDIR *dirp;\n+\tstruct dirent dirent_nfc;\n+} PRECOMPOSED_UTF_DIR;\n+\n+char *precompose_str(const char *in, iconv_t ic_precompose);\n+void precompose_argv(int argc, const char **argv);\n+void probe_utf8_pathname_composition(char *, int);\n+\n+PRECOMPOSED_UTF_DIR *precomposed_utf8_opendir(const char *dirname);\n+struct dirent *precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *dirp);\n+int precomposed_utf8_closedir(PRECOMPOSED_UTF_DIR *dirp);\n+\n+#ifndef __PRECOMPOSED_UNICODE_C__\n+#define opendir(n) precomposed_utf8_opendir(n)\n+#define readdir(d) precomposed_utf8_readdir(d)\n+#define closedir(d) precomposed_utf8_closedir(d)\n+#define DIR PRECOMPOSED_UTF_DIR\n+#endif /* __PRECOMPOSED_UNICODE_C__ */\n+\n+#define  __PRECOMPOSED_UNICODE_H__\n+#endif /* __PRECOMPOSED_UNICODE_H__ */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 230e198..8911743 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -153,6 +153,15 @@\n #include \"compat/msvc.h\"\n #endif\n \n+/* used on Mac OS X */\n+#ifdef PRECOMPOSED_UNICODE\n+#include \"compat/precomposed_utf8.h\"\n+#else\n+#define precompose_str(in,i_nfd2nfc)\n+#define precompose_argv(c,v)\n+#define probe_utf8_pathname_composition(a,b)\n+#endif\n+\n #ifndef NO_LIBGEN_H\n #include <libgen.h>\n #else\ndiff --git a/git.c b/git.c\nindex 8e34903..265db96 100644\n--- a/git.c\n+++ b/git.c\n@@ -298,6 +298,7 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \t\t    startup_info->have_repository) /* get_git_dir() may set up repo, avoid that */\n \t\t\ttrace_repo_setup(prefix);\n \t}\n+\tprecompose_argv(argc, argv);\n \tcommit_pager_choice();\n \n \tif (!help && p->option & NEED_WORK_TREE)\ndiff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh\nnew file mode 100755\nindex 0000000..ba3d83c\n--- /dev/null\n+++ b/t/t3910-mac-os-precompose.sh\n@@ -0,0 +1,117 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2012 Torsten Bögershausen\n+#\n+\n+test_description='utf-8 decomposed (nfd) converted to precomposed (nfc)'\n+\n+. ./test-lib.sh\n+\n+Adiarnfc=`printf '\\303\\204'`\n+Odiarnfc=`printf '\\303\\226'`\n+Adiarnfd=`printf 'A\\314\\210'`\n+Odiarnfd=`printf 'O\\314\\210'`\n+\n+mkdir junk &&\n+>junk/\"$Adiarnfc\" &&\n+case \"$(cd junk && echo *)\" in\n+\t\"$Adiarnfd\")\n+\ttest_nfd=1\n+\t;;\n+\t*)\t;;\n+esac\n+rm -rf junk\n+\n+if test \"$test_nfd\"\n+then\n+\ttest_expect_success \"detect if nfd needed\" '\n+\t\tprecomposedunicode=`git config  core.precomposedunicode` &&\n+\t\ttest \"$precomposedunicode\" = false &&\n+\t\tgit config  core.precomposedunicode true\n+\t'\n+\ttest_expect_success \"setup\" '\n+\t\t>x &&\n+\t\tgit add x &&\n+\t\tgit commit -m \"1st commit\" &&\n+\t\tgit rm x &&\n+\t\tgit commit -m \"rm x\"\n+\t'\n+\ttest_expect_success \"setup case mac\" '\n+\t\tgit checkout -b mac_os\n+\t'\n+\t# This will test nfd2nfc in readdir()\n+\ttest_expect_success \"add file Adiarnfc\" '\n+\t\techo f.Adiarnfc >f.$Adiarnfc &&\n+\t\tgit add f.$Adiarnfc &&\n+\t\tgit commit -m \"add f.$Adiarnfc\"\n+\t'\n+\t# This will test nfd2nfc in git stage()\n+\ttest_expect_success \"stage file d.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d.$Adiarnfd &&\n+\t\techo d.$Adiarnfd/f.$Adiarnfd >d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit stage d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d.$Adiarnfd/f.$Adiarnfd\"\n+\t'\n+\ttest_expect_success \"add link Adiarnfc\" '\n+\t\tln -s d.$Adiarnfd/f.$Adiarnfd l.$Adiarnfc &&\n+\t\tgit add l.$Adiarnfc &&\n+\t\tgit commit -m \"add l.Adiarnfc\"\n+\t'\n+\t# This will test git log\n+\ttest_expect_success \"git log f.Adiar\" '\n+\t\tgit log f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit log f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git ls-files\n+\ttest_expect_success \"git lsfiles f.Adiar\" '\n+\t\tgit ls-files f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit ls-files f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git mv\n+\ttest_expect_success \"git mv\" '\n+\t\tgit mv f.$Adiarnfd f.$Odiarnfc &&\n+\t\tgit mv d.$Adiarnfd d.$Odiarnfc &&\n+\t\tgit mv l.$Adiarnfd l.$Odiarnfc &&\n+\t\tgit commit -m \"mv Adiarnfd Odiarnfc\"\n+\t'\n+\t# Files can be checked out as nfc\n+\t# And the link has been corrected from nfd to nfc\n+\ttest_expect_success \"git checkout nfc\" '\n+\t\trm f.$Odiarnfc &&\n+\t\tgit checkout f.$Odiarnfc\n+\t'\n+\t# Make it possible to checkout files with their NFD names\n+\ttest_expect_success \"git checkout file nfd\" '\n+\t\trm -f f.* &&\n+\t\tgit checkout f.$Odiarnfd\n+\t'\n+\t# Make it possible to checkout links with their NFD names\n+\ttest_expect_success \"git checkout link nfd\" '\n+\t\trm l.* &&\n+\t\tgit checkout l.$Odiarnfd\n+\t'\n+\ttest_expect_success \"setup case mac2\" '\n+\t\tgit checkout master &&\n+\t\tgit reset --hard &&\n+\t\tgit checkout -b mac_os_2\n+\t'\n+\t# This will test nfd2nfc in git commit\n+\ttest_expect_success \"commit file d2.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d2.$Adiarnfd &&\n+\t\techo d2.$Adiarnfd/f.$Adiarnfd >d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit add d2.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d2.$Adiarnfd/f.$Adiarnfd\" -- d2.$Adiarnfd/f.$Adiarnfd\n+\t'\n+else\n+\t say \"Skipping nfc/nfd tests\"\n+fi\n+\n+test_done\n-- \n1.7.8.rc0.43.gb49a8\n"},{"id":"182912","messageId":"1327184934.31804.32.camel@centaur.lab.cmartin.tk","threadId":"29422","inReplyTo":"201201212036.57632.tboegi@web.de","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Carlos Martín Nieto","fromEmail":"cmn@elego.de","sentAt":"2012-01-21T22:28:54Z","receivedAt":"2012-01-21T22:28:54Z","isPatch":true,"sender":{"key":"cmn@elego.de","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"On Sat, 2012-01-21 at 20:36 +0100, Torsten Bögershausen wrote:\n> Allow git on Mac OS to store file names in the index in precomposed unicode,\n> while the file system uses decomposed unicode.\n> \n> The problem:\n> When a file called \"LATIN CAPITAL LETTER A WITH DIAERESIS\"\n> (in utf-8 encoded as 0xc3 0x84) is created, the Mac OS filesystem\n> converts \"precomposed unicode\" into \"decomposed unicode\".\n> This means that readdir() will return 0x41 0xcc 0x88.\n> \n> Git under Mac OS reverts the unicode decomposition of filenames.\n> \n> This is useful when pulling/pushing from repositories containing utf-8\n> encoded filenames using precomposed utf-8 like Linux or Windows (*).\n> \n> It allows sharing git repositories stored on a VFAT file system\n> (e.g. a USB stick), and mounted network share using samba.\n> \n> * (Not all Windows versions support UTF-8 yet:\n>    Msysgit needs the unicode branch, cygwin supports UTF-8 since 1.7)\n\nThis might be overly pedantic, but Windows doesn't really deal with\nUTF-8. To use Unicode you need to use the \"wide\" variant of the\nfunctions, and those take UTF-16.\n\n   cmn\n\n\n"},{"id":"182914","messageId":"7vsjj8acmh.fsf@alter.siamese.dyndns.org","threadId":"29422","inReplyTo":"201201212036.57632.tboegi@web.de","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-01-21T22:56:22Z","receivedAt":"2012-01-21T22:56:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"[Pinging Nguyen who has worked rather extensively on the start-up sequence\nfor ideas.]\n\nTorsten Bögershausen <tboegi@web.de> writes:\n\nI'll try to reword the log message a bit below.\n\n> When a file called \"LATIN CAPITAL LETTER A WITH DIAERESIS\" (in utf-8\n> encoded as 0xc3 0x84) is created, the Mac OS filesystem converts\n> \"precomposed unicode\" into \"decomposed unicode\". readdir() will return\n> 0x41 0xcc 0x88 for such a file, that does not match what the caller\n> thought it created.\n>\n> To work around this braindamage, allow git on Mac OS to optionally use a\n> wrapper for readdir() that converts decomposed unicode back into the\n> precomposed form, which most other platforms use natively. This makes it\n> easier for Mac OS users to work together on the same project with people\n> on other platforms (Note that not all Windows versions support UTF-8\n> yet. Msysgit needs the unicode branch, cygwin supports UTF-8 since\n> 1.7). This allows sharing git repositories stored on a VFAT file system\n> (e.g. a USB stick), and mounted network share using samba.\n>\n> This new feature is controlled by setting a new configuration variable\n> \"core.precomposedunicode\" to \"true\". Unless the variable is set to \"true\",\n> Git on Mac OS behaves exactly as before, for backward compatiblity.\n>\n> The code in compat/precomposed_utf8.c implements basically 4 new\n> functions: precomposed_utf8_opendir(), precomposed_utf8_readdir(),\n> precomposed_utf8_closedir() precompose_argv()\n>\n> In order to prevent that ever a file name in decomposed unicode is\n> entering the index, a \"brute force\" attempt is taken: all arguments into\n> git (argv[1]..argv[n]) are converted into precomposed unicode.  This is\n> done in git.c by calling precompose_argv().  This function is actually a\n> #define, and it is only defined under Mac OS.  Nothing is converted on\n> any other platforms.\n\nIt may be just me, but the above looks more in line with the usual style\nof writing in our existing log messages.\n\nIs this UTF-8 decomposition only an issue on HFS+, or does it happen on\nany filesystem mounted on a MacOS box? If the former, then the second line\nof the first paragraph needs further rephrasing, e.g. \"... is created,\nHFS+, the primary filesystem on the Mac OS, converts ...\".\n\n> Auto sensing:\n> When creating a new git repository with \"git init\" or \"git clone\",\n> \"core.precomposedunicode\" will be set \"false\".\n>\n> The user needs to activate this feature manually.\n> She typically sets core.precomposedunicode to \"true\" on HFS and VFAT,\n> or file systems mounted via SAMBA onto a Linux box.\n\nI am not sure about this design decision.\n\nI agree that it is prudent to introduce a new feature disabled by default,\nand I can understand that you tried to make the feature more discoverable\nby setting it explicitly to \"false\".\n\nBut I do not think it is a good idea. If a user is on MacOS and has only\nHFS+, then it would be more convenient to have the configuration set to\ntrue in $HOME/.gitconfig once and for all, to affect all repositories on\nthe box. \"git init\" dropping the explicit \"false\" to any new repositories\ndefeats that.\n\nWouldn't it make more sense if your \"git init\" did it this way?\n\n    * Do not do anything, if you know core.precomposedunicode is already\n      set (in /etc/gitconfig or $HOME/.gitconfig);\n\n    * Otherwise, if the \"probe\" says \"yes, we are on HFS+\", issue an\n      advice message to suggest the user to set it either in the\n      repository specific .git/config or in $HOME/.gitconfig file.\n\n> +core.precomposedunicode::\n> +\tThis option is only used by Mac OS implementation of git.\n> +\tWhen core.precomposedunicode=true,\n> +\tgit reverts the unicode decomposition of filenames done by Mac OS.\n> +\tThis is useful when pulling/pushing from repositories containing utf-8\n> +\tencoded filenames using precomposed unicode (like Linux).\n\nI would imagine that if the caller of creat(2) named the path in the\ndecomposed form, Mac OS would store it unaltered; strictly speaking, we\nshouldn't say \"reverts\". How about:\n\n    When set to true, pathnames in decomposed UTF-8 read from the\n    filesystem are converted to precomposed UTF-8 before they are used by\n    Git, to improve interoperability with other platforms.\n\n> +void precompose_argv(int argc, const char **argv)\n> +{\n> +\tint i = 0;\n> +\tconst char *oldarg;\n> +\tchar *newarg;\n> +\ticonv_t ic_precompose;\n> +\n> +\tgit_config(precomposed_unicode_config, NULL);\n\nAs the first thing called after main(), I still doubt this is a safe thing\nto do (Pinging Nguyen who has worked rather extensively on the start-up\nsequence for ideas). This is ifdefed away and will not break things on\nother platforms, which may make it even harder to diagnose breakages.\n"},{"id":"182920","messageId":"CACsJy8BKQHLdoXfSKsULkWWbWjWEuZgr=bVNKmgCSArvwbf2UA@mail.gmail.com","threadId":"29422","inReplyTo":"7vsjj8acmh.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-01-22T09:58:30Z","receivedAt":"2012-01-22T09:58:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jan 22, 2012 at 5:56 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> [Pinging Nguyen who has worked rather extensively on the start-up sequence\n> for ideas.]\n>\n> Torsten Bögershausen <tboegi@web.de> writes:\n>\n> I'll try to reword the log message a bit below.\n>\n>> When a file called \"LATIN CAPITAL LETTER A WITH DIAERESIS\" (in utf-8\n>> encoded as 0xc3 0x84) is created, the Mac OS filesystem converts\n>> \"precomposed unicode\" into \"decomposed unicode\". readdir() will return\n>> 0x41 0xcc 0x88 for such a file, that does not match what the caller\n>> thought it created.\n>>\n>> To work around this braindamage, allow git on Mac OS to optionally use a\n>> wrapper for readdir() that converts decomposed unicode back into the\n>> precomposed form, which most other platforms use natively. This makes it\n>> easier for Mac OS users to work together on the same project with people\n>> on other platforms (Note that not all Windows versions support UTF-8\n>> yet. Msysgit needs the unicode branch, cygwin supports UTF-8 since\n>> 1.7). This allows sharing git repositories stored on a VFAT file system\n>> (e.g. a USB stick), and mounted network share using samba.\n\nI just have a quick look, you reencode opendir, readdir, and\nclosedir() to precomposed form. But files are still in decomposed\nform, does open(<precomposed file>) work when only <decomposed file>\nexists?\n\n>> In order to prevent that ever a file name in decomposed unicode is\n>> entering the index, a \"brute force\" attempt is taken: all arguments into\n>> git (argv[1]..argv[n]) are converted into precomposed unicode.  This is\n>> done in git.c by calling precompose_argv().  This function is actually a\n>> #define, and it is only defined under Mac OS.  Nothing is converted on\n>> any other platforms.\n\nThis is not entirely safe. Filenames can be taken from a file for\nexample (--stdin option or similar). Unless I'm mistaken, all file\nnames must enter git through the index, the conversion at read-cache.c\nmay be a better option.\n\n>> Auto sensing:\n>> When creating a new git repository with \"git init\" or \"git clone\",\n>> \"core.precomposedunicode\" will be set \"false\".\n\nThis is a general comment on init auto detection feature. Perhaps we\nshould allow detection to be done when reinitializing a repo. Or at\nleast make an option to auto detect, print out new config values and\nuser can decide if they want to change current values themselves.\n\n>> +void precompose_argv(int argc, const char **argv)\n>> +{\n>> +     int i = 0;\n>> +     const char *oldarg;\n>> +     char *newarg;\n>> +     iconv_t ic_precompose;\n>> +\n>> +     git_config(precomposed_unicode_config, NULL);\n>\n> As the first thing called after main(), I still doubt this is a safe thing\n> to do (Pinging Nguyen who has worked rather extensively on the start-up\n> sequence for ideas). This is ifdefed away and will not break things on\n> other platforms, which may make it even harder to diagnose breakages.\n\nThis can't be worse than current state of pager and alias settings,\nwhich need to be detected even before setup_git_directory* is run.\n\nI'd rather encode at index level and read_directory() than at argv[].\nBut if reencoding argv is the only feasible way, perhaps put the\nconversion in parse_options()? Config reading is usually done by then,\nand we can move precomposed config reading to git_default_config. If\nsome commands do not use parse_options yet, this is a good opportunity\nto do so (I'll send pack-objects's parse_options() patch soon).\n-- \nDuy\n"},{"id":"182921","messageId":"CACsJy8AucS9ez=-zej=72dr+0AVvGFR_eZgQcabitXgmQTJOCA@mail.gmail.com","threadId":"29422","inReplyTo":"CACsJy8BKQHLdoXfSKsULkWWbWjWEuZgr=bVNKmgCSArvwbf2UA@mail.gmail.com","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-01-22T10:03:30Z","receivedAt":"2012-01-22T10:03:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2012/1/22 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:\n>>> In order to prevent that ever a file name in decomposed unicode is\n>>> entering the index, a \"brute force\" attempt is taken: all arguments into\n>>> git (argv[1]..argv[n]) are converted into precomposed unicode.\n\nForgot one more thing. We have case-insensitive support in place\nalready, we can hook precomposed form conversion there before\ncomparing. In other words we just need to support\n{pre,de}composed-insensitive string compare.\n-- \nDuy\n"},{"id":"183250","messageId":"4F251FA1.80400@web.de","threadId":"29422","inReplyTo":"CACsJy8BKQHLdoXfSKsULkWWbWjWEuZgr=bVNKmgCSArvwbf2UA@mail.gmail.com","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-01-29T10:29:53Z","receivedAt":"2012-01-29T10:29:53Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 22.01.12 10:58, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Jan 22, 2012 at 5:56 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>> [Pinging Nguyen who has worked rather extensively on the start-up sequence\n>> for ideas.]\n>>\n[snip]\n> \n> I just have a quick look, you reencode opendir, readdir, and\n> closedir() to precomposed form. But files are still in decomposed\n> form, does open(<precomposed file>) work when only <decomposed file>\n> exists?\n\nYes. All function like stat(), lstat(), open(), fopen(), unlink() behave the same\nfor precomped or decomposed. This is similar to the ignore case feature.\nAnd because the default HFS+ is case preserving, case insenstive and unicode decomposing\nall at the same time, a file name \"Ä\" could be reached under 4 different names.\nPlease see the output of the test script:\n(which is at the end of this email)\n\ntests/Darwin_i386/NFC file name created as nfc is readable as nfd\ntests/Darwin_i386/NFC readdir returns nfd but expected is nfc\ntests/Darwin_i386/NFD file name created as nfd is readable as nfc\ntests/Darwin_i386/NFCNFD 1 file found in directory, but there should be 2\ntests/Darwin_i386/NFCNFD nfc is missing, nfd is present\ntests/Darwin_i386/NFCNFD nfc File content overwritten by nfd\ntests/Darwin_i386/NFDNFC 1 file found in directory, but there should be 2\ntests/Darwin_i386/NFDNFC nfc is missing, nfd is present\ntests/Darwin_i386/NFDNFC nfd File content overwritten by nfc\n\n\n> \n>>> In order to prevent that ever a file name in decomposed unicode is\n>>> entering the index, a \"brute force\" attempt is taken: all arguments into\n>>> git (argv[1]..argv[n]) are converted into precomposed unicode.  This is\n>>> done in git.c by calling precompose_argv().  This function is actually a\n>>> #define, and it is only defined under Mac OS.  Nothing is converted on\n>>> any other platforms.\n> \n> This is not entirely safe. Filenames can be taken from a file for\n> example (--stdin option or similar). Unless I'm mistaken, all file\n> names must enter git through the index, the conversion at read-cache.c\n> may be a better option.\nGood point, thanks. \nI added some code to read-cache.c, and it works for files, but not for directories.\nI looked through the code for \"case-ignoring\" directory names, and couldn't\nfind something obvious. More work is to be done.\n \n\n[snip]\n> I'd rather encode at index level and read_directory() than at argv[].\n>But if reencoding argv is the only feasible way, perhaps put the\n>conversion in parse_options()?\n\nI tried that, and found that git-lsfiles.c doesn't use parse_options.\n\n[snip]\n\nOn the long run I want to get rid of the argv[] conversion completely,\nbut I'm not there yet.\n\nThanks for all comments and inspiration!\n\n(and apologies for my long response times I use to have)\n/Torsten\n\n\n\nPS:\nHere the script.\nMac OS writes decomposd unicode to HFS+, precomposed unicode to VFAT and SAMBA.\nIn any case readdir() returns decomposed.\n\n=================\n#!/bin/sh\nerrorandout() {\n  echo Error: The shell can not handle nfd\n  echo try to run /bin/bash $0\n  rm -rf $DIR\n  exit 1\n}\n\ncheckDirNfcOrNfd() {\n  DDNFCNFD=$1\n  readdirexp=$2\n  if test -r $DDNFCNFD/$aumlnfc; then\n    x=`cat $DDNFCNFD/$aumlnfc`\n    if test \"$x\" = nfd; then\n      echo $DDNFCNFD file name created as nfd is readable as nfc\n    fi\n  fi\n  if test -r $DDNFCNFD/$aumlnfd; then\n    x=`cat $DDNFCNFD/$aumlnfd 2>/dev/null` || {\n      echo $DDNFCNFD nfd is not readable, but readdir says that it exist\n    }\n    if test \"$x\" = nfc; then\n      echo $DDNFCNFD file name created as nfc is readable as nfd\n    fi\n  fi\n  readdirres=`echo $DDNFCNFD/*`\n  if test \"$readdirres\" != \"$DDNFCNFD/$readdirexp\"; then\n    if test \"$readdirres\" = $DDNFCNFD/$aumlnfd; then\n      echo $DDNFCNFD readdir returns nfd but expected is nfc\n    fi\n    if test \"$readdirres\" = $DDNFCNFD/$aumlnfc; then\n      echo $DDNFCNFD readdir returns nfc but expected is nfd\n    fi\n  fi\n}\n\ncheckdirnfcnfd() {\n  DDNFCNFD=$1\n  if test `ls -1 $DDNFCNFD | wc -l` != 2; then\n    if test `ls -1 $DDNFCNFD | wc -l` == 1; then\n      echo $DDNFCNFD 1 file found in directory, but there should be 2\n    else\n      echo $DDNFCNFD 2 files should be in directory\n    fi  \n  fi\n\n  x=`echo $DDNFCNFD/*`\n  a=`echo $DDNFCNFD/$aumlnfd $DDNFCNFD/$aumlnfc`\n  b=`echo $DDNFCNFD/$aumlnfc $DDNFCNFD/$aumlnfd`\n  c=`echo $DDNFCNFD/$aumlnfc $DDNFCNFD/$aumlnfc`\n  d=`echo $DDNFCNFD/$aumlnfd $DDNFCNFD/$aumlnfd`\n  e=`echo $DDNFCNFD/$aumlnfc`\n  f=`echo $DDNFCNFD/$aumlnfd`\n  case \"$x\" in\n    $a)\n    ;;      \n    $b)\n    ;;\n    $c)\n    echo $DDNFCNFD nfd is hidden, nfc is listed twice\n    ;;\n    $d)\n    echo $DDNFCNFD nfc is hidden, nfd is listed twice\n    ;;\n    $e)\n    echo $DDNFCNFD nfd is missing, nfc is present\n    ;;      \n    $f)\n    echo $DDNFCNFD nfc is missing, nfd is present\n    ;;      \n    *)\n    echo $DDNFCNFD x`echo $x | xxd`\n    ;;\n  esac\n\n  if ! test -r $DDNFCNFD/$aumlnfc; then\n    echo $DDNFCNFD/nfc File does not exist\n  else\n    x=`cat $DDNFCNFD/$aumlnfc`\n    if test \"$x\" != nfc; then\n      echo $DDNFCNFD nfc File content overwritten by $x\n    fi\n  fi\n  \n  if ! test -r $DDNFCNFD/$aumlnfd; then\n    echo $DDNFCNFD/nfd File does not exist\n  else\n    x=`cat $DDNFCNFD/$aumlnfd`\n    if test \"$x\" != nfd; then\n      echo $DDNFCNFD nfd File content overwritten by $x\n    fi\n  fi\n}\n\n\naumlnfc=$(printf '\\303\\204')\naumlnfd=$(printf '\\101\\314\\210')\n\nDIR=tests/`uname -s`_`uname -m`\necho \"DIR=$DIR\"\n\nrm -rf $DIR/NFC &&\nrm -rf $DIR/NFD &&\nrm -rf $DIR/NFCNFD &&\nrm -rf $DIR/NFDNFC &&\nmkdir -p $DIR/NFC &&\nmkdir -p $DIR/NFD &&\nmkdir -p $DIR/NFDNFC &&\nmkdir -p $DIR/NFCNFD &&\necho nfc > $DIR/NFC/$aumlnfc &&\necho nfd > $DIR/NFD/$aumlnfd &&\necho nfd > $DIR/NFDNFC/$aumlnfd &&\necho nfc > $DIR/NFDNFC/$aumlnfc &&\necho nfc > $DIR/NFCNFD/$aumlnfc &&\necho nfd > $DIR/NFCNFD/$aumlnfd && {\n    # test 1: basic if the shell handles nfd\n    if ! test -r $DIR/NFD/$aumlnfd; then\n      errorandout\n    fi\n\n  for DD in tests/*; do\n    checkDirNfcOrNfd $DD/NFC  $aumlnfc\n    checkDirNfcOrNfd $DD/NFD  $aumlnfd\n\n    checkdirnfcnfd $DD/NFCNFD\n    checkdirnfcnfd $DD/NFDNFC\n  done\n} || errorandout\n"},{"id":"183253","messageId":"4F25423F.7080006@web.de","threadId":"29422","inReplyTo":"4F251FA1.80400@web.de","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-01-29T12:57:35Z","receivedAt":"2012-01-29T12:57:35Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n> I tried that, and found that git-lsfiles.c doesn't use parse_options.\nOops, I shouldn't have written that: git-lsfiles uses parse_options.\n\nSorry for the noise.\n/torsten\n"},{"id":"183257","messageId":"CABPQNSYwuo0vXUj2ZWPHihZFpvwao+G8nWTEiBFFUm2xWdd5nQ@mail.gmail.com","threadId":"29422","inReplyTo":"1327184934.31804.32.camel@centaur.lab.cmartin.tk","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2012-01-29T16:26:52Z","receivedAt":"2012-01-29T16:26:52Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Sat, Jan 21, 2012 at 11:28 PM, Carlos Martín Nieto <cmn@elego.de> wrote:\n> On Sat, 2012-01-21 at 20:36 +0100, Torsten Bögershausen wrote:\n>> * (Not all Windows versions support UTF-8 yet:\n>>    Msysgit needs the unicode branch, cygwin supports UTF-8 since 1.7)\n>\n> This might be overly pedantic, but Windows doesn't really deal with\n> UTF-8. To use Unicode you need to use the \"wide\" variant of the\n> functions, and those take UTF-16.\n\nThis is exactly what the 'unicode'-branch in msysGit does, so the\ncomment is not incorrect at all.\n"},{"id":"194165","messageId":"4FE73675.2000901@web.de","threadId":"29422","inReplyTo":"CACsJy8AucS9ez=-zej=72dr+0AVvGFR_eZgQcabitXgmQTJOCA@mail.gmail.com","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-06-24T15:47:01Z","receivedAt":"2012-06-24T15:47:01Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 22.01.12 11:03, Nguyen Thai Ngoc Duy wrote:\n> 2012/1/22 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:\n>>>> In order to prevent that ever a file name in decomposed unicode is\n>>>> entering the index, a \"brute force\" attempt is taken: all arguments into\n>>>> git (argv[1]..argv[n]) are converted into precomposed unicode.\n> \n> Forgot one more thing. We have case-insensitive support in place\n> already, we can hook precomposed form conversion there before\n> comparing. In other words we just need to support\n> {pre,de}composed-insensitive string compare.\n> \n>Forgot one more thing. We have case-insensitive support in place\n>already, we can hook precomposed form conversion there before\n>comparing. In other words we just need to support\n>{pre,de}composed-insensitive string compare.\n>-- Duy \n\nYes, I like that idea.\nAfter doing some experiments with precomposed and decomposed file names,\nthe motvation for the fix, or so to say the root cause, changed.\n\nThe Mac OS X file system on VFAT has a kind of schizophrenia:\n- unicode file names are written as precomposed onto the disk\n  and Linux/Windows see them as precomposed.\n- readdir() returns always decomposed.\n- open()/fopen() stat() lstat() works for both pre- and decomposed\n\nTherefore a repository on VFAT under Mac OS X looks as follows:\n\n- file names on disk are precomposed\n- file names in the index are decomposed\n\nAs long as only Mac OS uses that device, there is no problem.\n\nWhen we now move the e.g. USB stick using VFAT to Linux\nthe \"decomposed\" in the index seem to be deleted and the\nprecomposed on disk are untracked.\nA complete desaster.\n\nTo keep the file names in the index and how they are stored\non disk the same, we can can set \"core.precomposedunicode\" = true.\n(Side note: should we rename it to \"i18n.precomposedunicode\" ?)\n\nNow we keep the index and file names on disk the same, and can move the\nUSB stick between Linux, Mac OS X, or Windows (since msysGit-1.7.10-\nnow called Git for Windows, or git under cygwin 1.7)\n\nThe same problem occurs when Mac OS mounts a network share from linux\nusing SAMBA:\nreaddir() returns decomposed, creat() stores file names in precomposed\non the remote linux machine.\n\nIf we put a file name with decomposed unicode on the linux machine,\nit will be listed as decomposed by readdir() on the Mac OS side.\nTrying to access this file failes, because Mac OS tries to open\nthe precomposed version, and that does not exist.\n\n\nAnother thing:\nThere are some reasons to avoid decomposed unicode in Linux and Windows:\nMany user space programs don't handle decomposed unicode very well.\nWhen e.g. an \"û\" should be displayed, the output looks like \"u^\" in many\nprograms.\nAnd if we need more motivations: decomposed unicode is hard to enter on\nthe keyboard.\n\n\n\nThen I went back to my original problem \n(versioning the \"Documents\" folder on my Linux $HOME under git\nand access it from Windows and Mac OS using SAMBA, or cloning it\nto a laptop...)\n\nKnowing that\na) Mac OS X handles precomposed and decomposed the same in open()...\nb) The user space program on Mac OS handle precomposed just fine\nc) Many user space programs don't presentate decomposed as it should be\nd) It is hard to enter decomposed unicode at the keyboard\ne) and therefore decomposed unicode is seldom used on Linux\nf) Mac OS using SAMBA puts file names in precomposed unicode on the \n   remote side\n\nDo we have a motivation for pushing a solution that ignores \nthe unicode composition ?\n\nI'll send a V5 version with hopefully a better motivation\n"},{"id":"195784","messageId":"50105AEB.9050905@dewire.com","threadId":"29422","inReplyTo":"4FE73675.2000901@web.de","subject":"Re: [PATCH V4] git on Mac OS and precomposed unicode","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2012-07-25T20:45:31Z","receivedAt":"2012-07-25T20:45:31Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Torsten Bögershausen skrev 2012-06-24 17.47:\n\n> Do we have a motivation for pushing a solution that ignores\n> the unicode composition ?\n\nI say we do. I tried your patch and it worked fine.\n\n> I'll send a V5 version with hopefully a better motivation\n\n-- robin\n"}]}