{"thread":{"id":"29312","subject":"[PATCH][RFC] git on Mac OS and precomposed unicode","startedAt":"2012-01-07T19:59:22Z","lastAt":"2012-01-09T16:42:00Z","messageCount":3,"participants":["Torsten Bögershausen","Miles Bader"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"182078","messageId":"201201072059.23074.tboegi@web.de","threadId":"29312","inReplyTo":null,"subject":"[PATCH][RFC] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-01-07T19:59:22Z","receivedAt":"2012-01-07T19:59:22Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Allow git on Mac OS to store file names in the index in precomposed unicode,\nwhile the file system used decomposed unicode.\n\nWhen a file called \"LATIN CAPITAL LETTER A WITH DIAERESIS\"\n(in utf-8 encoded as 0xc3 0x84) is created,\nthe filesystem converts \"precomposed unicode\" into \"decomposed unicode\",\nwhich means that readdir() will return 0x41 0xcc 0x88.\nWhen true, git reverts the unicode decomposition of filenames.\nThis is useful when pulling/pushing from repositories containing utf-8\nencoded filenames using precomposed utf-8 (like Linux).\n\nThis feature is automatically switched on when \"git init\" is run,\nand the file system is doing UTF-8 decompostion.\n(Which has been observed on HFS+, SMBFS and VFAT, but not on NFS)\nIt can be switched off by setting core.macosforcenfc=false\n\nIt is implemented by re-defining the readdir() functions.\nFile names are converted into precomposed UTF-8.\n\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\n Documentation/config.txt     |    9 ++\n Makefile                     |    3 +\n builtin/init-db.c            |   22 +++++\n compat/darwin.c              |  200 ++++++++++++++++++++++++++++++++++++++++++\n compat/darwin.h              |   31 +++++++\n git-compat-util.h            |    8 ++\n git.c                        |    1 +\n t/t0050-filesystem.sh        |    1 +\n t/t3910-mac-os-precompose.sh |  104 ++++++++++++++++++++++\n 9 files changed, 379 insertions(+), 0 deletions(-)\n create mode 100644 compat/darwin.c\n create mode 100644 compat/darwin.h\n create mode 100755 t/t3910-mac-os-precompose.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2959390..01b9465 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -175,6 +175,15 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignorecase true if appropriate when the repository\n is created.\n \n+core.precomposedunicode::\n+\tThis option is only used by Mac OS implementation of git.\n+\tWhen core.precomposedunicode=true,\n+\tgit reverts the unicode decomposition of filenames done by Mac OS.\n+\tThis is useful when pulling/pushing from repositories containing utf-8\n+\tencoded filenames using precomposed unicode (like Linux).\n+\tWhen false, file names are handled fully transparent by git.\n+\tIf in doubt, set core.precomposedunicode=false.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Makefile b/Makefile\nindex b21d2f1..596900e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -519,6 +519,7 @@ LIB_H += compat/bswap.h\n LIB_H += compat/cygwin.h\n LIB_H += compat/mingw.h\n LIB_H += compat/obstack.h\n+LIB_H += compat/darwin.h\n LIB_H += compat/win32/pthread.h\n LIB_H += compat/win32/syslog.h\n LIB_H += compat/win32/poll.h\n@@ -884,6 +885,8 @@ ifeq ($(uname_S),Darwin)\n \tendif\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\n+\tCOMPAT_OBJS += compat/darwin.o\n+\tBASIC_CFLAGS += -DPRECOMPOSED_UNICODE\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 0dacb8b..88c9de1 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -290,6 +290,28 @@ static int create_default_files(const char *template_path)\n \t\tstrcpy(path + len, \"CoNfIg\");\n \t\tif (!access(path, F_OK))\n \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n+#if defined (PRECOMPOSED_UNICODE)\n+\t\t{\n+\t\t\tconst static char *auml_nfc = \"\\xc3\\xa4\";\n+\t\t\tconst static char *auml_nfd = \"\\x61\\xcc\\x88\";\n+\t\t\tint output_fd;\n+\t\t\tpath[len] = 0;\n+\t\t\tstrcpy(path + len, auml_nfc);\n+\t\t\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n+\t\t\tif (output_fd >=0) {\n+\t\t\t\tclose(output_fd);\n+\t\t\t\tpath[len] = 0;\n+\t\t\t\tstrcpy(path + len, auml_nfd);\n+\t\t\t\tif (0 == access(path, R_OK))\n+\t\t\t\t\tgit_config_set(\"core.precomposedunicode\", \"true\");\n+\t\t\t\telse\n+\t\t\t\t\tgit_config_set(\"core.precomposedunicode\", \"false\");\n+\t\t\t\tpath[len] = 0;\n+\t\t\t\tstrcpy(path + len, auml_nfc);\n+\t\t\t\tunlink(path);\n+\t\t\t}\n+\t\t}\n+#endif\n \t}\n \n \treturn reinit;\ndiff --git a/compat/darwin.c b/compat/darwin.c\nnew file mode 100644\nindex 0000000..15de7c2\n--- /dev/null\n+++ b/compat/darwin.c\n@@ -0,0 +1,200 @@\n+#define __DARWIN_C__\n+\n+#include <stdlib.h>\n+#include <string.h>\n+#include <stdio.h>\n+#include <stdint.h>\n+\n+#include \"../cache.h\"\n+#include \"../utf8.h\"\n+\n+#include \"darwin.h\"\n+\n+static int mac_os_precomposed_unicode;\n+const static char *repo_encoding = \"UTF-8\";\n+const static char *path_encoding = \"UTF-8-MAC\";\n+\n+\n+/* Code borrowed from utf8.c */\n+#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))\n+\ttypedef const char * iconv_ibp;\n+#else\n+\ttypedef char * iconv_ibp;\n+#endif\n+static char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)\n+{\n+\tsize_t outsz, outalloc;\n+\tchar *out, *outpos;\n+\ticonv_ibp cp;\n+\n+\toutsz = insz;\n+\toutalloc = outsz + 1; /* for terminating NUL */\n+\tout = xmalloc(outalloc);\n+\toutpos = out;\n+\tcp = (iconv_ibp)in;\n+\n+\twhile (1) {\n+\t\tsize_t cnt = iconv(conv, &cp, &insz, &outpos, &outsz);\n+\n+\t\tif (cnt == -1) {\n+\t\t\tsize_t sofar;\n+\t\t\tif (errno != E2BIG) {\n+\t\t\t\tfree(out);\n+\t\t\t\ticonv_close(conv);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t\t/* insz has remaining number of bytes.\n+\t\t\t * since we started outsz the same as insz,\n+\t\t\t * it is likely that insz is not enough for\n+\t\t\t * converting the rest.\n+\t\t\t */\n+\t\t\tsofar = outpos - out;\n+\t\t\toutalloc = sofar + insz * 2 + 32;\n+\t\t\tout = xrealloc(out, outalloc);\n+\t\t\toutpos = out + sofar;\n+\t\t\toutsz = outalloc - sofar - 1;\n+\t\t}\n+\t\telse {\n+\t\t\t*outpos = '\\0';\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn out;\n+}\n+\n+static size_t\n+has_utf8(const char *s, size_t maxlen, size_t *strlen_c)\n+{\n+\tconst uint8_t *utf8p = (const uint8_t*) s;\n+\tsize_t strlen_chars = 0;\n+\tsize_t ret = 0;\n+\n+\tif ((!utf8p) || (!*utf8p))\n+\t\treturn 0;\n+\n+\twhile((*utf8p) && maxlen) {\n+\t\tif (*utf8p & 0x80)\n+\t\t\tret++;\n+\t\tstrlen_chars++;\n+\t\tutf8p++;\n+\t\tmaxlen--;\n+\t}\n+\tif (strlen_c)\n+\t\t*strlen_c = strlen_chars;\n+\n+\treturn ret;\n+}\n+\n+static int\n+precomposed_unicode_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcasecmp(var, \"core.precomposedunicode\")) {\n+\t\tmac_os_precomposed_unicode = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+void\n+argv_precompose(int argc, const char **argv)\n+{\n+\tint i = 0;\n+\tconst char *oldarg;\n+\tchar *newarg;\n+\ticonv_t ic_precompose;\n+\n+\tif (!strcmp(\"commit\", argv[0]))\n+\t\treturn;\n+\n+\tgit_config(precomposed_unicode_config, NULL);\n+\tif (!mac_os_precomposed_unicode)\n+\t\treturn;\n+\n+\tic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (ic_precompose == (iconv_t) -1)\n+\t\treturn;\n+\n+\twhile (i < argc) {\n+\t\tsize_t namelen;\n+\t\toldarg = argv[i];\n+\n+\t\tif (has_utf8(oldarg, (size_t)-1, &namelen)) {\n+\t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_precompose);\n+\t\t\tif (newarg)\n+\t\t\t\targv[i] = newarg;\n+\t\t}\n+\t\ti++;\n+\t}\n+\ticonv_close(ic_precompose);\n+}\n+\n+\n+DARWIN_DIR *\n+darwin_opendir(const char *dirname)\n+{\n+\tDARWIN_DIR *darwin_dir;\n+\tdarwin_dir = malloc(sizeof(DARWIN_DIR));\n+\tif (!darwin_dir)\n+\t\treturn NULL;\n+\n+\tdarwin_dir->dirp = opendir(dirname);\n+\tif (!darwin_dir->dirp) {\n+\t\tfree(darwin_dir);\n+\t\treturn NULL;\n+\t}\n+\tdarwin_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);\n+\tif (darwin_dir->ic_precompose == (iconv_t) -1) {\n+\t\tclosedir(darwin_dir->dirp);\n+\t\tfree(darwin_dir);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn darwin_dir;\n+}\n+\n+struct dirent *\n+darwin_readdir(DARWIN_DIR *darwin_dirp)\n+{\n+\tstruct dirent *res;\n+\tsize_t namelen = 0;\n+\n+\tres = readdir(darwin_dirp->dirp);\n+\tif (!res || !mac_os_precomposed_unicode || !has_utf8(res->d_name, (size_t)-1, &namelen))\n+\t\treturn res;\n+\telse {\n+\t\tint olderrno = errno;\n+\t\tsize_t outsz = sizeof(darwin_dirp->dirent_nfc.d_name) - 1; /* one for \\0 */\n+\t\tchar *outpos = darwin_dirp->dirent_nfc.d_name;\n+\t\ticonv_ibp cp;\n+\t\tsize_t cnt;\n+\t\tsize_t insz = namelen;\n+\t\tcp = (iconv_ibp)res->d_name;\n+\n+\t\t/* Copy all data except the name */\n+\t\tmemcpy(&darwin_dirp->dirent_nfc,\n+\t\t\t\t\t res,\n+\t\t\t\t\t sizeof(darwin_dirp->dirent_nfc)-sizeof(darwin_dirp->dirent_nfc.d_name));\n+\t\terrno = 0;\n+\n+\t\tcnt = iconv(darwin_dirp->ic_precompose, &cp, &insz, &outpos, &outsz);\n+\t\tif (cnt < sizeof(darwin_dirp->dirent_nfc.d_name) -1) {\n+\t\t\t*outpos = 0;\n+\t\t\terrno = olderrno;\n+\t\t\treturn &darwin_dirp->dirent_nfc;\n+\t\t}\n+\t\terrno = olderrno;\n+\t\treturn res;\n+\t}\n+}\n+\n+\n+int\n+darwin_closedir(DARWIN_DIR *darwin_dirp)\n+{\n+\tint ret_value;\n+\tret_value = closedir(darwin_dirp->dirp);\n+\tif (darwin_dirp->ic_precompose != (iconv_t)-1)\n+\t\ticonv_close(darwin_dirp->ic_precompose);\n+\tfree(darwin_dirp);\n+\treturn ret_value;\n+}\ndiff --git a/compat/darwin.h b/compat/darwin.h\nnew file mode 100644\nindex 0000000..094f930\n--- /dev/null\n+++ b/compat/darwin.h\n@@ -0,0 +1,31 @@\n+#ifndef __DARWIN_H__\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <dirent.h>\n+#include <iconv.h>\n+\n+\n+typedef struct {\n+\ticonv_t ic_precompose;\n+\tDIR *dirp;\n+\tstruct dirent dirent_nfc;\n+} DARWIN_DIR;\n+\n+char *str_precompose(const char *in, iconv_t ic_precompose);\n+\n+void argv_precompose(int argc, const char **argv);\n+\n+DARWIN_DIR *darwin_opendir(const char *dirname);\n+struct dirent *darwin_readdir(DARWIN_DIR *dirp);\n+int darwin_closedir(DARWIN_DIR *dirp);\n+\n+#ifndef __DARWIN_C__\n+#define opendir(n) darwin_opendir(n)\n+#define readdir(d) darwin_readdir(d)\n+#define closedir(d) darwin_closedir(d)\n+#define DIR DARWIN_DIR\n+\n+#endif  /* __DARWIN_C__ */\n+\n+#define  __DARWIN_H__\n+#endif /* __DARWIN_H__ */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 230e198..859dfcf 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -90,6 +90,14 @@\n #include <windows.h>\n #endif\n \n+#if defined (PRECOMPOSED_UNICODE)\n+#include \"compat/darwin.h\"\n+#else\n+#define str_precompose(in,i_nfd2nfc) (NULL)\n+#define argv_precompose(c,v)\n+\n+#endif\n+\n #include <unistd.h>\n #include <stdio.h>\n #include <sys/stat.h>\ndiff --git a/git.c b/git.c\nindex 8e34903..6b2ffb7 100644\n--- a/git.c\n+++ b/git.c\n@@ -298,6 +298,7 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \t\t    startup_info->have_repository) /* get_git_dir() may set up repo, avoid that */\n \t\t\ttrace_repo_setup(prefix);\n \t}\n+\targv_precompose(argc, argv);\n \tcommit_pager_choice();\n \n \tif (!help && p->option & NEED_WORK_TREE)\ndiff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh\nindex 1542cf6..befe39e 100755\n--- a/t/t0050-filesystem.sh\n+++ b/t/t0050-filesystem.sh\n@@ -126,6 +126,7 @@ test_expect_success \"setup unicode normalization tests\" '\n \n   test_create_repo unicode &&\n   cd unicode &&\n+  git config core.precomposedunicode false &&\n   touch \"$aumlcdiar\" &&\n   git add \"$aumlcdiar\" &&\n   git commit -m initial &&\ndiff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh\nnew file mode 100755\nindex 0000000..d4763c5\n--- /dev/null\n+++ b/t/t3910-mac-os-precompose.sh\n@@ -0,0 +1,104 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2012 Torsten Bögershausen\n+#\n+\n+test_description='utf-8 decomposed (nfd) converted to precomposed (nfc)'\n+\n+. ./test-lib.sh\n+\n+Adiarnfc=`printf '\\303\\204'`\n+Odiarnfc=`printf '\\303\\226'`\n+Adiarnfd=`printf 'A\\314\\210'`\n+Odiarnfd=`printf 'O\\314\\210'`\n+\n+mkdir junk &&\n+>junk/\"$Adiarnfc\" &&\n+case \"$(cd junk && echo *)\" in\n+\t\"$Adiarnfd\")\n+\ttest_nfd=1\n+\t;;\n+\t*)\t;;\n+esac\n+rm -rf junk\n+\n+if test \"$test_nfd\"\n+then\n+\ttest_expect_success \"detect if nfd needed\" '\n+\t\tprecomposedunicode=`git config --bool core.precomposedunicode` &&\n+\t\ttest \"$precomposedunicode\" = true\n+\t'\n+\ttest_expect_success \"setup\" '\n+\t\t>x &&\n+\t\tgit add x &&\n+\t\tgit commit -m \"1st commit\" &&\n+\t\tgit rm x &&\n+\t\tgit commit -m \"rm x\"\n+\t'\n+\ttest_expect_success \"setup case mac\" '\n+\tgit checkout -b mac_os\n+\t'\n+\t# This will test nfd2nfc in readdir()\n+\ttest_expect_success \"add file Adiarnfc\" '\n+\t\techo f.Adiarnfc >f.$Adiarnfc &&\n+\t\tgit add f.$Adiarnfc &&\n+\t\tgit commit -m \"add f.$Adiarnfc\"\n+\t'\n+\t# This will test nfd2nfc in git add()\n+\ttest_expect_success \"stage file d.Adiarnfd/f.Adiarnfd\" '\n+\t\tmkdir d.$Adiarnfd &&\n+\t\techo d.$Adiarnfd/f.$Adiarnfd >d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit stage d.$Adiarnfd/f.$Adiarnfd &&\n+\t\tgit commit -m \"add d.$Adiarnfd/f.$Adiarnfd\"\n+\t'\n+\ttest_expect_success \"add link Adiarnfc\" '\n+\t\tln -s d.$Adiarnfd/f.$Adiarnfd l.$Adiarnfc &&\n+\t\tgit add l.$Adiarnfc &&\n+\t\tgit commit -m \"add l.Adiarnfc\"\n+\t'\n+\t# This will test git log\n+\ttest_expect_success \"git log f.Adiar\" '\n+\t\tgit log f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit log f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git ls-files\n+\ttest_expect_success \"git lsfiles f.Adiar\" '\n+\t\tgit ls-files f.$Adiarnfc > f.Adiarnfc.log &&\n+\t\tgit ls-files f.$Adiarnfd > f.Adiarnfd.log &&\n+\t\ttest -s f.Adiarnfc.log &&\n+\t\ttest -s f.Adiarnfd.log &&\n+\t\ttest_cmp f.Adiarnfc.log f.Adiarnfd.log &&\n+\t\trm f.Adiarnfc.log f.Adiarnfd.log\n+\t'\n+\t# This will test git mv\n+\ttest_expect_success \"git mv\" '\n+\t\tgit mv f.$Adiarnfd f.$Odiarnfc &&\n+\t\tgit mv d.$Adiarnfd d.$Odiarnfc &&\n+\t\tgit mv l.$Adiarnfd l.$Odiarnfc &&\n+\t\tgit commit -m \"mv Adiarnfd Odiarnfc\"\n+\t'\n+\t# Files can be checked out as nfc\n+\t# And the link has been corrected from nfd to nfc\n+\ttest_expect_success \"git checkout nfc\" '\n+\t\trm f.$Odiarnfc &&\n+\t\tgit checkout f.$Odiarnfc\n+\t'\n+\t# Make it possible to checkout files with their NFD names\n+\ttest_expect_success \"git checkout file nfd\" '\n+\t\trm -f f.* &&\n+\t\tgit checkout f.$Odiarnfd\n+\t'\n+\t# Make it possible to checkout links with their NFD names\n+\ttest_expect_success \"git checkout link nfd\" '\n+\t\trm l.* &&\n+\t\tgit checkout l.$Odiarnfd\n+\t'\n+else\n+\t say \"Skipping nfc/nfd tests\"\n+fi\n+\n+test_done\n-- \n1.7.8.rc0.43.gb49a8\n"},{"id":"182087","messageId":"87vcomravs.fsf@catnip.gol.com","threadId":"29312","inReplyTo":"201201072059.23074.tboegi@web.de","subject":"Re: [PATCH][RFC] git on Mac OS and precomposed unicode","fromName":"Miles Bader","fromEmail":"miles@gnu.org","sentAt":"2012-01-08T06:01:11Z","receivedAt":"2012-01-08T06:01:11Z","isPatch":true,"sender":{"key":"miles@gnu.org","avatar":"https://gravatar.com/avatar/01069b69593af7bff28e2f97afeb3644ae6fe2f5f56cb3a8cf34c5fb8c36efe5?d=mp&s=160"},"body":"BTW, about the names, e.g. \"darwin.c\" etc -- is this code actually\nDarwin-specific, or simply Systems-that-happen-to-force-decomposed-\nunicode specific?\n\nIf the latter, maybe more generic names might be better.\n\nThanks,\n\n-Miles\n\n-- \n`To alcohol!  The cause of, and solution to,\n all of life's problems' --Homer J. Simpson\n"},{"id":"182170","messageId":"4F0B18D8.1010406@web.de","threadId":"29312","inReplyTo":"87vcomravs.fsf@catnip.gol.com","subject":"Re: [PATCH][RFC] git on Mac OS and precomposed unicode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-01-09T16:42:00Z","receivedAt":"2012-01-09T16:42:00Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 08.01.12 07:01, Miles Bader wrote:\n> BTW, about the names, e.g. \"darwin.c\" etc -- is this code actually\n> Darwin-specific, or simply Systems-that-happen-to-force-decomposed-\n> unicode specific?\n> \n> If the latter, maybe more generic names might be better.\n> \n> Thanks,\n> \n> -Miles\n> \nAs far as I know, Mac OS (darwin) is the only existing OS which likes\ndecomposed unicode so much, that forces decomposed unicode that way.\n/Torsten\n\n\n \n"}]}