{"thread":{"id":"31402","subject":"[RFC] i18n.pathencoding","startedAt":"2012-09-01T06:11:33Z","lastAt":"2012-09-08T10:09:14Z","messageCount":11,"participants":["Torsten Bögershausen","Robin Rosenberg","Nguyen Thai Ngoc Duy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"198176","messageId":"201209010811.33994.tboegi@web.de","threadId":"31402","inReplyTo":null,"subject":"[RFC] i18n.pathencoding","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-09-01T06:11:33Z","receivedAt":"2012-09-01T06:11:33Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Allow path names to be encoded in UTF-8 in the repository\nand checkout out as e.g. ISO-8859-1 in the working tree.\n\nIntroduce a config variable i18n.pathEncoding.\nIf empty, no re-encoding of path names is done.\n\nAdd t3911 to test encoding back and forth\n\nThe re-encoding is done in compat/reencode_pathname.c,\nwhere all file system functions like open(), stat(),\nreaddir() are re-defined.\n\nreencode_pathname.c includes all functionality from\nprecompose_utf8.c, which should be removed\n\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\nPlease read this as an RFC, so there several limitations:\n\n compat/reencode_pathname.h defines struct dirent_psx with d_name[2].\n   This is done to test renc_pn_readdir() in compat/reencode_pathname.c\n\n test case t1450 failes even on one of my linux machines. At first glance\n  it looks as the same failure which has been sometimes observed on Mac OS X.\n\n compat/precompose_utf8.[ch] had been integrated into reencode_pathname.[ch],\n  and should be removed.\n\n The patch should work on v7.1.12, it's not tested against latest master \n\n Comments are welcome.\n\n\n Documentation/config.txt      |  10 +\n Makefile                      |  11 +-\n builtin/init-db.c             |   3 +\n cache.h                       |   1 +\n compat/reencode_pathname.c    | 441 ++++++++++++++++++++++++++++++++++++++++++\n compat/reencode_pathname.h    |  72 +++++++\n config.c                      |   3 +\n environment.c                 |   1 +\n git-compat-util.h             |  20 +-\n parse-options.c               |   2 +-\n t/t3911-i18n-filename-8859.sh | 251 ++++++++++++++++++++++++\n wt-status.c                   |  21 +-\n 12 files changed, 827 insertions(+), 9 deletions(-)\n create mode 100644 compat/reencode_pathname.c\n create mode 100644 compat/reencode_pathname.h\n create mode 100755 t/t3911-i18n-filename-8859.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex a95e5a4..d633d54 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1442,6 +1442,16 @@ i18n.logOutputEncoding::\n \tCharacter encoding the commit messages are converted to when\n \trunning 'git log' and friends.\n \n+i18n.pathEncoding::\n+\tThis option is only used by some implementations of git.\n+\tWhen \"git init\" sets core.supportspathencoding to true,\n+\ti18n.pathEncoding can be set to re-encode path names when\n+\ta working tree is checked out.\n+\tPath names may be e.g. encoded in ISO-8859-1 and are stored as\n+\tUTF-8 encoded in the repository.\n+\tWhen not set, the encoding of path names is the same in working tree\n+\tand the repository.\n+\n imap::\n \tThe configuration variables in the 'imap' section are described\n \tin linkgit:git-imap-send[1].\ndiff --git a/Makefile b/Makefile\nindex 6b0c961..141562e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -143,6 +143,9 @@ all::\n #\n # Define NEEDS_LIBICONV if linking with libc is not enough (Darwin).\n #\n+# Define PATH_ENCODING if the encoding of file names\n+# differs from the encoding in the git repo\n+#\n # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\n # Patrick Mauritz).\n #\n@@ -595,6 +598,7 @@ LIB_H += compat/bswap.h\n LIB_H += compat/cygwin.h\n LIB_H += compat/mingw.h\n LIB_H += compat/obstack.h\n+LIB_H += compat/reencode_pathname.h\n LIB_H += compat/precompose_utf8.h\n LIB_H += compat/terminal.h\n LIB_H += compat/win32/dirent.h\n@@ -932,6 +936,7 @@ ifeq ($(uname_S),OSF1)\n \tNO_NSEC = YesPlease\n endif\n ifeq ($(uname_S),Linux)\n+\tPATH_ENCODING = YesPlease\n \tNO_STRLCPY = YesPlease\n \tNO_MKSTEMPS = YesPlease\n \tHAVE_PATHS_H = YesPlease\n@@ -999,7 +1004,7 @@ ifeq ($(uname_S),Darwin)\n \tNO_MEMMEM = YesPlease\n \tUSE_ST_TIMESPEC = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n-\tCOMPAT_OBJS += compat/precompose_utf8.o\n+\tCOMPAT_OBJS += compat/reencode_pathname.o\n \tBASIC_CFLAGS += -DPRECOMPOSE_UNICODE\n endif\n ifeq ($(uname_S),SunOS)\n@@ -1591,6 +1596,10 @@ ifdef FREAD_READS_DIRECTORIES\n \tCOMPAT_CFLAGS += -DFREAD_READS_DIRECTORIES\n \tCOMPAT_OBJS += compat/fopen.o\n endif\n+ifdef PATH_ENCODING\n+\tCOMPAT_CFLAGS += -DPATH_ENCODING\n+\tCOMPAT_OBJS += compat/reencode_pathname.o\n+endif\n ifdef NO_SYMLINK_HEAD\n \tBASIC_CFLAGS += -DNO_SYMLINK_HEAD\n endif\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 244fb7f..f159d43 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -291,6 +291,9 @@ static int create_default_files(const char *template_path)\n \t\tif (!access(path, F_OK))\n \t\t\tgit_config_set(\"core.ignorecase\", \"true\");\n \t\tprobe_utf8_pathname_composition(path, len);\n+#ifdef PATH_ENCODING\n+\t\tgit_config_set(\"core.supportspathencoding\", \"true\");\n+#endif\n \t}\n \n \treturn reinit;\ndiff --git a/cache.h b/cache.h\nindex 67f28b4..8023767 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1160,6 +1160,7 @@ extern int user_ident_sufficiently_given(void);\n extern const char *git_commit_encoding;\n extern const char *git_log_output_encoding;\n extern const char *git_mailmap_file;\n+extern const char *wt_path_encoding;\n \n /* IO helper functions */\n extern void maybe_flush_or_die(FILE *, const char *);\ndiff --git a/compat/reencode_pathname.c b/compat/reencode_pathname.c\nnew file mode 100644\nindex 0000000..3bdc776\n--- /dev/null\n+++ b/compat/reencode_pathname.c\n@@ -0,0 +1,441 @@\n+/*\n+ * Converts pathnames from one encoding into another.\n+ * The pathnames are stored as UTF-8 in the repository,\n+ * and might be checkout out as e.g. ISO-8859-1 in the working tree\n+ *\n+ * On MacOS X decomposed unicode is converted into precomposed unicode.\n+ */\n+\n+#define REENCODE_PATHNAME_C\n+#include \"cache.h\"\n+#include \"utf8.h\"\n+#include \"reencode_pathname.h\"\n+\n+#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))\n+\ttypedef const char *iconv_ibp;\n+#else\n+\ttypedef char *iconv_ibp;\n+#endif\n+\n+const static char *repo_path_encoding = \"UTF-8\";\n+\n+static iconv_t iconv_open_or_die(const char *tocode, const char *fromcode)\n+{\n+\ticonv_t my_iconv;\n+\tmy_iconv = iconv_open(tocode, fromcode);\n+\tif (my_iconv == (iconv_t) -1)\n+\t\tdie_errno(_(\"iconv_open(%s,%s) failed\"), tocode, fromcode);\n+\treturn my_iconv;\n+}\n+\n+static size_t has_non_ascii(const char *s, size_t maxlen, size_t *strlen_c)\n+{\n+\tconst uint8_t *ptr = (const uint8_t *)s;\n+\tsize_t strlen_chars = 0;\n+\tsize_t ret = 0;\n+\n+\tif (!ptr || !*ptr)\n+\t\treturn 0;\n+\n+\twhile (*ptr && maxlen) {\n+\t\tif (*ptr & 0x80)\n+\t\t\tret++;\n+\t\tstrlen_chars++;\n+\t\tptr++;\n+\t\tmaxlen--;\n+\t}\n+\tif (strlen_c)\n+\t\t*strlen_c = strlen_chars;\n+\n+\treturn ret;\n+}\n+\n+#ifdef PRECOMPOSE_UNICODE\n+void probe_utf8_pathname_composition(char *path, int len)\n+{\n+\tstatic const char *auml_nfc = \"\\xc3\\xa4\";\n+\tstatic const char *auml_nfd = \"\\x61\\xcc\\x88\";\n+\tint output_fd;\n+\tif (precomposed_unicode != -1)\n+\t\treturn; /* We found it defined in the global config, respect it */\n+\tstrcpy(path + len, auml_nfc);\n+\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n+\tif (output_fd >= 0) {\n+\t\tclose(output_fd);\n+\t\tstrcpy(path + len, auml_nfd);\n+\t\t/* Indicate to the user, that we can configure it to true */\n+\t\tif (!access(path, R_OK))\n+\t\t\tgit_config_set(\"core.precomposeunicode\", \"false\");\n+\t\t/* To be backward compatible, set precomposed_unicode to 0 */\n+\t\tprecomposed_unicode = 0;\n+\t\tstrcpy(path + len, auml_nfc);\n+\t\tif (unlink(path))\n+\t\t\tdie_errno(_(\"failed to unlink '%s'\"), path);\n+\t}\n+}\n+#endif\n+\n+void reencode_argv(int argc, const char **argv)\n+{\n+\tint i = 0;\n+\tconst char *oldarg;\n+\tchar *newarg;\n+\ticonv_t ic_wt_to_repo;\n+\n+#ifdef PRECOMPOSE_UNICODE\n+\tif (precomposed_unicode == 1)\n+\t\twt_path_encoding = \"UTF-8-MAC\";\n+#endif\n+\n+\tif (!wt_path_encoding || !*wt_path_encoding)\n+\t\treturn;\n+\n+\tic_wt_to_repo = iconv_open_or_die(repo_path_encoding, wt_path_encoding);\n+\n+\twhile (i < argc) {\n+\t\tsize_t namelen;\n+\t\toldarg = argv[i];\n+\t\tif (has_non_ascii(oldarg, (size_t)-1, &namelen)) {\n+\t\t\tnewarg = reencode_string_iconv(oldarg, namelen, ic_wt_to_repo);\n+\t\t\tif (newarg)\n+\t\t\t\targv[i] = newarg;\n+\t\t}\n+\t\ti++;\n+\t}\n+\ticonv_close(ic_wt_to_repo);\n+}\n+\n+#ifdef PATH_ENCODING\n+char *str_repo2worktree(const char *in)\n+{\n+\tint olderrno = errno;\n+\tchar *retvalue = NULL;\n+\tsize_t inlen;\n+\n+\tif (!wt_path_encoding || !*wt_path_encoding)\n+\t\treturn NULL;\n+\n+\tif (!in)\n+\t\tdie(\"str_repo2worktree in == NULL\\n\");\n+\n+\tif (has_non_ascii(in, (size_t)-1, &inlen)) {\n+\t\ticonv_t my_iconv_repo2worktree;\n+\t\tmy_iconv_repo2worktree = iconv_open_or_die(wt_path_encoding,\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t repo_path_encoding);\n+\n+\t\tretvalue = reencode_string_iconv(in, inlen, my_iconv_repo2worktree);\n+\t\ticonv_close(my_iconv_repo2worktree);\n+\t\tif (retvalue)\n+\t\t\terrno = olderrno;\n+\t} else\n+\t\terrno = olderrno;\n+\n+\treturn retvalue;\n+}\n+\n+char *str_worktree2repolen(const char *in, size_t insz)\n+{\n+\tchar *retvalue = NULL;\n+\tsize_t inlen;\n+\tif (!wt_path_encoding || !*wt_path_encoding)\n+\t\treturn NULL;\n+\n+\tif (has_non_ascii(in, insz, &inlen)) {\n+\tint olderrno = errno;\n+\t\ticonv_t my_iconv_worktree2repo;\n+\t\tmy_iconv_worktree2repo = iconv_open_or_die(repo_path_encoding,\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t wt_path_encoding);\n+\t\tretvalue = reencode_string_iconv(in, insz, my_iconv_worktree2repo);\n+\t\ticonv_close(my_iconv_worktree2repo);\n+\t\tif (retvalue)\n+\t\t\terrno = olderrno;\n+\t}\n+\treturn retvalue;\n+}\n+\n+char *str_worktree2repo(const char *in)\n+{\n+\treturn str_worktree2repolen(in, strlen(in));\n+}\n+#endif\n+\n+#define RENC_PN_DECL_SAVERRNO_PATH1(path) \\\n+\tint olderrno = errno; \\\n+\tconst char *path1_enc = path; \\\n+\tchar *path1_malloc_wt_encoded = NULL\n+\n+#define RENC_PN_DECL_PATH2(path) \\\n+\tconst char *path2_enc = path; \\\n+\tchar *path2_malloc_wt_encoded = NULL\n+\n+\n+#define RENC_PN_CONV_PATH1(path, erroret) \\\n+\terrno=0; \\\n+\tpath1_malloc_wt_encoded = str_repo2worktree(path); \\\n+\tif (!path1_malloc_wt_encoded && errno) { \\\n+\t\treturn erroret; \\\n+\t} \\\n+\tif (path1_malloc_wt_encoded) \\\n+\t\tpath1_enc = path1_malloc_wt_encoded; \\\n+\terrno = olderrno;\n+\n+#define RENC_PN_CONV_PATH2(path) \\\n+\terrno=0; \\\n+\tpath2_malloc_wt_encoded = str_repo2worktree(path); \\\n+\tif (!path2_malloc_wt_encoded && errno) { \\\n+\t\treturn -1; \\\n+\t} \\\n+\tif (path2_malloc_wt_encoded) \\\n+\t\tpath2_enc = path2_malloc_wt_encoded; \\\n+\terrno = olderrno;\n+\n+\n+RENC_FN_DIR *renc_pn_opendir(const char *dirname)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(dirname);\n+\tRENC_FN_DIR *renc_pn_dir = xmalloc(sizeof(RENC_FN_DIR));\n+\n+#ifdef PRECOMPOSE_UNICODE\n+\tif (precomposed_unicode == 1)\n+\t\twt_path_encoding = \"UTF-8-MAC\";\n+#endif\n+\n+\trenc_pn_dir->dirent_utf8 = xmalloc(sizeof(dirent_psx));\n+\trenc_pn_dir->dirent_utf8->max_name_len = sizeof(renc_pn_dir->dirent_utf8->d_name);\n+\n+\tRENC_PN_CONV_PATH1(dirname, NULL);\n+\n+\trenc_pn_dir->dirp = opendir(path1_enc);\n+\tolderrno = errno;\n+\tif (!renc_pn_dir->dirp) {\n+\t\tfree(path1_malloc_wt_encoded);\n+\t\tfree(renc_pn_dir->dirent_utf8);\n+\t\tfree(renc_pn_dir);\n+\t\treturn NULL;\n+\t} else\n+\t\trenc_pn_dir->ic_wt_to_repo = (iconv_t)-1;\n+\n+\tfree(path1_malloc_wt_encoded);\n+\terrno = olderrno;\n+\treturn renc_pn_dir;\n+}\n+\n+struct dirent_psx *renc_pn_readdir(RENC_FN_DIR *renc_pn_dir)\n+{\n+\tstruct dirent *res;\n+\tres = readdir(renc_pn_dir->dirp);\n+\tif (res) {\n+\t\tsize_t namelenz = strlen(res->d_name) + 1; /* \\0 */\n+\t\tsize_t new_len_needed = 0;\n+\t\tint ret_errno = errno;\n+\n+\t\trenc_pn_dir->dirent_utf8->d_ino\t = res->d_ino;\n+\t\trenc_pn_dir->dirent_utf8->d_type = res->d_type;\n+\tdo {\n+\t\t if (new_len_needed > renc_pn_dir->dirent_utf8->max_name_len) {\n+\t\t\t\tsize_t new_len = sizeof(dirent_psx) + new_len_needed -\n+\t\t\t\t\tsizeof(renc_pn_dir->dirent_utf8->d_name);\n+\n+\t\t\t\trenc_pn_dir->dirent_utf8 = xrealloc(renc_pn_dir->dirent_utf8, new_len);\n+\t\t\t\trenc_pn_dir->dirent_utf8->max_name_len = new_len_needed;\n+\t\t\t}\n+\n+\t\t\tif (wt_path_encoding && has_non_ascii(res->d_name, (size_t)-1, NULL)) {\n+\t\t\t\ticonv_ibp cp = (iconv_ibp)res->d_name;\n+\t\t\t\tsize_t inleft = namelenz;\n+\t\t\t\tchar *outpos = &renc_pn_dir->dirent_utf8->d_name[0];\n+\t\t\t\tsize_t outsz = renc_pn_dir->dirent_utf8->max_name_len;\n+\t\t\t\terrno = 0;\n+\t\t\t\tif (renc_pn_dir->ic_wt_to_repo == (iconv_t)-1)\n+\t\t\t\t\trenc_pn_dir->ic_wt_to_repo = iconv_open_or_die(repo_path_encoding,\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t wt_path_encoding);\n+\t\t\t\tif (-1 != iconv(renc_pn_dir->ic_wt_to_repo,\n+\t\t\t\t\t\t\t\t\t\t\t\t&cp, &inleft,\t&outpos, &outsz))\n+\t\t\t\t\tbreak; /* Conversion OK, we are done */\n+\t\t\t\tif (errno == E2BIG) {\n+\t\t\t\t\tchar *tmp = reencode_string_iconv(res->d_name, namelenz,\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\trenc_pn_dir->ic_wt_to_repo);\n+\t\t\t\t\tif (tmp) {\n+\t\t\t\t\t\tnew_len_needed = strlen(tmp) + 1; /* \\0 */\n+\t\t\t\t\t\tfree(tmp);\n+\t\t\t\t\t}\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * iconv() failed and errno could be EILSEQ, EINVAL, EBADF\n+\t\t\t\t\t * In general we avoid illegal byte sequences.\n+\t\t\t\t\t * If they occur on a mounted drive (e.g. NFS) it is not worth to\n+\t\t\t\t\t * die() for that, but rather let the user see the original name\n+\t\t\t\t\t */\n+\t\t\t\t\tnamelenz = 0; /* trigger strlcpy */\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tif (namelenz > renc_pn_dir->dirent_utf8->max_name_len)\n+\t\t\t\t\tnew_len_needed = namelenz; /* need to re-allocate */\n+\t\t\t\telse\n+\t\t\t\t\tnamelenz = 0;\t /* trigger strlcpy */\n+\t\t\t}\n+\t\t} while (new_len_needed > renc_pn_dir->dirent_utf8->max_name_len);\n+\n+\t\tif (!namelenz)\n+\t\t\tstrlcpy(renc_pn_dir->dirent_utf8->d_name, res->d_name,\n+\t\t\t\t\t\t\trenc_pn_dir->dirent_utf8->max_name_len);\n+\n+\t\terrno = ret_errno;\n+\t\treturn renc_pn_dir->dirent_utf8;\n+\t}\n+\treturn NULL;\n+}\n+\n+int renc_pn_closedir(RENC_FN_DIR *renc_pn_dir)\n+{\n+\tint ret_value;\n+\tint ret_errno;\n+\tret_value = closedir(renc_pn_dir->dirp);\n+\tret_errno = errno;\n+\tif (renc_pn_dir->ic_wt_to_repo != (iconv_t)-1)\n+\t\ticonv_close(renc_pn_dir->ic_wt_to_repo);\n+\tfree(renc_pn_dir->dirent_utf8);\n+\tfree(renc_pn_dir);\n+\terrno = ret_errno;\n+\treturn ret_value;\n+}\n+\n+int renc_pn_mkdir(const char *path, mode_t mode)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tint ret;\n+\n+\tRENC_PN_CONV_PATH1(path, -1);\n+\n+\tret = mkdir(path1_enc, mode);\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+int renc_pn_lstat(const char *path, struct stat *buf)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tint ret;\n+\n+\tRENC_PN_CONV_PATH1(path, -1);\n+\n+\tret = lstat(path1_enc, buf);\n+\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+int renc_pn_stat(const char *path, struct stat *buf)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tint ret;\n+\n+\tRENC_PN_CONV_PATH1(path, -1);\n+\n+\tret = stat(path1_enc, buf);\n+\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+int renc_pn_open(const char *path, int oflag, ...\t )\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tva_list params;\n+\tint mode;\n+\tint ret;\n+\n+\tva_start(params, oflag);\n+\tmode = va_arg(params, int);\n+\tva_end(params);\n+\n+\tRENC_PN_CONV_PATH1(path, -1);\n+\n+\tret = open(path1_enc, oflag, mode);\n+\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+int renc_pn_unlink(const char *path)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tint ret;\n+\n+\tRENC_PN_CONV_PATH1(path, -1);\n+\n+\tret = unlink(path1_enc);\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+FILE *renc_pn_fopen(const char *path, const char *mode)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tFILE *ret;\n+\n+\tRENC_PN_CONV_PATH1(path,NULL);\n+\n+\tret = fopen(path1_enc,mode);\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+\n+ssize_t renc_pn_readlink(const char *path, char *buf, size_t bufsiz)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(path);\n+\tssize_t ret;\n+\n+\tRENC_PN_CONV_PATH1(path, -1);\n+\n+\tret = readlink(path1_enc, buf, bufsiz);\n+\n+\tif (ret > 0) {\n+\t\tchar *new_buf = NULL;\n+\t\terrno = 0;\n+\t\tnew_buf = str_worktree2repolen(buf, ret);\n+\t\tif (new_buf) {\n+\t\t\tsize_t newlen = strlen(new_buf);\n+\t\t\tif (newlen > bufsiz)\n+\t\t\t\tnewlen = bufsiz;\n+\t\t\tmemcpy(buf, new_buf, newlen);\n+\t\t\tret = newlen;\n+\t\t\tfree(new_buf);\n+\t\t} else if (!errno)\n+\t\t\terrno = olderrno;\n+\t}\n+\tfree(path1_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+int renc_pn_symlink(const char *oldname, const char *newname)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(oldname);\n+\tRENC_PN_DECL_PATH2(newname);\n+\tint ret;\n+\n+\tRENC_PN_CONV_PATH1(oldname, -1);\n+\tRENC_PN_CONV_PATH2(newname);\n+\n+\tret = symlink(path1_enc, path2_enc);\n+\tfree(path1_malloc_wt_encoded);\n+\tfree(path2_malloc_wt_encoded);\n+\treturn ret;\n+}\n+\n+int renc_pn_rename(const char *oldname, const char *newname)\n+{\n+\tRENC_PN_DECL_SAVERRNO_PATH1(oldname);\n+\tRENC_PN_DECL_PATH2(newname);\n+\tint ret;\n+\n+\tRENC_PN_CONV_PATH1(oldname, -1);\n+\tRENC_PN_CONV_PATH2(newname);\n+\n+\tret = rename(path1_enc, path2_enc);\n+\tfree(path1_malloc_wt_encoded);\n+\tfree(path2_malloc_wt_encoded);\n+\n+\treturn ret;\n+}\ndiff --git a/compat/reencode_pathname.h b/compat/reencode_pathname.h\nnew file mode 100644\nindex 0000000..9300ba4\n--- /dev/null\n+++ b/compat/reencode_pathname.h\n@@ -0,0 +1,70 @@\n+#ifndef REENCODE_PATHNAME_H\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <dirent.h>\n+#include <fcntl.h>\n+#include <stdio.h>\n+#include <iconv.h>\n+\n+typedef struct dirent_psx {\n+\tino_t d_ino;            /* Posix */\n+\tsize_t max_name_len;    /* See below */\n+\tunsigned char d_type;   /* available on all systems git runs on */\n+\n+\t/*\n+\t * See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html\n+\t * NAME_MAX + 1 should be enough, but some systems have\n+\t * NAME_MAX=255 and strlen(d_name) may return 508 or 510\n+\t * Solution: allocate more when needed, see renc_pn_readdir()\n+\t */\n+\tchar   d_name[/* NAME_MAX */ 1+1];\n+} dirent_psx;\n+\n+typedef struct {\n+\ticonv_t ic_wt_to_repo;\n+\tDIR *dirp;\n+\tstruct dirent_psx *dirent_utf8;\n+} RENC_FN_DIR;\n+\n+void reencode_argv(int argc, const char **argv);\n+void probe_utf8_pathname_composition(char *, int);\n+\n+RENC_FN_DIR *renc_pn_opendir(const char *dirname);\n+struct dirent_psx *renc_pn_readdir(RENC_FN_DIR *dirp);\n+int renc_pn_closedir(RENC_FN_DIR *dirp);\n+\n+#ifdef PATH_ENCODING\n+char* str_repo2worktree(const char *in);\n+int renc_pn_mkdir(const char *path, mode_t mode);\n+int renc_pn_lstat(const char *path, struct stat *buf);\n+int renc_pn_stat(const char *path, struct stat *buf);\n+int renc_pn_open(const char *path, int oflag, ...  );\n+int renc_pn_unlink(const char *path);\n+FILE *renc_pn_fopen(const char *path, const char *mode);\n+ssize_t renc_pn_readlink(const char *path, char *buf, size_t bufsiz);\n+int renc_pn_symlink(const char *oldname, const char *newname);\n+int renc_pn_rename(const char *oldname, const char *newname);\n+#endif\n+\n+#ifndef REENCODE_PATHNAME_C\n+#define opendir(n) renc_pn_opendir(n)\n+#define readdir(d) renc_pn_readdir(d)\n+#define closedir(d) renc_pn_closedir(d)\n+#define dirent dirent_psx\n+#define DIR RENC_FN_DIR\n+\n+#ifdef PATH_ENCODING\n+#define mkdir(a,b) renc_pn_mkdir((a),(b))\n+#define lstat(a,b) renc_pn_lstat((a),(b))\n+#define stat(a,b) renc_pn_stat((a),(b))\n+#define open renc_pn_open\n+#define unlink renc_pn_unlink\n+#define fopen(a,b) renc_pn_fopen((a),(b))\n+#define readlink(a,b,c) renc_pn_readlink(a,b,c)\n+#define symlink(a,b) renc_pn_symlink(a,b)\n+#define rename(a,b) renc_pn_rename(a,b)\n+#endif\n+\n+#endif  /* REENCODE_PATHNAME_C */\n+#define  REENCODE_PATHNAME_H\n+#endif /* REENCODE_PATHNAME_H */\ndiff --git a/config.c b/config.c\nindex 2b706ea..d591c09 100644\n--- a/config.c\n+++ b/config.c\n@@ -775,6 +775,9 @@ static int git_default_i18n_config(const char *var, const char *value)\n \tif (!strcmp(var, \"i18n.logoutputencoding\"))\n \t\treturn git_config_string(&git_log_output_encoding, var, value);\n \n+\tif (!strcmp(var, \"i18n.pathencoding\"))\n+\t\treturn git_config_string(&wt_path_encoding, var, value);\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 85edd7f..ba81575 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -59,6 +59,7 @@ int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n+const char *wt_path_encoding = NULL;\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 35b095e..877b060 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -153,13 +153,21 @@\n #endif\n #endif\n \n-/* used on Mac OS X */\n-#ifdef PRECOMPOSE_UNICODE\n-#include \"compat/precompose_utf8.h\"\n+#if defined(PATH_ENCODING) || defined(PRECOMPOSE_UNICODE)\n+#include \"compat/reencode_pathname.h\"\n #else\n-#define precompose_str(in,i_nfd2nfc)\n-#define precompose_argv(c,v)\n-#define probe_utf8_pathname_composition(a,b)\n+#define reencode_argv(c,v)\n+#endif\n+\n+/* needed for Mac OS X */\n+#ifndef PRECOMPOSE_UNICODE\n+#define probe_utf8_pathname_composition(a,b);\n+#endif\n+\n+#ifndef PATH_ENCODING\n+#define str_worktree2repolen(in, insz) (NULL)\n+#define str_repo2worktree(in) (NULL)\n+#define str_worktree2repo(in) (NULL)\n #endif\n \n #ifndef NO_LIBGEN_H\ndiff --git a/parse-options.c b/parse-options.c\nindex c1c66bd..5840c18 100644\n--- a/parse-options.c\n+++ b/parse-options.c\n@@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n \t\tusage_with_options(usagestr, options);\n \t}\n \n-\tprecompose_argv(argc, argv);\n+\treencode_argv(argc, argv);\n \treturn parse_options_end(&ctx);\n }\n \ndiff --git a/t/t3911-i18n-filename-8859.sh b/t/t3911-i18n-filename-8859.sh\nnew file mode 100755\nindex 0000000..aa2be57\n--- /dev/null\n+++ b/t/t3911-i18n-filename-8859.sh\n@@ -0,0 +1,251 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Torsten Bögershausen\n+#\n+\n+test_description='file system encodings UTF-8 ISO8859-1'\n+\n+. ./test-lib.sh\n+\n+fname_UTF_8=`printf '\\303\\206\\302\\242'`\n+fname_ISO8859_1=`printf '\\306\\242'`\n+Euro_utf8=`printf '\\342\\202\\254'`\n+supportspathencoding=`git config core.supportspathencoding` || :\n+\n+\n+add_file_dir_link() {\n+\tlocal bname=$1\n+\tlocal fname=$2\n+\ttest_expect_success \"add file $fname.f $bname\" '\n+\t\tgit checkout master &&\n+\t\tgit checkout -b add_f_$bname &&\n+\t\t>$fname.f &&\n+\t\tgit add $fname.f &&\n+\t\tgit commit -m \"add fname\"\n+\t'\n+\n+\ttest_expect_success \"add dir $fname.d $bname\" '\n+\t\tgit checkout master &&\n+\t\tgit checkout -b add_d_$bname &&\n+\t\tmkdir $fname.d &&\n+\t\ttouch $fname.d/$fname.f &&\n+\t\tgit add $fname.d/$fname.f &&\n+\t\tgit commit -m \"add fname.d/fname\"\n+\t'\n+\n+\ti=0\n+\tfor src in x $fname; do\n+\t\tfor dst in x $fname; do\n+\t\t\ttest_expect_success \"add link $dst.l->$src.f on branch add_l_${i}_$bname\" '\n+\t\t\t\tgit checkout master &&\n+\t\t\t\tgit checkout -b add_l_${i}_$bname &&\n+\t\t\t\tln -s $src.f $dst.l &&\n+\t\t\t\tgit add $dst.l &&\n+\t\t\t\tgit commit -m \"add fname.l $i\"\n+\t\t\t'\n+\t\t\ti=$(($i+1))\n+\t\tdone\n+\tdone\n+}\n+\n+test_expect_success \"setup add rm x\" '\n+\t>x &&\n+\tgit add x &&\n+\tgit commit -m \"1st commit\" &&\n+\tgit rm x &&\n+\tgit commit -m \"rm x\"\n+'\n+\n+#combinations to be tested:\n+# UTF-8     -> ISO8859-1\n+# ISO8859-1 -> UTF-8\n+\n+if test \"$supportspathencoding\"\n+then\n+\tsrcencodings=\"ISO8859-1 UTF-8\"\n+\tfor srcenc in $srcencodings\n+\tdo\n+\t\tcase $srcenc in\n+\t\tISO8859-1)\n+\t\t\tdstenc=UTF-8\n+\t\t;;\n+\t\tUTF-8)\n+\t\t\tdstenc=ISO8859-1\n+\t\t;;\n+\t\tUTF-8-MAC)\n+\t\t\tdstenc=UTF-8\n+\t\t;;\n+\t\t*)\n+\t\t\techo >&2 \"Wrong encoding $srcenc\"\n+\t\t\texit 1\n+\t\t;;\n+\t\tesac\n+\t\teval fname_src=\\$fname_$(echo $srcenc | sed -e 's/-/_/g' -e 's/_MAC//')\n+\t\teval fname_dst=\\$fname_$(echo $dstenc | sed -e 's/-/_/g')\n+\t\ttest_expect_success \"setup $srcenc\" '\n+\t\t\tgit checkout master &&\n+\t\t\tgit config i18n.pathencoding $srcenc\n+\t\t'\n+\t\tadd_file_dir_link $srcenc $fname_src\n+\n+\t\ttest_expect_success \"setup $dstenc\" '\n+\t\t\tgit checkout master &&\n+\t\t\techo \"git checkout Master\" >&2\n+\t\t\tls -l >&2\n+\t\t\tgit config i18n.pathencoding $dstenc\n+\t\t'\n+\n+\t\ttest_expect_success \"checkout file $dstenc (was $srcenc)\" '\n+\t\t\tgit checkout add_f_$srcenc\n+\t\t'\n+\n+\t\ttest_expect_success \"exists file $dstenc (was $srcenc)\" '\n+\t\t\ttest -f $fname_dst.f\n+\t\t'\n+\n+\t\ttest_expect_success \"log file $dstenc (was $srcenc)\" '\n+\t\t\tgit log $fname_dst.f\n+\t\t'\n+\n+\t\ttest_expect_success \"git mv\" '\n+\t\t\tgit checkout -b mv_file_$srcenc &&\n+\t\t\tgit mv $fname_dst.f XX.f &&\n+\t\t\tgit commit -m \"git mv fname_dst.f XX.f\"\n+\t\t'\n+\n+\t\ttest_expect_success \"checkout dir $dstenc (was $srcenc)\" '\n+\t\t\tgit checkout add_d_$srcenc\n+\t\t'\n+\n+\t\ttest_expect_success \"exist dir $dstenc (was $srcenc)\" '\n+\t\t\ttest -d $fname_dst.d\n+\t\t'\n+\n+\t\ttest_expect_success \"log dir $dstenc (was $srcenc)\" '\n+\t\t\tgit log $fname_dst.d\n+\t\t'\n+\n+\t\ti=0\n+\t\tfor src in x $fname_dst; do\n+\t\t\tfor dst in x $fname_dst; do\n+\t\t\t\ttest_expect_success \"checkout link $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n+\t\t\t\t\tgit checkout add_l_${i}_$srcenc\n+\t\t\t\t'\n+\t\t\t\ttest_expect_success \"exist link $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n+\t\t\t\t\ttest -L $dst.l\n+\t\t\t\t'\n+\t\t\t\ttest_expect_success \"log link $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n+\t\t\t\t\tgit log $dst.l\n+\t\t\t\t'\n+\t\t\t\ttest_expect_success \"readlink $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n+\t\t\t\t\techo \"$src.f\" >expect &&\n+\t\t\t\t\treadlink \"$dst.l\" > actual &&\n+\t\t\t\t\ttest_cmp expect actual &&\n+\t\t\t\t\trm expect actual\n+\t\t\t\t'\n+\t\t\t\ti=$(($i+1))\n+\t\t\tdone\n+\t\tdone\n+\tdone\n+\t# Make sure that Euro sign can NOT be checked out in 8859\n+\t#fname_src=Euro\n+\ttest_expect_success \"setup UTF-8\" '\n+\t\tgit checkout master &&\n+\t\tgit config i18n.pathencoding UTF-8\n+\t'\n+\tadd_file_dir_link Euro $Euro_utf8\n+\n+\ttest_expect_success \"setup ISO8859-1\" '\n+\t\tgit checkout master &&\n+\t\trm -rf * &&\n+\t\tgit config i18n.pathencoding ISO8859-1\n+\t'\n+\ttest_expect_success \"checkout file Euro branch add_f_Euro\" '\n+\t\tgit checkout add_f_Euro\n+\t\techo *  >actual &&\n+\t\techo \"*\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\t\trm expect actual\n+\t'\n+\n+\ttest_expect_success \"checkout dir Euro branch add_d_Euro\" '\n+\t\trm -rf * &&\n+\t\ttest_must_fail git checkout add_d_Euro\n+\t'\n+\n+\ttest_expect_success \"Cleanup\" '\n+\t\tgit config i18n.pathencoding UTF-8 &&\n+\t\tgit checkout master &&\n+\t\trm -rf * &&\n+\t\tgit reset --hard &&\n+\t\tgit config i18n.pathencoding ISO8859-1\n+\t'\n+\n+\ttest_expect_success \"checkout link Euro.l->x.f branch add_l_1_Euro\" '\n+\t\t! git checkout add_l_1_Euro\n+\t'\n+\n+\ttest_expect_success \"No link Euro.l->x.f\" '\n+\t\techo *  >actual &&\n+\t\techo \"*\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\t\trm expect actual\n+\t'\n+\n+\ttest_expect_success \"Cleanup after Euro.l->x.f\" '\n+\t\tgit config i18n.pathencoding UTF-8 &&\n+\t\tgit checkout master &&\n+\t\trm -rf * &&\n+\t\tgit reset --hard &&\n+\t\tgit config i18n.pathencoding ISO8859-1\n+\t'\n+\n+\t# Checkoing out a soft link pointing to a filename outside\n+\t# 8859-1 should fail\n+\ttest_expect_failure \"checkout link x.l->Euro.f branch add_l_2_Euro\" '\n+\t\t! git checkout add_l_2_Euro\n+\t'\n+\n+\ttest_expect_success \"No link x.f->Euro.l\" '\n+\t\techo *  >actual &&\n+\t\techo \"*\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\t\trm expect actual\n+\t'\n+\n+\ttest_expect_success \"Cleanup after link x.l->Euro.f branch\" '\n+\t\tgit config i18n.pathencoding UTF-8 &&\n+\t\tgit checkout master &&\n+\t\trm -rf * &&\n+\t\tgit reset --hard &&\n+\t\tgit config i18n.pathencoding ISO8859-1\n+\t'\n+\n+\ttest_expect_success \"checkout link Euro.l->Euro.f branch add_l_3_Euro\" '\n+\t\t! git checkout add_l_3_Euro\n+\t'\n+\n+\ttest_expect_success \"No link Euro.l->Euro.f\" '\n+\t\techo *  >actual &&\n+\t\techo \"*\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\t\trm expect actual\n+\t'\n+\n+else\n+\ttest_expect_success \"setup 8859\" '\n+\t\tgit config i18n.pathencoding ISO8859-1 &&\n+\t\tgit checkout -b add_file_8859 &&\n+\t\t> $fname_src.f &&\n+\t\tgit add $fname_src.f &&\n+\t\tgit commit -m \"add fname_src\" &&\n+\t\tgit config i18n.pathencoding UTF-8 &&\n+\t\trm -rf * &&\n+\t\tgit reset --hard\n+\t'\n+\ttest_expect_success \"Silent support of pathencoding\" '\n+\t\ttest_must_fail test -f $fname_UTF_8.f\n+\t'\n+fi\n+\n+test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex c110cbc..1590caa 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -233,7 +233,26 @@ static void wt_status_print_trailer(struct wt_status *s)\n \tstatus_printf_ln(s, color(WT_STATUS_HEADER, s), \"\");\n }\n \n-#define quote_path quote_path_relative\n+#ifdef PATH_ENCODING\n+char *quote_path_repo2worktree(const char *in, int len,\n+\t\t\t  struct strbuf *out, const char *prefix)\n+{\n+\tconst char *in_encoded = in;\n+\tchar *in_worktree_encoded = str_repo2worktree(in);\n+\tchar *ret;\n+\t(void)len;\n+\n+\tif (in_worktree_encoded)\n+\t\tin_encoded = in_worktree_encoded;\n+\tret = quote_path_relative(in_encoded , -1, out, prefix);\n+\tfree(in_worktree_encoded);\n+\treturn ret;\n+\n+}\n+\t#define quote_path quote_path_repo2worktree\n+#else\n+\t#define quote_path quote_path_relative\n+#endif\n \n static void wt_status_print_unmerged_data(struct wt_status *s,\n \t\t\t\t\t  struct string_list_item *it)\n-- \n1.7.12\n"},{"id":"198237","messageId":"5043E4B4.9050801@dewire.com","threadId":"31402","inReplyTo":"201209010811.33994.tboegi@web.de","subject":"Re: [RFC] i18n.pathencoding","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2012-09-02T22:59:00Z","receivedAt":"2012-09-02T22:59:00Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Torsten Bögershausen skrev 2012-09-01 08.11:> Allow path names to be encoded in UTF-8 in the repository\n > and checkout out as e.g. ISO-8859-1 in the working tree.\n\nAck for attempting this.\n\nDid it myself if 2007, but times weren't ripe then, I guess.\n\n > +i18n.pathEncoding::\n > +\tThis option is only used by some implementations of git.\n > +\tWhen \"git init\" sets core.supportspathencoding to true,\n > +\ti18n.pathEncoding can be set to re-encode path names when\n > +\ta working tree is checked out.\n > +\tPath names may be e.g. encoded in ISO-8859-1 and are stored as\n > +\tUTF-8 encoded in the repository.\n > +\tWhen not set, the encoding of path names is the same in working tree\n > +\tand the repository.\n\n\"If set, then core.precomposeunicode is ignored on Mac OS X.\"\n\n > diff --git a/compat/reencode_pathname.c b/compat/reencode_pathname.c\n > new file mode 100644\n > index 0000000..3bdc776\n > --- /dev/null\n > +++ b/compat/reencode_pathname.c\n > @@ -0,0 +1,441 @@\n > +/*\n > + * Converts pathnames from one encoding into another.\n > + * The pathnames are stored as UTF-8 in the repository,\n > + * and might be checkout out as e.g. ISO-8859-1 in the working tree\n > + *\n > + * On MacOS X decomposed unicode is converted into precomposed unicode.\n, ignoring the setting of core.precomposeunicode.\n\n[...]\n > + */\n > +\n > +#define REENCODE_PATHNAME_C\n > +#include \"cache.h\"\n > +#include \"utf8.h\"\n > +#include \"reencode_pathname.h\"\n > +\n > +#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))\n > +\ttypedef const char *iconv_ibp;\n > +#else\n > +\ttypedef char *iconv_ibp;\n > +#endif\n > +\n > +const static char *repo_path_encoding = \"UTF-8\";\n > +\n > +static iconv_t iconv_open_or_die(const char *tocode, const char *fromcode)\n > +{\n > +\ticonv_t my_iconv;\n > +\tmy_iconv = iconv_open(tocode, fromcode);\njoin these two lines\n\n > +\tif (my_iconv == (iconv_t) -1)\n > +\t\tdie_errno(_(\"iconv_open(%s,%s) failed\"), tocode, fromcode);\n > +\treturn my_iconv;\n > +}\n > +\n > +static size_t has_non_ascii(const char *s, size_t maxlen, size_t *strlen_c)\n > +{\n > +\tconst uint8_t *ptr = (const uint8_t *)s;\n > +\tsize_t strlen_chars = 0;\n > +\tsize_t ret = 0;\n > +\n > +\tif (!ptr || !*ptr)\n > +\t\treturn 0;\n > +\n > +\twhile (*ptr && maxlen) {\n > +\t\tif (*ptr & 0x80)\n > +\t\t\tret++;\n > +\t\tstrlen_chars++;\n > +\t\tptr++;\n > +\t\tmaxlen--;\n > +\t}\n > +\tif (strlen_c)\n > +\t\t*strlen_c = strlen_chars;\n > +\n > +\treturn ret;\n > +}\n > +\n > +#ifdef PRECOMPOSE_UNICODE\n > +void probe_utf8_pathname_composition(char *path, int len)\n > +{\n > +\tstatic const char *auml_nfc = \"\\xc3\\xa4\";\n > +\tstatic const char *auml_nfd = \"\\x61\\xcc\\x88\";\n > +\tint output_fd;\n > +\tif (precomposed_unicode != -1)\n > +\t\treturn; /* We found it defined in the global config, respect it */\na bland line here would be nice\n\n > +\tstrcpy(path + len, auml_nfc);\n > +\toutput_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);\n > +\tif (output_fd >= 0) {\n > +\t\tclose(output_fd);\n > +\t\tstrcpy(path + len, auml_nfd);\n > +\t\t/* Indicate to the user, that we can configure it to true */\n > +\t\tif (!access(path, R_OK))\n > +\t\t\tgit_config_set(\"core.precomposeunicode\", \"false\");\n > +\t\t/* To be backward compatible, set precomposed_unicode to 0 */\n > +\t\tprecomposed_unicode = 0;\n > +\t\tstrcpy(path + len, auml_nfc);\n > +\t\tif (unlink(path))\n > +\t\t\tdie_errno(_(\"failed to unlink '%s'\"), path);\n > +\t}\n > +}\n > +#endif\n\n[...]\n\n > +struct dirent_psx *renc_pn_readdir(RENC_FN_DIR *renc_pn_dir)\n > +{\n > +\tstruct dirent *res;\n > +\tres = readdir(renc_pn_dir->dirp);\n > +\tif (res) {\n > +\t\tsize_t namelenz = strlen(res->d_name) + 1; /* \\0 */\n > +\t\tsize_t new_len_needed = 0;\n > +\t\tint ret_errno = errno;\n > +\n > +\t\trenc_pn_dir->dirent_utf8->d_ino\t = res->d_ino;\n > +\t\trenc_pn_dir->dirent_utf8->d_type = res->d_type;\n > +\tdo {\n > +\t\t if (new_len_needed > renc_pn_dir->dirent_utf8->max_name_len) {\nindent\n\n[...]\n\n > diff --git a/environment.c b/environment.c\n > index 85edd7f..ba81575 100644\n > --- a/environment.c\n > +++ b/environment.c\n > @@ -59,6 +59,7 @@ int grafts_replace_parents = 1;\n >   int core_apply_sparse_checkout;\n >   int merge_log_config = -1;\n >   int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n > +const char *wt_path_encoding = NULL;\nindent\n\n >   struct startup_info *startup_info;\n >   unsigned long pack_size_limit_cfg;\n >\n > diff --git a/git-compat-util.h b/git-compat-util.h\n > index 35b095e..877b060 100644\n > --- a/git-compat-util.h\n > +++ b/git-compat-util.h\n > @@ -153,13 +153,21 @@\n >   #endif\n >   #endif\n >\n > -/* used on Mac OS X */\n > -#ifdef PRECOMPOSE_UNICODE\n > -#include \"compat/precompose_utf8.h\"\n > +#if defined(PATH_ENCODING) || defined(PRECOMPOSE_UNICODE)\n > +#include \"compat/reencode_pathname.h\"\n >   #else\n > -#define precompose_str(in,i_nfd2nfc)\n > -#define precompose_argv(c,v)\n > -#define probe_utf8_pathname_composition(a,b)\n > +#define reencode_argv(c,v)\n > +#endif\n > +\n > +/* needed for Mac OS X */\n > +#ifndef PRECOMPOSE_UNICODE\n > +#define probe_utf8_pathname_composition(a,b);\n > +#endif\n > +\n > +#ifndef PATH_ENCODING\n > +#define str_worktree2repolen(in, insz) (NULL)\n > +#define str_repo2worktree(in) (NULL)\n > +#define str_worktree2repo(in) (NULL)\n >   #endif\n >\n >   #ifndef NO_LIBGEN_H\n > diff --git a/parse-options.c b/parse-options.c\n > index c1c66bd..5840c18 100644\n > --- a/parse-options.c\n > +++ b/parse-options.c\n > @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n >   \t\tusage_with_options(usagestr, options);\n >   \t}\n >\n > -\tprecompose_argv(argc, argv);\n > +\treencode_argv(argc, argv);\n >   \treturn parse_options_end(&ctx);\n >   }\n >\n > diff --git a/t/t3911-i18n-filename-8859.sh b/t/t3911-i18n-filename-8859.sh\n > new file mode 100755\n > index 0000000..aa2be57\n > --- /dev/null\n > +++ b/t/t3911-i18n-filename-8859.sh\n > @@ -0,0 +1,251 @@\n > +#!/bin/sh\n > +#\n > +# Copyright (c) 2010 Torsten Bögershausen\n > +#\n > +\n > +test_description='file system encodings UTF-8 ISO8859-1'\n > +\n > +. ./test-lib.sh\n > +\n > +fname_UTF_8=`printf '\\303\\206\\302\\242'`\n > +fname_ISO8859_1=`printf '\\306\\242'`\n > +Euro_utf8=`printf '\\342\\202\\254'`\n > +supportspathencoding=`git config core.supportspathencoding` || :\n > +\n > +\n > +add_file_dir_link() {\n > +\tlocal bname=$1\n > +\tlocal fname=$2\n > +\ttest_expect_success \"add file $fname.f $bname\" '\n > +\t\tgit checkout master &&\n > +\t\tgit checkout -b add_f_$bname &&\n > +\t\t>$fname.f &&\n > +\t\tgit add $fname.f &&\n > +\t\tgit commit -m \"add fname\"\n > +\t'\n > +\n > +\ttest_expect_success \"add dir $fname.d $bname\" '\n > +\t\tgit checkout master &&\n > +\t\tgit checkout -b add_d_$bname &&\n > +\t\tmkdir $fname.d &&\n > +\t\ttouch $fname.d/$fname.f &&\n > +\t\tgit add $fname.d/$fname.f &&\n > +\t\tgit commit -m \"add fname.d/fname\"\n > +\t'\n > +\n > +\ti=0\n > +\tfor src in x $fname; do\n > +\t\tfor dst in x $fname; do\n > +\t\t\ttest_expect_success \"add link $dst.l->$src.f on branch add_l_${i}_$bname\" '\n > +\t\t\t\tgit checkout master &&\n > +\t\t\t\tgit checkout -b add_l_${i}_$bname &&\n > +\t\t\t\tln -s $src.f $dst.l &&\n > +\t\t\t\tgit add $dst.l &&\n > +\t\t\t\tgit commit -m \"add fname.l $i\"\n > +\t\t\t'\n > +\t\t\ti=$(($i+1))\n > +\t\tdone\n > +\tdone\n > +}\n > +\n > +test_expect_success \"setup add rm x\" '\n > +\t>x &&\n > +\tgit add x &&\n > +\tgit commit -m \"1st commit\" &&\n > +\tgit rm x &&\n > +\tgit commit -m \"rm x\"\n > +'\n > +\n > +#combinations to be tested:\n > +# UTF-8     -> ISO8859-1\n > +# ISO8859-1 -> UTF-8\n > +\n > +if test \"$supportspathencoding\"\n > +then\n > +\tsrcencodings=\"ISO8859-1 UTF-8\"\n > +\tfor srcenc in $srcencodings\n > +\tdo\n > +\t\tcase $srcenc in\n > +\t\tISO8859-1)\n > +\t\t\tdstenc=UTF-8\n > +\t\t;;\n > +\t\tUTF-8)\n > +\t\t\tdstenc=ISO8859-1\n > +\t\t;;\n > +\t\tUTF-8-MAC)\n > +\t\t\tdstenc=UTF-8\n > +\t\t;;\n > +\t\t*)\n > +\t\t\techo >&2 \"Wrong encoding $srcenc\"\n > +\t\t\texit 1\n > +\t\t;;\n > +\t\tesac\n > +\t\teval fname_src=\\$fname_$(echo $srcenc | sed -e 's/-/_/g' -e 's/_MAC//')\n > +\t\teval fname_dst=\\$fname_$(echo $dstenc | sed -e 's/-/_/g')\n > +\t\ttest_expect_success \"setup $srcenc\" '\n > +\t\t\tgit checkout master &&\n > +\t\t\tgit config i18n.pathencoding $srcenc\n > +\t\t'\n > +\t\tadd_file_dir_link $srcenc $fname_src\n > +\n > +\t\ttest_expect_success \"setup $dstenc\" '\n > +\t\t\tgit checkout master &&\n > +\t\t\techo \"git checkout Master\" >&2\n > +\t\t\tls -l >&2\n > +\t\t\tgit config i18n.pathencoding $dstenc\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"checkout file $dstenc (was $srcenc)\" '\n > +\t\t\tgit checkout add_f_$srcenc\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"exists file $dstenc (was $srcenc)\" '\n > +\t\t\ttest -f $fname_dst.f\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"log file $dstenc (was $srcenc)\" '\n > +\t\t\tgit log $fname_dst.f\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"git mv\" '\n > +\t\t\tgit checkout -b mv_file_$srcenc &&\n > +\t\t\tgit mv $fname_dst.f XX.f &&\n > +\t\t\tgit commit -m \"git mv fname_dst.f XX.f\"\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"checkout dir $dstenc (was $srcenc)\" '\n > +\t\t\tgit checkout add_d_$srcenc\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"exist dir $dstenc (was $srcenc)\" '\n > +\t\t\ttest -d $fname_dst.d\n > +\t\t'\n > +\n > +\t\ttest_expect_success \"log dir $dstenc (was $srcenc)\" '\n > +\t\t\tgit log $fname_dst.d\n > +\t\t'\n > +\n > +\t\ti=0\n > +\t\tfor src in x $fname_dst; do\n > +\t\t\tfor dst in x $fname_dst; do\n > +\t\t\t\ttest_expect_success \"checkout link $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n > +\t\t\t\t\tgit checkout add_l_${i}_$srcenc\n > +\t\t\t\t'\n > +\t\t\t\ttest_expect_success \"exist link $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n > +\t\t\t\t\ttest -L $dst.l\n > +\t\t\t\t'\n > +\t\t\t\ttest_expect_success \"log link $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n > +\t\t\t\t\tgit log $dst.l\n > +\t\t\t\t'\n > +\t\t\t\ttest_expect_success \"readlink $dst.l->$src.f branch add_l_${i}_$srcenc\" '\n > +\t\t\t\t\techo \"$src.f\" >expect &&\n > +\t\t\t\t\treadlink \"$dst.l\" > actual &&\n > +\t\t\t\t\ttest_cmp expect actual &&\n > +\t\t\t\t\trm expect actual\n > +\t\t\t\t'\n > +\t\t\t\ti=$(($i+1))\n > +\t\t\tdone\n > +\t\tdone\n > +\tdone\n > +\t# Make sure that Euro sign can NOT be checked out in 8859\n\"8859-1\", The euro sign exists in 8859-15.\n\n > +\t#fname_src=Euro\n > +\ttest_expect_success \"setup UTF-8\" '\n > +\t\tgit checkout master &&\n > +\t\tgit config i18n.pathencoding UTF-8\n > +\t'\n > +\tadd_file_dir_link Euro $Euro_utf8\n > +\n > +\ttest_expect_success \"setup ISO8859-1\" '\n > +\t\tgit checkout master &&\n > +\t\trm -rf * &&\n > +\t\tgit config i18n.pathencoding ISO8859-1\n > +\t'\n > +\ttest_expect_success \"checkout file Euro branch add_f_Euro\" '\n > +\t\tgit checkout add_f_Euro\nMissing && ?\n\n > +\t\techo *  >actual &&\n > +\t\techo \"*\" >expect &&\n > +\t\ttest_cmp expect actual &&\n > +\t\trm expect actual\n > +\t'\n > +\n > +\ttest_expect_success \"checkout dir Euro branch add_d_Euro\" '\n > +\t\trm -rf * &&\n > +\t\ttest_must_fail git checkout add_d_Euro\n > +\t'\n > +\n > +\ttest_expect_success \"Cleanup\" '\n > +\t\tgit config i18n.pathencoding UTF-8 &&\n > +\t\tgit checkout master &&\n > +\t\trm -rf * &&\n > +\t\tgit reset --hard &&\n > +\t\tgit config i18n.pathencoding ISO8859-1\n > +\t'\n > +\n > +\ttest_expect_success \"checkout link Euro.l->x.f branch add_l_1_Euro\" '\n > +\t\t! git checkout add_l_1_Euro\n > +\t'\n > +\n > +\ttest_expect_success \"No link Euro.l->x.f\" '\n > +\t\techo *  >actual &&\n > +\t\techo \"*\" >expect &&\n > +\t\ttest_cmp expect actual &&\n > +\t\trm expect actual\n > +\t'\n > +\n > +\ttest_expect_success \"Cleanup after Euro.l->x.f\" '\n > +\t\tgit config i18n.pathencoding UTF-8 &&\n > +\t\tgit checkout master &&\n > +\t\trm -rf * &&\n > +\t\tgit reset --hard &&\n > +\t\tgit config i18n.pathencoding ISO8859-1\n > +\t'\n > +\n > +\t# Checkoing out a soft link pointing to a filename outside\n\"checking\"\n\n > +\t# 8859-1 should fail\n > +\ttest_expect_failure \"checkout link x.l->Euro.f branch add_l_2_Euro\" '\n > +\t\t! git checkout add_l_2_Euro\n > +\t'\n > +\n > +\ttest_expect_success \"No link x.f->Euro.l\" '\n > +\t\techo *  >actual &&\n > +\t\techo \"*\" >expect &&\n > +\t\ttest_cmp expect actual &&\n > +\t\trm expect actual\n > +\t'\n > +\n > +\ttest_expect_success \"Cleanup after link x.l->Euro.f branch\" '\n > +\t\tgit config i18n.pathencoding UTF-8 &&\n > +\t\tgit checkout master &&\n > +\t\trm -rf * &&\n > +\t\tgit reset --hard &&\n > +\t\tgit config i18n.pathencoding ISO8859-1\n > +\t'\n > +\n > +\ttest_expect_success \"checkout link Euro.l->Euro.f branch add_l_3_Euro\" '\n > +\t\t! git checkout add_l_3_Euro\n > +\t'\n > +\n > +\ttest_expect_success \"No link Euro.l->Euro.f\" '\n > +\t\techo *  >actual &&\n > +\t\techo \"*\" >expect &&\n > +\t\ttest_cmp expect actual &&\n > +\t\trm expect actual\n > +\t'\n > +\n > +else\n > +\ttest_expect_success \"setup 8859\" '\n\"8859-1\"\n\n > +\t\tgit config i18n.pathencoding ISO8859-1 &&\n > +\t\tgit checkout -b add_file_8859 &&\n > +\t\t> $fname_src.f &&\n > +\t\tgit add $fname_src.f &&\n > +\t\tgit commit -m \"add fname_src\" &&\n > +\t\tgit config i18n.pathencoding UTF-8 &&\n > +\t\trm -rf * &&\n > +\t\tgit reset --hard\n > +\t'\n > +\ttest_expect_success \"Silent support of pathencoding\" '\n > +\t\ttest_must_fail test -f $fname_UTF_8.f\n > +\t'\n > +fi\n > +\n > +test_done\n\n-- robin\n"},{"id":"198298","messageId":"CACsJy8A1GnhTeMzwXwA1C96pp0ERskxZC=SO+QE2__pfwmChow@mail.gmail.com","threadId":"31402","inReplyTo":"201209010811.33994.tboegi@web.de","subject":"Re: [RFC] i18n.pathencoding","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-09-04T12:23:35Z","receivedAt":"2012-09-04T12:23:35Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Sep 1, 2012 at 1:11 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n> diff --git a/parse-options.c b/parse-options.c\n> index c1c66bd..5840c18 100644\n> --- a/parse-options.c\n> +++ b/parse-options.c\n> @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n>                 usage_with_options(usagestr, options);\n>         }\n>\n> -       precompose_argv(argc, argv);\n> +       reencode_argv(argc, argv);\n>         return parse_options_end(&ctx);\n>  }\n\nIf you have to re-encode command line arguments, what about paths\ncoming --stdin or a file? I guess that paths that are argument to an\noption are not re-encoded too, which may put git in an inconsistent\nstate where it cannot compare paths to paths because some may be\nre-encoded above, some not.\n-- \nDuy\n"},{"id":"198311","messageId":"7vvcftbt8o.fsf@alter.siamese.dyndns.org","threadId":"31402","inReplyTo":"CACsJy8A1GnhTeMzwXwA1C96pp0ERskxZC=SO+QE2__pfwmChow@mail.gmail.com","subject":"Re: [RFC] i18n.pathencoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-04T17:19:51Z","receivedAt":"2012-09-04T17:19:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n\n> On Sat, Sep 1, 2012 at 1:11 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n>> diff --git a/parse-options.c b/parse-options.c\n>> index c1c66bd..5840c18 100644\n>> --- a/parse-options.c\n>> +++ b/parse-options.c\n>> @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n>>                 usage_with_options(usagestr, options);\n>>         }\n>>\n>> -       precompose_argv(argc, argv);\n>> +       reencode_argv(argc, argv);\n>>         return parse_options_end(&ctx);\n>>  }\n>\n> If you have to re-encode command line arguments, what about paths\n> coming --stdin or a file?\n\nThat problem is inherited from the MacOS precompose topic this one\nbuilds on.  Not that it is unimportant to fix, though.\n"},{"id":"198338","messageId":"50465BBC.30203@web.de","threadId":"31402","inReplyTo":"7vvcftbt8o.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] i18n.pathencoding","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-09-04T19:51:24Z","receivedAt":"2012-09-04T19:51:24Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 04.09.12 19:19, Junio C Hamano wrote:\n> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n> \n>> On Sat, Sep 1, 2012 at 1:11 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n>>> diff --git a/parse-options.c b/parse-options.c\n>>> index c1c66bd..5840c18 100644\n>>> --- a/parse-options.c\n>>> +++ b/parse-options.c\n>>> @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n>>>                 usage_with_options(usagestr, options);\n>>>         }\n>>>\n>>> -       precompose_argv(argc, argv);\n>>> +       reencode_argv(argc, argv);\n>>>         return parse_options_end(&ctx);\n>>>  }\n>>\n>> If you have to re-encode command line arguments, what about paths\n>> coming --stdin or a file?\n> \n> That problem is inherited from the MacOS precompose topic this one\n> builds on.  Not that it is unimportant to fix, though.\n\nThanks for the comments, \n(actually the precompose feature started as \"fully\" reencode,\nand was downsized to only do the readdir() and argv[] conversion)\n\nThis leads to 2 questions:\na) Do we want the reencode feature in git, so that I continue to work on it?\n   From a performance point of view that would probably OK:\n   The \"git status\" on a linux tree was slightly slower on my PC when measured with time.\n   From the user experience there was not a difference.\nb) If yes,\n   I have to admit that I don't use paths from --stdin or file so much,\n   except \"git am\" or \"git format-patch\"\n   Which commands are affected?\n   \n"},{"id":"198339","messageId":"7vy5kp8s3h.fsf@alter.siamese.dyndns.org","threadId":"31402","inReplyTo":"50465BBC.30203@web.de","subject":"Re: [RFC] i18n.pathencoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-04T20:12:50Z","receivedAt":"2012-09-04T20:12:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> This leads to 2 questions:\n> a) Do we want the reencode feature in git, so that I continue to work on it?\n>    From a performance point of view that would probably OK:\n>    The \"git status\" on a linux tree was slightly slower on my PC when measured with time.\n>    From the user experience there was not a difference.\n\nI am not fundamentally opposed to such a change, as long as the\nchange does not affect performance at all when the feature is not\nused, and the resulting code does not become too ugly.\n\nUse of the \"reencode\" feature may have to make things slow, but you\nhave to spend some cycle to do what the feature has to do anyway,\nso...\n\n> b) If yes,\n>    I have to admit that I don't use paths from --stdin or file so much,\n>    except \"git am\" or \"git format-patch\"\n>    Which commands are affected?\n\n$ git grep -l -e '--stdin' Documentation/git*\n\nmay be a good starting point.\n"},{"id":"198369","messageId":"CACsJy8ARmPwELFhdvu_Y2kZNbYNgTzoqLzriH2oB7WtxM8D7hg@mail.gmail.com","threadId":"31402","inReplyTo":"7vvcftbt8o.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] i18n.pathencoding","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-09-05T11:11:12Z","receivedAt":"2012-09-05T11:11:12Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Sep 5, 2012 at 12:19 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n>> On Sat, Sep 1, 2012 at 1:11 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n>>> @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n>>>                 usage_with_options(usagestr, options);\n>>>         }\n>>>\n>>> -       precompose_argv(argc, argv);\n>>> +       reencode_argv(argc, argv);\n>>>         return parse_options_end(&ctx);\n>>>  }\n>>\n>> If you have to re-encode command line arguments, what about paths\n>> coming --stdin or a file?\n>\n> That problem is inherited from the MacOS precompose topic this one\n> builds on.  Not that it is unimportant to fix, though.\n\nOn fixing that. 76759c7 describes the change as:\n\n    The argv[] conversion allows to use the TAB filename completion done\n    by the shell on command line.  It tolerates other tools which use\n    readdir() to feed decomposed file names into git.\n\nHow come git's internal conversion helps TAB filename completion,\nwhich is done before git is executed? If it's the pathname output from\ngit that helps tab completion, then perhaps we should convert at\noutput time rather than input, by piping all the output through a\nconverter when we know the output is textual. That keeps all paths in\nthe same encoding internally (I hope).\n\nJust two cents. And I'm not against i18n.pathencoding or anything.\n-- \nDuy\n"},{"id":"198400","messageId":"5047ACE7.3050000@web.de","threadId":"31402","inReplyTo":"CACsJy8ARmPwELFhdvu_Y2kZNbYNgTzoqLzriH2oB7WtxM8D7hg@mail.gmail.com","subject":"Re: [RFC] i18n.pathencoding","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-09-05T19:49:59Z","receivedAt":"2012-09-05T19:49:59Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 05.09.12 13:11, Nguyen Thai Ngoc Duy wrote:\n> On Wed, Sep 5, 2012 at 12:19 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n>>> On Sat, Sep 1, 2012 at 1:11 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n>>>> @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n>>>>                 usage_with_options(usagestr, options);\n>>>>         }\n>>>>\n>>>> -       precompose_argv(argc, argv);\n>>>> +       reencode_argv(argc, argv);\n>>>>         return parse_options_end(&ctx);\n>>>>  }\n>>>\n>>> If you have to re-encode command line arguments, what about paths\n>>> coming --stdin or a file?\n>>\n>> That problem is inherited from the MacOS precompose topic this one\n>> builds on.  Not that it is unimportant to fix, though.\n> \n> On fixing that. 76759c7 describes the change as:\n> \n>     The argv[] conversion allows to use the TAB filename completion done\n>     by the shell on command line.  It tolerates other tools which use\n>     readdir() to feed decomposed file names into git.\n> \n> How come git's internal conversion helps TAB filename completion,\n> which is done before git is executed?\n\nBecause the shell use readdir() itself when <TAB> is pressed\n\n$echo Bö > Bö\n$echo Ba > Ba\n$ls \nBo ̈\nBa\n\n# The terminal program of Mac OS shows the decomposed \"ö\" correctly,\n\"Bö\" is shown as \"Bö\", while xterm shows \"Bö\" as \"Bo ̈\",\nthat is \"o\" followod by a \"combining diaresis\".\n\nThe shell seems to do the same:\nto add Bö to git,\n\n$git add Bo<TAB>\n\nneeds to be used.\nAnd as a result, the sheel will pass the \"o\" in decomposed unicode to git,\nwhich is corrected in precompose_argv().\n"},{"id":"198401","messageId":"5047AD8A.3020203@web.de","threadId":"31402","inReplyTo":"7vy5kp8s3h.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] i18n.pathencoding","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-09-05T19:52:42Z","receivedAt":"2012-09-05T19:52:42Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 04.09.12 22:12, Junio C Hamano wrote:\n> Torsten Bögershausen <tboegi@web.de> writes:\n> \n>> This leads to 2 questions:\n>> a) Do we want the reencode feature in git, so that I continue to work on it?\n>>    From a performance point of view that would probably OK:\n>>    The \"git status\" on a linux tree was slightly slower on my PC when measured with time.\n>>    From the user experience there was not a difference.\n> \n> I am not fundamentally opposed to such a change, as long as the\n> change does not affect performance at all when the feature is not\n> used, and the resulting code does not become too ugly.\n> \n> Use of the \"reencode\" feature may have to make things slow, but you\n> have to spend some cycle to do what the feature has to do anyway,\n> so...\n> \n>> b) If yes,\n>>    I have to admit that I don't use paths from --stdin or file so much,\n>>    except \"git am\" or \"git format-patch\"\n>>    Which commands are affected?\n> \n> $ git grep -l -e '--stdin' Documentation/git*\n> \n> may be a good starting point.\n\n\n------------------------------\nI'll have a look into the --stdin stuff later, thanks Junio\n-------------------------------\n\nAnd here some benchmarks on my PC,\nfirst run is using git v1.7.12,\nsecond run with i18n.pathencoding applied \n===================\n for f in 1 2 3 4 5; do time git status  ; done 2>&1 | egrep \"^user|^real|^sys\"\nreal    0m0.492s\nuser    0m0.300s\nsys     0m0.200s\nreal    0m0.448s\nuser    0m0.260s\nsys     0m0.190s\nreal    0m0.443s\nuser    0m0.250s\nsys     0m0.200s\nreal    0m0.451s\nuser    0m0.260s\nsys     0m0.190s\nreal    0m0.429s\nuser    0m0.230s\nsys     0m0.200s\n=========================\nfor f in 1 2 3 4 5; do time ~/projects/git/tb.localfname/git status  ; done 2>&1 | egrep \"^user|^real|^sys\"\nreal    0m0.580s\nuser    0m0.320s\nsys     0m0.230s\nreal    0m0.461s\nuser    0m0.260s\nsys     0m0.200s\nreal    0m0.463s\nuser    0m0.250s\nsys     0m0.220s\nreal    0m0.445s\nuser    0m0.290s\nsys     0m0.160s\nreal    0m0.443s\nuser    0m0.240s\nsys     0m0.210s\n=======================\n\nIt seems that we have a little penalty on performance.\nI'll have a look at the readdir() function.\n"},{"id":"198413","messageId":"7vwr076dg7.fsf@alter.siamese.dyndns.org","threadId":"31402","inReplyTo":"5047ACE7.3050000@web.de","subject":"Re: [RFC] i18n.pathencoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-09-06T03:24:24Z","receivedAt":"2012-09-06T03:24:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> On 05.09.12 13:11, Nguyen Thai Ngoc Duy wrote:\n>> On Wed, Sep 5, 2012 at 12:19 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>>> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n>>>> On Sat, Sep 1, 2012 at 1:11 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n>>>>> @@ -476,7 +476,7 @@ int parse_options(int argc, const char **argv, const char *prefix,\n>>>>>                 usage_with_options(usagestr, options);\n>>>>>         }\n>>>>>\n>>>>> -       precompose_argv(argc, argv);\n>>>>> +       reencode_argv(argc, argv);\n>>>>>         return parse_options_end(&ctx);\n>>>>>  }\n>>>>\n>>>> If you have to re-encode command line arguments, what about paths\n>>>> coming --stdin or a file?\n>>>\n>>> That problem is inherited from the MacOS precompose topic this one\n>>> builds on.  Not that it is unimportant to fix, though.\n>> \n>> On fixing that. 76759c7 describes the change as:\n>> \n>>     The argv[] conversion allows to use the TAB filename completion done\n>>     by the shell on command line.  It tolerates other tools which use\n>>     readdir() to feed decomposed file names into git.\n>> \n>> How come git's internal conversion helps TAB filename completion,\n>> which is done before git is executed?\n>\n> Because the shell use readdir() itself when <TAB> is pressed\n>\n> $echo Bö > Bö\n> $echo Ba > Ba\n> $ls \n> Bo ̈\n> Ba\n>\n> # The terminal program of Mac OS shows the decomposed \"ö\" correctly,\n> \"Bö\" is shown as \"Bö\", while xterm shows \"Bö\" as \"Bo ̈\",\n> that is \"o\" followod by a \"combining diaresis\".\n>\n> The shell seems to do the same:\n> to add Bö to git,\n>\n> $git add Bo<TAB>\n>\n> needs to be used.\n> And as a result, the sheel will pass the \"o\" in decomposed unicode to git,\n> which is corrected in precompose_argv().\n\nWell explained; this does not have anything to do with what Git\noutputs.\n"},{"id":"198563","messageId":"504B194A.6050009@web.de","threadId":"31402","inReplyTo":"5043E4B4.9050801@dewire.com","subject":"Re: [RFC] i18n.pathencoding","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2012-09-08T10:09:14Z","receivedAt":"2012-09-08T10:09:14Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 03.09.12 00:59, Robin Rosenberg wrote:\n> Torsten Bögershausen skrev 2012-09-01 08.11:> Allow path names to be encoded in UTF-8 in the repository\n>> and checkout out as e.g. ISO-8859-1 in the working tree.\n> \n> Ack for attempting this.\n> \n> Did it myself if 2007, but times weren't ripe then, I guess.\n> \n>> +i18n.pathEncoding::\n>> +    This option is only used by some implementations of git.\n>> +    When \"git init\" sets core.supportspathencoding to true,\n>> +    i18n.pathEncoding can be set to re-encode path names when\n>> +    a working tree is checked out.\n>> +    Path names may be e.g. encoded in ISO-8859-1 and are stored as\n>> +    UTF-8 encoded in the repository.\n>> +    When not set, the encoding of path names is the same in working tree\n>> +    and the repository.\n> \n> \"If set, then core.precomposeunicode is ignored on Mac OS X.\"\n\nThanks for comments, \nI will integrate them (but don't expect that too soon ;-).\n\nBefore that I want to find & fix the performance issues, and the\nt1450 breakage asks for my attention again.\n\nHowever, the combination of i18n.pathEncoding and core.precomposeunicode under Mac OS X \nneeds to be clarified.\nMac OS X only allows unicode (an \"Ä\" encoded in ISO-8859-1 gives a single byte),\nand the HFS+ filesystem will list it as %C4.\n\nSo the i18.pathencoding does not make sence under Mac OS X, and should be ignored.\nIt is not ignored today, since the characters use in t3911 are choosen to be both valid UTF-8\nand ISO-8859-1 code points/characters, so that the feature can be tested on Mac OS X.\n\n\n\n\n\n  \n"}]}