{"thread":{"id":"19325","subject":"[RFC 1/8] UTF helpers","startedAt":"2009-05-12T22:50:23Z","lastAt":"2009-05-14T13:57:30Z","messageCount":20,"participants":["Robin Rosenberg","Johannes Schindelin","Esko Luontola","Andreas Ericsson","Alex Riesen","Junio C Hamano","Jay Soffian"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"113700","messageId":"1242168631-30753-1-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":null,"subject":"[RFC 0/8] Antique UTF-8 filename support","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:23Z","receivedAt":"2009-05-12T22:50:23Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"From: Robin Rosenberg <robin.rosenberg@gmail.com>\n\nSince there is some interest in the topic, now, I can republish my old 2 ½ year old\npatches so there is some real code to comment on. They apply on top of\n6dcfa306f2b67b733a7eb2d7ded1bc9987809edb, For completness I send\nall patches, but the interesing stuff is in patch 4 and 5. Beware of encoding\nissues with the test cases.\n\nThey do not handle Windows UTF-16 at all, but I think that is just a matter of writing\nwindows specifc wrappers for the filename and directory handling routines.\n\nFeel free to rewamp and steal ideas and add constructive criticism. Don't even \nthink of cherry-picking and rebasing, It's careful handpicking with copy/paste at \nbest, but mostly it's fuel for discussions.\n\nI'd admit some parts are quite kludgy and probably slow. as I was primarily \ninterested to see if it was even feasible, which it was. however there was simply\nno interest, which meant there was no point in optimizing it. It was simply the\nwrong problem at the time.\n\nDisclaimer: A problem with this approach is that, although it does character\nconversion, if you are on a non-UTF-8 locale it will not let you mange\nany repository. That is basically impossible and hence not the goal. It does\nhelp people with the same (or close) languages to cooperate without enforcing\na common encoding as long as stick to the common characters, i.e. the ones\nthat can be converted between the locales involved.\n\nThis is probably the most out-dated patch series ever. \n\n-- robin\n\nRobin Rosenberg (8):\n(mostly obsolete)\n  UTF helpers\n  Messages in locale.\n  Extend tests to cover locale wrt to commit messages.\n\nThe interesing stuff (patch 4 & 5)\n  UTF file names.\n  Extend all tests to work on UTF-8 filenames.\n\nold wip\n  test of utf_locallinks\n  Convert symlink dest in diff\n  UTF-8 in non-SHA1-objects\n\n Makefile                            |    8 +-\n builtin-add.c                       |    5 +-\n builtin-cat-file.c                  |    6 +-\n builtin-checkout-index.c            |   46 +++-\n builtin-commit-tree.c               |    9 +-\n builtin-ls-files.c                  |   26 ++-\n builtin-ls-tree.c                   |   16 +-\n builtin-rev-parse.c                 |    7 +-\n builtin-update-index.c              |   18 +-\n builtin-write-tree.c                |    5 +-\n diff.c                              |  111 ++++++--\n dir.c                               |   22 +-\n git-commit.sh                       |    5 +\n git-compat-util.h                   |   43 +++\n git-rebase.sh                       |    1 +\n git.c                               |    9 +\n log-tree.c                          |    4 +-\n merge-index.c                       |   25 ++-\n read-cache.c                        |    8 +-\n refs.c                              |   11 +-\n setup.c                             |   28 ++-\n t/lib-read-tree-m-3way.sh           |   38 ++--\n t/t-utf-filenames.sh                |   95 +++++++\n t/t-utf-msg.sh                      |   43 +++\n t/t0000-basic.sh                    |  117 ++++----\n t/t0010-racy-git.sh                 |   10 +-\n t/t1000-read-tree-m-3way.sh         |  240 +++++++++---------\n t/t1001-read-tree-m-2way.sh         |   56 ++--\n t/t1020-subdirectory.sh             |   63 +++---\n t/t1100-commit-tree-options.sh      |   12 +-\n t/t1400-update-ref.sh               |   10 +-\n t/t2000-checkout-cache-clash.sh     |   18 +-\n t/t2001-checkout-cache-clash.sh     |   30 +-\n t/t2002-checkout-cache-u.sh         |    8 +-\n t/t2003-checkout-cache-mkdir.sh     |  118 ++++----\n t/t2004-checkout-cache-temp.sh      |  144 +++++-----\n t/t2100-update-cache-badpath.sh     |   48 ++--\n t/t2101-update-index-reupdate.sh    |   56 ++--\n t/t3000-ls-files-others.sh          |   36 ++--\n t/t3002-ls-files-dashpath.sh        |   24 +-\n t/t3010-ls-files-killed-modified.sh |  104 ++++----\n t/t3020-ls-files-error-unmatch.sh   |   10 +-\n t/t3100-ls-tree-restrict.sh         |  122 +++++-----\n t/t3101-ls-tree-dirname.sh          |   88 +++---\n t/t3400-rebase.sh                   |   18 +-\n t/t3401-rebase-partial.sh           |   24 +-\n t/t3402-rebase-merge.sh             |   17 +-\n t/t3403-rebase-skip.sh              |   10 +-\n t/t3500-cherry.sh                   |   26 +-\n t/t3600-rm.sh                       |   28 +-\n t/t3700-add.sh                      |   30 +-\n t/t4000-diff-format.sh              |   26 +-\n t/t4001-diff-rename.sh              |   20 +-\n t/t4002-diff-basic.sh               |  160 ++++++------\n t/t4003-diff-rename-1.sh            |   66 +++---\n t/t4004-diff-rename-symlink.sh      |   40 ++--\n t/t4005-diff-rename-2.sh            |   54 ++--\n t/t4006-diff-mode.sh                |   14 +-\n t/t4008-diff-break-rewrite.sh       |  100 ++++----\n t/t4009-diff-rename-4.sh            |   63 +++---\n t/t4011-diff-symlink.sh             |   38 ++--\n t/t4012-diff-binary.sh              |   16 +-\n t/t7301-rev-parse.sh                |   20 ++\n t/test-lib.sh                       |   13 +-\n test-utf.c                          |   61 +++++\n utf.c                               |  501 +++++++++++++++++++++++++++++++++++\n utf.h                               |   27 ++\n 67 files changed, 2133 insertions(+), 1142 deletions(-)\n create mode 100755 t/t-utf-filenames.sh\n create mode 100755 t/t-utf-msg.sh\n create mode 100755 t/t7301-rev-parse.sh\n create mode 100644 test-utf.c\n create mode 100644 utf.c\n create mode 100644 utf.h\n"},{"id":"113699","messageId":"1242168631-30753-2-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-1-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 1/8] UTF helpers","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:24Z","receivedAt":"2009-05-12T22:50:24Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n Makefile          |    8 ++-\n git-compat-util.h |    1 +\n git.c             |    9 +++\n t/test-lib.sh     |    4 +-\n test-utf.c        |   61 ++++++++++++++++\n utf.c             |  207 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n utf.h             |   27 +++++++\n 7 files changed, 313 insertions(+), 4 deletions(-)\n create mode 100644 test-utf.c\n create mode 100644 utf.c\n create mode 100644 utf.h\n\ndiff --git a/Makefile b/Makefile\nindex 2d62efb..2d71f01 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -259,7 +259,7 @@ LIB_OBJS = \\\n \tobject.o pack-check.o patch-delta.o path.o pkt-line.o sideband.o \\\n \tquote.o read-cache.o refs.o run-command.o dir.o object-refs.o \\\n \tserver-info.o setup.o sha1_file.o sha1_name.o strbuf.o \\\n-\ttag.o tree.o usage.o config.o environment.o ctype.o copy.o \\\n+\ttag.o tree.o utf.o usage.o config.o environment.o ctype.o copy.o \\\n \tfetch-clone.o revision.o pager.o tree-walk.o xdiff-interface.o \\\n \twrite_or_die.o trace.o list-objects.o grep.o \\\n \talloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \\\n@@ -564,6 +564,9 @@ ifdef NO_ACCURATE_DIFF\n endif\n \n # Shell quote (do not use $(call) to accommodate ancient setups);\n+ALL_CFLAGS += -DUTF8INTERNAL=1\n+ALL_CFLAGS += -DDEBUG=1\n+#ALL_CFLAGS += -DTEST=1\n \n SHA1_HEADER_SQ = $(subst ','\\'',$(SHA1_HEADER))\n \n@@ -811,6 +814,9 @@ export NO_SVN_TESTS\n test: all\n \t$(MAKE) -C t/ all\n \n+test-utf$X: test-utf.c ctype.o utf.o usage.o\n+\t$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) test-utf.c utf.c ctype.o usage.o\n+\n test-date$X: test-date.c date.o ctype.o\n \t$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) test-date.c date.o ctype.o\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 0272d04..f83352b 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -25,6 +25,7 @@\n #include <netinet/in.h>\n #include <sys/types.h>\n #include <dirent.h>\n+#include \"utf.h\"\n \n /* On most systems <limits.h> would have given us this, but\n  * not on some systems (e.g. GNU/Hurd).\ndiff --git a/git.c b/git.c\nindex 6475847..bd4e726 100644\n--- a/git.c\n+++ b/git.c\n@@ -272,6 +272,15 @@ static void handle_internal_command(int argc, const char **argv, char **envp)\n \t};\n \tint i;\n \n+#ifdef DEBUG\n+\tif (debug()) {\n+\t\tfprintf(stderr,\"GIT-\");\n+\t\tfor (i = 1; i<argc; ++i)\n+\t\t\tfprintf(stderr,\"%s\",argv[i]);\n+\t\tfprintf(stderr,\"\\n\");\n+\t}\n+#endif\n+\n \t/* Turn \"git cmd --help\" into \"git help cmd\" */\n \tif (argc > 1 && !strcmp(argv[1], \"--help\")) {\n \t\targv[1] = argv[0];\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 07cb706..e8aefd8 100755\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -4,11 +4,9 @@\n #\n \n # For repeatability, reset the environment to known value.\n-LANG=C\n-LC_ALL=C\n PAGER=cat\n TZ=UTC\n-export LANG LC_ALL PAGER TZ\n+export PAGER TZ\n EDITOR=:\n VISUAL=:\n unset AUTHOR_DATE\ndiff --git a/test-utf.c b/test-utf.c\nnew file mode 100644\nindex 0000000..133eea0\n--- /dev/null\n+++ b/test-utf.c\n@@ -0,0 +1,61 @@\n+#include <stdio.h>\n+#include <time.h>\n+#include <assert.h>\n+\n+#include \"cache.h\"\n+#include \"utf.h\"\n+\n+int main(int argc, char **argv)\n+{\n+\tint i;\n+\n+#if 0\n+\tfor (i = 1; i < argc; i++) {\n+\t\tchar result1[100];\n+\t\tchar result2[100];\n+\n+\t\tutfcpy(result1, argv[i], strlen(argv[i])+1);\n+\t\tlocalcpy(result2, result1, strlen(result1)+1);\n+\n+\t\tprintf(\"%s -> %s -> %s\\n\", argv[i], result1, result2);\n+\t}\n+\treturn 0;\n+#endif\n+\n+#define test(name) case __LINE__: current_name=name; n++; printf(\"Testing case #%d: %s\\n\", n, current_name);\n+#define end_test break;\n+#define begin_suite() char *current_name=0; int n=1; for (i=0; i<1000; ++i) { switch(i) { \n+#define concats(a,b) #a #b\n+\n+#undef strcmp\n+#define assertStringEquals(a,b) assert(#a #b && strcmp(a,b)==0)\n+#define assertIntEquals(a,b) assert(#a #b && (a)==(b))\n+\n+#define end_suite() }}\n+\n+\tbegin_suite();\n+\n+\ttest(\"utfcpy\") {\n+\t  char result[100];\n+\t  utfcpy(result,\"Ändrad\",7);\n+\t  assertStringEquals(result,\"\\303\\204ndrad\");\n+\t} end_test;\n+\n+\ttest(\"utflen\") {\n+\t  int result=utflen(\"Ändrad\",7);\n+\t  assertIntEquals(result,8);\n+\t} end_test;\n+\n+\ttest(\"localcpy\") {\n+\t  char result[100];\n+\t  localcpy(result,\"\\303\\204ndrad\",8);\n+\t  assertStringEquals(result,\"Ändrad\");\n+\t} end_test;\n+\n+\ttest(\"locallen\") {\n+\t  int result=locallen(\"\\303\\204ndrad\",8);\n+\t  assertIntEquals(result,7);\n+\t} end_test;\n+\n+\tend_suite();\n+}\ndiff --git a/utf.c b/utf.c\nnew file mode 100644\nindex 0000000..eb430b2\n--- /dev/null\n+++ b/utf.c\n@@ -0,0 +1,207 @@\n+#undef UTF8INTERNAL\n+\n+#include <langinfo.h>\n+#include <iconv.h>\n+#include \"cache.h\"\n+#include <locale.h>\n+#include <stdarg.h>\n+\n+static iconv_t local_to_utf8 = (iconv_t)-1;\n+static iconv_t utf8_to_local = (iconv_t)-1;\n+static iconv_t utf8_to_utf8 = (iconv_t)-1;\n+static int same = 0;\n+\n+#if TEST\n+#define die printf\n+#endif\n+\n+static void\tinitlocale()\n+{\n+#ifndef NO_ICONV\n+\tif (!same && local_to_utf8 == (iconv_t)-1) {\n+\t\tsetlocale(LC_CTYPE, \"\");\n+\t\tchar *local_encoding = nl_langinfo(CODESET);\n+#ifdef DEBUG\n+\t\tif (debug()) fprintf(stderr,\"encoding=%s\\n\", local_encoding);\n+#endif\n+\t\tif (strcmp(local_encoding,\"UTF-8\") == 0) {\n+\t\t\tsame = 1;\n+\t\t\treturn;\n+\t\t}\n+\t\tlocal_to_utf8 = iconv_open(\"UTF-8\",  local_encoding);\n+\t\tif (local_to_utf8 == (iconv_t)-1) {\n+\t\t\tdie(\"cannot setup locale conversion from %s: %s\", local_encoding, strerror(errno));\n+\t\t}\n+#ifdef DEBUG\n+\t\tif (debug()) fprintf(stderr,\"utf8_to_local = iconv_open(%s,UTF-8)\\n\",local_encoding);\n+#endif\n+\t\tutf8_to_local = iconv_open(local_encoding,  \"UTF-8\");\n+\t\tif (utf8_to_local == (iconv_t)-1) {\n+\t\t\tdie(\"cannot setup locale conversion from %s: %s\", local_encoding, strerror(errno));\n+\t\t}\n+\n+\t\tutf8_to_utf8 = iconv_open(\"UTF-8\",\"UTF-8\");\n+\t\tif (utf8_to_utf8 == (iconv_t)-1) {\n+\t\t\tdie(\"cannot setup locale conversion from UTF-8 to UTF-8: %s\",strerror(errno));\n+\t\t}\n+\t}\n+#endif\n+}\n+\n+int maybe_utf8(const char *local, size_t len)\n+{\n+  char *self = xcalloc(1,len+1);\n+  char *selfp = self;\n+  size_t outlen = len+1;\n+  int ret = iconv(utf8_to_utf8, (char**)&local, &len, &selfp, &outlen);\n+  free(self);\n+  P((\"maybelocal: %0.*s %s\\n\", len, local, ret!=-1 ? \"yes\" : \"no\"));\n+  return ret != -1;\n+}\n+\n+size_t utflen(const char *local, size_t locallen)\n+{\n+#ifndef NO_ICONV\n+\tinitlocale();\n+\tif (same) {\n+\t\treturn locallen;\n+\t}\n+\tif (maybe_utf8(local, locallen))\n+\t\treturn locallen;\n+\n+\tsize_t outlen=locallen*6;\n+\tchar *outbuf=xcalloc(outlen,1);\n+\tchar *out=outbuf;\n+\ticonv(local_to_utf8, NULL, NULL, NULL, NULL);\n+\tconst char *vlocal = local;\n+\tsize_t vlocallen = locallen;\n+\tif (iconv(local_to_utf8,  (char**)&vlocal,  &vlocallen,  &out,  &outlen) == -1) {\n+#if TEST\n+\t\tperror(\"failed\");\n+#endif\n+\t\tfree(outbuf);\n+\t\treturn locallen;\n+\t}\n+\t*out = 0;\n+\tfree(outbuf);\n+\treturn locallen*6 - outlen;\n+#else\n+\treturn locallen;\n+#endif\n+}\n+\n+/* Copy and transform */\n+void utfcpy(char *to_utf, char *from_local, size_t localsize)\n+{\n+#ifdef DEBUG\n+\tchar *a=to_utf,*b=from_local;\n+#endif\n+#ifndef NO_ICONV\n+\tinitlocale();\n+\tif (same) {\n+\t\tmemcpy(to_utf, from_local, localsize);\n+\t\treturn;\n+\t}\n+\tif (maybe_utf8(from_local, localsize)) {\n+\t\tmemcpy(to_utf, from_local, localsize);\n+\t\treturn;\n+\t}\n+\n+\tsize_t outlen=localsize*6;\n+\ticonv(local_to_utf8, NULL, NULL, NULL, NULL);\n+\tchar *vfrom_local = from_local;\n+\tchar *vto_utf = to_utf;\n+\tsize_t vlocalsize = localsize;\n+\tif (iconv(local_to_utf8,  &vfrom_local,  &vlocalsize,  &vto_utf,  &outlen) == -1) {\n+\t\tfprintf(stderr,\"Failed to convert %0.*s to UTF\\n\", localsize, from_local);\n+\t\tmemcpy(to_utf,  from_local,  localsize);\n+\t}\n+#else\n+\tmemcpy(to_utf, from_local, localsize);\n+#endif\n+#ifdef DEBUG\n+\tif (debug()) fprintf(stderr,\"%0.*s ->UTF %0.*s\\n\", localsize, b, localsize*6 - outlen, a);\n+#endif\n+}\n+\n+size_t locallen(const char *utf, size_t utflen)\n+{\n+#ifndef NO_ICONV\n+\tinitlocale();\n+\tif (same) {\n+\t\treturn utflen;\n+\t}\n+\tchar *outbuf=xcalloc(utflen*4,1); /* ??, can we be more specific? */\n+\tchar *out=outbuf;\n+\tsize_t outlen=utflen*4;\n+\ticonv(utf8_to_local, NULL, NULL, NULL, NULL);\n+\tchar *vutf = utf;\n+\tsize_t vutflen = utflen;\n+\tif (iconv(utf8_to_local,  (char**)&vutf,  &vutflen,  &out,  &outlen) == -1) {\n+#ifdef DEBUG\n+\t\tperror(\"failed\");\n+#endif\n+\t\tfree(outbuf);\n+\t\treturn utflen;\n+\t}\n+\t*out = 0;\n+\tfree(outbuf);\n+\treturn utflen*4 - outlen; \t\n+#else\n+\treturn utflen;\n+#endif\n+}\n+\n+void localcpy(char *tolocal, char *fromutf, size_t utflen)\n+{\n+#ifdef DEBUG\n+\tchar *a=tolocal,*b=fromutf;\n+#endif\n+\tinitlocale();\n+\tif (same) {\n+\t\tmemcpy(tolocal, fromutf, utflen);\n+\t\treturn;\n+\t}\n+#ifndef NO_ICONV\n+\ticonv(utf8_to_local,  NULL,  NULL,  NULL,  NULL);\n+\tsize_t outlen=utflen*4;\n+\tchar *vfromutf = fromutf;\n+\tchar *vtolocal = tolocal;\n+\tsize_t vutflen = utflen;\n+\tif (iconv(utf8_to_local,  &vfromutf,  &vutflen,  &vtolocal,  &outlen) == -1) {\n+\t\tfprintf(stderr,\"Failed to convert %0.*s to LOCAL\\n\", utflen, fromutf);\n+\t\tmemcpy(tolocal, fromutf, utflen);\n+\t}\n+#else\n+\tmemcpy(tolocale, fromutf, utflen);\n+#endif\t\n+#ifdef DEBUG\n+\tif (debug()) fprintf(stderr,\"%0.*s ->LOCAL %0.*s\\n\", utflen, b, utflen*4-outlen, a);\n+#endif\n+}\n+\n+int PP(const char *fmt,...)\n+{\n+  va_list va;\n+  va_start(va,fmt);\n+  int ret=vfprintf(stderr,fmt,va);\n+  va_end(va);\n+  return ret;\n+}\n+\n+int debugf=-1;\n+\n+int debug()\n+{\n+\tif (debugf == -1) {\n+\t\tchar *f = getenv(\"DEBUG\");\n+\t\tif (!f) {\n+\t\t\tdebugf = 0;\n+\t\t} else if (f[0] != 0) {\n+\t\t\tdebugf = 1;\n+\t\t} else\n+\t\t\tdebugf = 0;\n+\t}\n+\treturn debugf == 1;\n+}\n+\ndiff --git a/utf.h b/utf.h\nnew file mode 100644\nindex 0000000..c6c6224\n--- /dev/null\n+++ b/utf.h\n@@ -0,0 +1,27 @@\n+#ifndef UTF_H\n+#define UTF_H 1\n+\n+/** The number of octets 'local' would occupy encoded as utf8.\n+ *  The input format is assumed to be local\n+ */\n+extern size_t utflen(const char *local,size_t locallen);\n+extern size_t locallen(const char *utf,size_t utflen);\n+\n+/* Copy and transform */\n+extern void utfcpy(char *toutf,char *fromlocal,size_t localen);\n+\n+/* Copy and transform */\n+extern void localcpy(char *tolocal,char *fromutf,size_t utflen);\n+\n+#ifdef DEBUG\n+#define D(x) do { if (debug()) fprintf(stderr,\"%s:%d:%s\\n\",__FILE__,__LINE__,x); } while(0)\n+#define P(x) do { if (debug()) { fprintf(stderr,\"%s:%d:\",__FILE__,__LINE__); PP x; } } while(0)\n+int PP(const char *fmt,...);\n+int debug();\n+\n+#else\n+#define D(x)\n+#define P(x)\n+#endif\n+\n+#endif\n-- \n1.6.3.dirty\n"},{"id":"113703","messageId":"1242168631-30753-3-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-2-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 2/8] Messages in locale.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:25Z","receivedAt":"2009-05-12T22:50:25Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n builtin-cat-file.c    |    6 +++++-\n builtin-commit-tree.c |    9 ++++++---\n git-rebase.sh         |    1 +\n log-tree.c            |    4 +++-\n refs.c                |   11 ++++++++---\n t/t-utf-msg.sh        |   43 +++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 66 insertions(+), 8 deletions(-)\n create mode 100755 t/t-utf-msg.sh\n\ndiff --git a/builtin-cat-file.c b/builtin-cat-file.c\nindex 6c16bfa..ff275bf 100644\n--- a/builtin-cat-file.c\n+++ b/builtin-cat-file.c\n@@ -145,6 +145,10 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)\n \tif (!buf)\n \t\tdie(\"git-cat-file %s: bad file\", argv[2]);\n \n-\twrite_or_die(1, buf, size);\n+\tsize_t localsize = locallen(buf,size);\n+\tchar *localbuf = xcalloc(localsize+1,1);\n+\tlocalcpy(localbuf, buf, size+1);\n+\twrite_or_die(1, localbuf, localsize);\n+\tfree(localbuf);\n \treturn 0;\n }\ndiff --git a/builtin-commit-tree.c b/builtin-commit-tree.c\nindex e2e690a..8d87ec7 100644\n--- a/builtin-commit-tree.c\n+++ b/builtin-commit-tree.c\n@@ -23,16 +23,19 @@ static void init_buffer(char **bufp, unsigned int *sizep)\n static void add_buffer(char **bufp, unsigned int *sizep, const char *fmt, ...)\n {\n \tchar one_line[2048];\n+\tchar one_line_utf[2048];\n \tva_list args;\n-\tint len;\n+\tint len,len_utf;\n \tunsigned long alloc, size, newsize;\n \tchar *buf;\n \n \tva_start(args, fmt);\n \tlen = vsnprintf(one_line, sizeof(one_line), fmt, args);\n \tva_end(args);\n+\tutfcpy(one_line_utf, one_line, len + 1);\n+\tlen_utf = strlen(one_line_utf);\n \tsize = *sizep;\n-\tnewsize = size + len;\n+\tnewsize = size + len_utf;\n \talloc = (size + 32767) & ~32767;\n \tbuf = *bufp;\n \tif (newsize > alloc) {\n@@ -41,7 +44,7 @@ static void add_buffer(char **bufp, unsigned int *sizep, const char *fmt, ...)\n \t\t*bufp = buf;\n \t}\n \t*sizep = newsize;\n-\tmemcpy(buf + size, one_line, len);\n+\tmemcpy(buf + size, one_line_utf, len_utf);\n }\n \n static void check_valid(unsigned char *sha1, const char *expect)\ndiff --git a/git-rebase.sh b/git-rebase.sh\nindex 546fa44..939ac40 100755\n--- a/git-rebase.sh\n+++ b/git-rebase.sh\n@@ -296,6 +296,7 @@ fi\n \n if test -z \"$do_merge\"\n then\n+\tLC_CTYPE=sv_SE.UTF-8 \\\n \tgit-format-patch -k --stdout --full-index --ignore-if-in-upstream \"$upstream\"..ORIG_HEAD |\n \tgit am --binary -3 -k --resolvemsg=\"$RESOLVEMSG\" \\\n \t\t--reflog-action=rebase\ndiff --git a/log-tree.c b/log-tree.c\nindex fbe1399..7c2564d 100644\n--- a/log-tree.c\n+++ b/log-tree.c\n@@ -104,6 +104,7 @@ static int append_signoff(char *buf, int buf_sz, int at, const char *signoff)\n void show_log(struct rev_info *opt, const char *sep)\n {\n \tstatic char this_header[16384];\n+\tstatic char this_header_local[16384];\n \tstruct log_info *log = opt->loginfo;\n \tstruct commit *commit = log->commit, *parent = log->parent;\n \tint abbrev = opt->diffopt.abbrev;\n@@ -217,7 +218,8 @@ void show_log(struct rev_info *opt, const char *sep)\n \tif (opt->add_signoff)\n \t\tlen = append_signoff(this_header, sizeof(this_header), len,\n \t\t\t\t     opt->add_signoff);\n-\tprintf(\"%s%s%s\", this_header, extra, sep);\n+\tlocalcpy(this_header_local, this_header, len+1);\n+\tprintf(\"%s%s%s\", this_header_local, extra, sep);\n }\n \n int log_tree_diff_flush(struct rev_info *opt)\ndiff --git a/refs.c b/refs.c\nindex 98327d7..cfe2704 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -363,8 +363,9 @@ static int log_ref_write(struct ref_lock *lock,\n \tconst unsigned char *sha1, const char *msg)\n {\n \tint logfd, written, oflags = O_APPEND | O_WRONLY;\n-\tunsigned maxlen, len;\n+\tunsigned maxlen, len, len_utf;\n \tchar *logrec;\n+\tchar *logrec_utf;\n \tconst char *committer;\n \n \tif (log_all_ref_updates) {\n@@ -400,10 +401,14 @@ static int log_ref_write(struct ref_lock *lock,\n \t\t\tsha1_to_hex(sha1),\n \t\t\tcommitter);\n \t}\n-\twritten = len <= maxlen ? write(logfd, logrec, len) : -1;\n+\tlogrec_utf = xmalloc(len*6);\n+\tutfcpy(logrec_utf, logrec, len + 1);\n+\tlen_utf = strlen(logrec_utf);\n+\twritten = len_utf <= maxlen ? write(logfd, logrec_utf, len_utf) : -1;\n \tfree(logrec);\n+\tfree(logrec_utf);\n \tclose(logfd);\n-\tif (written != len)\n+\tif (written != len_utf)\n \t\treturn error(\"Unable to append to %s\", lock->log_file);\n \treturn 0;\n }\ndiff --git a/t/t-utf-msg.sh b/t/t-utf-msg.sh\nnew file mode 100755\nindex 0000000..727d497\n--- /dev/null\n+++ b/t/t-utf-msg.sh\n@@ -0,0 +1,43 @@\n+#!/bin/sh\n+\n+test_description='Test charset management.\n+\n+This assumes normal tests works fine\n+and concentrates commit messages and other\n+descriptive data.'\n+\n+. ./test-lib.sh\n+\n+export GIT_AUTHOR_NAME='Pär Nördsson'\n+export GIT_COMMITTER_NAME='Pär Nördsson'\n+export GIT_AUTHOR_DATE='Thu Sep 14 22:54:30 2006 +0000'\n+export GIT_COMMITTER_DATE='Thu Sep 14 22:54:30 2006 +0000'\n+\n+test_expect_success \\\n+    'add simple text file' \\\n+    'echo hej >aland.txt &&\n+    git-add aland.txt &&\n+    git-commit -a -m \"Ändrad\" &&\n+    echo test $(git-ls-files) = \"aland.txt\\\"\" &&\n+    LC_CTYPE=sv_SE.UTF-8 echo test $(git-ls-files) = \"aland.txt\\\"\"\n+    '\n+\n+cat >>expected <<EOF\n+commit 6905219c78beda5d5efd2a5fe4fbe0a8757bb355\n+Author: Pär Nördsson <author@example.com>\n+Date:   Thu Sep 14 22:54:30 2006 +0000\n+\n+    Ändrad\n+EOF\n+\n+test_expect_success \\\n+    'log' \\\n+    '\n+    git log >actual &&\n+    diff -u actual expected\n+    '\n+\n+# todo: git-cat-file commit xxxxxxxxxxxxx\n+\n+test_done\n+\n-- \n1.6.3.dirty\n"},{"id":"113701","messageId":"1242168631-30753-4-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-3-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 3/8] Extend tests to cover locale wrt to commit messages.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:26Z","receivedAt":"2009-05-12T22:50:26Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n t/t1100-commit-tree-options.sh |   12 ++++++------\n t/t1400-update-ref.sh          |   10 +++++-----\n t/t3400-rebase.sh              |    6 +++---\n t/t3401-rebase-partial.sh      |    8 ++++----\n t/t3402-rebase-merge.sh        |   17 ++++++++++-------\n t/t3403-rebase-skip.sh         |   10 +++++-----\n 6 files changed, 33 insertions(+), 30 deletions(-)\n\ndiff --git a/t/t1100-commit-tree-options.sh b/t/t1100-commit-tree-options.sh\nindex 19a0ed4..04f2e3b 100755\n--- a/t/t1100-commit-tree-options.sh\n+++ b/t/t1100-commit-tree-options.sh\n@@ -13,10 +13,10 @@ object by defining all environment variables that it understands.\n \n cat >expected <<EOF\n tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904\n-author Author Name <author@email> 1117148400 +0000\n-committer Committer Name <committer@email> 1117150200 +0000\n+author Authör Name <author@email> 1117148400 +0000\n+committer Committer Näme <committer@email> 1117150200 +0000\n \n-comment text\n+cömment text\n EOF\n \n test_expect_success \\\n@@ -25,11 +25,11 @@ test_expect_success \\\n \n test_expect_success \\\n     'construct commit' \\\n-    'echo comment text |\n-     GIT_AUTHOR_NAME=\"Author Name\" \\\n+    'echo cömment text |\n+     GIT_AUTHOR_NAME=\"Authör Name\" \\\n      GIT_AUTHOR_EMAIL=\"author@email\" \\\n      GIT_AUTHOR_DATE=\"2005-05-26 23:00\" \\\n-     GIT_COMMITTER_NAME=\"Committer Name\" \\\n+     GIT_COMMITTER_NAME=\"Committer Näme\" \\\n      GIT_COMMITTER_EMAIL=\"committer@email\" \\\n      GIT_COMMITTER_DATE=\"2005-05-26 23:30\" \\\n      TZ=GMT git-commit-tree `cat treeid` >commitid 2>/dev/null'\ndiff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\nindex b3b920e..0daff8a 100755\n--- a/t/t1400-update-ref.sh\n+++ b/t/t1400-update-ref.sh\n@@ -71,12 +71,12 @@ touch .git/logs/refs/heads/master\n test_expect_success \\\n \t\"create $m (logged by touch)\" \\\n \t'GIT_COMMITTER_DATE=\"2005-05-26 23:30\" \\\n-\t git-update-ref HEAD '\"$A\"' -m \"Initial Creation\" &&\n+\t git-update-ref HEAD '\"$A\"' -m \"Initial Creation. /mäster\" &&\n \t test '\"$A\"' = $(cat .git/'\"$m\"')'\n test_expect_success \\\n \t\"update $m (logged by touch)\" \\\n \t'GIT_COMMITTER_DATE=\"2005-05-26 23:31\" \\\n-\t git-update-ref HEAD'\" $B $A \"'-m \"Switch\" &&\n+\t git-update-ref HEAD'\" $B $A \"'-m \"Switch /mäster\" &&\n \t test '\"$B\"' = $(cat .git/'\"$m\"')'\n test_expect_success \\\n \t\"set $m (logged by touch)\" \\\n@@ -84,9 +84,9 @@ test_expect_success \\\n \t git-update-ref HEAD'\" $A &&\n \t test $A\"' = $(cat .git/'\"$m\"')'\n \n-cat >expect <<EOF\n-$Z $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150200 +0000\tInitial Creation\n-$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150260 +0000\tSwitch\n+iconv -f iso-8859-1 -t utf-8 >expect <<EOF\n+$Z $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150200 +0000\tInitial Creation. /mäster\n+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150260 +0000\tSwitch /mäster\n $B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000\n EOF\n test_expect_success \\\ndiff --git a/t/t3400-rebase.sh b/t/t3400-rebase.sh\nindex b9d3131..2982615 100755\n--- a/t/t3400-rebase.sh\n+++ b/t/t3400-rebase.sh\n@@ -15,15 +15,15 @@ test_expect_success \\\n     'prepare repository with topic branch, then rebase against master' \\\n     'echo First > A &&\n      git-update-index --add A &&\n-     git-commit -m \"Add A.\" &&\n+     git-commit -m \"Add Ä.\" &&\n      git checkout -b my-topic-branch &&\n      echo Second > B &&\n      git-update-index --add B &&\n-     git-commit -m \"Add B.\" &&\n+     git-commit -m \"Add ß.\" &&\n      git checkout -f master &&\n      echo Third >> A &&\n      git-update-index A &&\n-     git-commit -m \"Modify A.\" &&\n+     git-commit -m \"Modify ¢.\" &&\n      git checkout -f my-topic-branch &&\n      git rebase master'\n \ndiff --git a/t/t3401-rebase-partial.sh b/t/t3401-rebase-partial.sh\nindex 8b19d3c..801b631 100755\n--- a/t/t3401-rebase-partial.sh\n+++ b/t/t3401-rebase-partial.sh\n@@ -15,23 +15,23 @@ test_expect_success \\\n     'prepare repository with topic branch' \\\n     'echo First > A &&\n      git-update-index --add A &&\n-     git-commit -m \"Add A.\" &&\n+     git-commit -m \"Add Ä.\" &&\n \n      git-checkout -b my-topic-branch &&\n \n      echo Second > B &&\n      git-update-index --add B &&\n-     git-commit -m \"Add B.\" &&\n+     git-commit -m \"Add ß.\" &&\n \n      echo AnotherSecond > C &&\n      git-update-index --add C &&\n-     git-commit -m \"Add C.\" &&\n+     git-commit -m \"Add ¢.\" &&\n \n      git-checkout -f master &&\n \n      echo Third >> A &&\n      git-update-index A &&\n-     git-commit -m \"Modify A.\"\n+     git-commit -m \"Modify Ä.\"\n '\n \n test_expect_success \\\ndiff --git a/t/t3402-rebase-merge.sh b/t/t3402-rebase-merge.sh\nindex 0779aaa..8045995 100755\n--- a/t/t3402-rebase-merge.sh\n+++ b/t/t3402-rebase-merge.sh\n@@ -7,8 +7,11 @@ test_description='git rebase --merge test'\n \n . ./test-lib.sh\n \n-T=\"A quick brown fox\n-jumps over the lazy dog.\"\n+export GIT_AUTHOR_NAME=\"Pär Nördsson\"\n+\n+T=\"A quick brown föx\n+jumps over the läzy dog.\"\n+\n for i in 1 2 3 4 5 6 7 8 9 10\n do\n \techo \"$i $T\"\n@@ -19,24 +22,24 @@ test_expect_success setup '\n \tgit commit -m\"initial\" &&\n \tgit branch side &&\n \techo \"11 $T\" >>original &&\n-\tgit commit -a -m\"master updates a bit.\" &&\n+\tgit commit -a -m\"mäster updates a bit.\" &&\n \n \techo \"12 $T\" >>original &&\n-\tgit commit -a -m\"master updates a bit more.\" &&\n+\tgit commit -a -m\"mäster updates a bit more.\" &&\n \n \tgit checkout side &&\n \t(echo \"0 $T\" ; cat original) >renamed &&\n \tgit add renamed &&\n \tgit update-index --force-remove original &&\n-\tgit commit -a -m\"side renames and edits.\" &&\n+\tgit commit -a -m\"side renames and ädits.\" &&\n \n \ttr \"[a-z]\" \"[A-Z]\" <original >newfile &&\n \tgit add newfile &&\n-\tgit commit -a -m\"side edits further.\" &&\n+\tgit commit -a -m\"side edits fürther.\" &&\n \n \ttr \"[a-m]\" \"[A-M]\" <original >newfile &&\n \trm -f original &&\n-\tgit commit -a -m\"side edits once again.\" &&\n+\tgit commit -a -m\"side edits önce again.\" &&\n \n \tgit branch test-rebase side &&\n \tgit branch test-rebase-pick side &&\ndiff --git a/t/t3403-rebase-skip.sh b/t/t3403-rebase-skip.sh\nindex 977c498..84f14fd 100755\n--- a/t/t3403-rebase-skip.sh\n+++ b/t/t3403-rebase-skip.sh\n@@ -13,20 +13,20 @@ test_description='git rebase --merge --skip tests'\n test_expect_success setup '\n \techo hello > hello &&\n \tgit add hello &&\n-\tgit commit -m \"hello\" &&\n+\tgit commit -m \"hällo\" &&\n \tgit branch skip-reference &&\n \n \techo world >> hello &&\n-\tgit commit -a -m \"hello world\" &&\n+\tgit commit -a -m \"hällo world\" &&\n \techo goodbye >> hello &&\n-\tgit commit -a -m \"goodbye\" &&\n+\tgit commit -a -m \"göödbye\" &&\n \n \tgit checkout -f skip-reference &&\n \techo moo > hello &&\n-\tgit commit -a -m \"we should skip this\" &&\n+\tgit commit -a -m \"we shöld skip this\" &&\n \techo moo > cow &&\n \tgit add cow &&\n-\tgit commit -m \"this should not be skipped\" &&\n+\tgit commit -m \"this shöd not be skipped\" &&\n \tgit branch pre-rebase skip-reference &&\n \tgit branch skip-merge skip-reference\n \t'\n-- \n1.6.3.dirty\n"},{"id":"113702","messageId":"1242168631-30753-5-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-4-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 4/8] UTF file names.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:27Z","receivedAt":"2009-05-12T22:50:27Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n builtin-add.c            |    5 +-\n builtin-checkout-index.c |   46 ++++++---\n builtin-ls-files.c       |   26 +++++-\n builtin-ls-tree.c        |   16 +++-\n builtin-rev-parse.c      |    7 +-\n builtin-update-index.c   |   18 +++-\n builtin-write-tree.c     |    5 +-\n diff.c                   |   97 ++++++++++++++------\n git-commit.sh            |    5 +\n git-compat-util.h        |   42 +++++++++\n merge-index.c            |   25 ++++--\n read-cache.c             |    8 +-\n setup.c                  |   28 +++++-\n t/t-utf-filenames.sh     |   95 +++++++++++++++++++\n utf.c                    |  230 ++++++++++++++++++++++++++++++++++++++++++++++\n 15 files changed, 580 insertions(+), 73 deletions(-)\n create mode 100755 t/t-utf-filenames.sh\n\ndiff --git a/builtin-add.c b/builtin-add.c\nindex febb75e..62ea692 100644\n--- a/builtin-add.c\n+++ b/builtin-add.c\n@@ -131,9 +131,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\treturn 0;\n \t}\n \n-\tfor (i = 0; i < dir.nr; i++)\n+\tfor (i = 0; i < dir.nr; i++) {\n+\t  P((\"Adding '%s'\\n\", dir.entries[i]->name));\n \t\tadd_file_to_index(dir.entries[i]->name, verbose);\n-\n+\t}\n \tif (active_cache_changed) {\n \t\tif (write_cache(newfd, active_cache, active_nr) ||\n \t\t    close(newfd) || commit_lock_file(&lock_file))\ndiff --git a/builtin-checkout-index.c b/builtin-checkout-index.c\nindex b097c88..745bf9a 100644\n--- a/builtin-checkout-index.c\n+++ b/builtin-checkout-index.c\n@@ -57,16 +57,22 @@ static void write_tempfile_record(const char *name, int prefix_length)\n \t\tfor (i = 1; i < 4; i++) {\n \t\t\tif (i > 1)\n \t\t\t\tputchar(' ');\n-\t\t\tif (topath[i][0])\n-\t\t\t\tfputs(topath[i], stdout);\n-\t\t\telse\n+\t\t\tif (topath[i][0]) {\n+\t\t\t\tchar localbuf[MAXPATHLEN];\n+\t\t\t\tlocalcpy(localbuf, topath[i], strlen(topath[i])+1);\n+\t\t\t\tfputs(localbuf, stdout);\n+\t\t\t} else\n \t\t\t\tputchar('.');\n \t\t}\n-\t} else\n-\t\tfputs(topath[checkout_stage], stdout);\n-\n+\t} else {\n+\t\tchar localbuf[MAXPATHLEN];\n+\t\tlocalcpy(localbuf, topath[checkout_stage], strlen(topath[checkout_stage])+1);\n+\t\tfputs(localbuf, stdout);\n+\t}\n \tputchar('\\t');\n-\twrite_name_quoted(\"\", 0, name + prefix_length,\n+\tchar localbuf[MAXPATHLEN];\n+\tlocalcpy(localbuf, name + prefix_length, strlen(name + prefix_length) + 1);\n+\twrite_name_quoted(\"\", 0, localbuf, \n \t\tline_termination, stdout);\n \tputchar(line_termination);\n \n@@ -77,8 +83,11 @@ static void write_tempfile_record(const char *name, int prefix_length)\n \n static int checkout_file(const char *name, int prefix_length)\n {\n+\tchar utfname[MAXPATHLEN];\n \tint namelen = strlen(name);\n-\tint pos = cache_name_pos(name, namelen);\n+\tutfcpy(utfname, name, namelen + 1);\n+\tint utfnamelen = strlen(utfname);\n+\tint pos = cache_name_pos(utfname, utfnamelen);\n \tint has_same_name = 0;\n \tint did_checkout = 0;\n \tint errs = 0;\n@@ -88,8 +97,8 @@ static int checkout_file(const char *name, int prefix_length)\n \n \twhile (pos < active_nr) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n-\t\tif (ce_namelen(ce) != namelen ||\n-\t\t    memcmp(ce->name, name, namelen))\n+\t\tif (ce_namelen(ce) != utfnamelen ||\n+\t\t    memcmp(ce->name, utfname, utfnamelen))\n \t\t\tbreak;\n \t\thas_same_name = 1;\n \t\tpos++;\n@@ -224,7 +233,9 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strncmp(arg, \"--prefix=\", 9)) {\n-\t\t\tstate.base_dir = arg+9;\n+\t\t\tstatic char utfbasedir[MAXPATHLEN];\n+\t\t\tutfcpy(utfbasedir, arg+9, strlen(arg+9)+1);\n+\t\t\tstate.base_dir = utfbasedir;\n \t\t\tstate.base_dir_len = strlen(state.base_dir);\n \t\t\tcontinue;\n \t\t}\n@@ -262,13 +273,16 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t\tconst char *arg = argv[i];\n \t\tconst char *p;\n \n+\t\tchar utfarg[MAXPATHLEN];\n+\t\tutfcpy(utfarg, arg, strlen(arg)+1);\n+\n \t\tif (all)\n \t\t\tdie(\"git-checkout-index: don't mix '--all' and explicit filenames\");\n \t\tif (read_from_stdin)\n \t\t\tdie(\"git-checkout-index: don't mix '--stdin' and explicit filenames\");\n-\t\tp = prefix_path(prefix, prefix_length, arg);\n+\t\tp = prefix_path(prefix, prefix_length, utfarg);\n \t\tcheckout_file(p, prefix_length);\n-\t\tif (p < arg || p > arg + strlen(arg))\n+\t\tif (p < arg || p > arg + strlen(utfarg))\n \t\t\tfree((char*)p);\n \t}\n \n@@ -288,9 +302,11 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t\t\t\tpath_name = unquote_c_style(buf.buf, NULL);\n \t\t\telse\n \t\t\t\tpath_name = buf.buf;\n-\t\t\tp = prefix_path(prefix, prefix_length, path_name);\n+\t\t\tchar utfpath_name[MAXPATHLEN];\n+\t\t\tutfcpy(utfpath_name, path_name, strlen(path_name)+1);\n+\t\t\tp = prefix_path(prefix, prefix_length, utfpath_name);\n \t\t\tcheckout_file(p, prefix_length);\n-\t\t\tif (p < path_name || p > path_name + strlen(path_name))\n+\t\t\tif (p < utfpath_name || p > utfpath_name + strlen(utfpath_name))\n \t\t\t\tfree((char *)p);\n \t\t\tif (path_name != buf.buf)\n \t\t\t\tfree(path_name);\ndiff --git a/builtin-ls-files.c b/builtin-ls-files.c\nindex ad8c41e..babac34 100644\n--- a/builtin-ls-files.c\n+++ b/builtin-ls-files.c\n@@ -65,6 +65,7 @@ static int match(const char **spec, char *ps_matched,\n \t\t\tps_matched++;\n \t\tcontinue;\n \tmatched:\n+\t\tP((\"matched(%s,%s)\\n\",m+len, filename+len));\n \t\tif (ps_matched)\n \t\t\t*ps_matched = 1;\n \t\treturn 1;\n@@ -76,6 +77,9 @@ static void show_dir_entry(const char *tag, struct dir_entry *ent)\n {\n \tint len = prefix_len;\n \tint offset = prefix_offset;\n+\tchar localpath[MAXPATHLEN];\n+\n+\tP((\"show_dir_entry(\\\"%s\\\",\\\"%s\\\"\\n\",tag,ent->name));\n \n \tif (len >= ent->len)\n \t\tdie(\"git-ls-files: internal error - directory entry not superset of prefix\");\n@@ -84,7 +88,8 @@ static void show_dir_entry(const char *tag, struct dir_entry *ent)\n \t\treturn;\n \n \tfputs(tag, stdout);\n-\twrite_name_quoted(\"\", 0, ent->name + offset, line_terminator, stdout);\n+\tlocalcpy(localpath, ent->name + offset, strlen(ent->name + offset) + 1);\n+\twrite_name_quoted(\"\", 0, localpath, line_terminator, stdout);\n \tputchar(line_terminator);\n }\n \n@@ -165,6 +170,8 @@ static void show_ce_entry(const char *tag, struct cache_entry *ce)\n \tint len = prefix_len;\n \tint offset = prefix_offset;\n \n+\tP((\"show_ce_entry(\\\"%s\\\",\\\"%s\\\"\\n\",tag,ce->name));\n+\n \tif (len >= ce_namelen(ce))\n \t\tdie(\"git-ls-files: internal error - cache entry not superset of prefix\");\n \n@@ -189,19 +196,23 @@ static void show_ce_entry(const char *tag, struct cache_entry *ce)\n \t}\n \n \tif (!show_stage) {\n+\t\tchar localpath[MAXPATHLEN];\n \t\tfputs(tag, stdout);\n-\t\twrite_name_quoted(\"\", 0, ce->name + offset,\n+\t\tlocalcpy(localpath, ce->name + offset, strlen(ce->name + offset) + 1);\n+\t\twrite_name_quoted(\"\", 0, localpath,\n \t\t\t\t  line_terminator, stdout);\n \t\tputchar(line_terminator);\n \t}\n \telse {\n+\t\tchar localpath[MAXPATHLEN];\n \t\tprintf(\"%s%06o %s %d\\t\",\n \t\t       tag,\n \t\t       ntohl(ce->ce_mode),\n \t\t       abbrev ? find_unique_abbrev(ce->sha1,abbrev)\n \t\t\t\t: sha1_to_hex(ce->sha1),\n \t\t       ce_stage(ce));\n-\t\twrite_name_quoted(\"\", 0, ce->name + offset,\n+\t\tlocalcpy(localpath, ce->name + offset, strlen(ce->name + offset) + 1);\n+\t\twrite_name_quoted(\"\", 0, localpath,\n \t\t\t\t  line_terminator, stdout);\n \t\tputchar(line_terminator);\n \t}\n@@ -451,6 +462,12 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)\n \n \tpathspec = get_pathspec(prefix, argv + i);\n \n+#ifdef DEBUG\n+ \tif (pathspec) {\n+\t\tP((\"pathspec[0]=%s\\n\",pathspec[0]));\n+\t\tP((\"pathspec[1]=%s\\n\",pathspec[1]));\n+\t}\n+#endif\n \t/* Verify that the pathspec matches the prefix */\n \tif (pathspec)\n \t\tprefix = verify_pathspec(prefix);\n@@ -461,6 +478,7 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)\n \t\tfor (num = 0; pathspec[num]; num++)\n \t\t\t;\n \t\tps_matched = xcalloc(1, num);\n+\t\tP((\"allocated ps_matched, %d entries\\n\",num));\n \t}\n \n \tif (dir.show_ignored && !exc_given) {\n@@ -485,12 +503,14 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)\n \t\t */\n \t\tint num, errors = 0;\n \t\tfor (num = 0; pathspec[num]; num++) {\n+\t\t  \tP((\"ps_matched[%d]=%d\\n\",num,ps_matched[num]));\n \t\t\tif (ps_matched[num])\n \t\t\t\tcontinue;\n \t\t\terror(\"pathspec '%s' did not match any.\",\n \t\t\t      pathspec[num] + prefix_offset);\n \t\t\terrors++;\n \t\t}\n+\t\tP((\"return errors ? 1 : 0; => %d\\n\",errors ? 1 : 0));\n \t\treturn errors ? 1 : 0;\n \t}\n \ndiff --git a/builtin-ls-tree.c b/builtin-ls-tree.c\nindex 201defd..4f53b0d 100644\n--- a/builtin-ls-tree.c\n+++ b/builtin-ls-tree.c\n@@ -78,8 +78,14 @@ static int show_tree(const unsigned char *sha1, const char *base, int baselen,\n \t\tprintf(\"%06o %s %s\\t\", mode, type,\n \t\t\t\tabbrev ? find_unique_abbrev(sha1,abbrev)\n \t\t\t\t\t: sha1_to_hex(sha1));\n-\twrite_name_quoted(base + chomp_prefix, baselen - chomp_prefix,\n-\t\t\t  pathname,\n+\tchar localprefix[MAXPATHLEN];\n+\tint localprefixlen = locallen(base + chomp_prefix, baselen - chomp_prefix);\n+\tlocalcpy(localprefix, base + chomp_prefix, baselen - chomp_prefix);\n+\t\n+\tchar localpathname[MAXPATHLEN];\n+\tlocalcpy(localpathname, pathname, strlen(pathname)+1);\n+\twrite_name_quoted(localprefix, localprefixlen,\n+\t\t\t  localpathname,\n \t\t\t  line_termination, stdout);\n \tputchar(line_termination);\n \treturn retval;\n@@ -147,6 +153,12 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix)\n \t\tdie(\"Not a valid object name %s\", argv[1]);\n \n \tpathspec = get_pathspec(prefix, argv + 2);\n+#ifdef DEBUG\n+\tif (pathspec) {\n+\t  P((\"pathspec[0]=%s\\n\",pathspec[0]));\n+\t  P((\"pathspec[1]=%s\\n\",pathspec[1]));\n+\t}\n+#endif\n \ttree = parse_tree_indirect(sha1);\n \tif (!tree)\n \t\tdie(\"not a tree object\");\ndiff --git a/builtin-rev-parse.c b/builtin-rev-parse.c\nindex fd3ccc8..da03906 100644\n--- a/builtin-rev-parse.c\n+++ b/builtin-rev-parse.c\n@@ -315,8 +315,11 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-prefix\")) {\n-\t\t\t\tif (prefix)\n-\t\t\t\t\tputs(prefix);\n+\t\t\t\tif (prefix) {\n+\t\t\t\t\tchar localprefix[MAXPATHLEN];\n+\t\t\t\t\tlocalcpy(localprefix, prefix, strlen(prefix)+1);\n+\t\t\t\t\tputs(localprefix);\n+\t\t\t\t}\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-cdup\")) {\ndiff --git a/builtin-update-index.c b/builtin-update-index.c\nindex a3c0a45..cfea4cd 100644\n--- a/builtin-update-index.c\n+++ b/builtin-update-index.c\n@@ -274,7 +274,9 @@ static void read_index_info(int line_termination)\n \t\telse\n \t\t\tpath_name = ptr;\n \n-\t\tif (!verify_path(path_name)) {\n+\t\tchar utfpath_name[MAXPATHLEN];\n+\t\tutfcpy(utfpath_name, path_name, strlen(path_name) + 1);\n+\t\tif (!verify_path(utfpath_name)) {\n \t\t\tfprintf(stderr, \"Ignoring path %s\\n\", path_name);\n \t\t\tif (path_name != ptr)\n \t\t\t\tfree(path_name);\n@@ -284,7 +286,7 @@ static void read_index_info(int line_termination)\n \n \t\tif (!mode) {\n \t\t\t/* mode == 0 means there is no such path -- remove */\n-\t\t\tif (remove_file_from_cache(path_name))\n+\t\t\tif (remove_file_from_cache(utfpath_name))\n \t\t\t\tdie(\"git-update-index: unable to remove %s\",\n \t\t\t\t    ptr);\n \t\t}\n@@ -294,7 +296,7 @@ static void read_index_info(int line_termination)\n \t\t\t * ptr[-41] is at the beginning of sha1\n \t\t\t */\n \t\t\tptr[-42] = ptr[-1] = 0;\n-\t\t\tif (add_cacheinfo(mode, sha1, path_name, stage))\n+\t\t\tif (add_cacheinfo(mode, sha1, utfpath_name, stage))\n \t\t\t\tdie(\"git-update-index: unable to update %s\",\n \t\t\t\t    path_name);\n \t\t}\n@@ -616,7 +618,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t\tusage(update_index_usage);\n \t\t\tdie(\"unknown option %s\", path);\n \t\t}\n-\t\tupdate_one(path, prefix, prefix_length);\n+\t\tchar utfpath[MAXPATHLEN];\n+\t\tutfcpy(utfpath, path, strlen(path) + 1);\n+\t\tupdate_one(utfpath, prefix, prefix_length);\n \t\tif (set_executable_bit)\n \t\t\tchmod_path(set_executable_bit, path);\n \t}\n@@ -633,11 +637,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t\tpath_name = unquote_c_style(buf.buf, NULL);\n \t\t\telse\n \t\t\t\tpath_name = buf.buf;\n-\t\t\tp = prefix_path(prefix, prefix_length, path_name);\n+\t\t\tchar utfpath_name[MAXPATHLEN];\n+\t\t\tutfcpy(utfpath_name, path_name, strlen(path_name) + 1);\n+\t\t\tp = prefix_path(prefix, prefix_length, utfpath_name);\n \t\t\tupdate_one(p, NULL, 0);\n \t\t\tif (set_executable_bit)\n \t\t\t\tchmod_path(set_executable_bit, p);\n-\t\t\tif (p < path_name || p > path_name + strlen(path_name))\n+\t\t\tif (p < utfpath_name || p > utfpath_name + strlen(utfpath_name))\n \t\t\t\tfree((char*) p);\n \t\t\tif (path_name != buf.buf)\n \t\t\t\tfree(path_name);\ndiff --git a/builtin-write-tree.c b/builtin-write-tree.c\nindex 50670dc..84c370f 100644\n--- a/builtin-write-tree.c\n+++ b/builtin-write-tree.c\n@@ -80,7 +80,10 @@ int cmd_write_tree(int argc, const char **argv, const char *unused_prefix)\n \tif (argc > 2)\n \t\tdie(\"too many options\");\n \n-\tret = write_tree(sha1, missing_ok, prefix);\n+\tchar utfprefix[MAXPATHLEN];\n+\tif (prefix)\n+\t  utfcpy(utfprefix, prefix, strlen(prefix)+1);\n+\tret = write_tree(sha1, missing_ok, prefix!=NULL ? utfprefix : NULL);\n \tprintf(\"%s\\n\", sha1_to_hex(sha1));\n \n \treturn ret;\ndiff --git a/diff.c b/diff.c\nindex 3315378..170ec5a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -964,9 +964,14 @@ static void builtin_diff(const char *name_a,\n \tchar *a_one, *b_two;\n \tconst char *set = diff_get_color(o->color_diff, DIFF_METAINFO);\n \tconst char *reset = diff_get_color(o->color_diff, DIFF_RESET);\n+\tchar localname_a[MAXPATHLEN];\n+\tchar localname_b[MAXPATHLEN];\n \n-\ta_one = quote_two(\"a/\", name_a);\n-\tb_two = quote_two(\"b/\", name_b);\n+\tlocalcpy(localname_a, name_a, strlen(name_a) + 1);\n+\tlocalcpy(localname_b, name_b, strlen(name_b) + 1);\n+\n+\ta_one = quote_two(\"a/\", localname_a);\n+\tb_two = quote_two(\"b/\", localname_b);\n \tlbl[0] = DIFF_FILE_VALID(one) ? a_one : \"/dev/null\";\n \tlbl[1] = DIFF_FILE_VALID(two) ? b_two : \"/dev/null\";\n \tprintf(\"%sdiff --git %s %s%s\\n\", set, a_one, b_two, reset);\n@@ -995,7 +1000,7 @@ static void builtin_diff(const char *name_a,\n \t\tif ((one->mode ^ two->mode) & S_IFMT)\n \t\t\tgoto free_ab_and_return;\n \t\tif (complete_rewrite) {\n-\t\t\temit_rewrite_diff(name_a, name_b, one, two);\n+\t\t\temit_rewrite_diff(localname_a, localname_b, one, two);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n \t}\n@@ -1372,18 +1377,22 @@ static void prepare_temp_file(const char *name,\n \t    work_tree_matches(name, one->sha1)) {\n \t\tstruct stat st;\n \t\tif (lstat(name, &st) < 0) {\n+\t\t\tchar localname[MAXPATHLEN];\n \t\t\tif (errno == ENOENT)\n \t\t\t\tgoto not_a_valid_file;\n-\t\t\tdie(\"stat(%s): %s\", name, strerror(errno));\n+\t\t\tlocalcpy(localname, name, strlen(name) + 1);\n+\t\t\tdie(\"stat(%s): %s\", localname, strerror(errno));\n \t\t}\n \t\tif (S_ISLNK(st.st_mode)) {\n \t\t\tint ret;\n \t\t\tchar buf[PATH_MAX + 1]; /* ought to be SYMLINK_MAX */\n+\t\t\tchar localname[MAXPATHLEN];\n+\t\t\tlocalcpy(localname, name, strlen(name) + 1);\n \t\t\tif (sizeof(buf) <= st.st_size)\n-\t\t\t\tdie(\"symlink too long: %s\", name);\n+\t\t\t\tdie(\"symlink too long: %s\", localname);\n \t\t\tret = readlink(name, buf, st.st_size);\n \t\t\tif (ret < 0)\n-\t\t\t\tdie(\"readlink(%s)\", name);\n+\t\t\t\tdie(\"readlink(%s)\", localname);\n \t\t\tprep_temp_blob(temp, buf, st.st_size,\n \t\t\t\t       (one->sha1_valid ?\n \t\t\t\t\tone->sha1 : null_sha1),\n@@ -1522,7 +1531,9 @@ static void run_external_diff(const char *pgm,\n \tretval = spawn_prog(pgm, spawn_arg);\n \tremove_tempfile();\n \tif (retval) {\n-\t\tfprintf(stderr, \"external diff died, stopping at %s.\\n\", name);\n+\t  \tchar localname[MAXPATHLEN];\n+\t  \tlocalcpy(localname, name, strlen(name) + 1);\n+\t\tfprintf(stderr, \"external diff died, stopping at %s.\\n\", localname);\n \t\texit(1);\n \t}\n }\n@@ -1574,6 +1585,9 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)\n \tchar *name_munged, *other_munged;\n \tint complete_rewrite = 0;\n \tint len;\n+\tchar localname[MAXPATHLEN];\n+\tchar localotherbuf[MAXPATHLEN];\n+\tchar *localother;\n \n \tif (DIFF_PAIR_UNMERGED(p)) {\n \t\t/* unmerged */\n@@ -1583,8 +1597,14 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)\n \n \tname = p->one->path;\n \tother = (strcmp(name, p->two->path) ? p->two->path : NULL);\n-\tname_munged = quote_one(name);\n-\tother_munged = quote_one(other);\n+\tlocalcpy(localname, name, strlen(name) + 1);\n+\tif (other) {\n+\t\tlocalcpy(localotherbuf, other, strlen(other) + 1);\n+\t\tlocalother = localotherbuf;\n+\t} else\n+\t\tlocalother = NULL;\n+\tname_munged = quote_one(localname);\n+\tother_munged = quote_one(localother);\n \tone = p->one; two = p->two;\n \n \tdiff_fill_sha1_info(one);\n@@ -2093,12 +2113,15 @@ static void diff_flush_raw(struct diff_filepair *p,\n \tconst char *path_one, *path_two;\n \tint inter_name_termination = '\\t';\n \tint line_termination = options->line_termination;\n+\tchar localpath_one[MAXPATHLEN];\n+\tchar localpath_two[MAXPATHLEN];\n \n \tif (!line_termination)\n \t\tinter_name_termination = 0;\n-\n-\tpath_one = p->one->path;\n-\tpath_two = p->two->path;\n+\tlocalcpy(localpath_one, p->one->path, strlen(p->one->path) + 1);\n+\tlocalcpy(localpath_two, p->two->path, strlen(p->two->path) + 1);\n+\tpath_one = localpath_one;\n+\tpath_two = localpath_two;\n \tif (line_termination) {\n \t\tpath_one = quote_one(path_one);\n \t\tpath_two = quote_one(path_two);\n@@ -2135,20 +2158,22 @@ static void diff_flush_raw(struct diff_filepair *p,\n \tif (two_paths)\n \t\tprintf(\"%c%s\", inter_name_termination, path_two);\n \tputchar(line_termination);\n-\tif (path_one != p->one->path)\n+\tif (path_one != localpath_one)\n \t\tfree((void*)path_one);\n-\tif (path_two != p->two->path)\n+\tif (path_two != localpath_two)\n \t\tfree((void*)path_two);\n }\n \n static void diff_flush_name(struct diff_filepair *p, int line_termination)\n {\n-\tchar *path = p->two->path;\n+\tchar localpath_two[MAXPATHLEN];\n+\tchar *path = localpath_two;\n+\tlocalcpy(localpath_two, p->two->path, strlen(p->two->path) + 1);\n \n \tif (line_termination)\n-\t\tpath = quote_one(p->two->path);\n+\t\tpath = quote_one(localpath_two);\n \tprintf(\"%s%c\", path, line_termination);\n-\tif (p->two->path != path)\n+\tif (localpath_two != path)\n \t\tfree(path);\n }\n \n@@ -2325,8 +2350,10 @@ static void diff_resolve_rename_copy(void)\n \t\t\t/* This is a \"no-change\" entry and should not\n \t\t\t * happen anymore, but prepare for broken callers.\n \t\t\t */\n+\t\t        char localpath[MAXPATHLEN];\n+\t\t\tlocalcpy(localpath, p->one->path, strlen(p->one->path) + 1);\n \t\t\terror(\"feeding unmodified %s to diffcore\",\n-\t\t\t      p->one->path);\n+\t\t\t      localpath);\n \t\t\tp->status = DIFF_STATUS_UNKNOWN;\n \t\t}\n \t}\n@@ -2359,20 +2386,24 @@ static void flush_one_pair(struct diff_filepair *p, struct diff_options *opt)\n \n static void show_file_mode_name(const char *newdelete, struct diff_filespec *fs)\n {\n+        char localpath[MAXPATHLEN];\n+\tlocalcpy(localpath, fs->path, strlen(fs->path) + 1);\n \tif (fs->mode)\n-\t\tprintf(\" %s mode %06o %s\\n\", newdelete, fs->mode, fs->path);\n+\t\tprintf(\" %s mode %06o %s\\n\", newdelete, fs->mode, localpath);\n \telse\n-\t\tprintf(\" %s %s\\n\", newdelete, fs->path);\n+\t\tprintf(\" %s %s\\n\", newdelete, localpath);\n }\n \n \n static void show_mode_change(struct diff_filepair *p, int show_name)\n {\n \tif (p->one->mode && p->two->mode && p->one->mode != p->two->mode) {\n-\t\tif (show_name)\n+\t        if (show_name) {\n+\t\t \tchar localpath[MAXPATHLEN];\n+\t\t\tlocalcpy(localpath, p->two->path, strlen(p->two->path) + 1);\n \t\t\tprintf(\" mode change %06o => %06o %s\\n\",\n-\t\t\t       p->one->mode, p->two->mode, p->two->path);\n-\t\telse\n+\t\t\t       p->one->mode, p->two->mode, localpath);\n+\t\t} else\n \t\t\tprintf(\" mode change %06o => %06o\\n\",\n \t\t\t       p->one->mode, p->two->mode);\n \t}\n@@ -2381,10 +2412,16 @@ static void show_mode_change(struct diff_filepair *p, int show_name)\n static void show_rename_copy(const char *renamecopy, struct diff_filepair *p)\n {\n \tconst char *old, *new;\n+\tchar localpath_one[MAXPATHLEN];\n+\tchar localpath_two[MAXPATHLEN];\n+\tchar localpath_old[MAXPATHLEN];\n+\tchar localpath_new[MAXPATHLEN];\n \n \t/* Find common prefix */\n-\told = p->one->path;\n-\tnew = p->two->path;\n+\tlocalcpy(localpath_old, p->one->path, strlen(p->one->path) + 1);\n+\tlocalcpy(localpath_new, p->two->path, strlen(p->two->path) + 1);\n+\told = localpath_old;\n+\tnew = localpath_new;\n \twhile (1) {\n \t\tconst char *slash_old, *slash_new;\n \t\tslash_old = strchr(old, '/');\n@@ -2400,13 +2437,15 @@ static void show_rename_copy(const char *renamecopy, struct diff_filepair *p)\n \t/* p->one->path thru old is the common prefix, and old and new\n \t * through the end of names are renames\n \t */\n+\tlocalcpy(localpath_one, p->one->path, strlen(p->one->path) + 1);\n+\tlocalcpy(localpath_two, p->two->path, strlen(p->two->path) + 1);\n \tif (old != p->one->path)\n \t\tprintf(\" %s %.*s{%s => %s} (%d%%)\\n\", renamecopy,\n-\t\t       (int)(old - p->one->path), p->one->path,\n+\t\t       (int)(old - localpath_one), localpath_one,\n \t\t       old, new, (int)(0.5 + p->score * 100.0/MAX_SCORE));\n \telse\n \t\tprintf(\" %s %s => %s (%d%%)\\n\", renamecopy,\n-\t\t       p->one->path, p->two->path,\n+\t\t       localpath_one, localpath_two,\n \t\t       (int)(0.5 + p->score * 100.0/MAX_SCORE));\n \tshow_mode_change(p, 0);\n }\n@@ -2428,7 +2467,9 @@ static void diff_summary(struct diff_filepair *p)\n \t\tbreak;\n \tdefault:\n \t\tif (p->score) {\n-\t\t\tprintf(\" rewrite %s (%d%%)\\n\", p->two->path,\n+\t\t  \tchar localpath[MAXPATHLEN];\n+\t\t\tlocalcpy(localpath, p->two->path, strlen(p->two->path) + 1);\n+\t\t\tprintf(\" rewrite %s (%d%%)\\n\", localpath,\n \t\t\t\t(int)(0.5 + p->score * 100.0/MAX_SCORE));\n \t\t\tshow_mode_change(p, 0);\n \t\t} else\tshow_mode_change(p, 1);\ndiff --git a/git-commit.sh b/git-commit.sh\nindex 5b1cf85..e2c647a 100755\n--- a/git-commit.sh\n+++ b/git-commit.sh\n@@ -1,4 +1,9 @@\n #!/bin/sh\n+if grep -q \"^xALL_CFLAGS += -DDEBUG=1\" $(dirname $0)/Makefile\n+then\n+    set -x\n+    PS4='$0:$LINENO:'\n+fi\n #\n # Copyright (c) 2005 Linus Torvalds\n # Copyright (c) 2006 Junio C Hamano\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f83352b..6a61026 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -188,4 +188,46 @@ static inline int sane_case(int x, int high)\n \treturn x;\n }\n \n+#ifndef MAXPATHLEN\n+#define MAXPATHLEN 256\n+#endif\n+\n+#ifdef UTF8INTERNAL\n+#ifdef NO_ICONV\n+#error \"NO_ICONV must NOT be set when UTF8INTERNAL is set\"\n+#endif\n+extern int utf_lstat(const char *path,struct stat *buf);\n+extern int utf_stat(const char *path,struct stat *buf);\n+extern DIR *utf_opendir(const char *path);\n+extern struct dirent* utf_readdir(DIR *dir);\n+extern int utf_closedir(DIR *dir);\n+extern FILE *utf_fopen(const char *path,char *mode);\n+extern FILE *utf_freopen(const char *path,char *mode,FILE *stream);\n+extern int utf_unlink(const char *path);\n+extern int utf_rmdir(const char *path);\n+extern int utf_open(const char *path,int flags, ...) __nonnull((1));\n+extern int utf_access(const char *path, int mode);\n+extern char *utf_getcwd(char *buf,int bufsize);\n+extern int utf_creat(const char *path,int mode);\n+extern int utf_mkdir(const char *path,int mode);\n+extern ssize_t utf_readlink(const char *path,char *buf,size_t bufsiz);\n+\n+#define lstat(path,buf) utf_lstat(path,buf)\n+#define stat(path,buf) utf_stat(path,buf)\n+#define opendir(path) utf_opendir(path)\n+#define readdir(dir) utf_readdir(dir)\n+#define closedir(dir) utf_closedir(dir)\n+#define fopen(name,mode) utf_fopen(name,mode)\n+#define freopen(name,mode,stream) utf_freopen(name,mode,stream)\n+#define unlink(name) utf_unlink(name)\n+#define rmdir(name) utf_rmdir(name)\n+//#define open(name,flags,mode) utf_open(name,flags,mode)\n+#define open utf_open\n+#define access(path,mode) utf_access(path,mode)\n+#define getcwd(buf,bufsize) utf_getcwd(buf,bufsize)\n+#define creat(path,mode) utf_creat(path,mode)\n+#define mkdir(path,mode) utf_mkdir(path,mode)\n+#define readlink(path,buf,bufsiz) utf_readlink(path,buf,bufsiz)\n+#endif\n+\n #endif\ndiff --git a/merge-index.c b/merge-index.c\nindex 646d090..8a98b49 100644\n--- a/merge-index.c\n+++ b/merge-index.c\n@@ -17,14 +17,23 @@ static void run_program(void)\n \tif (pid < 0)\n \t\tdie(\"unable to fork\");\n \tif (!pid) {\n-\t\texeclp(pgm, arguments[0],\n-\t\t\t    arguments[1],\n-\t\t\t    arguments[2],\n-\t\t\t    arguments[3],\n-\t\t\t    arguments[4],\n-\t\t\t    arguments[5],\n-\t\t\t    arguments[6],\n-\t\t\t    arguments[7],\n+\t\tchar argbuf[8][MAXPATHLEN];\n+\t\tchar* args[8];\n+\t\tint i;\n+\t\tfor (i=0; i<8; ++i) {\n+\t\t\tif (arguments[i]) {\n+\t\t\t      args[i] = argbuf[i];\n+\t\t\t      localcpy(args[i], arguments[i], strlen(arguments[i]) + 1);\n+\t\t\t}\n+\t\t}\n+\t\texeclp(pgm, args[0],\n+\t\t\t    args[1],\n+\t\t\t    args[2],\n+\t\t\t    args[3],\n+\t\t\t    args[4],\n+\t\t\t    args[5],\n+\t\t\t    args[6],\n+\t\t\t    args[7],\n \t\t\t    NULL);\n \t\tdie(\"unable to execute '%s'\", pgm);\n \t}\ndiff --git a/read-cache.c b/read-cache.c\nindex 97c3867..f7642ca 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -701,7 +701,9 @@ int refresh_cache(unsigned int flags)\n \t\t\ti--;\n \t\t\tif (allow_unmerged)\n \t\t\t\tcontinue;\n-\t\t\tprintf(\"%s: needs merge\\n\", ce->name);\n+\t\t\tchar localname[MAXPATHLEN];\n+\t\t\tlocalcpy(localname, ce->name, strlen(ce->name) + 1);\n+\t\t\tprintf(\"%s: needs merge\\n\", localname);\n \t\t\thas_errors = 1;\n \t\t\tcontinue;\n \t\t}\n@@ -721,7 +723,9 @@ int refresh_cache(unsigned int flags)\n \t\t\t}\n \t\t\tif (quiet)\n \t\t\t\tcontinue;\n-\t\t\tprintf(\"%s: needs update\\n\", ce->name);\n+\t\t\tchar localname[MAXPATHLEN];\n+\t\t\tlocalcpy(localname, ce->name, strlen(ce->name) + 1);\n+\t\t\tprintf(\"%s: needs update\\n\", localname);\n \t\t\thas_errors = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/setup.c b/setup.c\nindex 9a46a58..9e500f1 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -43,6 +43,8 @@ const char *prefix_path(const char *prefix, int len, const char *path)\n \t\tmemcpy(n, prefix, len);\n \t\tmemcpy(n + len, path, speclen+1);\n \t\tpath = n;\n+\t} else {\n+\t\tpath = xstrdup(path);\n \t}\n \treturn path;\n }\n@@ -107,6 +109,8 @@ void verify_non_filename(const char *prefix, const char *arg)\n \n const char **get_pathspec(const char *prefix, const char **pathspec)\n {\n+\tchar utfprefixbuf[MAXPATHLEN];\n+\tchar *utfprefix;\n \tconst char *entry = *pathspec;\n \tconst char **p;\n \tint prefixlen;\n@@ -114,20 +118,36 @@ const char **get_pathspec(const char *prefix, const char **pathspec)\n \tif (!prefix && !entry)\n \t\treturn NULL;\n \n+\tif (prefix) {\n+\t  utfcpy(utfprefixbuf, prefix, strlen(prefix)+1);\n+\t  utfprefix = utfprefixbuf;\n+\t} else {\n+\t  utfprefix = NULL;\n+\t}\n+\n \tif (!entry) {\n \t\tstatic const char *spec[2];\n-\t\tspec[0] = prefix;\n+\t\tspec[0] = xstrdup(utfprefix);\n \t\tspec[1] = NULL;\n \t\treturn spec;\n \t}\n \n \t/* Otherwise we have to re-write the entries.. */\n+\tint n;\n+\tfor (n=0; pathspec[n]; ++n)\n+\t\t;\n+\tchar **ret = xcalloc(n+1,sizeof(char*));\n+\tchar **r = ret;\n \tp = pathspec;\n-\tprefixlen = prefix ? strlen(prefix) : 0;\n+\tprefixlen = utfprefix ? strlen(utfprefix) : 0;\n \tdo {\n-\t\t*p = prefix_path(prefix, prefixlen, entry);\n+\t        char utfentry[MAXPATHLEN];\n+\t\tutfcpy(utfentry, entry, strlen(entry)+1);\n+\t\t*r = prefix_path(utfprefix, prefixlen, utfentry);\n+\t\tP((\"*r=%s\\n\",*r));\n+\t\t++r;\n \t} while ((entry = *++p) != NULL);\n-\treturn (const char **) pathspec;\n+\treturn (const char**)ret;\n }\n \n /*\ndiff --git a/t/t-utf-filenames.sh b/t/t-utf-filenames.sh\nnew file mode 100755\nindex 0000000..0731086\n--- /dev/null\n+++ b/t/t-utf-filenames.sh\n@@ -0,0 +1,95 @@\n+#!/bin/sh\n+\n+test_description='Test charset management.\n+\n+This assumes normal tests works fine\n+and concentrates on filenames with non-ascii\n+characters.'\n+\n+. ./test-lib.sh\n+\n+test_expect_success \\\n+    'add simple text file' \\\n+    'mkdir å &&\n+    echo hej >å/åland.txt &&\n+    git-add å/åland.txt &&\n+    git-commit -a -m \"Changed\" &&\n+    echo test $(git-ls-files) = \"\\\"\\\\345/\\\\345land.txt\\\"\" &&\n+    LC_CTYPE=sv_SE.UTF-8 echo test $(git-ls-files) = \"\\\"\\\\345/\\\\345land.txt\\\"\"\n+    '\n+\n+test_expect_success \\\n+    'change single text file, first time' \\\n+    'echo san >>å/åland.txt &&\n+    git-commit -a -m \"Changed again\"\n+    '\n+test_expect_success \\\n+    'add simple binary file' \\\n+    'dd if=/dev/urandom of=å/åäö bs=1 count=312 &&\n+    git-add å/åäö &&\n+    git-commit -a -m \"Changed\" &&\n+    git-ls-files &&\n+    test \"$(git-ls-files)\" = \"\\\"\\\\345/\\\\345land.txt\\\"\n+\\\"\\\\345/\\\\345\\\\344\\\\366\\\"\"\n+    '\n+test_expect_success \\\n+    'change single text file, second time' \\\n+    'echo san >>å/åland.txt &&\n+    git-commit -a -m \"Changed igen\"\n+    '\n+test_expect_success \\\n+    'change single binary file' \\\n+    'dd if=/dev/urandom of=å/åäö bs=1 count=312 &&\n+    git-commit -a -m \"Changed igen\"\n+    '\n+\n+test_expect_success \\\n+    'branch and merge, new file' \\\n+    '\n+    git-tag -f HERE &&\n+    git-checkout -b \"gren1\" &&\n+    echo >å/öland.txt hej &&\n+    git-add . &&\n+    git-commit -a -m \"Ändrad\" &&\n+    git-checkout master &&\n+    git-pull . gren1 &&\n+    test \"$(git-ls-files)\" = \"\\\"\\\\345/\\\\345land.txt\\\"\n+\\\"\\\\345/\\\\345\\\\344\\\\366\\\"\n+\\\"\\\\345/\\\\366land.txt\\\"\"\n+    '\n+test_expect_success \\\n+    'merge old file' \\\n+    '\n+    git-checkout gren1 &&\n+    echo >å/öland.txt hejsan &&\n+    git-commit -a -m \"Ändrad\" &&\n+    git-checkout master &&\n+    git-pull . gren1\n+    test \"$(git-ls-files)\" = \"\\\"\\\\345/\\\\345land.txt\\\"\n+\\\"\\\\345/\\\\345\\\\344\\\\366\\\"\n+\\\"\\\\345/\\\\366land.txt\\\"\"\n+    '\n+\n+test_expect_success \\\n+    'merge in-tree file' \\\n+    '\n+    echo >>å/öland.txt in master &&\n+    git-commit -a -m \"in master\" && \n+    git-checkout gren1 &&\n+    echo >å/öland.txt in branch &&\n+    git-update-index å/öland.txt &&\n+    git-checkout -m master\n+    test \"$(git-ls-files)\" = \"\\\"\\\\345/\\\\345land.txt\\\"\n+\\\"\\\\345/\\\\345\\\\344\\\\366\\\"\n+\\\"\\\\345/\\\\366land.txt\\\"\"\n+    test $(echo $(wc -l <å/öland.txt)) = 6\n+    '\n+\n+test_expect_success \\\n+    'clone to UTF' \\\n+    '\n+    rm -rf ../trash2 &&\n+    LC_ALL=sv_SE.UTF-8 LC_CTYPE=sv_SE.UTF-8 git-clone . ../trash2\n+    '\n+\n+test_done\ndiff --git a/utf.c b/utf.c\nindex eb430b2..7c44cac 100644\n--- a/utf.c\n+++ b/utf.c\n@@ -180,6 +180,236 @@ void localcpy(char *tolocal, char *fromutf, size_t utflen)\n #endif\n }\n \n+#define MAXRESOURCES 50\n+struct resource {\n+  void *key;\n+  void *value;\n+};\n+\n+static struct resource resources[MAXRESOURCES];\n+static void put(void *key, void *value)\n+{\n+  int i;\n+  for (i=0; i<MAXRESOURCES; ++i) {\n+    if (resources[i].key == 0) {\n+      resources[i].key = key;\n+      resources[i].value = value;\n+      return;\n+    }\n+  }\n+}\n+\n+static void* get(void *key)\n+{\n+  int i;\n+  for (i=0; i<MAXRESOURCES; ++i) {\n+    if (resources[i].key == key) {\n+      return resources[i].value;\n+    }\n+  }\n+  return NULL;\n+}\n+\n+static void* getandremove(void *key)\n+{\n+  int i;\n+  for (i=0; i<MAXRESOURCES; ++i) {\n+    if (resources[i].key == key) {\n+      resources[i].key = NULL;\n+      return resources[i].value;\n+    }\n+  }\n+  return NULL;\n+}\n+ \n+static void zfree(void *ret)\n+{\n+  if (ret)\n+    free(ret);\n+}\n+\n+int utf_lstat(char *path, struct stat *buf)\n+{\n+  P((\"utf_lstat(\\\"%s\\\",buf)[\",path));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  int ret = lstat(localpath, buf);\n+  if (ret >= 0 && (buf->st_mode & S_IFMT) == S_IFLNK) {\n+    char sympath[MAXPATHLEN];\n+    int n = utf_readlink(path, sympath, sizeof sympath);\n+    if (n < 0)\n+      die(\"Panic, cannot read link %s in utf_lstat\\n\", path);\n+    buf->st_size = strlen(sympath);\n+  }\n+  return ret;\n+}\n+\n+int utf_stat(char *path, struct stat *buf)\n+{\n+  P((\"utf_stat(\\\"%s\\\",buf)[\",path));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return stat(localpath, buf);\n+}\n+\n+DIR *utf_opendir(char *path)\n+{\n+  P((\"utf_opendir(\\\"%s\\\")\\n\",path));\n+  char localpath[MAXPATHLEN];\n+  DIR *ret = NULL;\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  ret = opendir(localpath);\n+  if (ret)\n+    put(ret, NULL);\n+  return ret;\n+}\n+\n+struct dirent* utf_readdir(DIR *dir)\n+{\n+  P((\"utf_readdir(\\\"%p\\\")\",dir));\n+  struct dirent *ret;\n+  int len;\n+  char utfpath[256];\n+  struct dirent *myret;\n+\n+  zfree(getandremove(dir));\n+  ret = readdir(dir);\n+  if (ret != NULL) {\n+    utfcpy(utfpath, ret->d_name, strlen(ret->d_name)+1);\n+    len=sizeof(struct dirent)+strlen(utfpath)+1;\n+    myret=malloc(len);\n+    memcpy(myret, ret, sizeof (struct dirent));\n+    put(dir, myret);\n+    strcpy(myret->d_name, utfpath);\n+    P((\"=>\\\"%s\\\"\\n\",myret->d_name));\n+    return myret;\n+  } else {\n+    return NULL;\n+  }\n+}\n+\n+int utf_closedir(DIR *dir)\n+{\n+  P((\"utf_closedir(%p)\\n\",dir));\n+  zfree(getandremove(dir));\n+  return closedir(dir);\n+}\n+\n+FILE *utf_fopen(char *path, char *mode)\n+{\n+  P((\"utf_fopen(\\\"%s\\\",\\\"%s\\\")[\",path,mode));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return fopen(localpath, mode);\n+} \n+\n+FILE *utf_freopen(char *path, char *mode, FILE *stream)\n+{\n+  P((\"utf_freopen(\\\"%s\\\",\\\"%s\\\",%p)[\",path,mode,stream));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return freopen(localpath, mode, stream);\n+} \n+\n+int utf_unlink(char *path)\n+{\n+  P((\"utf_unlink(\\\"%s\\\")[\",path));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return unlink(localpath);\n+}\n+\n+int utf_rmdir(char *path)\n+{\n+  P((\"utf_rmdir(\\\"%s\\\")[\",path));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return rmdir(localpath);\n+}\n+\n+int utf_open(char *path, int flags,...)\n+{\n+  va_list va;\n+  int mode;\n+  va_start(va,flags);\n+  mode = va_arg(va,int);\n+  va_end(va);\n+  P((\"utf_open(\\\"%s\\\",%d,%d)[\",path,flags,mode));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return open(localpath, flags, mode);\n+}\n+\n+int utf_access(char *path, int mode)\n+{\n+  P((\"utf_access(\\\"%s\\\",%d)[\",path,mode));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return access(localpath,mode);\n+}\n+\n+char *utf_getcwd(char *buf,int bufsize)\n+{\n+  char localbuf[MAXPATHLEN];\n+  char *ret=getcwd(localbuf,sizeof localbuf);\n+  if (ret != NULL) {\n+    if (buf == NULL) {\n+      if (bufsize == 0) {\n+\tbuf = malloc(bufsize);\n+      } else {\n+\tbuf = malloc(utflen(localbuf,strlen(localbuf)) + 1);\n+      }\n+    }\n+    utfcpy(buf, localbuf, strlen(localbuf) + 1);\n+    return buf;\n+  } else {\n+    return NULL;    \n+  }\n+}\n+\n+int utf_creat(const char *path,int mode)\n+{\n+  P((\"utf_creat(\\\"%s\\\",%d)[\",path,mode));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return creat(localpath, mode);\n+}\n+\n+int utf_mkdir(const char *path,int mode)\n+{\n+  P((\"utf_mkdir(\\\"%s\\\",%d)[\",path,mode));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"\\\"%s\\\"]\\n\",localpath));\n+  return mkdir(localpath, mode);\n+}\n+\n+ssize_t utf_readlink(const char *path,char *buf,size_t bufsiz)\n+{\n+  P((\"utf_readlink(\\\"%s\\\",BUF,%d)[\",path,bufsiz));\n+  char localpath[MAXPATHLEN];\n+  localcpy(localpath, path, strlen(path)+1);\n+  P((\"readlink(%s)\\n\", localpath));\n+  char localret[MAXPATHLEN];\n+  ssize_t ret = readlink(localpath, localret, bufsiz);\n+  if (ret == -1)\n+\treturn ret;\n+  localret[ret] = 0;\n+  utfcpy(buf, localret, ret+1);\n+  P((\"\\\"%s\\\"]\\n\",buf));\n+  return strlen(buf);\n+}\n+\n int PP(const char *fmt,...)\n {\n   va_list va;\n-- \n1.6.3.dirty\n"},{"id":"113705","messageId":"1242168631-30753-7-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-6-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 6/8] test of utf_locallinks","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:29Z","receivedAt":"2009-05-12T22:50:29Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n t/t4004-diff-rename-symlink.sh |    2 +-\n t/t4011-diff-symlink.sh        |    8 ++++----\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t4004-diff-rename-symlink.sh b/t/t4004-diff-rename-symlink.sh\nindex 663096e..b70d4ae 100755\n--- a/t/t4004-diff-rename-symlink.sh\n+++ b/t/t4004-diff-rename-symlink.sh\n@@ -41,7 +41,7 @@ new file mode 120000\n --- /dev/null\n +++ \"b/b\\366zbar\"\n @@ -0,0 +1 @@\n-+xzzzyö\n++xzzzyö\n \\ No newline at end of file\n diff --git \"a/fr\\366tz\" \"b/nitf\\366l\"\n similarity index 100%\ndiff --git a/t/t4011-diff-symlink.sh b/t/t4011-diff-symlink.sh\nindex 0ee622f..b9a0cbf 100755\n--- a/t/t4011-diff-symlink.sh\n+++ b/t/t4011-diff-symlink.sh\n@@ -16,7 +16,7 @@ index 0000000..7c465af\n --- /dev/null\n +++ \"b/fr\\366tz\"\n @@ -0,0 +1 @@\n-+xüzzü\n++xüzzü\n \\ No newline at end of file\n EOF\n \n@@ -42,7 +42,7 @@ index 7c465af..0000000\n --- \"a/fr\\366tz\"\n +++ /dev/null\n @@ -1 +0,0 @@\n--xüzzü\n+-xüzzü\n \\ No newline at end of file\n EOF\n \n@@ -69,9 +69,9 @@ index 7c465af..df1db54 120000\n --- \"a/fr\\366tz\"\n +++ \"b/fr\\366tz\"\n @@ -1 +1 @@\n--xüzzü\n+-xüzzü\n \\ No newline at end of file\n-+üxüuz\n++üxüuz\n \\ No newline at end of file\n EOF\n \n-- \n1.6.3.dirty\n"},{"id":"113706","messageId":"1242168631-30753-8-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-7-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 7/8] Convert symlink dest in diff","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:30Z","receivedAt":"2009-05-12T22:50:30Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n diff.c |   14 ++++++++++++--\n 1 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 170ec5a..c8132a4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1319,8 +1319,18 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)\n \t\t\t\tlocate_size_cache(s->sha1, 0, s->size);\n \t\t}\n \t\telse {\n-\t\t\ts->data = read_sha1_file(s->sha1, type, &s->size);\n-\t\t\ts->should_free = 1;\n+\t\t\tif (S_ISLNK(s->mode)) {\n+\t\t\t\tint linksize;\n+\t\t\t        char *linkdata = read_sha1_file(s->sha1, type, &linksize);\n+\t\t\t\ts->size = locallen(linkdata, linksize);\n+\t\t\t\ts->data = xmalloc(s->size + 1);\n+\t\t\t\tlocalcpy(s->data, linkdata, linksize + 1);\n+\t\t\t\ts->should_free = 1;\n+\t\t\t\tfree(linkdata);\n+\t\t\t} else {\n+\t\t\t\ts->data = read_sha1_file(s->sha1, type, &s->size);\n+\t\t\t\ts->should_free = 1;\n+\t\t\t}\n \t\t}\n \t}\n \treturn 0;\n-- \n1.6.3.dirty\n"},{"id":"113704","messageId":"1242168631-30753-9-git-send-email-robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"1242168631-30753-8-git-send-email-robin.rosenberg@dewire.com","subject":"[RFC 8/8] UTF-8 in non-SHA1-objects","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-12T22:50:31Z","receivedAt":"2009-05-12T22:50:31Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n dir.c |   22 ++++++++++++++------\n utf.c |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 80 insertions(+), 8 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 96389b3..043df50 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -114,22 +114,26 @@ static int add_excludes_from_file_1(const char *fname,\n {\n \tstruct stat st;\n \tint fd, i;\n-\tlong size;\n-\tchar *buf, *entry;\n+\tlong insize, size;\n+\tchar *inbuf, *entry, *buf;\n \n \tfd = open(fname, O_RDONLY);\n \tif (fd < 0 || fstat(fd, &st) < 0)\n \t\tgoto err;\n-\tsize = st.st_size;\n-\tif (size == 0) {\n+\tinsize = st.st_size;\n+\tif (insize == 0) {\n \t\tclose(fd);\n \t\treturn 0;\n \t}\n-\tbuf = xmalloc(size+1);\n-\tif (read(fd, buf, size) != size)\n+\tinbuf = xmalloc(insize+1);\n+\tif (read(fd, inbuf, insize) != insize)\n \t\tgoto err;\n \tclose(fd);\n \n+\tsize = utflen(inbuf, insize+1);\n+\tbuf = xmalloc(size);\n+\tutfcpy(buf, inbuf, insize+1);\n+\n \tbuf[size++] = '\\n';\n \tentry = buf;\n \tfor (i = 0; i < size; i++) {\n@@ -141,9 +145,11 @@ static int add_excludes_from_file_1(const char *fname,\n \t\t\tentry = buf + i + 1;\n \t\t}\n \t}\n+\tfree(inbuf);\n \treturn 0;\n \n  err:\n+\t// bug: not freeing inbuf...\n \tif (0 <= fd)\n \t\tclose(fd);\n \treturn -1;\n@@ -185,6 +191,7 @@ static int excluded_1(const char *pathname,\n \t\t      int pathlen,\n \t\t      struct exclude_list *el)\n {\n+\tP((\"excluded_1(\\\"%0.*s\\\") = \", pathlen, pathname));\n \tint i;\n \n \tif (el->nr) {\n@@ -202,6 +209,7 @@ static int excluded_1(const char *pathname,\n \t\t\t\t/* match basename */\n \t\t\t\tconst char *basename = strrchr(pathname, '/');\n \t\t\t\tbasename = (basename) ? basename+1 : pathname;\n+\t\t\t\tP((\"fnmatch(\\\"%s\\\",\\\"%s\\\")\", exclude, basename));\n \t\t\t\tif (fnmatch(exclude, basename, 0) == 0)\n \t\t\t\t\treturn to_exclude;\n \t\t\t}\n@@ -218,7 +226,7 @@ static int excluded_1(const char *pathname,\n \t\t\t\t    (baselen && pathname[baselen-1] != '/') ||\n \t\t\t\t    strncmp(pathname, x->base, baselen))\n \t\t\t\t    continue;\n-\n+\t\t\t\tP((\"fnmatch(\\\"%s\\\",\\\"%s\\\")\", exclude, pathname+baselen));\n \t\t\t\tif (fnmatch(exclude, pathname+baselen,\n \t\t\t\t\t    FNM_PATHNAME) == 0)\n \t\t\t\t\treturn to_exclude;\ndiff --git a/utf.c b/utf.c\nindex 7c44cac..cca64dc 100644\n--- a/utf.c\n+++ b/utf.c\n@@ -228,9 +228,28 @@ static void zfree(void *ret)\n     free(ret);\n }\n \n+int isgitpath(char *path)\n+{\n+  char *gitdir=getenv(\"GIT_DIR\");\n+  if (!gitdir)\n+    gitdir=\".git\";\n+\n+  P((\"gitdir=%s\\n\",gitdir));\n+  if (strncmp(path, gitdir, strlen(gitdir)) == 0) {\n+    P((\"gitdir:YES\\n\"));\n+    return 1;\n+  } else {\n+    P((\"gitdir:NO\\n\"));\n+  }\n+  \n+  return 0;\n+}\n+\n int utf_lstat(char *path, struct stat *buf)\n {\n   P((\"utf_lstat(\\\"%s\\\",buf)[\",path));\n+  if (isgitpath(path))\n+    return lstat(path, buf);\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -248,6 +267,8 @@ int utf_lstat(char *path, struct stat *buf)\n int utf_stat(char *path, struct stat *buf)\n {\n   P((\"utf_stat(\\\"%s\\\",buf)[\",path));\n+  if (isgitpath(path))\n+    return stat(path, buf);\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -257,19 +278,27 @@ int utf_stat(char *path, struct stat *buf)\n DIR *utf_opendir(char *path)\n {\n   P((\"utf_opendir(\\\"%s\\\")\\n\",path));\n+  if (isgitpath(path)) {\n+    P((\"not transforming dir\"));\n+    return opendir(path);\n+  }\n   char localpath[MAXPATHLEN];\n   DIR *ret = NULL;\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n   ret = opendir(localpath);\n   if (ret)\n-    put(ret, NULL);\n+    put(ret, xmalloc(1));\n   return ret;\n }\n \n struct dirent* utf_readdir(DIR *dir)\n {\n   P((\"utf_readdir(\\\"%p\\\")\",dir));\n+  if (!get(dir)) {\n+    P((\"Skipping\\n\"));\n+    return readdir(dir);\n+  }\n   struct dirent *ret;\n   int len;\n   char utfpath[256];\n@@ -294,6 +323,9 @@ struct dirent* utf_readdir(DIR *dir)\n int utf_closedir(DIR *dir)\n {\n   P((\"utf_closedir(%p)\\n\",dir));\n+  if (!get(dir))\n+    return closedir(dir);\n+\n   zfree(getandremove(dir));\n   return closedir(dir);\n }\n@@ -301,6 +333,9 @@ int utf_closedir(DIR *dir)\n FILE *utf_fopen(char *path, char *mode)\n {\n   P((\"utf_fopen(\\\"%s\\\",\\\"%s\\\")[\",path,mode));\n+  if (isgitpath(path))\n+    return fopen(path,mode);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -310,6 +345,9 @@ FILE *utf_fopen(char *path, char *mode)\n FILE *utf_freopen(char *path, char *mode, FILE *stream)\n {\n   P((\"utf_freopen(\\\"%s\\\",\\\"%s\\\",%p)[\",path,mode,stream));\n+  if (isgitpath(path))\n+    return utf_freopen(path, mode, stream);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -319,6 +357,9 @@ FILE *utf_freopen(char *path, char *mode, FILE *stream)\n int utf_unlink(char *path)\n {\n   P((\"utf_unlink(\\\"%s\\\")[\",path));\n+  if (isgitpath(path))\n+    return unlink(path);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -328,6 +369,9 @@ int utf_unlink(char *path)\n int utf_rmdir(char *path)\n {\n   P((\"utf_rmdir(\\\"%s\\\")[\",path));\n+  if (isgitpath(path))\n+    return rmdir(path);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -342,6 +386,9 @@ int utf_open(char *path, int flags,...)\n   mode = va_arg(va,int);\n   va_end(va);\n   P((\"utf_open(\\\"%s\\\",%d,%d)[\",path,flags,mode));\n+  if (isgitpath(path))\n+    return open(path, flags, mode);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -351,6 +398,9 @@ int utf_open(char *path, int flags,...)\n int utf_access(char *path, int mode)\n {\n   P((\"utf_access(\\\"%s\\\",%d)[\",path,mode));\n+  if (isgitpath(path))\n+    return access(path, mode);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -369,6 +419,11 @@ char *utf_getcwd(char *buf,int bufsize)\n \tbuf = malloc(utflen(localbuf,strlen(localbuf)) + 1);\n       }\n     }\n+    if (isgitpath(localbuf)) {\n+      strcpy(buf, localbuf);\n+      return buf;\n+    }\n+\n     utfcpy(buf, localbuf, strlen(localbuf) + 1);\n     return buf;\n   } else {\n@@ -379,6 +434,9 @@ char *utf_getcwd(char *buf,int bufsize)\n int utf_creat(const char *path,int mode)\n {\n   P((\"utf_creat(\\\"%s\\\",%d)[\",path,mode));\n+  if (isgitpath(path))\n+    return creat(path, mode);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -388,6 +446,9 @@ int utf_creat(const char *path,int mode)\n int utf_mkdir(const char *path,int mode)\n {\n   P((\"utf_mkdir(\\\"%s\\\",%d)[\",path,mode));\n+  if (isgitpath(path))\n+    return mkdir(path,mode);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"\\\"%s\\\"]\\n\",localpath));\n@@ -397,6 +458,9 @@ int utf_mkdir(const char *path,int mode)\n ssize_t utf_readlink(const char *path,char *buf,size_t bufsiz)\n {\n   P((\"utf_readlink(\\\"%s\\\",BUF,%d)[\",path,bufsiz));\n+  if (isgitpath(path))\n+    return readlink(path, buf, bufsiz);\n+\n   char localpath[MAXPATHLEN];\n   localcpy(localpath, path, strlen(path)+1);\n   P((\"readlink(%s)\\n\", localpath));\n-- \n1.6.3.dirty\n"},{"id":"113713","messageId":"alpine.DEB.1.00.0905130215260.27348@pacific.mpi-cbg.de","threadId":"19325","inReplyTo":"1242168631-30753-2-git-send-email-robin.rosenberg@dewire.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-13T00:20:32Z","receivedAt":"2009-05-13T00:20:32Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 13 May 2009, Robin Rosenberg wrote:\n\n> ---\n\nNo SOB.\n\n> diff --git a/git.c b/git.c\n> index 6475847..bd4e726 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -272,6 +272,15 @@ static void handle_internal_command(int argc, const char **argv, char **envp)\n>  \t};\n>  \tint i;\n>  \n> +#ifdef DEBUG\n> +\tif (debug()) {\n> +\t\tfprintf(stderr,\"GIT-\");\n> +\t\tfor (i = 1; i<argc; ++i)\n> +\t\t\tfprintf(stderr,\"%s\",argv[i]);\n> +\t\tfprintf(stderr,\"\\n\");\n> +\t}\n> +#endif\n> +\n\nWhat does that have to do with UTF support?\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 07cb706..e8aefd8 100755\n> --- a/t/test-lib.sh\n> +++  b/t/test-lib.sh\n> @@ -4,11 +4,9 @@\n>  #\n>  \n>  # For repeatability, reset the environment to known value.\n> -LANG=C\n> -LC_ALL=C\n>  PAGER=cat\n>  TZ=UTC\n> -export LANG LC_ALL PAGER TZ\n> +export PAGER TZ\n>  EDITOR=:\n>  VISUAL=:\n>  unset AUTHOR_DATE\n\nLikewise.\n\n> diff --git a/test-utf.c b/test-utf.c\n> new file mode 100644\n> index 0000000..133eea0\n> --- /dev/null\n> +++ b/test-utf.c\n> @@ -0,0 +1,61 @@\n> +#include <stdio.h>\n> +#include <time.h>\n> +#include <assert.h>\n> +\n> +#include \"cache.h\"\n> +#include \"utf.h\"\n> +\n> +int main(int argc, char **argv)\n> +{\n> +\tint i;\n> +\n> +#if 0\n> +\tfor (i = 1; i < argc; i++) {\n> +\t\tchar result1[100];\n> +\t\tchar result2[100];\n> +\n> +\t\tutfcpy(result1, argv[i], strlen(argv[i])+1);\n> +\t\tlocalcpy(result2, result1, strlen(result1)+1);\n> +\n> +\t\tprintf(\"%s -> %s -> %s\\n\", argv[i], result1, result2);\n> +\t}\n> +\treturn 0;\n> +#endif\n> +\n> +#define test(name) case __LINE__: current_name=name; n++; printf(\"Testing case #%d: %s\\n\", n, current_name);\n> +#define end_test break;\n> +#define begin_suite() char *current_name=0; int n=1; for (i=0; i<1000; ++i) { switch(i) { \n> +#define concats(a,b) #a #b\n> +\n> +#undef strcmp\n> +#define assertStringEquals(a,b) assert(#a #b && strcmp(a,b)==0)\n> +#define assertIntEquals(a,b) assert(#a #b && (a)==(b))\n> +\n> +#define end_suite() }}\n> +\n> +\tbegin_suite();\n> +\n> +\ttest(\"utfcpy\") {\n> +\t  char result[100];\n> +\t  utfcpy(result,\"?ndrad\",7);\n> +\t  assertStringEquals(result,\"\\303\\204ndrad\");\n> +\t} end_test;\n> +\n> +\ttest(\"utflen\") {\n> +\t  int result=utflen(\"?ndrad\",7);\n> +\t  assertIntEquals(result,8);\n> +\t} end_test;\n> +\n> +\ttest(\"localcpy\") {\n> +\t  char result[100];\n> +\t  localcpy(result,\"\\303\\204ndrad\",8);\n> +\t  assertStringEquals(result,\"?ndrad\");\n> +\t} end_test;\n> +\n> +\ttest(\"locallen\") {\n> +\t  int result=locallen(\"\\303\\204ndrad\",8);\n> +\t  assertIntEquals(result,7);\n> +\t} end_test;\n> +\n> +\tend_suite();\n> +}\n\nShould the test-utf binary not rather perform _actions_ (i.e. \ntransformations) instead of checks?\n\n> diff --git a/utf.c b/utf.c\n> new file mode 100644\n> index 0000000..eb430b2\n> --- /dev/null\n> +++ b/utf.c\n> @@ -0,0 +1,207 @@\n> +#undef UTF8INTERNAL\n> +\n> +#include <langinfo.h>\n> +#include <iconv.h>\n> +#include \"cache.h\"\n> +#include <locale.h>\n> +#include <stdarg.h>\n> +\n> +static iconv_t local_to_utf8 = (iconv_t)-1;\n> +static iconv_t utf8_to_local = (iconv_t)-1;\n> +static iconv_t utf8_to_utf8 = (iconv_t)-1;\n> +static int same = 0;\n> +\n> +#if TEST\n> +#define die printf\n> +#endif\n\nThis is ugly.\n\n> +\n> +static void\tinitlocale()\n> +{\n> +#ifndef NO_ICONV\n> +\tif (!same && local_to_utf8 == (iconv_t)-1) {\n> +\t\tsetlocale(LC_CTYPE, \"\");\n> +\t\tchar *local_encoding = nl_langinfo(CODESET);\n> +#ifdef DEBUG\n> +\t\tif (debug()) fprintf(stderr,\"encoding=%s\\n\", local_encoding);\n> +#endif\n\nThis is ugly.\n\n> +\t\tif (strcmp(local_encoding,\"UTF-8\") == 0) {\n> +\t\t\tsame = 1;\n> +\t\t\treturn;\n> +\t\t}\n> +\t\tlocal_to_utf8 = iconv_open(\"UTF-8\",  local_encoding);\n> +\t\tif (local_to_utf8 == (iconv_t)-1) {\n> +\t\t\tdie(\"cannot setup locale conversion from %s: %s\", local_encoding, strerror(errno));\n> +\t\t}\n> +#ifdef DEBUG\n> +\t\tif (debug()) fprintf(stderr,\"utf8_to_local = iconv_open(%s,UTF-8)\\n\",local_encoding);\n> +#endif\n\nThis is ugly.\n\n> +\t\tutf8_to_local = iconv_open(local_encoding,  \"UTF-8\");\n> +\t\tif (utf8_to_local == (iconv_t)-1) {\n> +\t\t\tdie(\"cannot setup locale conversion from %s: %s\", local_encoding, strerror(errno));\n> +\t\t}\n> +\n> +\t\tutf8_to_utf8 = iconv_open(\"UTF-8\",\"UTF-8\");\n> +\t\tif (utf8_to_utf8 == (iconv_t)-1) {\n> +\t\t\tdie(\"cannot setup locale conversion from UTF-8 to UTF-8: %s\",strerror(errno));\n> +\t\t}\n> +\t}\n> +#endif\n> +}\n> +\n> +int maybe_utf8(const char *local, size_t len)\n> +{\n> +  char *self = xcalloc(1,len+1);\n> +  char *selfp = self;\n> +  size_t outlen = len+1;\n> +  int ret = iconv(utf8_to_utf8, (char**)&local, &len, &selfp, &outlen);\n> +  free(self);\n> +  P((\"maybelocal: %0.*s %s\\n\", len, local, ret!=-1 ? \"yes\" : \"no\"));\n> +  return ret != -1;\n> +}\n> +\n> +size_t utflen(const char *local, size_t locallen)\n> +{\n> +#ifndef NO_ICONV\n> +\tinitlocale();\n> +\tif (same) {\n> +\t\treturn locallen;\n> +\t}\n> +\tif (maybe_utf8(local, locallen))\n> +\t\treturn locallen;\n> +\n> +\tsize_t outlen=locallen*6;\n> +\tchar *outbuf=xcalloc(outlen,1);\n> +\tchar *out=outbuf;\n> +\ticonv(local_to_utf8, NULL, NULL, NULL, NULL);\n> +\tconst char *vlocal = local;\n> +\tsize_t vlocallen = locallen;\n> +\tif (iconv(local_to_utf8,  (char**)&vlocal,  &vlocallen,  &out,  &outlen) == -1) {\n> +#if TEST\n> +\t\tperror(\"failed\");\n> +#endif\n> +\t\tfree(outbuf);\n> +\t\treturn locallen;\n> +\t}\n> +\t*out = 0;\n> +\tfree(outbuf);\n> +\treturn locallen*6 - outlen;\n> +#else\n> +\treturn locallen;\n> +#endif\n> +}\n> +\n> +/* Copy and transform */\n> +void utfcpy(char *to_utf, char *from_local, size_t localsize)\n> +{\n> +#ifdef DEBUG\n> +\tchar *a=to_utf,*b=from_local;\n> +#endif\n> +#ifndef NO_ICONV\n> +\tinitlocale();\n> +\tif (same) {\n> +\t\tmemcpy(to_utf, from_local, localsize);\n> +\t\treturn;\n> +\t}\n> +\tif (maybe_utf8(from_local, localsize)) {\n> +\t\tmemcpy(to_utf, from_local, localsize);\n> +\t\treturn;\n> +\t}\n> +\n> +\tsize_t outlen=localsize*6;\n> +\ticonv(local_to_utf8, NULL, NULL, NULL, NULL);\n> +\tchar *vfrom_local = from_local;\n> +\tchar *vto_utf = to_utf;\n> +\tsize_t vlocalsize = localsize;\n> +\tif (iconv(local_to_utf8,  &vfrom_local,  &vlocalsize,  &vto_utf,  &outlen) == -1) {\n> +\t\tfprintf(stderr,\"Failed to convert %0.*s to UTF\\n\", localsize, from_local);\n> +\t\tmemcpy(to_utf,  from_local,  localsize);\n> +\t}\n> +#else\n> +\tmemcpy(to_utf, from_local, localsize);\n> +#endif\n> +#ifdef DEBUG\n> +\tif (debug()) fprintf(stderr,\"%0.*s ->UTF %0.*s\\n\", localsize, b, localsize*6 - outlen, a);\n> +#endif\n> +}\n\nOkay, I'll stop here.  You might want to clean up your patch series before \nresending.\n\nCiao,\nDscho\n"},{"id":"113718","messageId":"200905130724.44634.robin.rosenberg@dewire.com","threadId":"19325","inReplyTo":"alpine.DEB.1.00.0905130215260.27348@pacific.mpi-cbg.de","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-05-13T05:24:44Z","receivedAt":"2009-05-13T05:24:44Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"> This is ugly.\n\nI told you so. No news.\n\n> Okay, I'll stop here.  You might want to clean up your patch series before \n> resending.\n\nI also told you why why I stopped working on the patches. The patches are not part of\na beauty contest and not meant for inclusion as such. If you cannot see through the\nrubble, just ignore the patches. If the conclusion is that this is a way forward, then I\ncould start working on a completely new set of much cleaner patches.,\n\n-- robin\n"},{"id":"113751","messageId":"4A0A91CE.3080905@gmail.com","threadId":"19325","inReplyTo":"200905130724.44634.robin.rosenberg@dewire.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-13T09:24:30Z","receivedAt":"2009-05-13T09:24:30Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"Robin Rosenberg wrote on 13.5.2009 8:24:\n> If the conclusion is that this is a way forward, then I\n> could start working on a completely new set of much cleaner patches.,\n\nThat would be great!\n\nI see that in those early patches you took the approach of converting \nthe filenames from the local encoding to UTF-8 at the outer edges of \nGit. That obviously was the easiest way to make the changes with minimal \nchanges to Git.\n\nI've been thinking about a bit more extensive approach, which should \nserve the interest of all stakeholders:\n\n\nNow the tree object contains the following information for each file: \nfilename, mode, sha1. To that would be added one more string: filename \nencoding. Unless the encoding is specified (such as in old commits \nbefore the encoding information was added), the default encoding is \n\"binary\", which is the same as how Git works now (it thinks filenames as \nseries of bytes, ignoring their encoding completely).\n\nWhen a file is added/committed, the following things will happen:\n\n1. Git finds out what is the filename encoding used by the system. Git \nwill try to detect it automatically from the environment, and the \nautodetected value can be overridden by setting a config variable \n\"i18n.localFilenameEncoding\". If autodetection fails, it will default to \n\"binary\".\n\n2. Git reads the config variable \"i18n.commitFilenameEncoding\". If \nlocalFilenameEncoding equals commitFilenameEncoding, or if either of \nthem is \"binary\", go to step 3A. Otherwise go to step 3B.\n\n3A. Git saves the filename together with the local filename encoding. \nThe bytes of the filename are not changed when it is stored in the \nrepository (the same as now).\n\n3B. Git converts the filename from localFilenameEncoding to \ncommitFilenameEncoding. (The commitFilenameEncoding may also specify a \nnormalized form for UTF-8, for example \"UTF-8 NFC\". This is needed for \nMac OS X.) Then Git saves the filename together with the commit filename \nencoding.\n\n\nWhen a file is checked out, the following things will happen:\n\n1. Git reads the actual filename encoding from the repository. If it is \nnot specified, \"binary\" will be assumed.\n\n2. Git detects the local filename encoding, the same was as before. If \nthe actual filename encoding equals the local filename encoding, or if \neither of them is \"binary\", go to step 3A. Otherwise go to step 3B.\n\n3A. Git creates the file using the same bytes as filename as what is \nstored in the repository. This is the same as how Git works now.\n\n3B. Git converts the filename from the actual filename encoding to the \nlocal filename encoding, and creates the file using the encoding of the \nlocal platform.\n\n\nThis should fit in with Git's philosophy of not modifying the user's \ndata without the user's permission. The data will always be stored \nunchanged into the repository, unless the user specifies \n\"i18n.commitFilenameEncoding\". The conversions are by default done only \non checkout. Git will try to serve the needs of the user as well as it \ncan by detecting the local filename encoding, but if the user so \ndesires, he can disable the conversions by specifying \n\"i18n.localFilenameEncoding\" as \"binary\", in which case Git will work \nthe same way as it does today.\n\n\nI was browsing Git's code, and it seems that the encoding information \nwould need to be added to struct name_entry in tree-walk.h. A quick \nsearch reveals that name_entry is used in 15 files, out of which only 4 \nfiles use it more than once. It would probably make sense to create a \nnew datatype for the filename, for example \"struct encoded_path { const \nchar *path; const char *encoding; }\", and then provide functions for \naccessing the filename with the right encoding (commit or local).\n\nI might even myself be able to make that change, because Git is not \nlegacy software (it has tests) and the needed changes seem quite local. \nI would just need a way to detect the encodings (at first it could rely \non manually set config variables) and have a library for doing the \nencoding conversions.\n\nOne big question is, that will this change require a change to the \nrepository format? Will it be possible to add the encoding field to the \ntree object, without breaking compatibility with older Git clients? If \ncompatibility needs to be broken, how it can be done in a controlled \nfashion?\n\n-- \nEsko Luontola\nwww.orfjackal.net\n"},{"id":"113759","messageId":"4A0A9AA2.1000004@op5.se","threadId":"19325","inReplyTo":"4A0A91CE.3080905@gmail.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-13T10:02:10Z","receivedAt":"2009-05-13T10:02:10Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Esko Luontola wrote:\n> Robin Rosenberg wrote on 13.5.2009 8:24:\n>> If the conclusion is that this is a way forward, then I\n>> could start working on a completely new set of much cleaner patches.,\n> \n> That would be great!\n> \n> I see that in those early patches you took the approach of converting \n> the filenames from the local encoding to UTF-8 at the outer edges of \n> Git. That obviously was the easiest way to make the changes with minimal \n> changes to Git.\n> \n> I've been thinking about a bit more extensive approach, which should \n> serve the interest of all stakeholders:\n> \n> \n> Now the tree object contains the following information for each file: \n> filename, mode, sha1. To that would be added one more string: filename\n> encoding. Unless the encoding is specified (such as in old commits \n> before the encoding information was added), the default encoding is \n> \"binary\", which is the same as how Git works now (it thinks filenames as \n> series of bytes, ignoring their encoding completely).\n> \n\n[ long and incompatible plan removed ]\n\n> One big question is, that will this change require a change to the \n> repository format? Will it be possible to add the encoding field to the \n> tree object, without breaking compatibility with older Git clients? If \n> compatibility needs to be broken, how it can be done in a controlled \n> fashion?\n> \n\nGenerally when one wants to change one of the basic object types in\ngit, some extraordinary benefit has to be shown that is not aimed\nat just a few people. Academic benefits (ie, \"non-real-worldy\") do\nnot fall into that category. In fact, it's so rare for someone to\nprovide such enormous benefit that the only time a core object format\nin git has been incompatibly changed is when Linus decided that trees\nshould be able to have subtrees. The change reduced the repository\nsize for the early git-tracked Linux kernel to about 4% of its\noriginal size, so there was a clear, undisputable and obvious benefit\nhuge enough to warrant breaking the git repository format entirely\njust to get it in (I might have gotten those details entirely wrong,\nbut it was something along those lines).\n\nSo unless you can change tree objects in a way that lets older git\nclients understand them while still adding this encoding cruft\n(it's cruft to me), I think your chances of getting such a change\ninto the git core are about the size of the colour green.\n\nIf you're *really* serious about it though, here's how to go about\nit:\n\n1. Make the changes so that newer git can always read and operate\non trees without the encoding information, regardless of what the\nconfiguration says.\n2. Modify 1.4.x branch to support this new format too, at least\nfor reading trees with the information in it. Otherwise some\npackage maintainers will just ignore such compatibility.\n3. Modify 1.5.x branch similarly.\n5. Make it configurable, but turned off by default and with a big\nfat warning when its turned on.\n6. 2 years later, remove the warning.\n7. 2 years lter, turn it on by default.\n8. 2 years later, remove the config option and make it a new\nmajor release, but maintain the two codepaths forever.\n\n\n1.[45].x branches are imaginary. They represent the branch that\ngets created when a new release in that series is necessary for\nsome reason.\n\n\nI haven't perused Robin's patches enough to know how they would\ninteract with older git, and I'm not really interested in encoding\nissues. English being the lingua franca of internet and opensource\ndevelopment anyways, every project I've ever seen has only files\nnamed in a manner that would fit nicely into 7-bit ascii.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"113761","messageId":"alpine.DEB.1.00.0905131213310.5046@intel-tinevez-2-302","threadId":"19325","inReplyTo":"200905130724.44634.robin.rosenberg@dewire.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-13T10:14:02Z","receivedAt":"2009-05-13T10:14:02Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 13 May 2009, Robin Rosenberg wrote:\n\n> > Okay, I'll stop here.  You might want to clean up your patch series \n> > before resending.\n> \n> I also told you why why I stopped working on the patches. The patches \n> are not part of a beauty contest and not meant for inclusion as such. If \n> you cannot see through the rubble, just ignore the patches. If the \n> conclusion is that this is a way forward, then I could start working on \n> a completely new set of much cleaner patches.,\n\nYou know, I really should have gone to bed, because I managed to mistake \nthe \"RFC\" for \"PATCH\".\n\nVery sorry for that,\nDscho\n"},{"id":"113764","messageId":"4A0A9F23.8050804@gmail.com","threadId":"19325","inReplyTo":"4A0A9AA2.1000004@op5.se","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-13T10:21:23Z","receivedAt":"2009-05-13T10:21:23Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"Andreas Ericsson wrote on 13.5.2009 13:02:\n> If you're *really* serious about it though, here's how to go about\n> it:\n\nThanks for the pointers. I'll have a look at what the repository format \nis and how to create a migration path.\n\n-- \nEsko Luontola\nwww.orfjackal.net\n"},{"id":"113775","messageId":"81b0412b0905130444t3eee4c92r9e2d3d995bfccfe4@mail.gmail.com","threadId":"19325","inReplyTo":"4A0A9F23.8050804@gmail.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2009-05-13T11:44:22Z","receivedAt":"2009-05-13T11:44:22Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2009/5/13 Esko Luontola <esko.luontola@gmail.com>:\n> Andreas Ericsson wrote on 13.5.2009 13:02:\n>>\n>> If you're *really* serious about it though, here's how to go about\n>> it:\n>\n> Thanks for the pointers. I'll have a look at what the repository format is\n> and how to create a migration path.\n>\n\nWhile at it, think about how are you going to merge trees where filenames\nhave changed their encoding (because of local configurations, you know).\nAnd take a look at memory use in your new encoding-per-filename format\n"},{"id":"113825","messageId":"7vljp04z6j.fsf@alter.siamese.dyndns.org","threadId":"19325","inReplyTo":"4A0A91CE.3080905@gmail.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-13T18:48:04Z","receivedAt":"2009-05-13T18:48:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Esko Luontola <esko.luontola@gmail.com> writes:\n\n> Robin Rosenberg wrote on 13.5.2009 8:24:\n>> If the conclusion is that this is a way forward, then I\n>> could start working on a completely new set of much cleaner patches.,\n>\n> That would be great!\n>\n> I see that in those early patches you took the approach of converting\n> the filenames from the local encoding to UTF-8 at the outer edges of\n> Git. That obviously was the easiest way to make the changes with\n> minimal changes to Git.\n\nWhich would be the _only_ sane approach.\n\nIf you allow people to record otherwise exactly the same tree object in\ndifferent encoding, like you seem to have in mind, subtree comparision\nbased on the object name will not work and you will end up always\ntraversing down to the tip, because you won't know if your subtrees need\nfilename iconv until you recurse into them and actually take a look.\n\nDon't do it.\n"},{"id":"113828","messageId":"4A0B2007.70808@gmail.com","threadId":"19325","inReplyTo":"7vljp04z6j.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-13T19:31:19Z","receivedAt":"2009-05-13T19:31:19Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"Junio C Hamano wrote on 13.5.2009 21:48:\n> If you allow people to record otherwise exactly the same tree object in\n> different encoding, like you seem to have in mind, subtree comparision\n> based on the object name will not work and you will end up always\n> traversing down to the tip, because you won't know if your subtrees need\n> filename iconv until you recurse into them and actually take a look.\n\nCould you please educate me, that which operations depend on \"doing \nsubtree comparisons based on the object name\", and in about which \nfiles/functions those comparisons are done? Also, do you mean by \"object \nname\" the SHA1 of the object, the filename of a file/directory, or \nsomething else? I'd like to know, which parts of the code I should read, \nto get a better mental model on how Git works.\n\n-- \nEsko Luontola\nwww.orfjackal.net\n"},{"id":"113838","messageId":"7vtz3o2293.fsf@alter.siamese.dyndns.org","threadId":"19325","inReplyTo":"4A0B2007.70808@gmail.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-13T20:10:00Z","receivedAt":"2009-05-13T20:10:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Esko Luontola <esko.luontola@gmail.com> writes:\n\n> Junio C Hamano wrote on 13.5.2009 21:48:\n>> If you allow people to record otherwise exactly the same tree object in\n>> different encoding, like you seem to have in mind, subtree comparision\n>> based on the object name will not work and you will end up always\n>> traversing down to the tip, because you won't know if your subtrees need\n>> filename iconv until you recurse into them and actually take a look.\n>\n> Could you please educate me, that which operations depend on \"doing\n> subtree comparisons based on the object name\",\n\n\"diff-tree A B\" looks at corresponding subtrees of A and B and does not\nrecurse into the identical subdirectories; \"git log -- dir\" uses this fact\nto speed up the checking of differences.  When a typical commit touches\nonly a handful of paths in a project with 20k paths, this really matters.\n\n> files/functions those comparisons are done? Also, do you mean by\n> \"object name\" the SHA1 of the object, the filename of a\n> file/directory, or something else?\n\nWhat is colloquially known as SHA-1 has an official terminology; see\nDocumentation/glossary.txt.\n"},{"id":"113868","messageId":"7viqk4z4cv.fsf@alter.siamese.dyndns.org","threadId":"19325","inReplyTo":"200905130724.44634.robin.rosenberg@dewire.com","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-14T04:38:08Z","receivedAt":"2009-05-14T04:38:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg <robin.rosenberg@dewire.com> writes:\n\n>> This is ugly.\n>\n> I told you so. No news.\n>\n>> Okay, I'll stop here.  You might want to clean up your patch series before \n>> resending.\n>\n> I also told you why why I stopped working on the patches. The patches are not part of\n> a beauty contest and not meant for inclusion as such.\n\nIt is rather sad; I suspect that the core of the series is buried in too\nmuch cruft deep enough to discourage many potential reviewers.  I think\nthe entire series look incoherent because attacking two largely unrelated\nthings at once.\n\n (1) Normalizing pathnames internally to UTF-8 and possibly convert it\n     back to native upon use (e.g. creat(), lstat(), unlink()) and output.\n     As Linus analyzed, this shouldn't be done too early in the callchain\n     for performance reasons, but I think your patch would give us a good\n     set of starting points to follow where the result from readdir(),\n     user input and other things that are pathnames come from and go.\n\n     This part of the patch series was inspiring.  You have to worry about\n     gitignore, gitattributes and readlink() vs contents of a blob object\n     that records a symbolic link values, which I think either escaped\n     analysis people have done so far or being ignored as a small detail,\n     but they are important;\n\n (2) Passing cat-file output through iconv to convert it.\n\n     I think this is unwarranted, even if the object given to cat-file\n     happens to be a commit or a tag object and you want to convert their\n     messages in native encoding.\n\n     I am not sure what should happen to \"cat-file tree\", \"ls-files\" and\n     \"ls-tree\".  The output from these plumbing does show pathnames, but I\n     tend to think it is Porcelain's job to turn them into whatever\n     encoding they want to use.  So are input to \"update-index --stdin\",\n     but I am still just thinking out loud.\n"},{"id":"113932","messageId":"76718490905140657h4f4544fdrc5b9ce15ca36c8a5@mail.gmail.com","threadId":"19325","inReplyTo":"7viqk4z4cv.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC 1/8] UTF helpers","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2009-05-14T13:57:30Z","receivedAt":"2009-05-14T13:57:30Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Thu, May 14, 2009 at 12:38 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> It is rather sad; I suspect that the core of the series is buried in too\n> much cruft deep enough to discourage many potential reviewers.  I think\n> the entire series look incoherent because attacking two largely unrelated\n> things at once.\n>\n>  (1) Normalizing pathnames internally to UTF-8 and possibly convert it\n>     back to native upon use (e.g. creat(), lstat(), unlink()) and output.\n>     As Linus analyzed, this shouldn't be done too early in the callchain\n>     for performance reasons, but I think your patch would give us a good\n>     set of starting points to follow where the result from readdir(),\n>     user input and other things that are pathnames come from and go.\n>\n>     This part of the patch series was inspiring.  You have to worry about\n>     gitignore, gitattributes and readlink() vs contents of a blob object\n>     that records a symbolic link values, which I think either escaped\n>     analysis people have done so far or being ignored as a small detail,\n>     but they are important;\n>\n>  (2) Passing cat-file output through iconv to convert it.\n>\n>     I think this is unwarranted, even if the object given to cat-file\n>     happens to be a commit or a tag object and you want to convert their\n>     messages in native encoding.\n>\n>     I am not sure what should happen to \"cat-file tree\", \"ls-files\" and\n>     \"ls-tree\".  The output from these plumbing does show pathnames, but I\n>     tend to think it is Porcelain's job to turn them into whatever\n>     encoding they want to use.  So are input to \"update-index --stdin\",\n>     but I am still just thinking out loud.\n\nI definitely do not have the time to work on unicode/utf-8/i18n\nsupport for git right now, but as an OS X user, it is something that\ninterests me. When this topic periodically pops up, I squirrel away\nthe useful messages into my \"someday\" folder. So even though it may\nseem that reviewing these patches is wasted effort, comments like the\nabove are helpful. I say this because I expect someday someone will\nwork on this topic, even if it is not me, and hopefully they can\nlocate prior discussion in the mailing list archives and as such it\nwill be of some use.\n\nIOW, your comments are useful and appreciated, even if it doesn't lead\nto improved patches right away.\n\nj.\n"}]}