{"thread":{"id":"7776","subject":"[PATCH 0/2] Controversial blob munging series","startedAt":"2007-04-22T06:08:22Z","lastAt":"2007-04-23T18:49:46Z","messageCount":10,"participants":["Junio C Hamano","Johannes Schindelin","Julian Phillips"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"40092","messageId":"11772221041630-git-send-email-junkio@cox.net","threadId":"7776","inReplyTo":null,"subject":"[PATCH 0/2] Controversial blob munging series","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-22T06:08:22Z","receivedAt":"2007-04-22T06:08:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This is on top of 'next' I'll push out after I am done with\nv1.5.1.2 I am preparing today.\n\n[1/2] Add 'filter' attribute and external filter driver definition.\n[2/2] Add 'ident' conversion.\n\nI'll park them near the tip of 'pu', but consider they are\nprimarily for interested people to experiment with.\n\nI suspect this might have helped me (and other Asians) a year\nago.  I did not manage to configure my Emacs to work well with\nutf-8 encoded Japanese text, and had some difficulties in\nmaintaining documentation for git-lost-found (it has my name\nspelled in Japanese).\n\nI could have had:\n\n\t(in .git/info/attributes)\n\tDocumentation/git-lost-found.txt filter=eucjp-n-utf8\n\n\t(in config)\n\t[filter \"eucjp-n-utf8\"]\n\t\tclean  = nkf -E -w\n\t\tsmudge = nkf -W -e\n\nso that checked-out copy is after \"nkf -W -e\" (filter to EUC-JP,\ntreating the input as UTF-8) to allow my Emacs work with EUC-JP.\nCheck-in will be done after \"nkf -E -w\" (filter to UTF-8,\ntreating the input as EUC-JP), which would keep the \"official\"\nversion in the repository as UTF-8.  The best part of this is\nthat the configurations above are both private to me, and people\ndo not even have to know that I am suffering from the inability\nto use UTF-8 in my editor.\n\nThese days I configured my Emacs to deal with UTF-8 much better\nthan when I added git-lost-found manual page, so I would not\nneed the above hack, though.\n\nI also suspect a \"fun but probably not very useful in practice\"\napplication would be to have \"indent\" as the clean filter while\nleaving \"smudge\" filter empty.\n"},{"id":"40093","messageId":"11772221051597-git-send-email-junkio@cox.net","threadId":"7776","inReplyTo":"11772221041630-git-send-email-junkio@cox.net","subject":"[PATCH 1/2] Add 'filter' attribute and external filter driver definition.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-22T06:08:23Z","receivedAt":"2007-04-22T06:08:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The interface is similar to the custom low-level merge drivers.\n\nFirst you configure your filter driver by defining 'filter.<name>.*'\nvariables in the configuration.\n\n\tfilter.<name>.clean\tfilter command to run upon checkin\n\tfilter.<name>.smudge\tfilter command to run upon checkout\n\nThen you assign filter attribute to each path, whose name\nmatches the custom filter driver's name.\n\nExample:\n\n\t(in .gitattributes)\n\t*.c\tfilter=indent\n\n\t(in config)\n\t[filter \"indent\"]\n\t\tclean = indent\n\t\tsmudge = cat\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n convert.c         |  259 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n t/t0021-filter.sh |   36 ++++++++\n 2 files changed, 286 insertions(+), 9 deletions(-)\n create mode 100755 t/t0021-filter.sh\n\ndiff --git a/convert.c b/convert.c\nindex 37239ac..1fa8f2a 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"attr.h\"\n+#include \"run-command.h\"\n \n /*\n  * convert.c - convert a file when checking it out and checking it in.\n@@ -200,18 +201,214 @@ static char *crlf_to_worktree(const char *path, const char *src, unsigned long *\n \treturn buffer;\n }\n \n+static int filter_buffer(const char *path, const char *src,\n+\t\t\t unsigned long size, const char *cmd)\n+{\n+\t/*\n+\t * Spawn cmd and feed the buffer contents through its stdin.\n+\t */\n+\tstruct child_process child_process;\n+\tint pipe_feed[2];\n+\tint write_err, status;\n+\n+\tmemset(&child_process, 0, sizeof(child_process));\n+\n+\tif (pipe(pipe_feed) < 0) {\n+\t\terror(\"cannot create pipe to run external filter %s\", cmd);\n+\t\treturn 1;\n+\t}\n+\n+\tchild_process.pid = fork();\n+\tif (child_process.pid < 0) {\n+\t\terror(\"cannot fork to run external filter %s\", cmd);\n+\t\tclose(pipe_feed[0]);\n+\t\tclose(pipe_feed[1]);\n+\t\treturn 1;\n+\t}\n+\tif (!child_process.pid) {\n+\t\tdup2(pipe_feed[0], 0);\n+\t\tclose(pipe_feed[0]);\n+\t\tclose(pipe_feed[1]);\n+\t\texeclp(cmd, cmd, NULL);\n+\t\treturn 1;\n+\t}\n+\tclose(pipe_feed[0]);\n+\tclose(1);\n+\n+\twrite_err = (write_in_full(pipe_feed[1], src, size) < 0);\n+\tif (close(pipe_feed[1]))\n+\t\twrite_err = 1;\n+\tif (write_err)\n+\t\terror(\"cannot feed the input to external filter %s\", cmd);\n+\n+\tstatus = finish_command(&child_process);\n+\tif (status)\n+\t\terror(\"external filter %s failed %d\", cmd, -status);\n+\treturn (write_err || status);\n+}\n+\n+static char *apply_filter(const char *path, const char *src,\n+\t\t\t  unsigned long *sizep, const char *cmd)\n+{\n+\t/*\n+\t * Create a pipeline to have the command filter the buffer's\n+\t * contents.\n+\t *\n+\t * (child --> cmd) --> us\n+\t */\n+\tconst int SLOP = 4096;\n+\tint pipe_feed[2];\n+\tint status;\n+\tchar *dst;\n+\tunsigned long dstsize, dstalloc;\n+\tstruct child_process child_process;\n+\n+\tif (!cmd)\n+\t\treturn NULL;\n+\n+\tmemset(&child_process, 0, sizeof(child_process));\n+\n+\tif (pipe(pipe_feed) < 0) {\n+\t\terror(\"cannot create pipe to run external filter %s\", cmd);\n+\t\treturn NULL;\n+\t}\n+\n+\tchild_process.pid = fork();\n+\tif (child_process.pid < 0) {\n+\t\terror(\"cannot fork to run external filter %s\", cmd);\n+\t\tclose(pipe_feed[0]);\n+\t\tclose(pipe_feed[1]);\n+\t\treturn NULL;\n+\t}\n+\tif (!child_process.pid) {\n+\t\tdup2(pipe_feed[1], 1);\n+\t\tclose(pipe_feed[0]);\n+\t\tclose(pipe_feed[1]);\n+\t\texit(filter_buffer(path, src, *sizep, cmd));\n+\t}\n+\tclose(pipe_feed[1]);\n+\n+\tdstalloc = *sizep;\n+\tdst = xmalloc(dstalloc);\n+\tdstsize = 0;\n+\n+\twhile (1) {\n+\t\tssize_t numread = xread(pipe_feed[0], dst + dstsize,\n+\t\t\t\t\tdstalloc - dstsize);\n+\n+\t\tif (numread <= 0) {\n+\t\t\tif (!numread)\n+\t\t\t\tbreak;\n+\t\t\terror(\"read from external filter %s failed\", cmd);\n+\t\t\tfree(dst);\n+\t\t\tdst = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\tdstsize += numread;\n+\t\tif (dstalloc <= dstsize + SLOP) {\n+\t\t\tdstalloc = dstsize + SLOP;\n+\t\t\tdst = xrealloc(dst, dstalloc);\n+\t\t}\n+\t}\n+\tif (close(pipe_feed[0])) {\n+\t\terror(\"read from external filter %s failed\", cmd);\n+\t\tfree(dst);\n+\t\tdst = NULL;\n+\t}\n+\n+\tstatus = finish_command(&child_process);\n+\tif (status) {\n+\t\terror(\"external filter %s failed %d\", cmd, -status);\n+\t\tfree(dst);\n+\t\tdst = NULL;\n+\t}\n+\n+\tif (dst)\n+\t\t*sizep = dstsize;\n+\treturn dst;\n+}\n+\n+static struct convert_driver {\n+\tconst char *name;\n+\tstruct convert_driver *next;\n+\tchar *smudge;\n+\tchar *clean;\n+} *user_convert, **user_convert_tail;\n+\n+static int read_convert_config(const char *var, const char *value)\n+{\n+\tconst char *ep, *name;\n+\tint namelen;\n+\tstruct convert_driver *drv;\n+\n+\t/*\n+\t * External conversion drivers are configured using\n+\t * \"filter.<name>.variable\".\n+\t */\n+\tif (prefixcmp(var, \"filter.\") || (ep = strrchr(var, '.')) == var + 6)\n+\t\treturn 0;\n+\tname = var + 7;\n+\tnamelen = ep - name;\n+\tfor (drv = user_convert; drv; drv = drv->next)\n+\t\tif (!strncmp(drv->name, name, namelen) && !drv->name[namelen])\n+\t\t\tbreak;\n+\tif (!drv) {\n+\t\tchar *namebuf;\n+\t\tdrv = xcalloc(1, sizeof(struct convert_driver));\n+\t\tnamebuf = xmalloc(namelen + 1);\n+\t\tmemcpy(namebuf, name, namelen);\n+\t\tnamebuf[namelen] = 0;\n+\t\tdrv->name = namebuf;\n+\t\tdrv->next = NULL;\n+\t\t*user_convert_tail = drv;\n+\t\tuser_convert_tail = &(drv->next);\n+\t}\n+\n+\tep++;\n+\n+\t/*\n+\t * filter.<name>.smudge and filter.<name>.clean specifies\n+\t * the command line:\n+\t *\n+\t *\tcommand-line\n+\t *\n+\t * The command-line will not be interpolated in any way.\n+\t */\n+\n+\tif (!strcmp(\"smudge\", ep)) {\n+\t\tif (!value)\n+\t\t\treturn error(\"%s: lacks value\", var);\n+\t\tdrv->smudge = strdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(\"clean\", ep)) {\n+\t\tif (!value)\n+\t\t\treturn error(\"%s: lacks value\", var);\n+\t\tdrv->clean = strdup(value);\n+\t\treturn 0;\n+\t}\n+\treturn 0;\n+}\n+\n static void setup_convert_check(struct git_attr_check *check)\n {\n \tstatic struct git_attr *attr_crlf;\n+\tstatic struct git_attr *attr_filter;\n \n-\tif (!attr_crlf)\n+\tif (!attr_crlf) {\n \t\tattr_crlf = git_attr(\"crlf\", 4);\n-\tcheck->attr = attr_crlf;\n+\t\tattr_filter = git_attr(\"filter\", 6);\n+\t\tuser_convert_tail = &user_convert;\n+\t\tgit_config(read_convert_config);\n+\t}\n+\tcheck[0].attr = attr_crlf;\n+\tcheck[1].attr = attr_filter;\n }\n \n static int git_path_check_crlf(const char *path, struct git_attr_check *check)\n {\n-\tconst char *value = check->value;\n+\tconst char *value = check[0].value;\n \n \tif (ATTR_TRUE(value))\n \t\treturn CRLF_TEXT;\n@@ -224,26 +421,70 @@ static int git_path_check_crlf(const char *path, struct git_attr_check *check)\n \treturn CRLF_GUESS;\n }\n \n+static struct convert_driver *git_path_check_convert(const char *path,\n+\t\t\t\t\t     struct git_attr_check *check)\n+{\n+\tconst char *value = check[1].value;\n+\tstruct convert_driver *drv;\n+\n+\tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value))\n+\t\treturn NULL;\n+\tfor (drv = user_convert; drv; drv = drv->next)\n+\t\tif (!strcmp(value, drv->name))\n+\t\t\treturn drv;\n+\treturn NULL;\n+}\n+\n char *convert_to_git(const char *path, const char *src, unsigned long *sizep)\n {\n-\tstruct git_attr_check check[1];\n+\tstruct git_attr_check check[2];\n \tint crlf = CRLF_GUESS;\n+\tchar *filter = NULL;\n+\tchar *buf, *buf2;\n \n \tsetup_convert_check(check);\n-\tif (!git_checkattr(path, 1, check)) {\n+\tif (!git_checkattr(path, 2, check)) {\n+\t\tstruct convert_driver *drv;\n \t\tcrlf = git_path_check_crlf(path, check);\n+\t\tdrv = git_path_check_convert(path, check);\n+\t\tif (drv && drv->clean)\n+\t\t\tfilter = drv->clean;\n \t}\n-\treturn crlf_to_git(path, src, sizep, crlf);\n+\n+\tbuf = apply_filter(path, src, sizep, filter);\n+\n+\tbuf2 = crlf_to_git(path, buf ? buf : src, sizep, crlf);\n+\tif (buf2) {\n+\t\tfree(buf);\n+\t\tbuf = buf2;\n+\t}\n+\n+\treturn buf;\n }\n \n char *convert_to_working_tree(const char *path, const char *src, unsigned long *sizep)\n {\n-\tstruct git_attr_check check[1];\n+\tstruct git_attr_check check[2];\n \tint crlf = CRLF_GUESS;\n+\tchar *filter = NULL;\n+\tchar *buf, *buf2;\n \n \tsetup_convert_check(check);\n-\tif (!git_checkattr(path, 1, check)) {\n+\tif (!git_checkattr(path, 2, check)) {\n+\t\tstruct convert_driver *drv;\n \t\tcrlf = git_path_check_crlf(path, check);\n+\t\tdrv = git_path_check_convert(path, check);\n+\t\tif (drv && drv->smudge)\n+\t\t\tfilter = drv->smudge;\n \t}\n-\treturn crlf_to_worktree(path, src, sizep, crlf);\n+\n+\tbuf = crlf_to_worktree(path, src, sizep, crlf);\n+\n+\tbuf2 = apply_filter(path, buf ? buf : src, sizep, filter);\n+\tif (buf2) {\n+\t\tfree(buf);\n+\t\tbuf = buf2;\n+\t}\n+\n+\treturn buf;\n }\ndiff --git a/t/t0021-filter.sh b/t/t0021-filter.sh\nnew file mode 100755\nindex 0000000..d6bb1c6\n--- /dev/null\n+++ b/t/t0021-filter.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+test_description='external filter conversion'\n+\n+. ./test-lib.sh\n+\n+cat <<\\EOF >rot13.sh\n+tr '[a-zA-Z]' '[n-za-mN-ZA-M]'\n+EOF\n+chmod +x rot13.sh\n+\n+test_expect_success setup '\n+\tgit config filter.rot13.smudge ./rot13.sh &&\n+\tgit config filter.rot13.clean ./rot13.sh &&\n+\n+\techo \"*.t filter=rot13\" >.gitattributes &&\n+\n+\t{\n+\t    echo a b c d e f g h i j k l m\n+\t    echo n o p q r s t u v w x y z\n+\t} >test &&\n+\tcat test >test.t &&\n+\tcat test >test.o &&\n+\tgit add test test.t &&\n+\trm -f test test.t &&\n+\tgit checkout -- test test.t\n+'\n+\n+test_expect_success check '\n+\n+\tcmp test.o test &&\n+\tcmp test.o test.t\n+\n+'\n+\n+test_done\n-- \n1.5.1.2.919.g280f4\n"},{"id":"40094","messageId":"11772221052901-git-send-email-junkio@cox.net","threadId":"7776","inReplyTo":"11772221041630-git-send-email-junkio@cox.net","subject":"[PATCH 2/2] Add 'ident' conversion.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-22T06:08:24Z","receivedAt":"2007-04-22T06:08:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The 'ident' attribute set to path squashes \"$ident:<any bytes\nexcept dollor sign>$\" to \"$ident$\" upon checkin, and expands it\nto \"$ident: <blob SHA-1> $\" upon checkout.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n convert.c         |  186 +++++++++++++++++++++++++++++++++++++++++++++++++---\n t/t0021-filter.sh |   22 +++++--\n 2 files changed, 192 insertions(+), 16 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 1fa8f2a..0b85dd7 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -395,20 +395,161 @@ static void setup_convert_check(struct git_attr_check *check)\n {\n \tstatic struct git_attr *attr_crlf;\n \tstatic struct git_attr *attr_filter;\n+\tstatic struct git_attr *attr_ident;\n \n \tif (!attr_crlf) {\n \t\tattr_crlf = git_attr(\"crlf\", 4);\n \t\tattr_filter = git_attr(\"filter\", 6);\n+\t\tattr_ident = git_attr(\"ident\", 5);\n \t\tuser_convert_tail = &user_convert;\n \t\tgit_config(read_convert_config);\n \t}\n \tcheck[0].attr = attr_crlf;\n \tcheck[1].attr = attr_filter;\n+\tcheck[2].attr = attr_ident;\n+}\n+\n+static int count_ident(const char *cp, unsigned long size)\n+{\n+\t/*\n+\t * \"$ident: 0000000000000000000000000000000000000000 $\" <=> \"$ident$\"\n+\t */\n+\tint cnt = 0;\n+\tchar ch;\n+\n+\twhile (size) {\n+\t\tch = *cp++;\n+\t\tsize--;\n+\t\tif (ch != '$')\n+\t\t\tcontinue;\n+\t\tif (size < 6)\n+\t\t\tbreak;\n+\t\tif (memcmp(\"ident\", cp, 5))\n+\t\t\tcontinue;\n+\t\tch = cp[5];\n+\t\tcp += 6;\n+\t\tsize -= 6;\n+\t\tif (ch == '$')\n+\t\t\tcnt++; /* $ident$ */\n+\t\tif (ch != ':')\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * \"$ident: ... \"; scan up to the closing dollar sign and discard.\n+\t\t */\n+\t\twhile (size) {\n+\t\t\tch = *cp++;\n+\t\t\tsize--;\n+\t\t\tif (ch == '$') {\n+\t\t\t\tcnt++;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\treturn cnt;\n+}\n+\n+static char *ident_to_git(const char *path, const char *src, unsigned long *sizep, int ident)\n+{\n+\tint cnt;\n+\tunsigned long size;\n+\tchar *dst, *buf;\n+\n+\tif (!ident)\n+\t\treturn NULL;\n+\tsize = *sizep;\n+\tcnt = count_ident(src, size);\n+\tif (!cnt)\n+\t\treturn NULL;\n+\tbuf = xmalloc(size);\n+\n+\tfor (dst = buf; size; size--) {\n+\t\tchar ch = *src++;\n+\t\t*dst++ = ch;\n+\t\tif ((ch == '$') && (6 <= size) &&\n+\t\t    !memcmp(\"ident:\", src, 6)) {\n+\t\t\tunsigned long rem = size - 6;\n+\t\t\tconst char *cp = src + 6;\n+\t\t\tdo {\n+\t\t\t\tch = *cp++;\n+\t\t\t\tif (ch == '$')\n+\t\t\t\t\tbreak;\n+\t\t\t\trem--;\n+\t\t\t} while (rem);\n+\t\t\tif (!rem)\n+\t\t\t\tcontinue;\n+\t\t\tmemcpy(dst, \"ident$\", 6);\n+\t\t\tdst += 6;\n+\t\t\tsize -= (cp - src);\n+\t\t\tsrc = cp;\n+\t\t}\n+\t}\n+\n+\t*sizep = dst - buf;\n+\treturn buf;\n+}\n+\n+static char *ident_to_worktree(const char *path, const char *src, unsigned long *sizep, int ident)\n+{\n+\tint cnt;\n+\tunsigned long size;\n+\tchar *dst, *buf;\n+\tunsigned char sha1[20];\n+\n+\tif (!ident)\n+\t\treturn NULL;\n+\n+\tsize = *sizep;\n+\tcnt = count_ident(src, size);\n+\tif (!cnt)\n+\t\treturn NULL;\n+\n+\thash_sha1_file(src, size, \"blob\", sha1);\n+\tbuf = xmalloc(size + cnt * 43);\n+\n+\tfor (dst = buf; size; size--) {\n+\t\tconst char *cp;\n+\t\tchar ch = *src++;\n+\t\t*dst++ = ch;\n+\t\tif ((ch != '$') || (size < 6) || memcmp(\"ident\", src, 5))\n+\t\t\tcontinue;\n+\n+\t\tif (src[5] == ':') {\n+\t\t\t/* discard up to but not including the closing $ */\n+\t\t\tunsigned long rem = size - 6;\n+\t\t\tcp = src + 6;\n+\t\t\tdo {\n+\t\t\t\tch = *cp++;\n+\t\t\t\tif (ch == '$')\n+\t\t\t\t\tbreak;\n+\t\t\t\trem--;\n+\t\t\t} while (rem);\n+\t\t\tif (!rem)\n+\t\t\t\tcontinue;\n+\t\t\tsize -= (cp - src);\n+\t\t} else if (src[5] == '$')\n+\t\t\tcp = src + 5;\n+\t\telse\n+\t\t\tcontinue;\n+\n+\t\tmemcpy(dst, \"ident: \", 7);\n+\t\tdst += 7;\n+\t\tmemcpy(dst, sha1_to_hex(sha1), 40);\n+\t\tdst += 40;\n+\t\t*dst++ = ' ';\n+\t\tsize -= (cp - src);\n+\t\tsrc = cp;\n+\t\t*dst++ = *src++;\n+\t\tsize--;\n+\t}\n+\n+\t*sizep = dst - buf;\n+\treturn buf;\n }\n \n static int git_path_check_crlf(const char *path, struct git_attr_check *check)\n {\n-\tconst char *value = check[0].value;\n+\tconst char *value = check->value;\n \n \tif (ATTR_TRUE(value))\n \t\treturn CRLF_TEXT;\n@@ -424,7 +565,7 @@ static int git_path_check_crlf(const char *path, struct git_attr_check *check)\n static struct convert_driver *git_path_check_convert(const char *path,\n \t\t\t\t\t     struct git_attr_check *check)\n {\n-\tconst char *value = check[1].value;\n+\tconst char *value = check->value;\n \tstruct convert_driver *drv;\n \n \tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value))\n@@ -435,20 +576,29 @@ static struct convert_driver *git_path_check_convert(const char *path,\n \treturn NULL;\n }\n \n+static int git_path_check_ident(const char *path, struct git_attr_check *check)\n+{\n+\tconst char *value = check->value;\n+\n+\treturn !!ATTR_TRUE(value);\n+}\n+\n char *convert_to_git(const char *path, const char *src, unsigned long *sizep)\n {\n-\tstruct git_attr_check check[2];\n+\tstruct git_attr_check check[3];\n \tint crlf = CRLF_GUESS;\n+\tint ident = 0;\n \tchar *filter = NULL;\n \tchar *buf, *buf2;\n \n \tsetup_convert_check(check);\n-\tif (!git_checkattr(path, 2, check)) {\n+\tif (!git_checkattr(path, 3, check)) {\n \t\tstruct convert_driver *drv;\n-\t\tcrlf = git_path_check_crlf(path, check);\n-\t\tdrv = git_path_check_convert(path, check);\n+\t\tcrlf = git_path_check_crlf(path, check + 0);\n+\t\tdrv = git_path_check_convert(path, check + 1);\n \t\tif (drv && drv->clean)\n \t\t\tfilter = drv->clean;\n+\t\tident = git_path_check_ident(path, check + 2);\n \t}\n \n \tbuf = apply_filter(path, src, sizep, filter);\n@@ -459,26 +609,40 @@ char *convert_to_git(const char *path, const char *src, unsigned long *sizep)\n \t\tbuf = buf2;\n \t}\n \n+\tbuf2 = ident_to_git(path, buf ? buf : src, sizep, ident);\n+\tif (buf2) {\n+\t\tfree(buf);\n+\t\tbuf = buf2;\n+\t}\n+\n \treturn buf;\n }\n \n char *convert_to_working_tree(const char *path, const char *src, unsigned long *sizep)\n {\n-\tstruct git_attr_check check[2];\n+\tstruct git_attr_check check[3];\n \tint crlf = CRLF_GUESS;\n+\tint ident = 0;\n \tchar *filter = NULL;\n \tchar *buf, *buf2;\n \n \tsetup_convert_check(check);\n-\tif (!git_checkattr(path, 2, check)) {\n+\tif (!git_checkattr(path, 3, check)) {\n \t\tstruct convert_driver *drv;\n-\t\tcrlf = git_path_check_crlf(path, check);\n-\t\tdrv = git_path_check_convert(path, check);\n+\t\tcrlf = git_path_check_crlf(path, check + 0);\n+\t\tdrv = git_path_check_convert(path, check + 1);\n \t\tif (drv && drv->smudge)\n \t\t\tfilter = drv->smudge;\n+\t\tident = git_path_check_ident(path, check + 2);\n \t}\n \n-\tbuf = crlf_to_worktree(path, src, sizep, crlf);\n+\tbuf = ident_to_worktree(path, src, sizep, ident);\n+\n+\tbuf2 = crlf_to_worktree(path, buf ? buf : src, sizep, crlf);\n+\tif (buf2) {\n+\t\tfree(buf);\n+\t\tbuf = buf2;\n+\t}\n \n \tbuf2 = apply_filter(path, buf ? buf : src, sizep, filter);\n \tif (buf2) {\ndiff --git a/t/t0021-filter.sh b/t/t0021-filter.sh\nindex d6bb1c6..725a425 100755\n--- a/t/t0021-filter.sh\n+++ b/t/t0021-filter.sh\n@@ -13,24 +13,36 @@ test_expect_success setup '\n \tgit config filter.rot13.smudge ./rot13.sh &&\n \tgit config filter.rot13.clean ./rot13.sh &&\n \n-\techo \"*.t filter=rot13\" >.gitattributes &&\n+\t{\n+\t    echo \"*.t filter=rot13\"\n+\t    echo \"*.i ident\"\n+\t} >.gitattributes &&\n \n \t{\n \t    echo a b c d e f g h i j k l m\n \t    echo n o p q r s t u v w x y z\n+\t    echo '\\''$ident$'\\''\n \t} >test &&\n \tcat test >test.t &&\n \tcat test >test.o &&\n-\tgit add test test.t &&\n-\trm -f test test.t &&\n-\tgit checkout -- test test.t\n+\tcat test >test.i &&\n+\tgit add test test.t test.i &&\n+\trm -f test test.t test.i &&\n+\tgit checkout -- test test.t test.i\n '\n \n+script='s/^\\$ident: \\([0-9a-f]*\\) \\$/\\1/p'\n+\n test_expect_success check '\n \n \tcmp test.o test &&\n-\tcmp test.o test.t\n+\tcmp test.o test.t &&\n \n+\t# ident should be stripped in the repository\n+\tgit diff --raw --exit-code :test :test.i &&\n+\tid=$(git rev-parse --verify :test) &&\n+\tembedded=$(sed -ne \"$script\" test.i) &&\n+\ttest \"z$id\" = \"z$embedded\"\n '\n \n test_done\n-- \n1.5.1.2.919.g280f4\n"},{"id":"40199","messageId":"Pine.LNX.4.64.0704231540580.8822@racer.site","threadId":"7776","inReplyTo":"11772221041630-git-send-email-junkio@cox.net","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-04-23T13:50:19Z","receivedAt":"2007-04-23T13:50:19Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 21 Apr 2007, Junio C Hamano wrote:\n\n> This is on top of 'next' I'll push out after I am done with\n> v1.5.1.2 I am preparing today.\n> \n> [1/2] Add 'filter' attribute and external filter driver definition.\n> [2/2] Add 'ident' conversion.\n\nI think this is great work! And it is useful, too. Let me describe a usage \nscenario I have in mind.\n\nBeing stuck with Pine, which still does not do Maildir, and wanting \nto be able to read my mails as distributed as I am working on documents \nand software projects, I always dreamt of having all my mail in Git.\n\nWith filters, it should be relatively easy to do that. Before checking in, \nthe individual mailbox files are split, the contents are put into the \nobject database, and the mailbox file is replaced by a text file \nconsisting of the SHA1s of the mails.\n\nIdeally, I would eventually not only teach Pine to understand Maildir \nformat, but read and store the mails in a Git backend. Alas, I am way too \nlazy for that.\n\nSo, with filters I'd do the cheap and easy thing.\n\nYou may not be able to appreciate the advantages of my scenario, but this \nkind of flexibility is what makes Git so useful.\n\nCiao,\nDscho\n"},{"id":"40209","messageId":"Pine.LNX.4.64.0704231725520.22673@reaper.quantumfyre.co.uk","threadId":"7776","inReplyTo":"Pine.LNX.4.64.0704231540580.8822@racer.site","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2007-04-23T16:29:44Z","receivedAt":"2007-04-23T16:29:44Z","isPatch":true,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"On Mon, 23 Apr 2007, Johannes Schindelin wrote:\n\n> Being stuck with Pine, which still does not do Maildir, and wanting\n> to be able to read my mails as distributed as I am working on documents\n> and software projects, I always dreamt of having all my mail in Git.\n>\n> With filters, it should be relatively easy to do that. Before checking in,\n> the individual mailbox files are split, the contents are put into the\n> object database, and the mailbox file is replaced by a text file\n> consisting of the SHA1s of the mails.\n>\n> Ideally, I would eventually not only teach Pine to understand Maildir\n> format, but read and store the mails in a Git backend. Alas, I am way too\n> lazy for that.\n\nYou do know about Eduardo Chappa's patches for pine?  In particular \nhttp://staff.washington.edu/chappa/pine/info/maildir.html.\n\n-- \nJulian\n\n  ---\nWindows contains FAT.\nUse Linux -- you won't ever have to worry about your weight.\n\n    -- Ewout Stam\n"},{"id":"40211","messageId":"Pine.LNX.4.64.0704231853390.8822@racer.site","threadId":"7776","inReplyTo":"Pine.LNX.4.64.0704231725520.22673@reaper.quantumfyre.co.uk","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-04-23T16:54:01Z","receivedAt":"2007-04-23T16:54:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 23 Apr 2007, Julian Phillips wrote:\n\n> On Mon, 23 Apr 2007, Johannes Schindelin wrote:\n> \n> > Being stuck with Pine, which still does not do Maildir, and wanting\n> > to be able to read my mails as distributed as I am working on documents\n> > and software projects, I always dreamt of having all my mail in Git.\n> >\n> > With filters, it should be relatively easy to do that. Before checking in,\n> > the individual mailbox files are split, the contents are put into the\n> > object database, and the mailbox file is replaced by a text file\n> > consisting of the SHA1s of the mails.\n> >\n> > Ideally, I would eventually not only teach Pine to understand Maildir\n> > format, but read and store the mails in a Git backend. Alas, I am way too\n> > lazy for that.\n> \n> You do know about Eduardo Chappa's patches for pine?  In particular\n> http://staff.washington.edu/chappa/pine/info/maildir.html.\n\nI did not.\n\nThanks,\nDscho\n"},{"id":"40214","messageId":"7vzm4zf1zx.fsf@assigned-by-dhcp.cox.net","threadId":"7776","inReplyTo":"Pine.LNX.4.64.0704231540580.8822@racer.site","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-23T17:13:06Z","receivedAt":"2007-04-23T17:13:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> On Sat, 21 Apr 2007, Junio C Hamano wrote:\n>\n>> This is on top of 'next' I'll push out after I am done with\n>> v1.5.1.2 I am preparing today.\n>> \n>> [1/2] Add 'filter' attribute and external filter driver definition.\n>> [2/2] Add 'ident' conversion.\n>\n> I think this is great work! And it is useful, too. Let me describe a usage \n> scenario I have in mind.\n>\n> Being stuck with Pine, which still does not do Maildir, and wanting \n> to be able to read my mails as distributed as I am working on documents \n> and software projects, I always dreamt of having all my mail in Git.\n>\n> With filters, it should be relatively easy to do that. Before checking in, \n> the individual mailbox files are split, the contents are put into the \n> object database, and the mailbox file is replaced by a text file \n> consisting of the SHA1s of the mails.\n>\n> Ideally, I would eventually not only teach Pine to understand Maildir \n> format, but read and store the mails in a Git backend. Alas, I am way too \n> lazy for that.\n>\n> So, with filters I'd do the cheap and easy thing.\n>\n> You may not be able to appreciate the advantages of my scenario, but this \n> kind of flexibility is what makes Git so useful.\n\nAn earlier message $gmane/44896 from Linus comes to my mind.  An excerpt:\n\n   The thing is, it's easy enough (although potentially _very_ expensive) to \n   run some per-file script at each commit and at each checkout. But there \n   are some fundamental operations that are even more common:\n\n    - checking for \"file changed\", aka the \"git status\" kind of thing\n\n      Anything we do would have to follow the same \"stat\" rules, at a \n      minimum. You can *not* afford to have to check the file manually.\n\n      So especially if you combine several pieces into one, or split one file \n      into several pieces, your index would have to contain the entry \n      that matches the _filesystem_ (because that's what the index is all \n      about), but then the *tree* would contain the pieces (or the single \n      entry that matches several filesystem entries).\n\nand I am inclined to think that this is quite fundamental.  I\nthink you just fell into category who want \"extended semantics\"\nLinus talked about in $gmane/45214:\n\n  I suspect that this gets some complaining off our back, but I *also* \n  suspect that people will actually end up really screwing themselves with \n  something like this and then blaming us and causing a huge pain down the \n  line when we've supported this and people want \"extended semantics\" that \n  are no longer clean.\n\nwhich is kind of dissapointing.\n\nEven if you somehow solved the issue of \"stat\" rule, I do not\nknow what your plans are to manage the blobs that you drop in\nthe object store.  The list of object names in the mail-index\nfile you are generating do not count as connectivity for the\npurpose of fetch/push/fsck/prune.\n"},{"id":"40220","messageId":"Pine.LNX.4.64.0704231933560.8822@racer.site","threadId":"7776","inReplyTo":"7vzm4zf1zx.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-04-23T17:35:33Z","receivedAt":"2007-04-23T17:35:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 23 Apr 2007, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > On Sat, 21 Apr 2007, Junio C Hamano wrote:\n> >\n> >> This is on top of 'next' I'll push out after I am done with\n> >> v1.5.1.2 I am preparing today.\n> >> \n> >> [1/2] Add 'filter' attribute and external filter driver definition.\n> >> [2/2] Add 'ident' conversion.\n> >\n> > I think this is great work! And it is useful, too. Let me describe a usage \n> > scenario I have in mind.\n> >\n> > Being stuck with Pine, which still does not do Maildir, and wanting \n> > to be able to read my mails as distributed as I am working on documents \n> > and software projects, I always dreamt of having all my mail in Git.\n> >\n> > With filters, it should be relatively easy to do that. Before checking in, \n> > the individual mailbox files are split, the contents are put into the \n> > object database, and the mailbox file is replaced by a text file \n> > consisting of the SHA1s of the mails.\n> >\n> > Ideally, I would eventually not only teach Pine to understand Maildir \n> > format, but read and store the mails in a Git backend. Alas, I am way too \n> > lazy for that.\n> >\n> > So, with filters I'd do the cheap and easy thing.\n> >\n> > You may not be able to appreciate the advantages of my scenario, but this \n> > kind of flexibility is what makes Git so useful.\n> \n> An earlier message $gmane/44896 from Linus comes to my mind.  An excerpt:\n> \n>    The thing is, it's easy enough (although potentially _very_ expensive) to \n>    run some per-file script at each commit and at each checkout. But there \n>    are some fundamental operations that are even more common:\n> \n>     - checking for \"file changed\", aka the \"git status\" kind of thing\n> \n>       Anything we do would have to follow the same \"stat\" rules, at a \n>       minimum. You can *not* afford to have to check the file manually.\n> \n>       So especially if you combine several pieces into one, or split one file \n>       into several pieces, your index would have to contain the entry \n>       that matches the _filesystem_ (because that's what the index is all \n>       about), but then the *tree* would contain the pieces (or the single \n>       entry that matches several filesystem entries).\n> \n> and I am inclined to think that this is quite fundamental.  I\n> think you just fell into category who want \"extended semantics\"\n> Linus talked about in $gmane/45214:\n> \n>   I suspect that this gets some complaining off our back, but I *also* \n>   suspect that people will actually end up really screwing themselves with \n>   something like this and then blaming us and causing a huge pain down the \n>   line when we've supported this and people want \"extended semantics\" that \n>   are no longer clean.\n> \n> which is kind of dissapointing.\n> \n> Even if you somehow solved the issue of \"stat\" rule, I do not\n> know what your plans are to manage the blobs that you drop in\n> the object store.  The list of object names in the mail-index\n> file you are generating do not count as connectivity for the\n> purpose of fetch/push/fsck/prune.\n\nI had the idea to update a ref, which holds \"trees\" of message-id -> blob \npairs, and get updated at the same time.\n\nIf Git were libified already, I might have tried to go for direct storage \nin .git/objects/ instead.\n\nCiao,\nDscho\n"},{"id":"40227","messageId":"7virbnexuc.fsf@assigned-by-dhcp.cox.net","threadId":"7776","inReplyTo":"Pine.LNX.4.64.0704231933560.8822@racer.site","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-23T18:42:51Z","receivedAt":"2007-04-23T18:42:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> On Mon, 23 Apr 2007, Junio C Hamano wrote:\n> ...\n>> ... I am inclined to think that this is quite fundamental.  I\n>> think you just fell into category who want \"extended semantics\"\n>> Linus talked about in $gmane/45214:\n>> \n>>   I suspect that this gets some complaining off our back, but I *also* \n>>   suspect that people will actually end up really screwing themselves with \n>>   something like this and then blaming us and causing a huge pain down the \n>>   line when we've supported this and people want \"extended semantics\" that \n>>   are no longer clean.\n>> \n>> which is kind of dissapointing.\n\nI think this was the biggest worry.  If even Dscho, who is among\na dozen people with the most intimate knowledge of git on the\nplanet, gets it wrong, I can almost guarantee that we will get\ninto the mess Linus predicted above.\n\n>> Even if you somehow solved the issue of \"stat\" rule, I do not\n>> know what your plans are to manage the blobs that you drop in\n>> the object store.  The list of object names in the mail-index\n>> file you are generating do not count as connectivity for the\n>> purpose of fetch/push/fsck/prune.\n>\n> I had the idea to update a ref, which holds \"trees\" of message-id -> blob \n> pairs, and get updated at the same time.\n\nI somehow thought this mailbox thing was because you wanted to\ntransfer mailboxes across repositories.  How would you prevent\nthat ref from getting out of sync with the mail-index file git\nknows nothing about its involvement in connectivity?\n"},{"id":"40229","messageId":"Pine.LNX.4.64.0704232047480.8822@racer.site","threadId":"7776","inReplyTo":"7virbnexuc.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 0/2] Controversial blob munging series","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-04-23T18:49:46Z","receivedAt":"2007-04-23T18:49:46Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 23 Apr 2007, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > On Mon, 23 Apr 2007, Junio C Hamano wrote:\n> > ...\n> >> ... I am inclined to think that this is quite fundamental.  I\n> >> think you just fell into category who want \"extended semantics\"\n> >> Linus talked about in $gmane/45214:\n> >> \n> >>   I suspect that this gets some complaining off our back, but I *also* \n> >>   suspect that people will actually end up really screwing themselves with \n> >>   something like this and then blaming us and causing a huge pain down the \n> >>   line when we've supported this and people want \"extended semantics\" that \n> >>   are no longer clean.\n> >> \n> >> which is kind of dissapointing.\n> \n> I think this was the biggest worry.  If even Dscho, who is among\n> a dozen people with the most intimate knowledge of git on the\n> planet, gets it wrong, I can almost guarantee that we will get\n> into the mess Linus predicted above.\n\nFlattering always works :-)\n\n> >> Even if you somehow solved the issue of \"stat\" rule, I do not\n> >> know what your plans are to manage the blobs that you drop in\n> >> the object store.  The list of object names in the mail-index\n> >> file you are generating do not count as connectivity for the\n> >> purpose of fetch/push/fsck/prune.\n> >\n> > I had the idea to update a ref, which holds \"trees\" of message-id -> blob \n> > pairs, and get updated at the same time.\n> \n> I somehow thought this mailbox thing was because you wanted to\n> transfer mailboxes across repositories.  How would you prevent\n> that ref from getting out of sync with the mail-index file git\n> knows nothing about its involvement in connectivity?\n\nIf your suspicion was that I did not really think it through, then you're \ncorrect. Of course, I would have transferred _all_ refs anyway, since the \nwhole point of the exercise is to lose nothing.\n\nHowever, I see where your argument is going.\n\nSince Julian pointed out that there is a maildir patch for pine, I'll \nprobably go for that one, since it is hanging lower.\n\nCiao,\nDscho\n"}]}