{"thread":{"id":"42634","subject":"[PATCH 0/4] extend smudge/clean filters with direct file access","startedAt":"2016-06-16T20:33:44Z","lastAt":"2016-06-17T18:26:34Z","messageCount":22,"participants":["Joey Hess","Junio C Hamano","Eric Sunshine","Michael J Gruber"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"289398","messageId":"20160616203259.5886-1-joeyh@joeyh.name","threadId":"42634","inReplyTo":null,"subject":"[PATCH 0/4] extend smudge/clean filters with direct file access","fromName":"Joey Hess","fromEmail":"joeyh@joeyh.name","sentAt":"2016-06-16T20:32:55Z","receivedAt":"2016-06-16T20:33:44Z","isPatch":true,"sender":{"key":"joeyh@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"As discussed in this thread:\nhttp://thread.gmane.org/gmane.comp.version-control.git/294425\nThis adds smudge-to-file and clean-from-file commands supplimenting the\nsmudge and clean filters.\n\nThis interface can be much more efficient when operating on large files,\nbecause the whole file content does not need to be streamed through the\nfilter. It even allows for things like clean-from-file commands that avoid\nreading the whole content of the file, and for smudge-to-file commands that\npopulate a work tree file using an efficient Copy On Write operation.\n\nJoey Hess (4):\n  clarify %f documentation\n  add smudge-to-file and clean-from-file filter configuration\n  use clean-from-file in git add\n  use smudge-to-file in git checkout etc\n\n Documentation/config.txt        |  27 +++++++---\n Documentation/gitattributes.txt |  44 ++++++++++++++++-\n convert.c                       | 107 ++++++++++++++++++++++++++++++++++------\n convert.h                       |  10 ++++\n entry.c                         |  34 ++++++++++---\n sha1_file.c                     |  42 +++++++++++++---\n t/t0021-conversion.sh           |  64 ++++++++++++++++++++++++\n 7 files changed, 292 insertions(+), 36 deletions(-)\n\n-- \n2.9.0.4.g2856e74.dirty\n\n"},{"id":"289399","messageId":"20160616203259.5886-4-joeyh@joeyh.name","threadId":"42634","inReplyTo":"20160616203259.5886-1-joeyh@joeyh.name","subject":"[PATCH 3/4] use clean-from-file in git add","fromName":"Joey Hess","fromEmail":"joeyh@joeyh.name","sentAt":"2016-06-16T20:32:58Z","receivedAt":"2016-06-16T20:33:46Z","isPatch":true,"sender":{"key":"joeyh@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Includes test cases.\n\nSigned-off-by: Joey Hess <joeyh@joeyh.name>\n---\n sha1_file.c           | 42 ++++++++++++++++++++++++++++++++++++------\n t/t0021-conversion.sh | 36 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 72 insertions(+), 6 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d5e1121..8df86a0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3329,6 +3329,29 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n \treturn ret;\n }\n \n+static int index_from_file_convert_blob(unsigned char *sha1,\n+\t\t\t\t      const char *path, unsigned flags)\n+{\n+\tint ret;\n+\tconst int write_object = flags & HASH_WRITE_OBJECT;\n+\tstruct strbuf sbuf = STRBUF_INIT;\n+\n+\tassert(path);\n+\tassert(can_clean_from_file(path));\n+\n+\tconvert_to_git_filter_from_file(path, &sbuf,\n+\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n+\n+\tif (write_object)\n+\t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n+\t\t\t\t      sha1);\n+\telse\n+\t\tret = hash_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n+\t\t\t\t     sha1);\n+\tstrbuf_release(&sbuf);\n+\treturn ret;\n+}\n+\n static int index_pipe(unsigned char *sha1, int fd, enum object_type type,\n \t\t      const char *path, unsigned flags)\n {\n@@ -3421,12 +3444,19 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned\n \n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFREG:\n-\t\tfd = open(path, O_RDONLY);\n-\t\tif (fd < 0)\n-\t\t\treturn error_errno(\"open(\\\"%s\\\")\", path);\n-\t\tif (index_fd(sha1, fd, st, OBJ_BLOB, path, flags) < 0)\n-\t\t\treturn error(\"%s: failed to insert into database\",\n-\t\t\t\t     path);\n+\t\tif (can_clean_from_file(path)) {\n+\t\t\tif (index_from_file_convert_blob(sha1, path, flags) < 0)\n+\t\t\t\treturn error(\"%s: failed to insert into database\",\n+\t\t\t\t\t     path);\n+\t\t}\n+\t\telse {\n+\t\t\tfd = open(path, O_RDONLY);\n+\t\t\tif (fd < 0)\n+\t\t\t\treturn error_errno(\"open(\\\"%s\\\")\", path);\n+\t\t\tif (index_fd(sha1, fd, st, OBJ_BLOB, path, flags) < 0)\n+\t\t\t\treturn error(\"%s: failed to insert into database\",\n+\t\t\t\t\t     path);\n+\t\t}\n \t\tbreak;\n \tcase S_IFLNK:\n \t\tif (strbuf_readlink(&sb, path, st->st_size))\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 7bac2bc..1043ea5 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -12,6 +12,14 @@ tr \\\n EOF\n chmod +x rot13.sh\n \n+cat <<EOF >rot13-from-file.sh\n+#!$SHELL_PATH\n+fsfile=\"\\$1\"\n+touch rot13-from-file.ran\n+cat \"\\$fsfile\" | ./rot13.sh\n+EOF\n+chmod +x rot13-from-file.sh\n+\n test_expect_success setup '\n \tgit config filter.rot13.smudge ./rot13.sh &&\n \tgit config filter.rot13.clean ./rot13.sh &&\n@@ -268,4 +276,32 @@ test_expect_success 'disable filter with empty override' '\n \ttest_must_be_empty err\n '\n \n+test_expect_success 'clean-from-file filter is used when adding a file' '\n+\ttest_config filter.rot13.clean-from-file \"./rot13-from-file.sh %p\" &&\n+\n+\techo \"*.t filter=rot13\" >.gitattributes &&\n+\n+\tcat test.t >fstest.t &&\n+\tgit add fstest.t &&\n+\ttest -e rot13-from-file.ran &&\n+\trm -f rot13-from-file.ran &&\n+\n+\trm -f fstest.t &&\n+\tgit checkout -- fstest.t &&\n+\tcmp test.t fstest.t\n+'\n+\n+test_expect_success 'clean-from-file filter is not used when clean filter is not configured' '\n+\ttest_config filter.no.smudge ./rot13.sh &&\n+\ttest_config filter.no.clean-from-file \"./rot13-from-file.sh %p\" &&\n+\n+\techo \"*.no filter=no\" >.gitattributes &&\n+\n+\tcat test.t >test.no &&\n+\tgit add test.no &&\n+\ttest ! -e rot13-from-file.ran &&\n+\tgit cat-file blob :test.no >actual &&\n+\tcmp test.t actual\n+'\n+\n test_done\n-- \n2.9.0.4.g2856e74.dirty\n\n"},{"id":"289400","messageId":"20160616203259.5886-3-joeyh@joeyh.name","threadId":"42634","inReplyTo":"20160616203259.5886-1-joeyh@joeyh.name","subject":"[PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Joey Hess","fromEmail":"joeyh@joeyh.name","sentAt":"2016-06-16T20:32:57Z","receivedAt":"2016-06-16T20:33:53Z","isPatch":true,"sender":{"key":"joeyh@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"This adds new smudge-to-file and clean-from-file filter commands,\nwhich are similar to smudge and clean but allow direct access to files on\ndisk.\n\nIn smudge-to-file and clean-from-file, \"%p\" is expanded to the path to the\nfile that should be cleaned from, or smudged to.\n\nThis interface can be much more efficient when operating on large files,\nbecause the whole file content does not need to be streamed through the\nfilter. It even allows for things like clean-from-file commands that avoid\nreading the whole content of the file, and for smudge-to-file commands that\npopulate a work tree file using an efficient Copy On Write operation.\n\nThe new filter commands will not be used for all filtering. They are\nefficient to use when git add is adding a file, or when the work tree is\nbeing updated, but not a good fit when git is internally filtering blob\nobjects in memory for eg, a diff.\n\nSo, a user who wants to use smudge-to-file should also provide a smudge\ncommand to be used in cases where smudge-to-file is not used. And ditto\nwith clean-from-file and clean. To avoid foot-shooting configurations, the\nnew commands are not used unless the old commands are also configured.\n\nThat also ensures that a filter driver configuration that includes these\nnew commands will work, although less efficiently, when used with an older\nversion of git that does not support them.\n\nSigned-off-by: Joey Hess <joeyh@joeyh.name>\n---\n Documentation/config.txt        |  27 +++++++---\n Documentation/gitattributes.txt |  37 ++++++++++++++\n convert.c                       | 107 ++++++++++++++++++++++++++++++++++------\n convert.h                       |  10 ++++\n 4 files changed, 160 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2e1b2e4..bbb9296 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1299,14 +1299,29 @@ format.useAutoBase::\n \tformat-patch by default.\n \n filter.<driver>.clean::\n-\tThe command which is used to convert the content of a worktree\n-\tfile to a blob upon checkin.  See linkgit:gitattributes[5] for\n-\tdetails.\n+\tThe command which is used on checkin to convert the content of\n+\ta worktree file (provided on stdin) to a blob (written to stdout).\n+\tSee linkgit:gitattributes[5] for details.\n \n filter.<driver>.smudge::\n-\tThe command which is used to convert the content of a blob\n-\tobject to a worktree file upon checkout.  See\n-\tlinkgit:gitattributes[5] for details.\n+\tThe command which is used on checkout to convert the content of a\n+\tblob object (provided on stdin) to a worktree file (written to\n+\tstdout).\n+\tSee linkgit:gitattributes[5] for details.\n+\n+filter.<driver>.clean-from-file::\n+\tOptional command which is used on checkin to convert the content\n+\tof a worktree file, which can be read from disk, to a blob\n+\t(written to stdout).\n+\tOnly used when filter.<driver>.clean is also configured.\n+\tSee linkgit:gitattributes[5] for details.\n+\n+filter.<driver>.smudge-to-file::\n+\tOptional command which is used to convert the content of a blob\n+\tobject (provided on stdin) to a worktree file, writing directly\n+\tto the file.\n+\tOnly used when filter.<driver>.clean is also configured.\n+\tSee linkgit:gitattributes[5] for details.\n \n fsck.<msg-id>::\n \tAllows overriding the message type (error, warn or ignore) of a\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex e077cc9..32621e7 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -378,6 +378,43 @@ Note that the \"%f\" is the name of a file in the git repository; the\n corresponding file on disk may not exist, or may have unrelated contents to\n what git is filtering.\n \n+There are two extra commands \"clean-from-file\" and \"smudge-to-file\", which\n+can optionally be set in a filter driver. These are similar to the \"clean\"\n+and \"smudge\" commands, but avoid needing to pipe the contents of files\n+through the filters, and instead reading/writing files in the filesystem.\n+This can be more efficient when using filters with large files that are not\n+directly stored in the repository.\n+\n+Sequence \"%p\" on the \"clean-from-file\" and \"smudge-to-file\" command line\n+is replaced with the path to a file that the filter can access.\n+\n+In the \"clean-from-file\" command, \"%p\" is the path to the file that\n+it should clean. Like the \"clean\" command, it should output the cleaned\n+version to standard output.\n+\n+In the \"smudge-from-file\" command, \"%p\" is the path to the file that it\n+should write to. (This file will already exist, as an empty file that can\n+be written to or replaced.) Like the \"smudge\" command, \"smudge-from-file\"\n+is fed the blob object from its standard input.\n+\n+Some git operations that need to apply filters cannot use \"clean-from-file\"\n+and \"smudge-to-file\", since the files are not present to disk. So, to avoid\n+inconsistent behavior, \"clean-from-file\" will only be used if \"clean\" is\n+also configured, and \"smudge-to-file\" will only be used if \"smudge\" is also\n+configured.\n+\n+An example large file storage filter driver using \"*-from-file\" follows:\n+\n+------------------------\n+[filter \"bigfiles\"]\n+\tclean = store-bigfile --from-stdin\n+\tclean-from-file = store-bigfile %p\n+\tsmudge = retrieve-bigfile --to-stdout\n+\tsmudge-to-file = retrieve-bigfile %p\n+\trequired\n+\n+------------------------\n+\n Interaction between checkin/checkout attributes\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \ndiff --git a/convert.c b/convert.c\nindex b1614bf..bfb7578 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -360,7 +360,8 @@ struct filter_params {\n \tunsigned long size;\n \tint fd;\n \tconst char *cmd;\n-\tconst char *path;\n+\tconst char *path; /* Path within the git repository */\n+\tconst char *fspath; /* Path to file on disk */\n };\n \n static int filter_buffer_or_fd(int in, int out, void *data)\n@@ -376,16 +377,22 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \t/* apply % substitution to cmd */\n \tstruct strbuf cmd = STRBUF_INIT;\n \tstruct strbuf path = STRBUF_INIT;\n+\tstruct strbuf fspath = STRBUF_INIT;\n \tstruct strbuf_expand_dict_entry dict[] = {\n \t\t{ \"f\", NULL, },\n+\t\t{ (params->fspath ? \"p\" : NULL), NULL, },\n \t\t{ NULL, NULL, },\n \t};\n \n \t/* quote the path to preserve spaces, etc. */\n \tsq_quote_buf(&path, params->path);\n \tdict[0].value = path.buf;\n+\tif (params->fspath) {\n+\t\tsq_quote_buf(&fspath, params->fspath);\n+\t\tdict[1].value = fspath.buf;\n+\t}\n \n-\t/* expand all %f with the quoted path */\n+\t/* expand all %f and %p with the quoted path and fspath */\n \tstrbuf_expand(&cmd, params->cmd, strbuf_expand_dict_cb, &dict);\n \tstrbuf_release(&path);\n \n@@ -427,7 +434,8 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n \treturn (write_err || status);\n }\n \n-static int apply_filter(const char *path, const char *src, size_t len, int fd,\n+static int apply_filter(const char *path, const char *fspath,\n+\t\t\tconst char *src, size_t len, int fd,\n                         struct strbuf *dst, const char *cmd)\n {\n \t/*\n@@ -456,6 +464,7 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n \tparams.fd = fd;\n \tparams.cmd = cmd;\n \tparams.path = path;\n+\tparams.fspath = fspath;\n \n \tfflush(NULL);\n \tif (start_async(&async))\n@@ -486,6 +495,8 @@ static struct convert_driver {\n \tstruct convert_driver *next;\n \tconst char *smudge;\n \tconst char *clean;\n+\tconst char *smudge_to_file;\n+\tconst char *clean_from_file;\n \tint required;\n } *user_convert, **user_convert_tail;\n \n@@ -512,8 +523,9 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n \t}\n \n \t/*\n-\t * filter.<name>.smudge and filter.<name>.clean specifies\n-\t * the command line:\n+\t * filter.<name>.smudge, filter.<name>.clean,\n+\t * filter.<name>.smudge-to-file, filter.<name>.clean-from-file\n+\t * specifies the command line:\n \t *\n \t *\tcommand-line\n \t *\n@@ -526,6 +538,12 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(\"clean\", key))\n \t\treturn git_config_string(&drv->clean, var, value);\n \n+\tif (!strcmp(\"smudge-to-file\", key))\n+\t\treturn git_config_string(&drv->smudge_to_file, var, value);\n+\n+\tif (!strcmp(\"clean-from-file\", key))\n+\t\treturn git_config_string(&drv->clean_from_file, var, value);\n+\n \tif (!strcmp(\"required\", key)) {\n \t\tdrv->required = git_config_bool(var, value);\n \t\treturn 0;\n@@ -823,7 +841,35 @@ int would_convert_to_git_filter_fd(const char *path)\n \tif (!ca.drv->required)\n \t\treturn 0;\n \n-\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n+\treturn apply_filter(path, NULL, NULL, 0, -1, NULL, ca.drv->clean);\n+}\n+\n+int can_clean_from_file(const char *path)\n+{\n+\tstruct conv_attrs ca;\n+\n+\tconvert_attrs(&ca, path);\n+\tif (!ca.drv)\n+\t\treturn 0;\n+\n+\t/* Only use the clean-from-file filter when the clean filter is also\n+\t * configured.\n+\t */\n+\treturn (ca.drv->clean_from_file && ca.drv->clean);\n+}\n+\n+int can_smudge_to_file(const char *path)\n+{\n+\tstruct conv_attrs ca;\n+\n+\tconvert_attrs(&ca, path);\n+\tif (!ca.drv)\n+\t\treturn 0;\n+\n+\t/* Only use the smudge-to-file filter when the smudge filter is also\n+\t * configured.\n+\t */\n+\treturn (ca.drv->smudge_to_file && ca.drv->smudge);\n }\n \n const char *get_convert_attr_ascii(const char *path)\n@@ -866,7 +912,7 @@ int convert_to_git(const char *path, const char *src, size_t len,\n \t\trequired = ca.drv->required;\n \t}\n \n-\tret |= apply_filter(path, src, len, -1, dst, filter);\n+\tret |= apply_filter(path, NULL, src, len, -1, dst, filter);\n \tif (!ret && required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n@@ -891,14 +937,33 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n \tassert(ca.drv);\n \tassert(ca.drv->clean);\n \n-\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n+\tif (!apply_filter(path, NULL, NULL, 0, fd, dst, ca.drv->clean))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n \n-static int convert_to_working_tree_internal(const char *path, const char *src,\n+void convert_to_git_filter_from_file(const char *path, struct strbuf *dst,\n+\t\t\t\t   enum safe_crlf checksafe)\n+{\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(&ca, path);\n+\n+\tassert(ca.drv);\n+\tassert(ca.drv->clean);\n+\tassert(ca.drv->clean_from_file);\n+\n+\tif (!apply_filter(path, path, \"\", 0, -1, dst, ca.drv->clean_from_file))\n+\t\tdie(\"%s: clean-from-file filter '%s' failed\", path, ca.drv->name);\n+\n+\tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n+\tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n+}\n+\n+static int convert_to_working_tree_internal(const char *path,\n+\t\t\t\t\t    const char *destpath,\n+\t\t\t\t\t    const char *src,\n \t\t\t\t\t    size_t len, struct strbuf *dst,\n \t\t\t\t\t    int normalizing)\n {\n@@ -909,7 +974,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \n \tconvert_attrs(&ca, path);\n \tif (ca.drv) {\n-\t\tfilter = ca.drv->smudge;\n+\t\tif (destpath)\n+\t\t\tfilter = ca.drv->smudge_to_file;\n+\t\telse\n+\t\t\tfilter = ca.drv->smudge;\n \t\trequired = ca.drv->required;\n \t}\n \n@@ -920,7 +988,7 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t}\n \t/*\n \t * CRLF conversion can be skipped if normalizing, unless there\n-\t * is a smudge filter.  The filter might expect CRLFs.\n+\t * is a filter.  The filter might expect CRLFs.\n \t */\n \tif (filter || !normalizing) {\n \t\tret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);\n@@ -930,21 +998,30 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n-\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n+\tret_filter = apply_filter(path, destpath, src, len, -1, dst, filter);\n \tif (!ret_filter && required)\n-\t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n+\t\tdie(\"%s: %s filter %s failed\", path, destpath ? \"smudge-to-file\" : \"smudge\", ca.drv->name);\n \n \treturn ret | ret_filter;\n }\n \n int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n-\treturn convert_to_working_tree_internal(path, src, len, dst, 0);\n+\treturn convert_to_working_tree_internal(path, NULL, src, len, dst, 0);\n+}\n+\n+int convert_to_working_tree_filter_to_file(const char *path, const char *destpath, const char *src, size_t len)\n+{\n+\tstruct strbuf output = STRBUF_INIT;\n+\tint ret = convert_to_working_tree_internal(path, destpath, src, len, &output, 0);\n+\t/* The smudge-to-file filter stdout is not used. */\n+\tstrbuf_release(&output);\n+\treturn ret;\n }\n \n int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n-\tint ret = convert_to_working_tree_internal(path, src, len, dst, 1);\n+\tint ret = convert_to_working_tree_internal(path, NULL, src, len, dst, 1);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\ndiff --git a/convert.h b/convert.h\nindex ccf436b..53e1474 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -41,6 +41,10 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n \t\t\t  struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t   size_t len, struct strbuf *dst);\n+extern int convert_to_working_tree_filter_to_file(const char *path,\n+\t\t\t\t\t\t  const char *destpath,\n+\t\t\t\t\t\t  const char *src,\n+\t\t\t\t\t\t  size_t len);\n extern int renormalize_buffer(const char *path, const char *src, size_t len,\n \t\t\t      struct strbuf *dst);\n static inline int would_convert_to_git(const char *path)\n@@ -52,6 +56,12 @@ extern void convert_to_git_filter_fd(const char *path, int fd,\n \t\t\t\t     struct strbuf *dst,\n \t\t\t\t     enum safe_crlf checksafe);\n extern int would_convert_to_git_filter_fd(const char *path);\n+/* Precondition: can_clean_from_file(path) == true */\n+extern void convert_to_git_filter_from_file(const char *path,\n+\t\t\t\t\t    struct strbuf *dst,\n+\t\t\t\t\t    enum safe_crlf checksafe);\n+extern int can_clean_from_file(const char *path);\n+extern int can_smudge_to_file(const char *path);\n \n /*****************************************************************\n  *\n-- \n2.9.0.4.g2856e74.dirty\n\n"},{"id":"289401","messageId":"20160616203259.5886-2-joeyh@joeyh.name","threadId":"42634","inReplyTo":"20160616203259.5886-1-joeyh@joeyh.name","subject":"[PATCH 1/4] clarify %f documentation","fromName":"Joey Hess","fromEmail":"joeyh@joeyh.name","sentAt":"2016-06-16T20:32:56Z","receivedAt":"2016-06-16T20:33:57Z","isPatch":true,"sender":{"key":"joeyh@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"It's natural to expect %f to be an actual file on disk; help avoid that\nmistake.\n\nSigned-off-by: Joey Hess <joeyh@joeyh.name>\n---\n Documentation/gitattributes.txt | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex e3b1de8..e077cc9 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -365,8 +365,8 @@ you can declare that the filter is `required`, in the configuration:\n ------------------------\n \n Sequence \"%f\" on the filter command line is replaced with the name of\n-the file the filter is working on.  A filter might use this in keyword\n-substitution.  For example:\n+the file in the git repository the filter is working on.\n+A filter might use this in keyword substitution.  For example:\n \n ------------------------\n [filter \"p4\"]\n@@ -374,6 +374,9 @@ substitution.  For example:\n \tsmudge = git-p4-filter --smudge %f\n ------------------------\n \n+Note that the \"%f\" is the name of a file in the git repository; the\n+corresponding file on disk may not exist, or may have unrelated contents to\n+what git is filtering.\n \n Interaction between checkin/checkout attributes\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n-- \n2.9.0.4.g2856e74.dirty\n\n"},{"id":"289402","messageId":"20160616203259.5886-5-joeyh@joeyh.name","threadId":"42634","inReplyTo":"20160616203259.5886-1-joeyh@joeyh.name","subject":"[PATCH 4/4] use smudge-to-file in git checkout etc","fromName":"Joey Hess","fromEmail":"joeyh@joeyh.name","sentAt":"2016-06-16T20:32:59Z","receivedAt":"2016-06-16T20:34:00Z","isPatch":true,"sender":{"key":"joeyh@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"This makes git checkout, git reset, etc use smudge-to-file.\n\nIncludes test cases.\n\n(There's a call to convert_to_working_tree in merge-recursive.c\nthat could also be made to use smudge-to-file as well.)\n\nSigned-off-by: Joey Hess <joeyh@joeyh.name>\n---\n entry.c               | 34 +++++++++++++++++++++++++++-------\n t/t0021-conversion.sh | 44 ++++++++++++++++++++++++++++++++++++--------\n 2 files changed, 63 insertions(+), 15 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 519e042..6a23159 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -175,8 +175,12 @@ static int write_entry(struct cache_entry *ce,\n \n \t\t/*\n \t\t * Convert from git internal format to working tree format\n+\t\t * unless the smudge-to-file filter can write to the\n+\t\t * file directly.\n \t\t */\n+\t\tint smudge_to_file = can_smudge_to_file(ce->name);\n \t\tif (ce_mode_s_ifmt == S_IFREG &&\n+\t\t    ! smudge_to_file &&\n \t\t    convert_to_working_tree(ce->name, new, size, &buf)) {\n \t\t\tfree(new);\n \t\t\tnew = strbuf_detach(&buf, &newsize);\n@@ -189,13 +193,29 @@ static int write_entry(struct cache_entry *ce,\n \t\t\treturn error_errno(\"unable to create file %s\", path);\n \t\t}\n \n-\t\twrote = write_in_full(fd, new, size);\n-\t\tif (!to_tempfile)\n-\t\t\tfstat_done = fstat_output(fd, state, &st);\n-\t\tclose(fd);\n-\t\tfree(new);\n-\t\tif (wrote != size)\n-\t\t\treturn error(\"unable to write file %s\", path);\n+\t\tif (! can_smudge_to_file(ce->name)) {\n+\t\t\twrote = write_in_full(fd, new, size);\n+\t\t\tif (!to_tempfile)\n+\t\t\t\tfstat_done = fstat_output(fd, state, &st);\n+\t\t\tclose(fd);\n+\t\t\tfree(new);\n+\t\t\tif (wrote != size)\n+\t\t\t\treturn error(\"unable to write file %s\", path);\n+\t\t}\n+\t\telse {\n+\t\t\tclose(fd);\n+\t\t\tconvert_to_working_tree_filter_to_file(ce->name, path, new, size);\n+\t\t\tfree(new);\n+\t\t\tif (!to_tempfile) {\n+\t\t\t\t/* Re-open the file to stat it; the\n+\t\t\t\t * smudge-to-file filter may have replaced\n+\t\t\t\t * the file. */\n+\t\t\t\tfd = open(path, O_RDONLY);\n+\t\t\t\tif (fd < 0) {\n+\t\t\t\t\treturn error_errno(\"unable to create file %s\", path);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n \t\tbreak;\n \tcase S_IFGITLINK:\n \t\tif (to_tempfile)\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 1043ea5..b0e2e5e 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -14,12 +14,20 @@ chmod +x rot13.sh\n \n cat <<EOF >rot13-from-file.sh\n #!$SHELL_PATH\n-fsfile=\"\\$1\"\n+srcfile=\"\\$1\"\n touch rot13-from-file.ran\n-cat \"\\$fsfile\" | ./rot13.sh\n+cat \"\\$srcfile\" | ./rot13.sh\n EOF\n chmod +x rot13-from-file.sh\n \n+cat <<EOF >rot13-to-file.sh\n+#!$SHELL_PATH\n+destfile=\"\\$1\"\n+touch rot13-to-file.ran\n+./rot13.sh > \"\\$destfile\"\n+EOF\n+chmod +x rot13-to-file.sh\n+\n test_expect_success setup '\n \tgit config filter.rot13.smudge ./rot13.sh &&\n \tgit config filter.rot13.clean ./rot13.sh &&\n@@ -291,17 +299,37 @@ test_expect_success 'clean-from-file filter is used when adding a file' '\n \tcmp test.t fstest.t\n '\n \n+test_expect_success 'smudge-to-file filter is used when checking out a file' '\n+\ttest_config filter.rot13.smudge-to-file \"./rot13-to-file.sh %p\" &&\n+\n+\trm -f fstest.t &&\n+\tgit checkout -- fstest.t &&\n+\tcmp test.t fstest.t &&\n+\n+\ttest -e rot13-to-file.ran &&\n+\trm -f rot13-to-file.ran\n+'\n+\n test_expect_success 'clean-from-file filter is not used when clean filter is not configured' '\n-\ttest_config filter.no.smudge ./rot13.sh &&\n-\ttest_config filter.no.clean-from-file \"./rot13-from-file.sh %p\" &&\n+\ttest_config filter.noclean.smudge ./rot13.sh &&\n+\ttest_config filter.noclean.clean-from-file \"./rot13-from-file.sh %p\" &&\n \n-\techo \"*.no filter=no\" >.gitattributes &&\n+\techo \"*.no filter=noclean\" >.gitattributes &&\n \n \tcat test.t >test.no &&\n \tgit add test.no &&\n-\ttest ! -e rot13-from-file.ran &&\n-\tgit cat-file blob :test.no >actual &&\n-\tcmp test.t actual\n+\ttest ! -e rot13-from-file.ran\n+'\n+\n+test_expect_success 'smudge-to-file filter is not used when smudge filter is not configured' '\n+\ttest_config filter.nosmudge.clean ./rot13.sh &&\n+\ttest_config filter.nosmudge.smudge-to-file \"./rot13-to-file.sh %p\" &&\n+\n+\techo \"*.no filter=nosmudge\" >.gitattributes &&\n+\n+\trm -f fstest.t &&\n+\tgit checkout -- fstest.t &&\n+\ttest ! -e rot13-to-file.ran\n '\n \n test_done\n-- \n2.9.0.4.g2856e74.dirty\n\n"},{"id":"289405","messageId":"20160616205553.GA23176@kitenet.net","threadId":"42634","inReplyTo":"20160616203259.5886-5-joeyh@joeyh.name","subject":"Re: [PATCH 4/4] use smudge-to-file in git checkout etc","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-16T20:55:53Z","receivedAt":"2016-06-16T20:55:58Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Joey Hess wrote:\n> +\t\tint smudge_to_file = can_smudge_to_file(ce->name);\n>  \t\tif (ce_mode_s_ifmt == S_IFREG &&\n> +\t\t    ! smudge_to_file &&\n>  \t\t    convert_to_working_tree(ce->name, new, size, &buf)) {\n>  \t\t\tfree(new);\n>  \t\t\tnew = strbuf_detach(&buf, &newsize);\n> @@ -189,13 +193,29 @@ static int write_entry(struct cache_entry *ce,\n\n> +\t\tif (! can_smudge_to_file(ce->name)) {\n> +\t\t}\n> +\t\telse {\n> +\t\t\tclose(fd);\n> +\t\t\tconvert_to_working_tree_filter_to_file(ce->name, path, new, size);\n\nOops, I had meant to avoid using smudge-to-file when\ne_mode_s_ifmt != S_IFREG, and forgot it in this patch, so it does the\nwrong thing for symlinks.\n\nI'll send an updated patch set fixing this after any other review.\n\n-- \nsee shy jo\n"},{"id":"289409","messageId":"xmqq8ty44y4t.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160616203259.5886-2-joeyh@joeyh.name","subject":"Re: [PATCH 1/4] clarify %f documentation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-16T21:33:38Z","receivedAt":"2016-06-16T21:33:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <joeyh@joeyh.name> writes:\n\n> It's natural to expect %f to be an actual file on disk; help avoid that\n> mistake.\n\nI agree that \"the name of the file\" can be interpreted in many ways,\nand I agree that it would be a good idea to find a better phrase to\nname the path that is being worked on, but I do not think \"the file\nin the git repository\" is that phrase.  When I first read the\nupdated text without the above two lines in the log message, I\nthought \"hmph, so we may use a temporary file somewhere in $GIT_DIR/\nand that would be pointed at by %f, not the actual path we are\nworking on???\", i.e. the rephrasing had exactly the opposite effect\non me.\n\nGiven that this being part of gitattributes(5) that begins with\n\n    A `gitattributes` file is a simple text file that gives\n    `attributes` to pathnames.\n\n    Each line in `gitattributes` file is of form:\n\n            pattern\tattr1 attr2 ...\n\n    That is, a pattern followed by an attributes list,\n    separated by whitespaces.  When the pattern matches the\n    path in question, the attributes listed on the line are given to\n    the path.\n\nand that the readers already read something like this in the\nparagraphs before the mention of '%f':\n\n    For example, in .gitattributes, you would assign the `filter`\n    attribute for paths.\n\n    ------------------------\n    *.c\tfilter=indent\n    ------------------------\n\n    Then you would define ...\n\nI think using the word \"path\" somewhere in the updated description\nis more likely to have the effect you desire.\n\n\n> Signed-off-by: Joey Hess <joeyh@joeyh.name>\n> ---\n>  Documentation/gitattributes.txt | 7 +++++--\n>  1 file changed, 5 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index e3b1de8..e077cc9 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -365,8 +365,8 @@ you can declare that the filter is `required`, in the configuration:\n>  ------------------------\n>  \n>  Sequence \"%f\" on the filter command line is replaced with the name of\n> -the file the filter is working on.  A filter might use this in keyword\n> -substitution.  For example:\n> +the file in the git repository the filter is working on.\n> +A filter might use this in keyword substitution.  For example:\n>  \n>  ------------------------\n>  [filter \"p4\"]\n> @@ -374,6 +374,9 @@ substitution.  For example:\n>  \tsmudge = git-p4-filter --smudge %f\n>  ------------------------\n>  \n> +Note that the \"%f\" is the name of a file in the git repository; the\n> +corresponding file on disk may not exist, or may have unrelated contents to\n> +what git is filtering.\n>  \n>  Interaction between checkin/checkout attributes\n>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n"},{"id":"289411","messageId":"xmqqy4643ig1.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160616203259.5886-3-joeyh@joeyh.name","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-16T21:57:50Z","receivedAt":"2016-06-16T21:57:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <joeyh@joeyh.name> writes:\n\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 2e1b2e4..bbb9296 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1299,14 +1299,29 @@ format.useAutoBase::\n>  \tformat-patch by default.\n>  \n>  filter.<driver>.clean::\n> -\tThe command which is used to convert the content of a worktree\n> -\tfile to a blob upon checkin.  See linkgit:gitattributes[5] for\n> -\tdetails.\n> +\tThe command which is used on checkin to convert the content of\n> +\ta worktree file (provided on stdin) to a blob (written to stdout).\n> +\tSee linkgit:gitattributes[5] for details.\n>  \n>  filter.<driver>.smudge::\n> -\tThe command which is used to convert the content of a blob\n> -\tobject to a worktree file upon checkout.  See\n> -\tlinkgit:gitattributes[5] for details.\n> +\tThe command which is used on checkout to convert the content of a\n> +\tblob object (provided on stdin) to a worktree file (written to\n> +\tstdout).\n> +\tSee linkgit:gitattributes[5] for details.\n\nA \"filter\" by definition reads from its standard input and writes\nthe result to its standard output, so I do not know if this change\nis necessary.  If you are going to do this, in documentation\n(i.e. not code), please spell standard input/output out, and avoid\nstdin/stdout.\n\nThere is what we would want to fix, though.  \"worktree file\" should\nbe spelled \"working tree file\".  This used not to matter before \"git\nworktree\" was invented (before that we used these two terms\ninterchangeably), but these days the distinction matters.\n\n> +filter.<driver>.clean-from-file::\n\nDocumentation/config.txt hopefully lists all the configuration, but\nI do not see anything that uses 'words-joined-with-dash' format.\nPlease do not invent new out-of-convention names.\n\n> +\tOptional command which is used on checkin to convert the content\n> +\tof a worktree file, which can be read from disk, to a blob\n> +\t(written to stdout).\n\nI am not sure why we want to say \"which can be read from disk\".\n\nI think what a reader needs to be told (but this paragraph is not\ntelling her) in order to understand what you meant to say is:\n\n\tcleanFromFile is asked to produce the \"cleaned\" content to\n\tits standard output (to be stored in the object database),\n\tbut unlike clean, it does not work as a filter and is not\n\tgiven a file descriptor to read the working tree contents\n\tfrom.  Instead, it is told the path for which the contents\n\tneed to be generated as the first parameter on its command\n\tline.\n\n(the above is deliberately made verbose and is not meant as a\nsuggestion to be literally used in your updated documentation).\n\nWith that understanding, the reader may be able to guess that \"can\nbe read from disk\" is a permission for her cleanFromFile filter\n(i.e. it does not necessarily have to read from it and produced its\noutput by some other magic); otherwise it can be misread as if the\ncontent of a working tree file is deposited on the disk to enable\nthe filter to read it, but the location of that on-disk file is\nsomewhere unspecified by this paragraph.\n\n> +\tOnly used when filter.<driver>.clean is also configured.\n> +\tSee linkgit:gitattributes[5] for details.\n> +\n> +filter.<driver>.smudge-to-file::\n> +\tOptional command which is used to convert the content of a blob\n> +\tobject (provided on stdin) to a worktree file, writing directly\n> +\tto the file.\n\nA similar comment applies.  With \", writing directly to the file.\"\nreplaced with something like \". The command is expected to write to\nthe working tree file at path given as the first parameter on the\ncommand line.\" it would be sufficient (i.e. it is clear enough that\nit does not give the smudged contents via its standard output, and\nno need to say \"unlike smudge filter, ...\" like we needed to for the\n\"clean\" side).\n\n"},{"id":"289419","messageId":"20160617024803.GB17088@kitenet.net","threadId":"42634","inReplyTo":"xmqq8ty44y4t.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/4] clarify %f documentation","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-17T02:48:03Z","receivedAt":"2016-06-17T02:48:20Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Junio C Hamano wrote:\n> I agree that \"the name of the file\" can be interpreted in many ways,\n> and I agree that it would be a good idea to find a better phrase to\n> name the path that is being worked on, but I do not think \"the file\n> in the git repository\" is that phrase.\n\n> I think using the word \"path\" somewhere in the updated description\n> is more likely to have the effect you desire.\n\n\"path\" is also very ambiguous. I see that \"tracked\" is often used to\ndescribe what %f is, so how about:\n \n+ Note that \"%f\" is the name of a file tracked by Git. Depending on the\n+ version that is being filtered, the corresponding file on disk may not\n+ exist, or may have different contents. So, smudge and clean commands should\n+ not try to access the file on disk.\n\n-- \nsee shy jo\n"},{"id":"289421","messageId":"xmqqfusc33aa.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160617024803.GB17088@kitenet.net","subject":"Re: [PATCH 1/4] clarify %f documentation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-17T03:25:17Z","receivedAt":"2016-06-17T03:25:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <id@joeyh.name> writes:\n\n> Junio C Hamano wrote:\n>> I agree that \"the name of the file\" can be interpreted in many ways,\n>> and I agree that it would be a good idea to find a better phrase to\n>> name the path that is being worked on, but I do not think \"the file\n>> in the git repository\" is that phrase.\n>\n>> I think using the word \"path\" somewhere in the updated description\n>> is more likely to have the effect you desire.\n>\n> \"path\" is also very ambiguous. I see that \"tracked\" is often used to\n> describe what %f is, so how about:\n>  \n> + Note that \"%f\" is the name of a file tracked by Git. Depending on the\n> + version that is being filtered, the corresponding file on disk may not\n> + exist, or may have different contents. So, smudge and clean commands should\n> + not try to access the file on disk.\n\nI think that places stress on a wrong point.\n\nI do have a preference between \"file\" or \"path\", merely because, as\nI showed already (go back and read what you are responding to), the\npreceding paragraphs all talk in terms of \"paths\".  But that is not\nthe important part.  \n\n\"tracked by Git\" is not all that interesting, compared to the fact\nthat your filter needs to give contents relevant to that path\nbecause that is what the command line argument Git gives you with\n'%f' means.  It is not a random filename \"tracked by Git\".  Among 47\nother files tracked by Git, the single one being given is the one\nthe code that drives the filter is WORKING ON, and I think that\nneeds to be written in the description, hence \"the path that is\nbeing worked on\" was my suggestion.\n\n\n\n"},{"id":"289422","messageId":"CAPig+cTk8NkNKRcDCWpOhh_g4kXUhHHfFdfPeeq1-zJTzxXt6w@mail.gmail.com","threadId":"42634","inReplyTo":"20160616203259.5886-3-joeyh@joeyh.name","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2016-06-17T06:05:43Z","receivedAt":"2016-06-17T06:05:48Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jun 16, 2016 at 4:32 PM, Joey Hess <joeyh@joeyh.name> wrote:\n> This adds new smudge-to-file and clean-from-file filter commands,\n> which are similar to smudge and clean but allow direct access to files on\n> disk.\n> [...]\n> Signed-off-by: Joey Hess <joeyh@joeyh.name>\n> ---\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> @@ -1299,14 +1299,29 @@ format.useAutoBase::\n> +filter.<driver>.clean-from-file::\n> +       Optional command which is used on checkin to convert the content\n> +       of a worktree file, which can be read from disk, to a blob\n> +       (written to stdout).\n> +       Only used when filter.<driver>.clean is also configured.\n> +       See linkgit:gitattributes[5] for details.\n> +\n> +filter.<driver>.smudge-to-file::\n> +       Optional command which is used to convert the content of a blob\n> +       object (provided on stdin) to a worktree file, writing directly\n> +       to the file.\n> +       Only used when filter.<driver>.clean is also configured.\n\ns/clean/smudge/\n\n> +       See linkgit:gitattributes[5] for details.\n"},{"id":"289433","messageId":"3c7349e7-34a1-ed7f-721e-e7a7b191bb3d@drmicha.warpmail.net","threadId":"42634","inReplyTo":"20160616203259.5886-3-joeyh@joeyh.name","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2016-06-17T09:36:11Z","receivedAt":"2016-06-17T09:36:20Z","isPatch":true,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Joey Hess venit, vidit, dixit 16.06.2016 22:32:\n> This adds new smudge-to-file and clean-from-file filter commands,\n> which are similar to smudge and clean but allow direct access to files on\n> disk.\n> \n> In smudge-to-file and clean-from-file, \"%p\" is expanded to the path to the\n> file that should be cleaned from, or smudged to.\n> \n> This interface can be much more efficient when operating on large files,\n> because the whole file content does not need to be streamed through the\n> filter. It even allows for things like clean-from-file commands that avoid\n> reading the whole content of the file, and for smudge-to-file commands that\n> populate a work tree file using an efficient Copy On Write operation.\n> \n> The new filter commands will not be used for all filtering. They are\n> efficient to use when git add is adding a file, or when the work tree is\n> being updated, but not a good fit when git is internally filtering blob\n> objects in memory for eg, a diff.\n> \n> So, a user who wants to use smudge-to-file should also provide a smudge\n> command to be used in cases where smudge-to-file is not used. And ditto\n> with clean-from-file and clean. To avoid foot-shooting configurations, the\n> new commands are not used unless the old commands are also configured.\n\nI'm not sure this will save all feet. I foresee \"why is smudge-to-file\nnot doing anything\" reports...\n\nIn addition, it opens the way to doing completely different things in\nsmudge and smudge-to-file - which partly is intentional, of course.\n\nDo you make any promises that %p is a seekable file?\n\n> That also ensures that a filter driver configuration that includes these\n> new commands will work, although less efficiently, when used with an older\n> version of git that does not support them.\n> \n> Signed-off-by: Joey Hess <joeyh@joeyh.name>\n> ---\n>  Documentation/config.txt        |  27 +++++++---\n>  Documentation/gitattributes.txt |  37 ++++++++++++++\n>  convert.c                       | 107 ++++++++++++++++++++++++++++++++++------\n>  convert.h                       |  10 ++++\n>  4 files changed, 160 insertions(+), 21 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 2e1b2e4..bbb9296 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1299,14 +1299,29 @@ format.useAutoBase::\n>  \tformat-patch by default.\n>  \n>  filter.<driver>.clean::\n> -\tThe command which is used to convert the content of a worktree\n> -\tfile to a blob upon checkin.  See linkgit:gitattributes[5] for\n> -\tdetails.\n> +\tThe command which is used on checkin to convert the content of\n> +\ta worktree file (provided on stdin) to a blob (written to stdout).\n> +\tSee linkgit:gitattributes[5] for details.\n>  \n>  filter.<driver>.smudge::\n> -\tThe command which is used to convert the content of a blob\n> -\tobject to a worktree file upon checkout.  See\n> -\tlinkgit:gitattributes[5] for details.\n> +\tThe command which is used on checkout to convert the content of a\n> +\tblob object (provided on stdin) to a worktree file (written to\n> +\tstdout).\n> +\tSee linkgit:gitattributes[5] for details.\n> +\n> +filter.<driver>.clean-from-file::\n> +\tOptional command which is used on checkin to convert the content\n> +\tof a worktree file, which can be read from disk, to a blob\n> +\t(written to stdout).\n> +\tOnly used when filter.<driver>.clean is also configured.\n> +\tSee linkgit:gitattributes[5] for details.\n> +\n> +filter.<driver>.smudge-to-file::\n> +\tOptional command which is used to convert the content of a blob\n> +\tobject (provided on stdin) to a worktree file, writing directly\n> +\tto the file.\n> +\tOnly used when filter.<driver>.clean is also configured.\n> +\tSee linkgit:gitattributes[5] for details.\n>  \n>  fsck.<msg-id>::\n>  \tAllows overriding the message type (error, warn or ignore) of a\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index e077cc9..32621e7 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -378,6 +378,43 @@ Note that the \"%f\" is the name of a file in the git repository; the\n>  corresponding file on disk may not exist, or may have unrelated contents to\n>  what git is filtering.\n>  \n> +There are two extra commands \"clean-from-file\" and \"smudge-to-file\", which\n> +can optionally be set in a filter driver. These are similar to the \"clean\"\n> +and \"smudge\" commands, but avoid needing to pipe the contents of files\n> +through the filters, and instead reading/writing files in the filesystem.\n> +This can be more efficient when using filters with large files that are not\n> +directly stored in the repository.\n> +\n> +Sequence \"%p\" on the \"clean-from-file\" and \"smudge-to-file\" command line\n> +is replaced with the path to a file that the filter can access.\n> +\n> +In the \"clean-from-file\" command, \"%p\" is the path to the file that\n> +it should clean. Like the \"clean\" command, it should output the cleaned\n> +version to standard output.\n> +\n> +In the \"smudge-from-file\" command, \"%p\" is the path to the file that it\n> +should write to. (This file will already exist, as an empty file that can\n> +be written to or replaced.) Like the \"smudge\" command, \"smudge-from-file\"\n> +is fed the blob object from its standard input.\n> +\n> +Some git operations that need to apply filters cannot use \"clean-from-file\"\n> +and \"smudge-to-file\", since the files are not present to disk. So, to avoid\n> +inconsistent behavior, \"clean-from-file\" will only be used if \"clean\" is\n> +also configured, and \"smudge-to-file\" will only be used if \"smudge\" is also\n> +configured.\n> +\n> +An example large file storage filter driver using \"*-from-file\" follows:\n> +\n> +------------------------\n> +[filter \"bigfiles\"]\n> +\tclean = store-bigfile --from-stdin\n> +\tclean-from-file = store-bigfile %p\n> +\tsmudge = retrieve-bigfile --to-stdout\n> +\tsmudge-to-file = retrieve-bigfile %p\n> +\trequired\n> +\n> +------------------------\n> +\n>  Interaction between checkin/checkout attributes\n>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>  \n> diff --git a/convert.c b/convert.c\n> index b1614bf..bfb7578 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -360,7 +360,8 @@ struct filter_params {\n>  \tunsigned long size;\n>  \tint fd;\n>  \tconst char *cmd;\n> -\tconst char *path;\n> +\tconst char *path; /* Path within the git repository */\n> +\tconst char *fspath; /* Path to file on disk */\n>  };\n>  \n>  static int filter_buffer_or_fd(int in, int out, void *data)\n> @@ -376,16 +377,22 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n>  \t/* apply % substitution to cmd */\n>  \tstruct strbuf cmd = STRBUF_INIT;\n>  \tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf fspath = STRBUF_INIT;\n>  \tstruct strbuf_expand_dict_entry dict[] = {\n>  \t\t{ \"f\", NULL, },\n> +\t\t{ (params->fspath ? \"p\" : NULL), NULL, },\n>  \t\t{ NULL, NULL, },\n>  \t};\n>  \n>  \t/* quote the path to preserve spaces, etc. */\n>  \tsq_quote_buf(&path, params->path);\n>  \tdict[0].value = path.buf;\n> +\tif (params->fspath) {\n> +\t\tsq_quote_buf(&fspath, params->fspath);\n> +\t\tdict[1].value = fspath.buf;\n> +\t}\n>  \n> -\t/* expand all %f with the quoted path */\n> +\t/* expand all %f and %p with the quoted path and fspath */\n>  \tstrbuf_expand(&cmd, params->cmd, strbuf_expand_dict_cb, &dict);\n>  \tstrbuf_release(&path);\n>  \n> @@ -427,7 +434,8 @@ static int filter_buffer_or_fd(int in, int out, void *data)\n>  \treturn (write_err || status);\n>  }\n>  \n> -static int apply_filter(const char *path, const char *src, size_t len, int fd,\n> +static int apply_filter(const char *path, const char *fspath,\n> +\t\t\tconst char *src, size_t len, int fd,\n>                          struct strbuf *dst, const char *cmd)\n>  {\n>  \t/*\n> @@ -456,6 +464,7 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,\n>  \tparams.fd = fd;\n>  \tparams.cmd = cmd;\n>  \tparams.path = path;\n> +\tparams.fspath = fspath;\n>  \n>  \tfflush(NULL);\n>  \tif (start_async(&async))\n> @@ -486,6 +495,8 @@ static struct convert_driver {\n>  \tstruct convert_driver *next;\n>  \tconst char *smudge;\n>  \tconst char *clean;\n> +\tconst char *smudge_to_file;\n> +\tconst char *clean_from_file;\n>  \tint required;\n>  } *user_convert, **user_convert_tail;\n>  \n> @@ -512,8 +523,9 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n>  \t}\n>  \n>  \t/*\n> -\t * filter.<name>.smudge and filter.<name>.clean specifies\n> -\t * the command line:\n> +\t * filter.<name>.smudge, filter.<name>.clean,\n> +\t * filter.<name>.smudge-to-file, filter.<name>.clean-from-file\n> +\t * specifies the command line:\n>  \t *\n>  \t *\tcommand-line\n>  \t *\n> @@ -526,6 +538,12 @@ static int read_convert_config(const char *var, const char *value, void *cb)\n>  \tif (!strcmp(\"clean\", key))\n>  \t\treturn git_config_string(&drv->clean, var, value);\n>  \n> +\tif (!strcmp(\"smudge-to-file\", key))\n> +\t\treturn git_config_string(&drv->smudge_to_file, var, value);\n> +\n> +\tif (!strcmp(\"clean-from-file\", key))\n> +\t\treturn git_config_string(&drv->clean_from_file, var, value);\n> +\n>  \tif (!strcmp(\"required\", key)) {\n>  \t\tdrv->required = git_config_bool(var, value);\n>  \t\treturn 0;\n> @@ -823,7 +841,35 @@ int would_convert_to_git_filter_fd(const char *path)\n>  \tif (!ca.drv->required)\n>  \t\treturn 0;\n>  \n> -\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n> +\treturn apply_filter(path, NULL, NULL, 0, -1, NULL, ca.drv->clean);\n> +}\n> +\n> +int can_clean_from_file(const char *path)\n> +{\n> +\tstruct conv_attrs ca;\n> +\n> +\tconvert_attrs(&ca, path);\n> +\tif (!ca.drv)\n> +\t\treturn 0;\n> +\n> +\t/* Only use the clean-from-file filter when the clean filter is also\n> +\t * configured.\n> +\t */\n> +\treturn (ca.drv->clean_from_file && ca.drv->clean);\n> +}\n> +\n> +int can_smudge_to_file(const char *path)\n> +{\n> +\tstruct conv_attrs ca;\n> +\n> +\tconvert_attrs(&ca, path);\n> +\tif (!ca.drv)\n> +\t\treturn 0;\n> +\n> +\t/* Only use the smudge-to-file filter when the smudge filter is also\n> +\t * configured.\n> +\t */\n> +\treturn (ca.drv->smudge_to_file && ca.drv->smudge);\n>  }\n>  \n>  const char *get_convert_attr_ascii(const char *path)\n> @@ -866,7 +912,7 @@ int convert_to_git(const char *path, const char *src, size_t len,\n>  \t\trequired = ca.drv->required;\n>  \t}\n>  \n> -\tret |= apply_filter(path, src, len, -1, dst, filter);\n> +\tret |= apply_filter(path, NULL, src, len, -1, dst, filter);\n>  \tif (!ret && required)\n>  \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n>  \n> @@ -891,14 +937,33 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,\n>  \tassert(ca.drv);\n>  \tassert(ca.drv->clean);\n>  \n> -\tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))\n> +\tif (!apply_filter(path, NULL, NULL, 0, fd, dst, ca.drv->clean))\n>  \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n>  \n>  \tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n>  \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n>  }\n>  \n> -static int convert_to_working_tree_internal(const char *path, const char *src,\n> +void convert_to_git_filter_from_file(const char *path, struct strbuf *dst,\n> +\t\t\t\t   enum safe_crlf checksafe)\n> +{\n> +\tstruct conv_attrs ca;\n> +\tconvert_attrs(&ca, path);\n> +\n> +\tassert(ca.drv);\n> +\tassert(ca.drv->clean);\n> +\tassert(ca.drv->clean_from_file);\n> +\n> +\tif (!apply_filter(path, path, \"\", 0, -1, dst, ca.drv->clean_from_file))\n> +\t\tdie(\"%s: clean-from-file filter '%s' failed\", path, ca.drv->name);\n> +\n> +\tcrlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n> +\tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n> +}\n> +\n> +static int convert_to_working_tree_internal(const char *path,\n> +\t\t\t\t\t    const char *destpath,\n> +\t\t\t\t\t    const char *src,\n>  \t\t\t\t\t    size_t len, struct strbuf *dst,\n>  \t\t\t\t\t    int normalizing)\n>  {\n> @@ -909,7 +974,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n>  \n>  \tconvert_attrs(&ca, path);\n>  \tif (ca.drv) {\n> -\t\tfilter = ca.drv->smudge;\n> +\t\tif (destpath)\n> +\t\t\tfilter = ca.drv->smudge_to_file;\n> +\t\telse\n> +\t\t\tfilter = ca.drv->smudge;\n>  \t\trequired = ca.drv->required;\n>  \t}\n>  \n> @@ -920,7 +988,7 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n>  \t}\n>  \t/*\n>  \t * CRLF conversion can be skipped if normalizing, unless there\n> -\t * is a smudge filter.  The filter might expect CRLFs.\n> +\t * is a filter.  The filter might expect CRLFs.\n>  \t */\n>  \tif (filter || !normalizing) {\n>  \t\tret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);\n> @@ -930,21 +998,30 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n>  \t\t}\n>  \t}\n>  \n> -\tret_filter = apply_filter(path, src, len, -1, dst, filter);\n> +\tret_filter = apply_filter(path, destpath, src, len, -1, dst, filter);\n>  \tif (!ret_filter && required)\n> -\t\tdie(\"%s: smudge filter %s failed\", path, ca.drv->name);\n> +\t\tdie(\"%s: %s filter %s failed\", path, destpath ? \"smudge-to-file\" : \"smudge\", ca.drv->name);\n>  \n>  \treturn ret | ret_filter;\n>  }\n>  \n>  int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)\n>  {\n> -\treturn convert_to_working_tree_internal(path, src, len, dst, 0);\n> +\treturn convert_to_working_tree_internal(path, NULL, src, len, dst, 0);\n> +}\n> +\n> +int convert_to_working_tree_filter_to_file(const char *path, const char *destpath, const char *src, size_t len)\n> +{\n> +\tstruct strbuf output = STRBUF_INIT;\n> +\tint ret = convert_to_working_tree_internal(path, destpath, src, len, &output, 0);\n> +\t/* The smudge-to-file filter stdout is not used. */\n> +\tstrbuf_release(&output);\n> +\treturn ret;\n>  }\n>  \n>  int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)\n>  {\n> -\tint ret = convert_to_working_tree_internal(path, src, len, dst, 1);\n> +\tint ret = convert_to_working_tree_internal(path, NULL, src, len, dst, 1);\n>  \tif (ret) {\n>  \t\tsrc = dst->buf;\n>  \t\tlen = dst->len;\n> diff --git a/convert.h b/convert.h\n> index ccf436b..53e1474 100644\n> --- a/convert.h\n> +++ b/convert.h\n> @@ -41,6 +41,10 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n>  \t\t\t  struct strbuf *dst, enum safe_crlf checksafe);\n>  extern int convert_to_working_tree(const char *path, const char *src,\n>  \t\t\t\t   size_t len, struct strbuf *dst);\n> +extern int convert_to_working_tree_filter_to_file(const char *path,\n> +\t\t\t\t\t\t  const char *destpath,\n> +\t\t\t\t\t\t  const char *src,\n> +\t\t\t\t\t\t  size_t len);\n>  extern int renormalize_buffer(const char *path, const char *src, size_t len,\n>  \t\t\t      struct strbuf *dst);\n>  static inline int would_convert_to_git(const char *path)\n> @@ -52,6 +56,12 @@ extern void convert_to_git_filter_fd(const char *path, int fd,\n>  \t\t\t\t     struct strbuf *dst,\n>  \t\t\t\t     enum safe_crlf checksafe);\n>  extern int would_convert_to_git_filter_fd(const char *path);\n> +/* Precondition: can_clean_from_file(path) == true */\n> +extern void convert_to_git_filter_from_file(const char *path,\n> +\t\t\t\t\t    struct strbuf *dst,\n> +\t\t\t\t\t    enum safe_crlf checksafe);\n> +extern int can_clean_from_file(const char *path);\n> +extern int can_smudge_to_file(const char *path);\n>  \n>  /*****************************************************************\n>   *\n> \n\n"},{"id":"289435","messageId":"20160617123209.GA24025@kitenet.net","threadId":"42634","inReplyTo":"xmqqfusc33aa.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/4] clarify %f documentation","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-17T12:32:09Z","receivedAt":"2016-06-17T12:32:26Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Junio C Hamano wrote:\n> \"tracked by Git\" is not all that interesting, compared to the fact\n> that your filter needs to give contents relevant to that path\n> because that is what the command line argument Git gives you with\n> '%f' means.  It is not a random filename \"tracked by Git\".  Among 47\n> other files tracked by Git, the single one being given is the one\n> the code that drives the filter is WORKING ON, and I think that\n> needs to be written in the description, hence \"the path that is\n> being worked on\" was my suggestion.\n\nAh, \"being worked on\" does clarify it well, I think:\n\n+ Note that \"%f\" is the name of the path that is being worked on. Depending\n+ on the version that is being filtered, the corresponding file on disk may\n+ not exist, or may have different contents. So, smudge and clean commands\n+ should not try to access the file on disk, but only act as filters on the\n+ content provided to them on standard input.\n\n-- \nsee shy jo\n"},{"id":"289436","messageId":"20160617124707.GB24025@kitenet.net","threadId":"42634","inReplyTo":"3c7349e7-34a1-ed7f-721e-e7a7b191bb3d@drmicha.warpmail.net","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-17T12:47:07Z","receivedAt":"2016-06-17T12:47:21Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Michael J Gruber wrote:\n> I'm not sure this will save all feet. I foresee \"why is smudge-to-file\n> not doing anything\" reports...\n\nIt could display a warning message if smudge-to-file is set and smudge is\nnot.\n\n> In addition, it opens the way to doing completely different things in\n> smudge and smudge-to-file - which partly is intentional, of course.\n\nThey can be implemented very differently, but need to provide the same\nfile content. Otherwise git checkout and git diff would show different\ncontent for the same file, for example. This is up to the implementor of\na filter driver to get right.\n\n> Do you make any promises that %p is a seekable file?\n\nYes, %p is a regular file and so is seekable, statable, etc.\n\n-- \nsee shy jo\n"},{"id":"289441","messageId":"20160617131311.GC24025@kitenet.net","threadId":"42634","inReplyTo":"xmqqy4643ig1.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-17T13:13:11Z","receivedAt":"2016-06-17T13:13:25Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Junio C Hamano wrote:\n> There is what we would want to fix, though.  \"worktree file\" should\n> be spelled \"working tree file\". This used not to matter before \"git\n> worktree\" was invented (before that we used these two terms\n> interchangeably), but these days the distinction matters.\n\nThe existing documentation that I am patching uses the term \"worktree\nfile\" which is why I continued to use that wording.\n\n(Unless this is a documentation transition that you want to happen\npeicemeal as documentation is updated for other reasons?)\n\n> > +filter.<driver>.clean-from-file::\n> \n> Documentation/config.txt hopefully lists all the configuration, but\n> I do not see anything that uses 'words-joined-with-dash' format.\n> Please do not invent new out-of-convention names.\n\nPoint taken; I'll use cleanFromFile and smudgeToFile.\n\nHere's a revised version of the documentation that I think takes the other\nsuggestions onboard. I emphasise that clean and smudge operate as filters,\nto contrast better with cleanFromFile and smudgeToFile not operating as\nregular stdio filters.\n\n filter.<driver>.clean::\n-       The command which is used to convert the content of a worktree\n+       The command which is used as a filter to convert the content of a worktree\n        file to a blob upon checkin.  See linkgit:gitattributes[5] for\n        details.\n \n filter.<driver>.smudge::\n-       The command which is used to convert the content of a blob\n+       The command which is used as a filter to convert the content of a blob\n        object to a worktree file upon checkout.  See\n        linkgit:gitattributes[5] for details.\n \n+filter.<driver>.cleanFromFile::\n+       Similar to filter.<driver>.clean but the specified command \n+       directly accesses a worktree file on disk, rather than\n+       receiving the file content from standard input. \n+       In the command, \"%p\" is replaced with the name of the file. \n+       Only used when filter.<driver>.clean is also configured.\n+       See linkgit:gitattributes[5] for details.\n+\n+filter.<driver>.smudgeToFile::\n+       Similar to filter.<driver>.smudge but the specified command\n+       writes the content of a blob directly to a worktree file,\n+       rather than to standard output.\n+       In the command, \"%p\" is replaced with the name of the file.\n+       Only used when filter.<driver>.smudge is also configured.\n+       See linkgit:gitattributes[5] for details.\n+\n\nThis could be extended more, but I think this should describe the config\nsettings concisely and point to the more involved discussion of filter drivers\nin gitattributes.\n\n-- \nsee shy jo\n"},{"id":"289448","messageId":"xmqqbn2z3j0c.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160617123209.GA24025@kitenet.net","subject":"Re: [PATCH 1/4] clarify %f documentation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-17T15:57:55Z","receivedAt":"2016-06-17T15:58:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <id@joeyh.name> writes:\n\n> Junio C Hamano wrote:\n>> \"tracked by Git\" is not all that interesting, compared to the fact\n>> that your filter needs to give contents relevant to that path\n>> because that is what the command line argument Git gives you with\n>> '%f' means.  It is not a random filename \"tracked by Git\".  Among 47\n>> other files tracked by Git, the single one being given is the one\n>> the code that drives the filter is WORKING ON, and I think that\n>> needs to be written in the description, hence \"the path that is\n>> being worked on\" was my suggestion.\n>\n> Ah, \"being worked on\" does clarify it well, I think:\n>\n> + Note that \"%f\" is the name of the path that is being worked on. Depending\n> + on the version that is being filtered, the corresponding file on disk may\n> + not exist, or may have different contents. So, smudge and clean commands\n> + should not try to access the file on disk, but only act as filters on the\n> + content provided to them on standard input.\n\nLooks much better than the original text.  Thanks.\n\nAnother reason why you do not want to say \"tracked\" is for \"git add\"\nthat adds a brand-new file that is not yet tracked.  Technically,\n\"trackable file\" may work but we have to add \"trackable\" to the\nglossary, but so far we did without having to, and I'm reluctant to\ninvent yet another word, so...\n"},{"id":"289452","messageId":"xmqq60t73igm.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160617124707.GB24025@kitenet.net","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-17T16:09:45Z","receivedAt":"2016-06-17T16:09:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <id@joeyh.name> writes:\n\n>> Do you make any promises that %p is a seekable file?\n>\n> Yes, %p is a regular file and so is seekable, statable, etc.\n\nI wonder if we prefer not to make this customizable (i.e. not having\nto use'%p').  Unlike '%f' that is optional, smudgeTo and cleanFrom\n\"filters\" are not filters and MUST read from the path given and not\nfrom any other paths.\n\nA misconfigured smudgeTo filter that uses %f would overwrite a wrong\nfile when used with checkout-index --prefix=<there>, right?\n\nThe only thing '%p' buys is that a \"filter\" could be written in such\na way that takes the pathname at arbitrary place on the command\nline.  A command line that does not have '%p' anywhere is invalid,\nwhich is quite different from how '%f' behaves.\n\nWould an interface that always appends the pathname at the end of\nthe command line string work?  E.g.\n\n\t[filter \"foo\"] smugeToFile = cmd --from-file\n\nwould run \"cmd --from-file PATH\" when Git wants it to read from\nPATH, and\n\n\t[filter \"bar\"] smugeToFile = sh -c 'cmd --from-file=\"$0\"'\n\nwould run\n\n\tsh -c 'cmd --from-file=\"$0\"' PATH\n\nwhich in turn becomes 'cmd --from-file=PATH'.\n\nOr something like that.\n\n"},{"id":"289460","messageId":"xmqqk2hn207d.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"xmqq60t73igm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-17T17:29:26Z","receivedAt":"2016-06-17T17:29:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> I wonder if we prefer not to make this customizable (i.e. not having\n> to use'%p').  Unlike '%f' that is optional, smudgeTo and cleanFrom\n> \"filters\" are not filters and MUST read from the path given and not\n> from any other paths.\n> ...\n> Would an interface that always appends the pathname at the end of\n> the command line string work?\n\nJust to avoid misunderstanding, when I say \"I wonder if we want to\ngo this route instead\", I am open to a response that says \"No, we do\nnot; going that route is worse because ...\".  Please do not take it\nas a suggestion or request to \"do it this way instead\".\n\nI may not even favor that other \"route\" I mention in my message with\n\"I wonder\"; I am merely saying that I cannot come up with that\n\"because ...\"  part to reject it myself, and asking for help from\nothers.\n\nIn this case, I would be happy to hear \"Yes, '%p' may merely be a\nway to allow taking the pathname at anywhere on the command line,\nbut that flexibility is important because...\".\n\nThanks.\n\n"},{"id":"289462","messageId":"20160617173725.GB15901@kitenet.net","threadId":"42634","inReplyTo":"xmqq60t73igm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-17T17:37:25Z","receivedAt":"2016-06-17T17:37:38Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Junio C Hamano wrote:\n> Would an interface that always appends the pathname at the end of\n> the command line string work?\n\nI'm ok with this, and like getting rid of %p as it's not distinguishable\nfrom %f without reading the documentation.\n\nThe sh -c trick can of course be used if some other ordering of\nparameters is needed. Probably anything using this interface is gonna be\nimplemented with the interface in mind from the beginning and won't need\nsuch a trick.\n\n-- \nsee shy jo\n"},{"id":"289466","messageId":"20160617180650.GA16404@kitenet.net","threadId":"42634","inReplyTo":"xmqq60t73igm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2016-06-17T18:06:50Z","receivedAt":"2016-06-17T18:07:08Z","isPatch":true,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Junio C Hamano wrote:\n> Would an interface that always appends the pathname at the end of\n> the command line string work?\n\nOne problem with this is that \"appends\" is subtly unclear in this case.\n\nWith the example of smugeToFile = cmd --to-file\nit seems that a space should be added by git before the filename.\n\nOn the other handle, consider smugeToFile = cmd --to-file=\nhere a space is not wanted before the filename.\n\nSo, either a space is automatically included before the filename\nand the second example breaks, or no space is included, and\nto make the first example work would need careful inclusion of the\ntrailing space with quoting to prevent it being elided\neg, smugeToFile = \"cmd --to-file \"\n\n%p does avoid this ambiguity. But as Junio noted, %p is mandatory in the\ncommand for it to possibly work. Git could refuse to use smugeToFile = cmd\nas not containing a %p and so not possibly being able to work.\n\nOr we could pick one of the two methods of appending the file\n(I prefer not including a space before it as more flexible), and\nanything using this interface would need to design its command line\nparsing with this interface in mind, and would probably choose to use\n--to-file=foo rather than --to-file foo.\n\n-- \nsee shy jo\n"},{"id":"289467","messageId":"xmqqa8ij1xne.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160617180650.GA16404@kitenet.net","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-17T18:24:37Z","receivedAt":"2016-06-17T18:24:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <id@joeyh.name> writes:\n\n> Or we could pick one of the two methods of appending the file\n> (I prefer not including a space before it as more flexible), and\n> anything using this interface would need to design its command line\n> parsing with this interface in mind, and would probably choose to use\n> --to-file=foo rather than --to-file foo.\n\nWhen I gave the example in the message you are responding to, I\nchose to assume that we will always have a space there, and this was\nto to avoid a problem that will become common if we did otherwise,\nnamely:\n\n    [filter \"foo\"] cleanFromFile = command\n\nwhich must be spelled as\n\n    [filter \"foo\"] cleanFromFile = \"command \"\n\nto separate the command name with it argument if we didn't add the\nspace there.\n\nI tend to agree with what you said in the other response, i.e.\n\"Probably anything using this interface is gonna be implemented with\nthe interface in mind from the beginning\", so they will not have an\nissue if the interface always added a space there, and there is the\n\"sh -c\" thing if they really want to do something unusual.\n"},{"id":"289468","messageId":"xmqq60t71xke.fsf@gitster.mtv.corp.google.com","threadId":"42634","inReplyTo":"20160617131311.GC24025@kitenet.net","subject":"Re: [PATCH 2/4] add smudge-to-file and clean-from-file filter configuration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-17T18:26:25Z","receivedAt":"2016-06-17T18:26:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Joey Hess <id@joeyh.name> writes:\n\n> Here's a revised version of the documentation that I think takes the other\n> suggestions onboard. I emphasise that clean and smudge operate as filters,\n> to contrast better with cleanFromFile and smudgeToFile not operating as\n> regular stdio filters.\n> ...\n> This could be extended more, but I think this should describe the config\n> settings concisely and point to the more involved discussion of filter drivers\n> in gitattributes.\n\nYup.  This version I can agree with.\n\nThanks.\n"}]}